Training an Open-Vocabulary Monocular 3D Detection Model without 3D Data

Authors: rui  huang
henry  zheng
yan  wang
zhuofan  xia
marco  pavone
gao  huang
monocular  image
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-04 08:59:43

Abstract: Open-vocabulary 3D object detection has recently attracted considerable attention due to its broad applications in autonomous driving and robotics, which aims to effectively recognize novel classes in previously unseen domains. However, existing point cloud-based open-vocabulary 3D detection models are limited by their high deployment costs. In this work, we propose a novel open-vocabulary monocular 3D object detection framework, dubbed OVM3D-Det, which trains detectors using only RGB images, making it both cost-effective and scalable to publicly available data. Unlike traditional methods, OVM3D-Det does not require high-precision LiDAR or 3D sensor data for either input or generating 3D bounding boxes. Instead, it employs open-vocabulary 2D models and pseudo-LiDAR to automatically label 3D objects in RGB images, fostering the learning of open-vocabulary monocular 3D detectors. However, training 3D models with labels directly derived from pseudo-LiDAR is inadequate due to imprecise boxes estimated from noisy point clouds and severely occluded objects. To address these issues, we introduce two innovative designs: adaptive pseudo-LiDAR erosion and bounding box refinement with prior knowledge from large language models. These techniques effectively calibrate the 3D labels and enable RGB-only training for 3D detectors. Extensive experiments demonstrate the superiority of OVM3D-Det over baselines in both indoor and outdoor scenarios. The code will be released.

Main paper: 
 introduction: 
 3D object detection has attracted substantial attention in recent years due to its broad applications in fields such as autonomous driving [13, 25, 26, 64, 55, 31] , augmented reality [17, 15, 22] , embodied AI [41, 7, 73, 61] , etc. Although many 3D detectors [42, 62, 51, 35, 32, 48, 50] have marked their impressive performance in the course of 3D perception development and consistently renew the stateof-the-art records on the popular benchmarks [14, 10, 52] , they often struggle to generalize to other object categories beyond their training data, severely limiting their applications in diverse real-world scenarios. To address this challenge, many previous works [30, 6, 59, 76, 71, 75] have taken steps forward to detect open-vocabulary objects in broader scenarios, i.e., focusing on identifying and localizing the objects that do not belong to the known categories during training.However, these open-vocabulary 3D detectors rely heavily on point clouds captured from highprecision LiDARs or 3D sensors, which are expensive to deploy. On the contrary, in another line of research, monocular 3D object detectors directly take in single-view images to localize and recognize 3D bounding boxes of the targets. Without the costly acquisition of point clouds, monocular 3D detectors broaden the horizon of deploying object detection models on more economical platforms [33, 9, 4, 29, 44] . Even though the monocular approaches have liberated 3D object detection from the expensive point clouds during inference, their training procedure still requires images and LiDAR or 3D sensor data pairs in the intensive labeling process, preventing further data scaling. Given the abundance of RGB images online, one might ask: Could the training of monocular 3D detectors benefit from the readily available RGB images to improve the open-vocabulary perception capability?In this paper, we answer this question by proposing a novel open-vocabulary monocular 3D object detection framework named OVM3D-Det, which requires only RGB images for training, enables the exploitation of a diverse array of data, and fosters its open-vocabulary potential upon deployment, as shown in Fig. 1 . To excavate valuable knowledge and explore the open-set objects from large amounts of images, we harness open-vocabulary 2D models to find the objects that belong to the novel categories. In order to obtain their corresponding 3D bounding boxes, we adopt pseudo-LiDAR [57] assisted with depth estimator to determine the location of the objects in 3D spaces.Nevertheless, a naive implementation of this framework leads to inaccurate 3D bounding boxes, impairing the performance of trained open-vocabulary 3D detectors (see Tab. 5(a), 7.3% vs. 18.5% AP). We attribute this to the highly noisy pseudo-LiDAR. Although recent advances in depth estimation models [16, 67, 40, 56, 65] showcase impressive zero-shot generalization across various structured environments and cameras with different intrinsics, pseudo-LiDAR can accumulate errors during the auto-labeling process: (1) the generated point clouds are noisy, making it difficult to distinguish targets from background artifacts (see Fig. 2 ); (2) target objects may be occluded whose actual sizes are hard to estimate, as the point clouds are projected from a single view.Facing these challenges, we design two innovative components in our OVM3D-Det framework. In order to mitigate the first issue, we propose an adaptive erosion method to filter artifacts in pseudo-LiDAR and reserve the target objects. This method adaptively removes noise points close to targets according to the object size, thereby improving the precision of 3D bounding boxes. For the second issue, if most points of the objects are missing, accurately estimating their dimensions is challenging. We introduce prior knowledge of objects in each category by prompting large language models such as GPT-4 [1] to determine reasonable bounding boxes and devise a bounding box search technique to optimize the unreasonable box. Finally, the open-vocabulary monocular 3D object detectors are trained to localize with the pseudo bounding boxes and to recognize by aligning with text embeddings.In summary, the contributions of our paper are threefold: (1) We present the first work targeting image-based open-vocabulary 3D detection tasks across both indoor and outdoor scenarios; (2) We introduce a framework, OVM3D-Det, that automatically labels 3D objects without relying on LiDAR point clouds, thereby expanding the capability to utilize internet-scale data; (3) Our proposed detector, trained on auto-labeled pseudo targets, significantly outperforms the baselines, showcasing the effectiveness of our approach.Pseudo-LiDAR Noisy RegionFigure 2 : Comparison between LiDAR data and pseudo-LiDAR. Although pseudo-LiDAR is much denser than LiDAR, it is highly noisy (as highlighted in the red boxes), making it inadequate for directly generating accurate 3D bounding boxes. 
related works: 
 Open-Vocabulary 3D Object Detection. Recently, there has been an increasing interest in 3D openvocabulary scene understanding. This emerging field takes advantage of the zero-shot recognition capabilities of 2D vision-language models [20, 45] , enabling the generalization to objects belonging to new classes [8, 11, 19, 39, 53, 70, 23] . In this context, several works [30, 6, 59, 12, 71, 75] explore open-vocabulary 3D object detection, targeting novel object identification. OV-3DET [30] introduces triplet cross-modal contrastive learning to link image, point cloud, and text modalities, allowing the point cloud detector to leverage vision-language pre-trained models [45] . CODA [6] trains a class-agnostic 3D detector on annotated base categories, gradually discovering novel objects and incorporating them into the training labels. However, previous methods require point clouds as input with high deployment costs, while we focus on open-vocabulary monocular 3D object detection.Monocular 3D Object Detection. Monocular 3D object detection aims to predict 3D bounding boxes and object classes from a single image, which is challenging due to the loss of depth information when converting 3D to 2D. Previous works [33, 9, 4, 29, 74, 44, 63, 37, 72] leverage additional prior information, such as geometric constraints or ground depth, to enhance performance. Pseudo-LiDAR [43, 57, 58, 68] generates pseudo LiDAR points with the off-the-shelf depth estimator and then employs the LiDAR-based 3D detector. Cube R-CNN [3] improves the model's generalization across datasets by utilizing virtual depth, transforming object depth with consistent virtual camera intrinsics. While these methods have yielded impressive results, they all require manually labeled 3D annotations for model training. To reduce the labor-intensive 3D data labeling process, some works [38, 18, 54] have explored weakly-supervised approaches for monocular 3D object detection. Although these approaches eliminate the need for 3D annotations, they still require raw LiDAR data.In [21] , to address the lack of 3D annotations, they use an off-the-shelf depth estimator to enhance 3D localization, transferring this information to 3D detectors through self-knowledge distillation. However, with the depth estimation model being pre-trained on the same dataset used for 3D detection, the generalization ability of this method cannot be guaranteed. Different from weakly-supervised methods, we seek to ease the burden of expensive 3D data acquisition, enabling the massive image data and increasing the model's open-vocabulary ability.Depth Estimation Model. Reconstructing depth from monocular RGB image is an ill-posed problem [68, 36] . MiDaS [46] introduces an affine-invariant target to ignore the varying depth scales and shifts across different datasets, estimating the relative depth. Recently, the achievements of visual foundation models [34, 24] trained on large-scale datasets have encouraged the development of various foundational depth estimation models [16, 67, 40, 56, 65] . Depth Anything [65] scales up to a large volume of unlabeled data, significantly expanding data coverage and thereby improving generalization. Metric3D [67] points out that the key to generalization lies in addressing the metric ambiguity of different camera models and proposes a canonical camera space transformation to handle the scale ambiguity problem. Unidepth [40] designs a camera module that generates dense camera representations to guide the depth prediction module, allowing it to zero-shot predict depth for a given image. The tremendous progress of these depth estimation models lays the foundation for generating practical pseudo-LiDAR in our method. Step ①: Generate per-instance pseudo-LiDAR.Step ②: Apply an adaptive erosion process to remove artifacts and noises.Step ③: Estimate the orientation.Step ④: Tightly fit a box and utilize object priors to assess the estimated box; if deemed unreasonable, search for the optimal box.Step ⑤: Train the model with pseudo labels. 
method: 
 Our goal is to train an open-vocabulary monocular 3D model in an unsupervised manner using image data, without the need for LiDAR point clouds. To achieve this, we propose an automatic labeling framework, OVM3D-Det, that generates 3D labels for RGB images. OVM3D-Det initially generates per-instance depth maps and unprojects them into pseudo-LiDAR point clouds. It subsequently post-processes the instance-level pseudo-LiDAR data to remove artifacts and noises (Sec. 3.1). Then, we fit a 3D bounding box closely to the refined pseudo-LiDAR and employ object priors to evaluate the box; if it is considered unreasonable, we search for the optimal box. (Sec. 3.2). Finally, we train a Cube R-CNN model on the auto-labeled data (Sec. 3.3). The overall framework is illustrated in Fig. 3 .Pseudo-LiDAR Generation. For a given image, we begin with using the off-the-shelf Grounded-SAM [47] to detect and segment novel objects. Grounded-SAM integrates Grounding DINO [28] as an open-set object detector, combined with the segment anything model (SAM) [24] , enabling simultaneous detection and segmentation of regions within images using arbitrary text inputs. After querying Grounded-SAM with a text list of novel objects of interest, we derive bounding boxes {B k } K k=1 and corresponding masks {M k } K k=1 for all the objects of interest in the image, where K denotes the number of detected objects. Then we use the pre-trained monocular deep estimation model Unidepth [40] to predict its depth map D(u, v). Note that we directly employ the pre-trained model for zero-shot estimation, and there is no overlap between the datasets used in our experiments and the training datasets of the pre-trained model. This ensures that our method has strong generalizability. Following [57] , we can obtain the 3D coordinates (x, y, z) of each pixel (u, v) within the camera's coordinate system:z = D(u, v) x = (u -c U ) × z/f U y = (v -c V ) × z/f V (1)Here, (c U , c V ) denotes the pixel coordinates corresponding to the camera center, while f U and f V represent the horizontal and vertical focal lengths, respectively. For each mask M i , applying eq. ( 1) to all pixels within the mask yields a 3D point cloud x (n) , y (n) , z (n) Ni n=1 representing the object, where N i denotes the total pixel count of the mask M i . The resulting point cloud is referred to as pseudo-LiDAR.Adaptive Pseudo-LiDAR Erosion. We visualize the pseudo-LiDAR and observe that, while it is generally impressive, it still exhibits considerable noise, making it challenging to directly obtain accurate 3D bounding boxes from it. As shown in Fig. 2 , compared to LiDAR data, pseudo-LiDAR is denser but also noisier. We identify that the primary source of inaccuracy in pseudo-LiDAR stems from projection errors at the edges of predicted masks. This issue arises because foreground objects and the background, which may have large disparities in depth values, can appear closely adjacent in images, thereby complicating depth estimation at object boundaries. Therefore, we propose employing the image erosion operation to eliminate inaccurate object mask edges while preserving other regions. Image erosion is the morphological operation in digital image processing. Despite its simplicity, the erosion procedure effectively reduces noise, resulting in a cleaner pseudo-LiDAR representation. As shown in Fig. 3(b) , after eroding the edges of the car's mask, the pseudo-LiDAR of the car no longer contains noise at the top and front of the vehicle. Please refer to Sec. C for details.On the other hand, we note that aggressive erosion may completely remove small objects like distant pedestrians, while mild erosion may leave considerable noise on large objects. Hence, we introduce an adaptive erosion module to address this issue. Specifically, this module adjusts the erosion technique based on the Grounding-SAM-generated mask size in the RGB image. Specifically, small objects will undergo less erosion, preserving more of their area, while larger objects will have more of their edges removed, ensuring more accurate pseudo-LiDAR.Box Orientation Estimation. Building upon the refined pseudo-LiDAR, we generate 3D bounding boxes as pseudo-labels to equip our monocular 3D object detection model with localization capabilities. Without losing generality, we assume that the 3D bounding boxes of objects in the scene are parallel to the ground. To generate such boxes, we begin by estimating the ground plane equation in the camera coordinate system C. The same method as in Sec. 3.1 is used to generate the pseudo-LiDAR for the ground. Specifically, Grounded-SAM [47] is prompted with "ground" for outdoor scenes or "floor" for indoor scenes to obtain their masks. We back-project these masks into 3D space using the estimated depth, and apply the least squares method to fit these points and derive the equation representing the ground. Then, the pseudo-LiDAR is transformed to the new coordinate system C ′ where the origin aligns with that of the camera coordinate system C but the horizontal plane is parallel to the ground.As presented in Fig. 3(c) , we further project the point cloud onto the ground plane to estimate the orientation of the boxes. Following the method described in [38] , we calculate the direction of each pair of points in the pseudo-LiDAR and construct a histogram of these calculated directions. The direction with the highest frequency in the histogram is selected as the object orientation. Additionally, we try Principal Component Analysis (PCA) to extract the main feature components of the pseudo-LiDAR through a linear transformation, assigning the direction with the greatest variance as the orientation of the bounding box. Through experiments, we observe that the simpler and more efficient PCA method can effectively extract the orientation of the objects (see Tab. 5(c)).Bounding Box Search. After estimating the orientation, we enclose the pseudo-LiDAR of each object with a tight 3D bounding box according to the object's direction. These bounding boxes are recorded as coarse pseudo boxes. However, due to the possible occlusion of objects in the images, the generated pseudo-LiDAR may only cover part of the objects. On the other hand, the refined pseudo-LiDAR may still contain noise. As a result, we observe that the enclosed boxes can be undersized or oversized. Therefore, to generate the proper pseudo labels for training, we further search for the optimal bounding box based on the coarse pseudo box.We first determine whether the dimensions of a coarse pseudo box are within a reasonable range. Large language models (LLMs), such as GPT-4 [1] , are employed to gather data on the typical sizes of objects. For instance, we prompt GPT-4 with "Please provide the (length, width, height) for objects of the <CLASS> category according to their typical sizes in real life." In our experiments, we find that GPT-4 can offer reliable information regarding the usual sizes of objects within specific categories, including their width, length, and height. When we replace the object sizes predicted by GPT-4 with the sizes statistically derived from the ground truth annotations in the training dataset, the performance is similar (see Tab. 5(d)). The predicted dimensions from GPT-4 are considered as the category priors. If the dimensions of a coarse pseudo box fall within the reasonable range of its corresponding category priors, the coarse pseudo box is considered a valid pseudo label. This range is set as τ 1 to τ 2 times the priors, where τ 1 represents the lower threshold and τ 2 is the upper threshold. Conversely, if it is too small or too large, it undergoes an optimal box search process.ℒ!"#$% = 𝑃 -𝑃&'!%" ( 𝑷 𝑷𝐢𝐧𝐭𝐞𝐫 𝑷 𝑷𝐢𝐧𝐭𝐞𝐫 Camera Camera (a) (b) Box A Box B Box C Box DBox A < Box B Ray Tracing Loss: Box C < Box D Point Ratio Loss: Box C > Box D = 1 - !"# %& '%()*+ ()+(,-*.-/%0 ( ) !"# %& 344 *.-'%()*+ ( 5 )Camera CameraPotential boxes are proposed based on the coarse pseudo box. Since the pseudo-LiDAR is constructed from the projection of points captured on the surface of the object, at least one edge of the coarse pseudo box should belong to the optimal pseudo box. Therefore, in the bird's-eye view, for each of the four corners of the coarse pseudo box, we propose two boxes for two possible directions with the class dimension priors (see the blue point and two blue boxes in Fig. 3(d) ). That is to say, there are 8 proposal boxes for each coarse pseudo box.We then select the best-fitting proposal box through a box-searching procedure. Given that monocular RGB cameras only capture the surfaces of objects facing the camera, a valid bounding box should have point clouds concentrated around the edges closer to the camera compared to those farther away. Therefore, we borrow the ray tracing loss from [38] to determine which box is proper. As shown in Fig. 4 (a), the loss calculates the distance between each LiDAR point and the nearest intersection point between the proposal box and the camera ray to that point. However, calculating the ray tracing loss alone is not sufficient. Since the calculation of the ray tracing loss only includes points where the camera ray intersects with the box, there can sometimes be shortcuts. As shown in Fig. 4 (b), Box C has less ray tracing loss, but it is clearly not what we want. Thus, we propose adding a point ratio loss that considers the ratio of the point cloud enclosed by the box to the overall object point cloud.The overall box-searching loss is the sum of these two losses:L trace = ∥P -P inter ∥ 2 ,if Ray P intersects with the proposal box, 0, otherwise.EQUATIONwhere N inside and N all denote the number of points inside the pseudo box and the total number of points, λ is the hyper-parameter that balances the contribution of two losses. For each of the objects, the proposal box with the lowest box-searching loss is considered the pseudo label for training.We adopt Cube R-CNN [3] as our basic 3D monocular detector, and replace its classification branch with a text-alignment head that aligns the output object features with text embeddings. The aligning loss L aligning = L CE (c i , y i ), where c i = f i • t represents the dot product of the object feature f i and text embeddings t, and y i denotes the pseudo label predicted by Grounded-SAM. The training objective is defined as: L train = L localization + L aligning . 
experiments: 
 We first evaluate the proposed OVM3D-Det model on outdoor datasets, KITTI [14] and nuScenes [5] , as well as indoor datasets, SUN RGB-D [52] and ARKitScenes [2] (Sec. 4.2). We then provide thorough ablation studies and detailed analysis to uncover the factors contributing to the effectiveness of OVM3D-Det (Sec. 4.3). Finally, we show qualitative results of our OVM3D-Det model (Sec. 4.4).Datasets. We evaluate OVM3D-Det on the KITTI [14] and nuScenes [5] datasets for outdoor settings, and on the SUN RGB-D [52] and ARKitScenes [2] datasets for indoor settings. Note that neither of these datasets appeared in the training data of Unidepth [40] , so there is no data leakage. KITTI [ and ARKitScenes has 8 base and 6 novel classes. Please refer to Sec. A for more details. We would like to note that we do not need to split the datasets into base and novel splits to train our model. We introduce this base/novel setting only to compare our method with the proposed baselines due to the lack of a suitable baseline before.Implementation Details. We prompt the off-the-shelf Grounded-SAM [47] to detect and segment objects in new categories. For the depth estimation, we adopt the state-of-state-art monocular depth estimation model, Unidepth [40] . Both models are used directly for inference without any fine-tuning. We train on the monocular 3D object detection model Cube R-CNN [3] with the auto-generated pseudo labels, modifying the classification branch to output a 768-dimensional feature and following most of its optimization hyper-parameters. See Sec. E for more details.Baselines. To the best of our knowledge, we are the first to explore open vocabulary monocular 3D object detection models. Therefore, to better evaluate the proposed task, we introduce the following baselines: 1) We train the Cube R-CNN [3] with all dataset annotations, including both base and novel classes, to establish the oracle baseline. We report the open-vocabulary monocular 3D object detection results on outdoor and indoor datasets in Tab. 1 and Tab. 2, respectively. It can be observed that the unsupervised OVM3D-Det model significantly outperforms the baseline on novel categories. Specifically, it achieves a 6.7% improvement in average precision (AP) on the KITTI dataset, 9.7% AP on the nuScenes dataset, 8.5% AP on the SUN RGB-D dataset, and 16.8% AP on the ARKitScenes dataset. Besides, the OVM3D-Det model trained on the ground truth annotations of the base class and the pseudo labels of the novel class also surpasses the baseline by a large margin on the novel category. This indicates that our framework does indeed generate accurate pseudo-labels for various classes, thereby enhancing the model's performance in the novel category.In Fig. 6 , we show the model performance varying with the amount of training data on KITTI. As the training data increases, we observe a continuous improvement in performance. This highlights the strong potential of our model, suggesting that it can be trained to achieve more generalized open-vocabulary capabilities when using more readily available image data.Comparison with Point Cloud-Based Methods. To compare our method with state-of-the-art point cloud-based open-vocabulary 3D detection approaches, we first simply evaluate existing open vocabulary detector OV-3DET [30] that is trained and tested on point cloud data using pseudo-LiDAR as input. Specifically, we only replace the point cloud input of OV-3DET with the pseudo-LiDAR which is unprojected from the RGB images and estimated depth maps. Other parts of the model in OV-3DET are kept untouched for fair comparison. As shown in Tab. 3, directly changing the input data format leads to poor performance with only 3.4% AP. This phenomenon is also observed and discussed in previous works [36, 66, 60] , which we attribute to the distribution shifts between real point clouds and pseudo-LiDARs.After being trained with pseudo-LiDAR, OV-3DET improves from 3.4% to 7.1% AP, marking a 2x improvement compared to the original model. CODA [6] shows a slight improvement compared to OV-3DET. Nonetheless, the 10.8% AP of our method still demonstrates our superiority, since our approach focuses on adapting to the noisy nature of pseudo-LiDAR, whereas previous openvocabulary 3D detectors are mainly designed for real point clouds to generate pseudo labels. The key designs that distinguish our method from other baselines are the adaptive pseudo-LiDAR erosion and bounding box refinement techniques, incorporating the prior knowledge of language models. Our method can generate pseudo labels more effectively for pseudo-LiDAR than other baselines.For outdoor 3D detection, we also try to establish another baseline since OV-3DET [30] only includes results on indoor data. We attempt to train OV-3DET on the KITTI dataset with pseudo-LiDAR. As presented in Tab. 4, simply training on pseudo-LiDAR points causes a rather weak performance of only 1.3% AP. This result reflects the nature of outdoor data that the vast range of spatial scale could lead to intolerable pseudo-label errors from tiny noise. Therefore, it is challenging to generate reasonable and reliable pseudo boxes due to the inaccurate object edges of the raw pseudo-LiDAR.We conduct ablation studies on the KITTI dataset. The results are shown in Tab. 5. We report the mean AP performance across all the classes for each model.Comparison with Naive Framework. If we remove the refinement applied to pseudo-LiDAR and bounding box (i.e., directly utilizing raw pseudo-LiDAR and tightly bounding it to obtain the box), there is a significant decrease of 11.2% AP in model performance. This result shows the effectiveness of our framework in addressing the noisy pseudo-LiDAR, where the refinement of both the pseudo-LiDAR data and the estimated bounding boxes proves to be pivotal.Mask Erosion. We replace the adaptive erosion process with a fixed kernel erosion, resulting in a decline of 2.1% AP. It demonstrates the effectiveness of our proposed adaptive erosion module, indicating that a standard fixed kernel may not be suitable for all cases, as it could overly erode small objects or be inadequate for larger objects with substantial noise.Orientation Estimation. Here, we compare two methods for determining the orientation of boxes: using Principal Component Analysis (PCA) or calculating the distribution of the directions of each pair of points in the pseudo-LiDAR. We find that the simple and efficient PCA method performs better, exceeding 1.8% AP.Dimension Priors. The dimension priors of objects are an essential component of our method. They not only assist in identifying unreasonable predictions of bounding boxes but also aid in correcting the sizes of the bounding boxes. As shown in Tab. 5(d), the absence of these priors leads to a significant decrease of 4.5% AP. Notably, when we utilize priors derived from a large language model or dataset statistics, the results are similar. In Sec. B, we provide a thorough comparison of LLM-generated priors and shape priors obtained from dataset statistics across all datasets. This suggests that the large language model possesses a strong commonsense understanding, and further underscores the potential of our approach to be transferred to a wide range of data. We also test the sensitivity of the lower threshold τ 1 and the upper threshold τ 2 used for determining the reasonability of a box. As shown in Tab. 5(f) and (g), they are stable within a wide range.Box Search Loss. We further explore the impact of different loss functions on evaluating the bounding box proposals. The results are presented in Tab. 5(e). Using ray tracing loss alone results in inadequate performance, yielding only a 12.2% AP. However, adding the point ratio loss significantly boosts performance to 18.5% AP.Self-Training. Additionally, we can refine the model using a self-training approach, which involves using the results from the previously well-trained model as pseudo labels in the subsequent training process. As shown in the Tab. 6, self-training can further enhance the quality of the initially generated pseudo boxes, improving detection performance for objects at various distances. The qualitative results are presented in Fig. 5 . We prompt the OVM3D-Det model with text queries of the desired categories, obtaining corresponding detection boxes. Results show its capability to locate and recognize objects of diverse types and sizes in both indoor and outdoor scenarios.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We have disclosed the implementation information necessary for reproducibility in the main paper and the appendix as thoroughly as possible.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
conclusion: 
 In Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: Code will be released. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: We discussed the detailed broader impact in the appendix. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for the responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: We believe no such risk is involved. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Besides Unidepth, we evaluate alternative depth estimation models, such as Metric3D [67] . As shown in Tab. 12, we find that the more accurate the depth estimation model we use, the better the performance of the trained open-vocabulary monocular 3D detection model, suggesting promising potential as depth estimation models advance. When searching for the optimal box, we set λ to 5 for indoor scenes and 10 for outdoor scenes to balance the ray tracing loss and point ratio loss. For the adaptive erosion module, for outdoor scenes, if the maximum width of the instance mask exceeds 10 pixels, image erosion is performed for 4 iterations; otherwise, 2 iterations. For indoor scenes, the values are 12 and 2 iterations, respectively. As we aim to evaluate the benchmarks, we utilize the label lists from the datasets to query the open-vocabulary 2D model for generating pseudo labels of the classes of interest. Following Cube R-CNN, we utilize the DLA34 backbone [69] -FPN [27] , which is pretrained on ImageNet [49] . We employ the SGD optimizer, with the learning rate decaying by a factor of 10 at 60% and 80% of the training process. During training, we apply random data augmentation techniques such as horizontal flipping and scaling within the range of [0.50, 1.25]. The model is trained for 29,000 iterations with a batch size of 32 and a learning rate of 0.02 on both SUN RGB-D [52] and ARKitScenes [2] , a batch size of 16 and a learning rate of 0.01 on KITTI [14] , and a batch size of 32 and a learning rate of 0.01 on nuScenes [5] . We train on the KITTI dataset for ∼ 8 hours with 2 A100 GPUs, on nuScenes for ∼ 6 hours with 4 A100 GPUs, on SUN RGB-D for ∼ 12 hours with 2 A100 GPUs, and on ARKitScenes for ∼ 20 hours with 2 A100 GPUs.F Failure Case We present some failure cases of OVM3D-Det in Fig. 8 . (a), (b) and (c) are instances of distant objects being missed by detection. In (d), the reflection of a car on the window is mistakenly detected as a vehicle. As our framework relies on foundational depth estimation models for generating pseudo-LiDAR for auto-labeling, the accuracy of depth estimation directly affects label precision. Particularly, errors increase quadratically with distance, resulting in lower-quality pseudo-labels for distant objects and thus limiting our model's performance at longer ranges. This issue merits further investigation in our future research endeavors.Table 13 : Licenses of assets used.Cube R-CNN [3] CC-BY-NC 4.0 Grounding DINO [28] Apache License 2.0 Segment Anything [24] Apache License 2.0 Grounded-SAM [47] Apache License 2.0 GPT-4 [1] MIT License Unidepth [40] CC-BY-NC 4.0 Metric3D [67] 2-Clause BSD License KITTI [14] CC-BY-NC-SA 3.0 DEED nuScenes [5] CC-BY-NC 4.0 SUN RGB-D [52] MIT License ARKitScenes [2] Apple License (Link)Our proposed OVM3D-Det framework relies on advanced depth estimation models to generate pseudo-LiDAR. Although state-of-the-art depth estimation models have demonstrated strong generalization capabilities, their predictions for distant objects are not yet accurate enough, which may lead to errors when generating pseudo labels for these distant objects. Additionally, the performance of monocular systems may degrade under adverse lighting or weather conditions, as these factors directly affect the quality and clarity of the input images. These limitations underscore the need for future research to enhance the robustness and versatility of open vocabulary monocular 3D detection systems, particularly in challenging and dynamic environments.The end results of this research are a novel approach to 3D detection that can be applied to autonomous driving, augmented reality, and other robotics applications. By enabling improved 3D object detection using monocular images, our framework has the potential to enhance the safety and accuracy of this system, contributing to the overall advancement of autonomous technologies. Before deployment, it is crucial to establish appropriate safety thresholds and conduct thorough testing to ensure reliability. While our approach does not explicitly exploit dataset biases, it is subject to the same limitations and potential biases as other machine learning techniques.• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 


Decision: Post-rebuttal, all four reviewers are in favor of acceptance. The AC has examined the paper, the reviews, rebuttal, and discussion and concurs. Since the rebuttal was critical for the paper's acceptance, the AC urges the authors to carefully incorporate all information from the rebuttal into the final version of the paper.

Reviews: review: paper_summary: paper_summary: This paper presents OVM3D-Det, a framework for generating 3D bounding box pseudo-labels from monocular RGB images using foundational vision models like GroundingSAM and UniDepth. Using these pseudo-labels, authors train an open-vocabulary version of Cube-RCNN. Authors demonstrate that their proposed approach generalizes to novel categories unseen during training.
strength_weakness: strength_weakness: Limited Evaluation. Authors claim that there are no prior methods that address this problem. However, since the proposed method trusts monocular depth estimates as if they were ground truth LiDAR, one can simply evaluate existing RGB + LiDAR open vocabulary detector [1] using pseudo-lidar. Notably, there has been significant prior work in open-vocabulary 3D detection which should be acknowledged in this work. 

"Closed World" Setup. Splitting datasets into base vs. novel splits goes against the setup of open-vocabulary detection. Notably, in 2D open vocabulary detection, methods pre-train on diverse datasets like V3Det, Objects365, ODinW, and report zero-shot results on all classes in COCO. A similar setup here would be to train on diverse 3D datasets and report zero-shot results on all classes in KITTI and SUNRGBD.

Pseudo-Labeler Heuristics. The design of the pseudo-labeler is extremely brittle, particularly because it relies on hand designed heuristics. I would not expect such a pseudo-labeler to work well on COCO (despite CubeRCNN doing a reasonable job). 

Small Scale Evaluation. Both SUNRGBD and KITTI are very small 3D detection datasets by modern standards. These datasets alone are not enough to study open-vocabulary detection. I would encourage authors to train on and evaluate with more diverse datasets. Notably, Omni3D, the dataset the accompanies CubeRCNN, already provides large-scale training data. Why were the other datasets in Omni3D not used?

[1] Open-Vocabulary Point Cloud Object Detection without 3D Annotation. Lu et. al. CVPR 2023.
questions: questions: Limitations of Base and Novel Splits. Splitting individual datasets into base and novel splits no longer makes sense in the era of foundation models. Explicitly trying to avoid data leakage will prevent methods from using the latest and greatest models, artificially hampering progress. The best way to address this issue is by evaluating on a sufficiently large and diverse test set such that even if a small part of the data has already been seen by some model (e.g. like UniDepth), the aggregate performance is still a reliable indicator of overall test performance. 

Realistic Benchmarking on Out of Distribution Datasets. Similar to 2D open-vocabulary detectors, it would be interesting to benchmark the performance of this approach on unseen datasets (not just unseen classes). For example, how might the proposed model perform on nuScenes if it was trained on KITTI? Notably, nuScenes has many classes not included in KITTI like construction vehicle and motorcycle. 

Dimension Prior vs. Real Prior. Using LLMs to generate 3D shape priors is an interesting idea. Intuitively, LLMs seen many descriptions of object shapes on the web, so it makes sense that they can provide reasonable shape priors. How do the LLM shape priors (e.g. L, W, H) actually compare with the real shape priors learned from dataset statistics?

Why Pseudo-Label 3D Datasets? Although the premise of the paper makes sense, the execution is a bit confusing. In particular, both KITTI and SUNRGBD both already have 3D bounding box annotations. It doesn't make sense to pseudo-label these datasets. Instead, it makes more sense to try to pseudo-label datasets like COCO that cannot currently be used by existing 3D detectors.
limitations: limitations: Yes, authors highlight that their proposed approach is reliant on depth estimation models, and is susceptible to their failure modes.
paper_summary: paper_summary: The paper presents a novel open-vocabulary monocular 3D object detection framework named OVM3D-Det. This approach aims to train 3D object detectors using only RGB images, making it cost-effective and scalable. The framework utilizes pseudo-LiDAR and large language models to generate pseudo 3D labels, enabling training without high-precision LiDAR data. The authors propose adaptive pseudo-LiDAR erosion and bounding box refinement to improve label accuracy. Extensive experimentation demonstrates the superiority of OVM3D-Det over existing baselines in both indoor and outdoor scenarios.
strength_weakness: strength_weakness: - **Dependence on Depth Estimation Quality**: The performance of the proposed method heavily relies on the quality of the depth estimation model, which may not always be accurate, especially for distant objects.  
 &nbsp; 
- **Limited Evaluation on Adverse Conditions**: The paper does not extensively compare the method with other indoor OV-3Det methods [1, 2]  
 &nbsp; 
- **Complexity of Implementation**: The framework involves several sophisticated components, including depth estimation, pseudo-LiDAR generation, and large language models, which might complicate implementation and reproducibility.  
 &nbsp;

[1] Yuheng Lu, Chenfeng Xu, Xiaobao Wei, Xiaodong Xie, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. Open-vocabulary point-cloud object detection without 3d annotation. In CVPR, 2023. 1, 3  
[2] Yang Cao, Zeng Yihan, Hang Xu, and Dan Xu. Coda: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection. In NeurIPS, 2023
questions: questions: - How does the performance of the proposed method vary with the quality of the depth estimation model? Can the authors provide more insights or quantitative analysis on this aspect?  
 &nbsp; 
- Have the authors considered comparing the indoor OV-3Det methods[1, 2]?  
 &nbsp; 
- Can the authors provide more details on the computational requirements and efficiency of the proposed framework? How does it compare with point cloud-based methods in terms of computational costs?  
 &nbsp; 
- How generalizable is the proposed method to other types of data or domains beyond those tested in the experiments?  
 &nbsp; 
- Are there any specific challenges or limitations in integrating the proposed method into real-world applications such as autonomous driving systems?  
 &nbsp;

[1] Yuheng Lu, Chenfeng Xu, Xiaobao Wei, Xiaodong Xie, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. Open-vocabulary point-cloud object detection without 3d annotation. In CVPR, 2023. 1, 3  
[2] Yang Cao, Zeng Yihan, Hang Xu, and Dan Xu. Coda: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection. In NeurIPS, 2023
limitations: limitations: The authors have provided discussions about the limitations and broader impact.
paper_summary: paper_summary: This paper proposes OVM3D-Det, an open-vocabulary monocular 3D object detection framework that trains detectors using only RGB images. It introduces two key designs: adaptive pseudo-LiDAR erosion and bounding box refinement, addressing challenges arising from the absence of LiDAR point clouds.
strength_weakness: strength_weakness: 1.	The KITTI dataset has a limited number of classes (5 classes). It would be beneficial to provide results on the NuScenes dataset (23 classes) to better demonstrate the open-vocabulary performance.
2.	The adaptive pseudo-LiDAR erosion module is not illustrated in sufficient detail. The method section discussing this module is vague; please provide more comprehensive details on its design.
3.	More visualizations of the pseudo-LiDAR erosion module are needed. Currently, the visualization of this module is only included in the overall framework figure. Additional visualizations would better demonstrate its effectiveness.
4.	The related work section on monocular 3D object detection is not up-to-date. It is recommended to cite recent papers accepted by CVPR 2024:
a)	MonoCD: Monocular 3D Object Detection with Complementary Depths, CVPR24
b)	Learning Occupancy for Monocular 3D Object Detection, CVPR24
c)	Weakly Supervised Monocular 3D Detection with a Single-View Image, CVPR24
d)	Decoupled Pseudo-labeling for Semi-Supervised Monocular 3D Object Detection, CVPR24
5.	For Table 1 and Table 2, it would be helpful to include a baseline trained without ground-truth annotations for base classes to provide a better comparison.
questions: questions: Please refer to the weakness section.
limitations: limitations: N/A
paper_summary: paper_summary: This paper introduces OVM3D-Det, an open-vocabulary monocular 3D object detection framework that utilizes only RGB images for training. It leverages open-vocabulary 2D models and pseudo-LiDAR to automatically label 3D objects in these images. Additionally, it incorporates pseudo-LiDAR erosion and bounding box refinement modules, enhanced by prior knowledge from large language models. Experimental results demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. Although the experiments indicate improvements with the proposed method, the overall AP remains low, and the final performance heavily depends on the Depth Estimator used. Moreover, the experiments seem to lack performance metrics for Unidepth.

2. The effect of GPT-4 in this method is minimal, as it only provides object scaling. The costs and benefits of using it are not well-delineated.
questions: questions: 1. Does this framework heavily depend on the performance of Unidepth? What is the detection performance of Unidepth? How significant is the impact of missed objects, such as those at a distant range, on the final training performance?

2. What are the costs and benefits of using GPT-4?  Is the improvement from using LLM minimal in the ablation study? The performance is very close to using an average vehicle scaling size in the ablation study?

3. The ablation studies are confusing. What are the improvements of each component compared to the baseline?
limitations: limitations: According to the ablation study, the proposed modules show improvements.

Comments: title: Final score
Further reply
comment: Thank you for your valuable feedback and support. We are committed to incorporating the revisions in the final version.
Thank you for your valuable feedback and support. We are committed to incorporating the revisions in the next manuscript.
Thanks to the authors for conducting more experiments to answer my questions. Please include all the results in the final version, which will benefit the community and inspire more work. Totally speaking, most of my questions are addressed well. I raise my score to 6. Good luck :)
Thanks for your rebuttals. I have read all the contents, and most of my concerns have been addressed.
Thank you for your insightful comments. We have carefully considered your concerns and questions, and we have made every effort to address them thoroughly in our rebuttal. Since the discussion phase is halfway through, we would like to know if our responses have adequately addressed your concerns.

We would greatly appreciate it if you could reassess the paper in light of our responses, provided they have clarified your concerns. If not, we would be happy to provide further explanations or clarifications on any part of the paper. Your feedback is invaluable to us, and we are committed to addressing all your inquiries comprehensively.

Once again, thank you for your time and effort in reviewing our paper and for your valuable feedback.
We would like to express our gratitude once again for your time and valuable feedback. In our last communication, you mentioned the possibility of improving the rating for our paper. Is there any additional issue that requires our attention? We welcome any further discussion and suggestions.
Thank you very much for your insightful feedback. We are grateful for your recommendation to incorporate the new results into the final version, and we will certainly do so.

We would greatly appreciate it if you could kindly express a clearer indication of your acceptance intention, as it could significantly impact the outcome of our submission. Thank you again for your time and consideration.
We greatly appreciate your time and valuable suggestions. We have carefully and thoroughly revisited the CoDA [1] paper and noticed that it includes a comparative analysis with OV-3DET [2] in the final part of the experiments. Following its protocol, we conducted the experiment and present the results below. Our method still significantly outperforms CoDA, as we carefully considered the noisy nature of pseudo-LiDAR, leading to the generation of more accurate pseudo labels. 

Thank you once again for your insightful review, which has greatly enhanced the quality and clarity of our paper. We welcome any further discussion and suggestions.

| Model | Performance on SUN RGB-D (AP) |
| --- | --- |
| Cube R-CNN (fully-supervised) | 15.1 |
| OV-3DET | 7.1 |
| CoDA | 7.7 |
| Ours | 10.6 |

[1] CoDA: Collaborative Novel Box Discovery and Cross-Modal Alignment for Open-Vocabulary 3D Object Detection. Cao et al. NeurIPS 2023.

[2] Open-Vocabulary Point Cloud Object Detection without 3D Annotation. Lu et al. CVPR 2023.
Thank you for providing the additional experiments and detailed explanations in your rebuttal. It addressed my concerns effectively. I recommend the authors incorporate all new results in the rebuttal into the final version.
Thank the authors for the effort in the rebuttal. I have carefully read all the contents, and most of my concerns have been addressed. However, I have a question: as far as I know, CoDA doesn't strongly depend on ground truth and can also be trained using pseudo labels. Could you please provide the results for that?
Thanks for your positive feedback! We are glad that we have addressed your concerns.
Authors have sufficiently addressed my questions. I recommend accepting this paper.