Title: Training an Open-Vocabulary Monocular 3D Detection Model without 3D Data
Venue: NeurIPS 2024
Accepted: yes
Keywords: Monocular 3D Detection

Open-Vocabulary 3D Detection
Abstract: Open-vocabulary 3D object detection has recently attracted considerable attention due to its broad applications in autonomous driving and robotics, which aims to effectively recognize novel classes in previously unseen domains. However, existing point cloud-based open-vocabulary 3D detection models are limited by their high deployment costs. In this work, we propose a novel open-vocabulary monocular 3D object detection framework, dubbed OVM3D-Det, which trains detectors using only RGB images, making it both cost-effective and scalable to publicly available data. Unlike traditional methods, OVM3D-Det does not require high-precision LiDAR or 3D sensor data for either input or generating 3D bounding boxes. Instead, it employs open-vocabulary 2D models and pseudo-LiDAR to automatically label 3D objects in RGB images, fostering the learning of open-vocabulary monocular 3D detectors. However, training 3D models with labels directly derived from pseudo-LiDAR is inadequate due to imprecise boxes estimated from noisy point clouds and severely occluded objects. To address these issues, we introduce two innovative designs: adaptive pseudo-LiDAR erosion and bounding box refinement with prior knowledge from large language models. These techniques effectively calibrate the 3D labels and enable RGB-only training for 3D detectors. Extensive experiments demonstrate the superiority of OVM3D-Det over baselines in both indoor and outdoor scenarios. The code will be released.

Training an Open-Vocabulary Monocular 3D Object Detection Model without 3D Data

Abstract

Open-vocabulary 3D object detection has recently attracted considerable attention due to its broad applications in autonomous driving and robotics, which aims to effectively recognize novel classes in previously unseen domains. However, existing point cloud-based open-vocabulary 3D detection models are limited by their high deployment costs. In this work, we propose a novel open-vocabulary monocular 3D object detection framework, dubbed OVM3D-Det, which trains detectors using only RGB images, making it both cost-effective and scalable to publicly available data. Unlike traditional methods, OVM3D-Det does not require high-precision LiDAR or 3D sensor data for either input or generating 3D bounding boxes. Instead, it employs open-vocabulary 2D models and pseudoLiDAR to automatically label 3D objects in RGB images, fostering the learning of open-vocabulary monocular 3D detectors. However, training 3D models with labels directly derived from pseudo-LiDAR is inadequate due to imprecise boxes estimated from noisy point clouds and severely occluded objects. To address these issues, we introduce two innovative designs: adaptive pseudo-LiDAR erosion and bounding box refinement with prior knowledge from large language models. These techniques effectively calibrate the 3D labels and enable RGB-only training for 3D detectors. Extensive experiments demonstrate the superiority of OVM3D-Det over baselines in both indoor and outdoor scenarios. The code will be released.

1 Introduction

3D object detection has attracted substantial attention in recent years due to its broad applications in fields such as autonomous driving [13, 25, 26, 64, 55, 31], augmented reality [17, 15, 22], embodied AI [41, 7, 73, 61], etc . Although many 3D detectors [42, 62, 51, 35, 32, 48, 50] have marked their impressive performance in the course of 3D perception development and consistently renew the stateof-the-art records on the popular benchmarks [14, 10, 52], they often struggle to generalize to other object categories beyond their training data, severely limiting their applications in diverse real-world scenarios. To address this challenge, many previous works [30, 6, 59, 76, 71, 75] have taken steps forward to detect open-vocabulary objects in broader scenarios, i.e. , focusing on identifying and localizing the objects that do not belong to the known categories during training.

However, these open-vocabulary 3D detectors rely heavily on point clouds captured from highprecision LiDARs or 3D sensors, which are expensive to deploy. On the contrary, in another line of research, monocular 3D object detectors directly take in single-view images to localize and recognize

Training

Label

Generation

-Based Open-Vocabulary

Image

-Based Open-Vocabulary

Predictions

Inference

Easy Access

to Training

Data

Predictions

3D Detector

Point Clouds &

RGB Images

RGB Images

Pseudo

Pre-trained Depth

Estimation Model

Utilizing 2D & 3D

Correspondence

Cheap to

Deploy

Open-Vocabulary

2D Model

Open-Vocabulary

2D Model

Monocular

3D Detector

Pseudo

3D bounding boxes of the targets. Without the costly acquisition of point clouds, monocular 3D detectors broaden the horizon of deploying object detection models on more economical platforms [33, 9, 4, 29, 44]. Even though the monocular approaches have liberated 3D object detection from the expensive point clouds during inference, their training procedure still requires images and LiDAR or 3D sensor data pairs in the intensive labeling process, preventing further data scaling. Given the abundance of RGB images online, one might ask: Could the training of monocular 3D detectors benefit from the readily available RGB images to improve the open-vocabulary perception capability?

In this paper, we answer this question by proposing a novel open-vocabulary monocular 3D object detection framework named OVM3D-Det , which requires only RGB images for training, enables the exploitation of a diverse array of data, and fosters its open-vocabulary potential upon deployment, as shown in Fig. 1. To excavate valuable knowledge and explore the open-set objects from large amounts of images, we harness open-vocabulary 2D models to find the objects that belong to the novel categories. In order to obtain their corresponding 3D bounding boxes, we adopt pseudo-LiDAR [57] assisted with depth estimator to determine the location of the objects in 3D spaces.

Nevertheless, a naive implementation of this framework leads to inaccurate 3D bounding boxes, impairing the performance of trained open-vocabulary 3D detectors (see Tab. 5(a), 7.3% vs. 18.5% AP). We attribute this to the highly noisy pseudo-LiDAR. Although recent advances in depth estimation models [16, 67, 40, 56, 65] showcase impressive zero-shot generalization across various structured environments and cameras with different intrinsics, pseudo-LiDAR can accumulate errors during the auto-labeling process: (1) the generated point clouds are noisy, making it difficult to distinguish targets from background artifacts (see Fig. 2); (2) target objects may be occluded whose actual sizes are hard to estimate, as the point clouds are projected from a single view .

Facing these challenges, we design two innovative components in our OVM3D-Det framework. In order to mitigate the first issue, we propose an adaptive erosion method to filter artifacts in pseudoLiDAR and reserve the target objects. This method adaptively removes noise points close to targets according to the object size, thereby improving the precision of 3D bounding boxes. For the second issue, if most points of the objects are missing, accurately estimating their dimensions is challenging. We introduce prior knowledge of objects in each category by prompting large language models such as GPT-4 [1] to determine reasonable bounding boxes and devise a bounding box search technique to optimize the unreasonable box. Finally, the open-vocabulary monocular 3D object detectors are trained to localize with the pseudo bounding boxes and to recognize by aligning with text embeddings.

In summary, the contributions of our paper are threefold: (1) We present the first work targeting image-based open-vocabulary 3D detection tasks across both indoor and outdoor scenarios; (2) We introduce a framework, OVM3D-Det, that automatically labels 3D objects without relying on LiDAR point clouds, thereby expanding the capability to utilize internet-scale data; (3) Our proposed detector, trained on auto-labeled pseudo targets, significantly outperforms the baselines, showcasing the effectiveness of our approach.

RGB Image

LiDAR

Pseudo-LiDAR

Noisy Region

2 Related Works

Open-Vocabulary 3D Object Detection. Recently, there has been an increasing interest in 3D openvocabulary scene understanding. This emerging field takes advantage of the zero-shot recognition capabilities of 2D vision-language models [20, 45], enabling the generalization to objects belonging to new classes [8, 11, 19, 39, 53, 70, 23]. In this context, several works [30, 6, 59, 12, 71, 75] explore open-vocabulary 3D object detection, targeting novel object identification. OV-3DET [30] introduces triplet cross-modal contrastive learning to link image, point cloud, and text modalities, allowing the point cloud detector to leverage vision-language pre-trained models [45]. CODA [6] trains a class-agnostic 3D detector on annotated base categories, gradually discovering novel objects and incorporating them into the training labels. However, previous methods require point clouds as input with high deployment costs, while we focus on open-vocabulary monocular 3D object detection.

Monocular 3D Object Detection. Monocular 3D object detection aims to predict 3D bounding boxes and object classes from a single image, which is challenging due to the loss of depth information when converting 3D to 2D. Previous works [33, 9, 4, 29, 74, 44, 63, 37, 72] leverage additional prior information, such as geometric constraints or ground depth, to enhance performance. PseudoLiDAR [43, 57, 58, 68] generates pseudo LiDAR points with the off-the-shelf depth estimator and then employs the LiDAR-based 3D detector. Cube R-CNN [3] improves the model's generalization across datasets by utilizing virtual depth, transforming object depth with consistent virtual camera intrinsics. While these methods have yielded impressive results, they all require manually labeled 3D annotations for model training. To reduce the labor-intensive 3D data labeling process, some works [38, 18, 54] have explored weakly-supervised approaches for monocular 3D object detection. Although these approaches eliminate the need for 3D annotations, they still require raw LiDAR data. In [21], to address the lack of 3D annotations, they use an off-the-shelf depth estimator to enhance 3D localization, transferring this information to 3D detectors through self-knowledge distillation. However, with the depth estimation model being pre-trained on the same dataset used for 3D detection, the generalization ability of this method cannot be guaranteed. Different from weakly-supervised methods, we seek to ease the burden of expensive 3D data acquisition, enabling the massive image data and increasing the model's open-vocabulary ability.

Depth Estimation Model. Reconstructing depth from monocular RGB image is an ill-posed problem [68, 36]. MiDaS [46] introduces an affine-invariant target to ignore the varying depth scales and shifts across different datasets, estimating the relative depth. Recently, the achievements of visual foundation models [34, 24] trained on large-scale datasets have encouraged the development of various foundational depth estimation models [16, 67, 40, 56, 65]. Depth Anything[65] scales up to a large volume of unlabeled data, significantly expanding data coverage and thereby improving generalization. Metric3D [67] points out that the key to generalization lies in addressing the metric ambiguity of different camera models and proposes a canonical camera space transformation to handle the scale ambiguity problem. Unidepth [40] designs a camera module that generates dense camera representations to guide the depth prediction module, allowing it to zero-shot predict depth for a given image. The tremendous progress of these depth estimation models lays the foundation for generating practical pseudo-LiDAR in our method.

Depth

Estimator

Open-Vocabulary

2D Detector

Back

Project

Apply Masks to

Prompt with Texts

(a) Step

①

:

Pseudo-LiDAR Generation

Pseudo-LiDAR

(b) Step

②

:

Adaptive Pseudo-LiDAR Erosion

Erode Large Objects

(c) Step

③

:

Box Orientation Estimation

Erode

Erode

(d) Step

④

:

Bounding Box Search

Principal

Component

Analysis

Estimate the

Project Points

to the Ground

Tightly Bound the Projected Points

Propose

Potential Boxes

Partially

Seen

/ Noisy

Entirely

Seen

& Clean

①

②

What is the typical size ( length, width, height )

for objects of the <CLASS> category ?

Object

Priors

Search

Optimal

Box

Erode Small Objects

(e) Step

⑤

:

Training

Text

Features

Align

Object

Features

Is the size

reasonable?

Pseudo 3D

Box

Supervise

Localization

Capability

Recognition

Capability

Prompts:

X

+

3 Method

Our goal is to train an open-vocabulary monocular 3D model in an unsupervised manner using image data, without the need for LiDAR point clouds. To achieve this, we propose an automatic labeling framework, OVM3D-Det, that generates 3D labels for RGB images. OVM3D-Det initially generates per-instance depth maps and unprojects them into pseudo-LiDAR point clouds. It subsequently post-processes the instance-level pseudo-LiDAR data to remove artifacts and noises (Sec. 3.1). Then, we fit a 3D bounding box closely to the refined pseudo-LiDAR and employ object priors to evaluate the box; if it is considered unreasonable, we search for the optimal box. (Sec. 3.2). Finally, we train a Cube R-CNN model on the auto-labeled data (Sec. 3.3). The overall framework is illustrated in Fig. 3.

3.1 Generate Pseudo-LiDAR

Pseudo-LiDAR Generation. For a given image, we begin with using the off-the-shelf GroundedSAM [47] to detect and segment novel objects. Grounded-SAM integrates Grounding DINO [28] as an open-set object detector, combined with the segment anything model (SAM) [24], enabling simultaneous detection and segmentation of regions within images using arbitrary text inputs. After querying Grounded-SAM with a text list of novel objects of interest, we derive bounding boxes { B k } K k =1 and corresponding masks { M k } K k =1 for all the objects of interest in the image, where K denotes the number of detected objects. Then we use the pre-trained monocular deep estimation model Unidepth [40] to predict its depth map D ( u, v ) . Note that we directly employ the pre-trained model for zero-shot estimation, and there is no overlap between the datasets used in our experiments and the training datasets of the pre-trained model. This ensures that our method has strong generalizability. Following [57], we can obtain the 3D coordinates ( x, y, z ) of each pixel ( u, v ) within the camera's coordinate system:

Here, ( c U , c V ) denotes the pixel coordinates corresponding to the camera center, while f U and f V represent the horizontal and vertical focal lengths, respectively. For each mask M i , applying eq. (1) to all pixels within the mask yields a 3D point cloud {( x ( n ) , y ( n ) , z ( n ) )} N i n =1 representing the object, where N i denotes the total pixel count of the mask M i . The resulting point cloud is referred to as pseudo-LiDAR.

Adaptive Pseudo-LiDAR Erosion. We visualize the pseudo-LiDAR and observe that, while it is generally impressive, it still exhibits considerable noise, making it challenging to directly obtain accurate 3D bounding boxes from it. As shown in Fig. 2, compared to LiDAR data, pseudo-LiDAR is denser but also noisier. We identify that the primary source of inaccuracy in pseudo-LiDAR stems from projection errors at the edges of predicted masks. This issue arises because foreground objects and the background, which may have large disparities in depth values, can appear closely adjacent in images, thereby complicating depth estimation at object boundaries. Therefore, we propose employing the image erosion operation to eliminate inaccurate object mask edges while preserving other regions. Image erosion is the morphological operation in digital image processing. Despite its simplicity, the erosion procedure effectively reduces noise, resulting in a cleaner pseudo-LiDAR representation. As shown in Fig. 3(b), after eroding the edges of the car's mask, the pseudo-LiDAR of the car no longer contains noise at the top and front of the vehicle. Please refer to Sec. C for details.

On the other hand, we note that aggressive erosion may completely remove small objects like distant pedestrians, while mild erosion may leave considerable noise on large objects. Hence, we introduce an adaptive erosion module to address this issue. Specifically, this module adjusts the erosion technique based on the Grounding-SAM-generated mask size in the RGB image. Specifically, small objects will undergo less erosion, preserving more of their area, while larger objects will have more of their edges removed, ensuring more accurate pseudo-LiDAR.

3.2 Generate Pseudo 3D Bounding Boxes

Box Orientation Estimation. Building upon the refined pseudo-LiDAR, we generate 3D bounding boxes as pseudo-labels to equip our monocular 3D object detection model with localization capabilities. Without losing generality, we assume that the 3D bounding boxes of objects in the scene are parallel to the ground. To generate such boxes, we begin by estimating the ground plane equation in the camera coordinate system C . The same method as in Sec. 3.1 is used to generate the pseudo-LiDAR for the ground. Specifically, Grounded-SAM [47] is prompted with 'ground' for outdoor scenes or 'floor' for indoor scenes to obtain their masks. We back-project these masks into 3D space using the estimated depth, and apply the least squares method to fit these points and derive the equation representing the ground. Then, the pseudo-LiDAR is transformed to the new coordinate system C ′ where the origin aligns with that of the camera coordinate system C but the horizontal plane is parallel to the ground.

As presented in Fig. 3(c), we further project the point cloud onto the ground plane to estimate the orientation of the boxes. Following the method described in [38], we calculate the direction of each pair of points in the pseudo-LiDAR and construct a histogram of these calculated directions. The direction with the highest frequency in the histogram is selected as the object orientation. Additionally, we try Principal Component Analysis (PCA) to extract the main feature components of the pseudoLiDAR through a linear transformation, assigning the direction with the greatest variance as the orientation of the bounding box. Through experiments, we observe that the simpler and more efficient PCA method can effectively extract the orientation of the objects (see Tab. 5(c)).

Bounding Box Search. After estimating the orientation, we enclose the pseudo-LiDAR of each object with a tight 3D bounding box according to the object's direction. These bounding boxes are recorded as coarse pseudo boxes. However, due to the possible occlusion of objects in the images, the generated pseudo-LiDAR may only cover part of the objects. On the other hand, the refined pseudo-LiDAR may still contain noise. As a result, we observe that the enclosed boxes can be

undersized or oversized. Therefore, to generate the proper pseudo labels for training, we further search for the optimal bounding box based on the coarse pseudo box.

We first determine whether the dimensions of a coarse pseudo box are within a reasonable range. Large language models (LLMs), such as GPT-4 [1], are employed to gather data on the typical sizes of objects. For instance, we prompt GPT-4 with 'Please provide the (length, width, height) for objects of the <CLASS> category according to their typical sizes in real life.'

In our experiments, we find that GPT-4 can offer reliable information regarding the usual sizes of objects within specific categories, including their width, length, and height. When we replace the object sizes predicted by GPT-4 with the sizes statistically derived from the ground truth annotations in the training dataset, the performance is similar (see Tab. 5(d)). The predicted dimensions from GPT-4 are considered as the category priors. If the dimensions of a coarse pseudo box fall within the reasonable range of its corresponding category priors, the coarse pseudo box is considered a valid pseudo τ τ

Ray Tracing Loss

ℒ

!"#$%

=

𝑃 - 𝑃

&'!%"

(

𝑷

𝑷

𝐢𝐧𝐭𝐞𝐫

𝑷

𝑷

𝐢𝐧𝐭𝐞𝐫

Camera

Camera

(a)

(b)

Box A

Box C

Box D

Box A

<

Box C

<

Box D

Point Ratio Loss:

Box C

>

Box D

=

1 -

!"#	%&	'%()*+	()+(,-	*.-	/%0	(			)

!"#	%&	344	*.-	'%()*+	(			5			)

Point Ratio Loss

Camera

Camera

Figure 4: Ray tracing loss and point ratio loss. label. This range is set as 1 to 2 times the priors, where τ 1 represents the lower threshold and τ 2 is the upper threshold. Conversely, if it is too small or too large, it undergoes an optimal box search process.

Potential boxes are proposed based on the coarse pseudo box. Since the pseudo-LiDAR is constructed from the projection of points captured on the surface of the object, at least one edge of the coarse pseudo box should belong to the optimal pseudo box. Therefore, in the bird's-eye view, for each of the four corners of the coarse pseudo box, we propose two boxes for two possible directions with the class dimension priors (see the blue point and two blue boxes in Fig. 3(d)). That is to say, there are 8 proposal boxes for each coarse pseudo box.

We then select the best-fitting proposal box through a box-searching procedure. Given that monocular RGB cameras only capture the surfaces of objects facing the camera, a valid bounding box should have point clouds concentrated around the edges closer to the camera compared to those farther away. Therefore, we borrow the ray tracing loss from [38] to determine which box is proper. As shown in Fig. 4(a), the loss calculates the distance between each LiDAR point and the nearest intersection point between the proposal box and the camera ray to that point. However, calculating the ray tracing loss alone is not sufficient. Since the calculation of the ray tracing loss only includes points where the camera ray intersects with the box, there can sometimes be shortcuts. As shown in Fig. 4(b), Box C has less ray tracing loss, but it is clearly not what we want. Thus, we propose adding a point ratio loss that considers the ratio of the point cloud enclosed by the box to the overall object point cloud. The overall box-searching loss is the sum of these two losses:

where N inside and N all denote the number of points inside the pseudo box and the total number of points, λ is the hyper-parameter that balances the contribution of two losses. For each of the objects, the proposal box with the lowest box-searching loss is considered the pseudo label for training.

3.3 Training Detector on Auto-Generated Pseudo Labels

We adopt Cube R-CNN [3] as our basic 3D monocular detector, and replace its classification branch with a text-alignment head that aligns the output object features with text embeddings. The aligning loss L aligning = L CE ( c i , y i ) , where c i = f i · t represents the dot product of the object feature f i and text embeddings t , and y i denotes the pseudo label predicted by Grounded-SAM. The training objective is defined as: L train = L localization + L aligning .

4 Experiments

We first evaluate the proposed OVM3D-Det model on outdoor datasets, KITTI [14] and nuScenes [5], as well as indoor datasets, SUN RGB-D [52] and ARKitScenes [2] (Sec. 4.2). We then provide thorough ablation studies and detailed analysis to uncover the factors contributing to the effectiveness of OVM3D-Det (Sec. 4.3). Finally, we show qualitative results of our OVM3D-Det model (Sec. 4.4).

4.1 Experiment Setup

Datasets. We evaluate OVM3D-Det on the KITTI [14] and nuScenes [5] datasets for outdoor settings, and on the SUN RGB-D [52] and ARKitScenes [2] datasets for indoor settings. Note that neither of these datasets appeared in the training data of Unidepth [40], so there is no data leakage . KITTI [14] has 7,481 images for training and 7,518 images for testing. Since the official test set is unavailable, we follow [3] to resplit the training set into 3,321 training images, 391 validation images, and 3,769 test images. For nuScenes [5], we use 26,215 images for training, 1,915 images for validation, and 6,019 images for testing. SUN RGB-D [52] consists of a total of 10k samples, each annotated with oriented 3D bounding boxes, in which 4,929 samples are used for training, 356 samples for validation, and 5,050 samples for test following [3]. ARKitScenes [2] includes 48,046 images for training, 5,268 images for validation, and 7,610 images for testing.

Metrics. We report 3D average-precision ( AP 3D ) following Cube R-CNN [3] for fair comparison. Predictions are matched to ground truth by calculating the intersection-over-union ( IoU 3D ) of 3D bounding boxes. We report mAP 3D across all categories over the IoU 3D thresholds from 0.05 to 0.50 at a 0.05 step. We split the dataset into base and novel categories to evaluate open-vocabulary performance. In KITTI, there are 2 base classes (car, pedestrian) and 3 novel classes (cyclist, van, truck); nuScenes has 3 base and 6 novel classes; SUN RGB-D includes 20 base and 18 novel classes; and ARKitScenes has 8 base and 6 novel classes. Please refer to Sec. A for more details. We would like to note that we do not need to split the datasets into base and novel splits to train our model. We introduce this base/novel setting only to compare our method with the proposed baselines due to the lack of a suitable baseline before.

Implementation Details. We prompt the off-the-shelf Grounded-SAM [47] to detect and segment objects in new categories. For the depth estimation, we adopt the state-of-state-art monocular depth estimation model, Unidepth [40]. Both models are used directly for inference without any fine-tuning. We train on the monocular 3D object detection model Cube R-CNN [3] with the auto-generated pseudo labels, modifying the classification branch to output a 768-dimensional feature and following most of its optimization hyper-parameters. See Sec. E for more details.

4.2 Main Results

Baselines. To the best of our knowledge, we are the first to explore open vocabulary monocular 3D object detection models. Therefore, to better evaluate the proposed task, we introduce the following baselines: 1) We train the Cube R-CNN [3] with all dataset annotations, including both base and novel classes, to establish the oracle baseline. 2) We develop another baseline to enable Cube R-CNN with open vocabulary capabilities by utilizing Grounding DINO. Specifically, we first use the base class annotations of the training set to train a class-agnostic Cube R-CNN detection model. This enables it to localize the instances from the novel classes during testing, without being limited to a fixed set of categories. During testing, we use Cube R-CNN to predict 3D bounding box proposals and use Grounding DINO to obtain 2D detection results from the images. We then back-project the 3D bounding boxes onto the image frames and match them with the predictions of Grounding DINO. The matched 2D detection boxes from Grounding DINO are then used to assign categories to the corresponding 3D bounding boxes. 3) Although our OVM3D-Det model does not require any annotations for training, we provide a baseline for comparison that is trained using ground-truth annotations for base classes and pseudo labels for novel classes, denoted by OVM3D-Det ∗ .

Results. We report the open-vocabulary monocular 3D object detection results on outdoor and indoor datasets in Tab. 1 and Tab. 2, respectively. It can be observed that the unsupervised OVM3D-Det model significantly outperforms the baseline on novel categories. Specifically, it achieves a 6.7%

improvement in average precision (AP) on the KITTI dataset, 9.7% AP on the nuScenes dataset, 8.5% AP on the SUN RGB-D dataset, and 16.8% AP on the ARKitScenes dataset. Besides, the OVM3D-Det model trained on the ground truth annotations of the base class and the pseudo labels of the novel class also surpasses the baseline by a large margin on the novel category. This indicates that our framework does indeed generate accurate pseudo-labels for various classes, thereby enhancing the model's performance in the novel category.

In Fig. 6, we show the model performance varying with the amount of training data on KITTI. As the training data increases, we observe a continuous improvement in performance. This highlights the strong potential of our model, suggesting that it can be trained to achieve more generalized open-vocabulary capabilities when using more readily available image data.

Comparison with Point Cloud-Based Methods. To compare our method with state-of-the-art point cloud-based open-vocabulary 3D detection approaches, we first simply evaluate existing open vocabulary detector OV-3DET [30] that is trained and tested on point cloud data using pseudo-LiDAR as input. Specifically, we only replace the point cloud input of OV-3DET with the pseudo-LiDAR which is unprojected from the RGB images and estimated depth maps. Other parts of the model in OV-3DET are kept untouched for fair comparison. As shown in Tab. 3, directly changing the input data format leads to poor performance with only 3.4% AP. This phenomenon is also observed and discussed in previous works [36, 66, 60], which we attribute to the distribution shifts between real point clouds and pseudo-LiDARs.

After being trained with pseudo-LiDAR, OV-3DET improves from 3.4% to 7.1% AP, marking a 2x improvement compared to the original model. CODA [6] shows a slight improvement compared

to OV-3DET. Nonetheless, the 10.8% AP of our method still demonstrates our superiority, since our approach focuses on adapting to the noisy nature of pseudo-LiDAR, whereas previous openvocabulary 3D detectors are mainly designed for real point clouds to generate pseudo labels. The key designs that distinguish our method from other baselines are the adaptive pseudo-LiDAR erosion and bounding box refinement techniques, incorporating the prior knowledge of language models. Our method can generate pseudo labels more effectively for pseudo-LiDAR than other baselines.

For outdoor 3D detection, we also try to establish another baseline since OV-3DET [30] only includes results on indoor data. We attempt to train OV-3DET on the KITTI dataset with pseudo-LiDAR. As presented in Tab. 4, simply training on pseudo-LiDAR points causes a rather weak performance of only 1.3% AP. This result reflects the nature of outdoor data that the vast range of spatial scale could lead to intolerable pseudo-label errors from tiny noise. Therefore, it is challenging to generate reasonable and reliable pseudo boxes due to the inaccurate object edges of the raw pseudo-LiDAR.

4.3 Ablation Studies

We conduct ablation studies on the KITTI dataset. The results are shown in Tab. 5. We report the mean AP performance across all the classes for each model.

Comparison with Naive Framework. If we remove the refinement applied to pseudo-LiDAR and bounding box (i.e., directly utilizing raw pseudo-LiDAR and tightly bounding it to obtain the box), there is a significant decrease of 11.2% AP in model performance. This result shows the effectiveness of our framework in addressing the noisy pseudo-LiDAR, where the refinement of both the pseudo-LiDAR data and the estimated bounding boxes proves to be pivotal.

Mask Erosion. We replace the adaptive erosion process with a fixed kernel erosion, resulting in a decline of 2.1% AP. It demonstrates the effectiveness of our proposed adaptive erosion module, indicating that a standard fixed kernel may not be suitable for all cases, as it could overly erode small objects or be inadequate for larger objects with substantial noise.

Orientation Estimation. Here, we compare two methods for determining the orientation of boxes: using Principal Component Analysis (PCA) or calculating the distribution of the directions of each pair of points in the pseudo-LiDAR. We find that the simple and efficient PCA method performs better, exceeding 1.8% AP.

Dimension Priors. The dimension priors of objects are an essential component of our method. They not only assist in identifying unreasonable predictions of bounding boxes but also aid in correcting the sizes of the bounding boxes. As shown in Tab. 5(d), the absence of these priors leads to a significant decrease of 4.5% AP. Notably, when we utilize priors derived from a large language model or dataset statistics, the results are similar. In Sec. B, we provide a thorough comparison of LLM-generated priors and shape priors obtained from dataset statistics across all datasets. This suggests that the large language model possesses a strong commonsense understanding, and further underscores the

(b) KITTI

0

20

40

60

80

100

data percent (%)

14

15

16

17

18

19

AP

potential of our approach to be transferred to a wide range of data. We also test the sensitivity of the lower threshold τ 1 and the upper threshold τ 2 used for determining the reasonability of a box. As shown in Tab. 5(f) and (g), they are stable within a wide range.

Box Search Loss. We further explore the impact of different loss functions on evaluating the bounding box proposals. The results are presented in Tab. 5(e). Using ray tracing loss alone results in inadequate performance, yielding only a 12.2% AP. However, adding the point ratio loss significantly boosts performance to 18.5% AP.

Self-Training. Additionally, we can refine the model using a self-training approach, which involves using the results from the previously well-trained model as pseudo labels in the subsequent training process. As shown in the Tab. 6, self-training can further enhance the quality of the initially generated pseudo boxes, improving detection performance for objects at various distances.

4.4 Qualitative Results

The qualitative results are presented in Fig. 5. We prompt the OVM3D-Det model with text queries of the desired categories, obtaining corresponding detection boxes. Results show its capability to locate and recognize objects of diverse types and sizes in both indoor and outdoor scenarios.

5 Conclusion

In this paper, we introduced OVM3D-Det, the first framework targeting image-based 3D openvocabulary detection. OVM3D-Det automatically labels open-vocabulary 3D bounding boxes using only RGB images, exploring the great potential of leveraging omnipresent image data from the internet. To address the noisy point clouds and occluded target objects in predicted pseudo-LiDAR, we devise two innovative solutions: adaptive pseudo-LiDAR erosion and bounding box search using world knowledge. Our open-vocabulary detector trained under OVM3D-Det outperforms the baselines by at least +6.7% AP and up to +16.8% AP. In a word, we propose a simple, yet surprisingly effective baseline method for image-based 3D open-vocabulary detection and we hope it could be a new study point for future research.

Acknowledgement. This work is supported in part by the National Key R&D Program of China under grants 2022ZD0114903.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023. 2, 6, 19

Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, et al. Arkitscenes: A diverse realworld dataset for 3d indoor scene understanding using mobile rgb-d data. arXiv preprint arXiv:2111.08897 , 2021. 7, 18, 19

Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, and Georgia Gkioxari. Omni3d: A large benchmark and model for 3d object detection in the wild. In CVPR , 2023. 3, 6, 7, 8, 19

Garrick Brazil and Xiaoming Liu. M3d-rpn: Monocular 3d region proposal network for object detection. In ICCV , 2019. 2, 3

Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR , 2020. 7, 18, 19

Yang Cao, Zeng Yihan, Hang Xu, and Dan Xu. Coda: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection. In NeurIPS , 2023. 1, 3, 8

Devendra Singh Chaplot, Murtaza Dalal, Saurabh Gupta, Jitendra Malik, and Russ R Salakhutdinov. Seal: Self-supervised embodied active learning using exploration and 3d consistency. In NeurIPS , 2021. 1

Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang. Clip2scene: Towards label-efficient 3d scene understanding by clip. In CVPR , 2023. 3

Xiaozhi Chen, Kaustav Kundu, Ziyu Zhang, Huimin Ma, Sanja Fidler, and Raquel Urtasun. Monocular 3d object detection for autonomous driving. In CVPR , 2016. 2, 3

Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR , 2017. 1

Runyu Ding, Jihan Yang, Chuhui Xue, Wenqing Zhang, Song Bai, and Xiaojuan Qi. Pla: Language-driven open-vocabulary 3d scene understanding. In CVPR , 2023. 3

Djamahl Etchegaray, Zi Huang, Tatsuya Harada, and Yadan Luo. Find n'propagate: Openvocabulary 3d object detection in urban environments. arXiv preprint arXiv:2403.13556 , 2024. 3

Lue Fan, Xuan Xiong, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang. Rangedet: In defense of range view for lidar-based 3d object detection. In ICCV , 2021. 1

Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR , 2012. 1, 7, 18, 19

Yongjie Guan, Xueyu Hou, Nan Wu, Bo Han, and Tao Han. Deepmix: mobility-aware, lightweight, and hybrid 3d object detection for headsets. In MobiSys , 2022. 1

Vitor Guizilini, Igor Vasiljevic, Dian Chen, Rares , Ambrus , , and Adrien Gaidon. Towards zero-shot scale-aware monocular depth estimation. In ICCV , 2023. 2, 3

Lei Han, Tian Zheng, Yinheng Zhu, Lan Xu, and Lu Fang. Live semantic 3d perception for immersive augmented reality. TVCG , 2020. 1

Jiawei He, Yuqi Wang, Yuntao Chen, and Zhaoxiang Zhang. Weakly supervised 3d object detection with multi-stage generalization. arXiv preprint arXiv:2306.05418 , 2023. 3

Rui Huang, Songyou Peng, Ayca Takmaz, Federico Tombari, Marc Pollefeys, Shiji Song, Gao Huang, and Francis Engelmann. Segment3d: Learning fine-grained class-agnostic 3d segmentation without manual labels. In ECCV , 2024. 3

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML , 2021. 3

Xueying Jiang, Sheng Jin, Lewei Lu, Xiaoqin Zhang, and Shijian Lu. Weakly supervised monocular 3d detection with a single-view image. In CVPR , 2024. 3

Linh Kästner, Vlad Catalin Frasineanu, and Jens Lambrecht. A 3d-deep-learning-based augmented reality calibration method for robotic environments using depth sensor data. In ICRA , 2020. 1

Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In ICCV , 2023. 3

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In ICCV , 2023. 3, 4, 19

Zhichao Li, Feng Wang, and Naiyan Wang. Lidar r-cnn: An efficient and universal 3d object detector. In CVPR , 2021. 1

Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's-eye-view representation from multi-camera images via spatiotemporal transformers. In ECCV , 2022. 1

Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR , 2017. 18

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In ECCV , 2024. 4, 8, 19

Yuxuan Liu, Yuan Yixuan, and Ming Liu. Ground-aware monocular 3d object detection for autonomous driving. RA-L , 2021. 2, 3

Yuheng Lu, Chenfeng Xu, Xiaobao Wei, Xiaodong Xie, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. Open-vocabulary point-cloud object detection without 3d annotation. In CVPR , 2023. 1, 3, 8, 9

Chen Min, Dawei Zhao, Liang Xiao, Jian Zhao, Xinli Xu, Zheng Zhu, Lei Jin, Jianshu Li, Yulan Guo, Junliang Xing, et al. Driveworld: 4d pre-trained scene understanding via world models for autonomous driving. In CVPR , 2024. 1

Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-end transformer model for 3d object detection. In ICCV , 2021. 1

Arsalan Mousavian, Dragomir Anguelov, John Flynn, and Jana Kosecka. 3d bounding box estimation using deep learning and geometry. In CVPR , 2017. 2, 3

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. TMLR , 2023. 3

Xuran Pan, Zhuofan Xia, Shiji Song, Li Erran Li, and Gao Huang. 3d object detection with pointformer. In CVPR , 2021. 1

Dennis Park, Rares Ambrus, Vitor Guizilini, Jie Li, and Adrien Gaidon. Is pseudo-lidar needed for monocular 3d object detection? In ICCV , 2021. 3, 8

Liang Peng, Junkai Xu, Haoran Cheng, Zheng Yang, Xiaopei Wu, Wei Qian, Wenxiao Wang, Boxi Wu, and Deng Cai. Learning occupancy for monocular 3d object detection. In CVPR , 2024. 3

Liang Peng, Senbo Yan, Boxi Wu, Zheng Yang, Xiaofei He, and Deng Cai. Weakm3d: Towards weakly supervised monocular 3d object detection. In ICLR , 2022. 3, 5, 6

Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Marc Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In CVPR , 2023. 3

Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In CVPR , 2024. 2, 3, 4, 7, 19

Mihir Prabhudesai, Hsiao-Yu Fish Tung, Syed Ashar Javed, Maximilian Sieb, Adam W Harley, and Katerina Fragkiadaki. Embodied language grounding with 3d visual feature representations. In CVPR , 2020. 1

Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. In ICCV , 2019. 1

Rui Qian, Divyansh Garg, Yan Wang, Yurong You, Serge Belongie, Bharath Hariharan, Mark Campbell, Kilian Q Weinberger, and Wei-Lun Chao. End-to-end pseudo-lidar for image-based 3d object detection. In CVPR , 2020. 3

Zequn Qin and Xi Li. Monoground: Detecting monocular 3d objects from the ground. In CVPR , 2022. 2, 3

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021. 3

René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. TPAMI , 2020. 3

Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159 , 2024. 4, 5, 7, 19

Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Imvoxelnet: Image to voxels projection for monocular and multi-view general-purpose 3d object detection. In WACV , 2022. 1

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV , 2015. 18

Yichao Shen, Zigang Geng, Yuhui Yuan, Yutong Lin, Ze Liu, Chunyu Wang, Han Hu, Nanning Zheng, and Baining Guo. V-DETR: DETR with vertex relative position encoding for 3d object detection. In ICLR , 2024. 1

Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-rcnn: Point-voxel feature set abstraction for 3d object detection. In CVPR , 2020. 1

Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In CVPR , 2015. 1, 7, 18, 19

Ayça Takmaz, Elisabetta Fedele, Robert W Sumner, Marc Pollefeys, Federico Tombari, and Francis Engelmann. Openmask3d: Open-vocabulary 3d instance segmentation. In NeurIPS , 2023. 3

Runzhou Tao, Wencheng Han, Zhongying Qiu, Cheng-Zhong Xu, and Jianbing Shen. Weakly supervised monocular 3d object detection using multi-view projection and direction consistency. In CVPR , 2023. 3

Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xiangyu Zhang. Exploring object-centric temporal modeling for efficient multi-view 3d object detection. In ICCV , 2023. 1

Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In CVPR , 2023. 2, 3

Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger. Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In CVPR , 2019. 2, 3, 4

Yan Wang, Bin Yang, Rui Hu, Ming Liang, and Raquel Urtasun. Plumenet: Efficient 3d object detection from stereo images. In IROS , 2021. 3

Zhenyu Wang, Yali Li, Taichi Liu, Hengshuang Zhao, and Shengjin Wang. Ov-uni3detr: Towards unified open-vocabulary 3d object detection via cycle-modality propagation. In ECCV , 2024. 1, 3

Xinshuo Weng and Kris Kitani. Monocular 3d object detection with pseudo-lidar point cloud. In ICCVW , 2019. 8

Zhenyu Wu, Ziwei Wang, Xiuwei Xu, Jiwen Lu, and Haibin Yan. Embodied task planning with large language models. arXiv preprint arXiv:2307.01848 , 2023. 1

Qian Xie, Yu-Kun Lai, Jing Wu, Zhoutao Wang, Yiming Zhang, Kai Xu, and Jun Wang. Mlcvnet: Multi-level context votenet for 3d object detection. In CVPR , 2020. 1

Longfei Yan, Pei Yan, Shengzhou Xiong, Xuanyu Xiang, and Yihua Tan. Monocd: Monocular 3d object detection with complementary depths. In CVPR , 2024. 3

Chenyu Yang, Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang, Hongyang Li, Yu Qiao, Lewei Lu, et al. Bevformer v2: Adapting modern image backbones to bird's-eye-view recognition via perspective supervision. In CVPR , 2023. 1

Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR , 2024. 2, 3

Xiaoqing Ye, Liang Du, Yifeng Shi, Yingying Li, Xiao Tan, Jianfeng Feng, Errui Ding, and Shilei Wen. Monocular 3d object detection via feature domain adaptation. In ECCV , 2020. 8

Wei Yin, Chi Zhang, Hao Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang, Xiaozhi Chen, and Chunhua Shen. Metric3d: Towards zero-shot metric 3d prediction from a single image. In ICCV , 2023. 2, 3, 18, 19

Yurong You, Yan Wang, Wei-Lun Chao, Divyansh Garg, Geoff Pleiss, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger. Pseudo-lidar++: Accurate depth for 3d object detection in autonomous driving. In ICLR , 2020. 3

Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In CVPR , 2018. 18

Yihan Zeng, Chenhan Jiang, Jiageng Mao, Jianhua Han, Chaoqiang Ye, Qingqiu Huang, Dit-Yan Yeung, Zhen Yang, Xiaodan Liang, and Hang Xu. Clip2: Contrastive language-image-point pretraining from real-world point cloud data. In CVPR , 2023. 3

Hu Zhang, Jianhua Xu, Tao Tang, Haiyang Sun, Xin Yu, Zi Huang, and Kaicheng Yu. Opensight: A simple open-vocabulary framework for lidar-based object detection. In ECCV , 2024. 1, 3

Jiacheng Zhang, Jiaming Li, Xiangru Lin, Wei Zhang, Xiao Tan, Junyu Han, Errui Ding, Jingdong Wang, and Guanbin Li. Decoupled pseudo-labeling for semi-supervised monocular 3d object detection. In CVPR , 2024. 3

Qianfan Zhao, Lu Zhang, Lingxi Wu, Hong Qiao, and Zhiyong Liu. A real 3d embodied dataset for robotic active visual learning. RA-L , 2022. 1

Yunsong Zhou, Quan Liu, Hongzi Zhu, Yunzhe Li, Shan Chang, and Minyi Guo. Mogde: Boosting mobile monocular 3d object detection with ground depth estimation. In NeurIPS , 2022. 3

Chenming Zhu, Wenwei Zhang, Tai Wang, Xihui Liu, and Kai Chen. Object2scene: Putting objects in context for open-vocabulary 3d detection. arXiv preprint arXiv:2309.09456 , 2023. 1, 3

Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyu Guo, Ziyao Zeng, Zipeng Qin, Shanghang Zhang, and Peng Gao. Pointclip v2: Prompting clip and gpt for powerful 3d open-world learning. In ICCV , 2023. 1

Appendix

A Category Splits

The category splits for all datasets are shown in Tab. 7.

B Dimension Priors

Tab. 8 to 11 provide a comprehensive comparison between LLM-generated priors and shape priors derived from dataset statistics across all datasets. For clarity, we keep one decimal place. As we use shape priors solely to filter out pseudo boxes that may be unreliable due to noise or occlusion and to refine them, it suffices as long as the shape priors fall within a reasonable range.

C Image Erosion Process

We propose to employ the image erosion operation to eliminate inaccurate object mask edges, thereby obtaining accurate pseudo-LiDAR after unprojection. Image erosion is a fundamental operation in digital image processing that involves shrinking or wearing away the boundaries of foreground objects in a binary image. As shown in Fig. 7, M i is the binary mask of one object we interested, where each pixel M i ( x, y ) is either 0 (background) or 1 (foreground). Let B ( u, v ) represent the elements of the structuring element B , which is a small binary matrix of size m × n . Erosion of the mask M i by the structuring element B is denoted as M i ⊖ B .

In our experiment, we employ a 3x3 structuring element B where all elements are set to 1. The erosion operation can be applied repeatedly. The intensity of erosion can be controlled by adjusting the number of erosion iterations. We note that aggressive erosion may completely remove small objects like distant pedestrians, while mild erosion may leave considerable noise on large objects. Therefore, for masks with a smaller area, we use a smaller number of iterations; conversely, for masks with a larger area, we use a larger number of iterations, ensuring more accurate pseudo-LiDAR.

Eroded by a

3 x 3 kernel

𝑀

!

𝐵

D Performance Varies with Depth Estimation Quality

Besides Unidepth, we evaluate alternative depth estimation models, such as Metric3D [67]. As shown in Tab. 12, we find that the more accurate the depth estimation model we use, the better the performance of the trained open-vocabulary monocular 3D detection model, suggesting promising potential as depth estimation models advance.

E Implementation Details

When searching for the optimal box, we set λ to 5 for indoor scenes and 10 for outdoor scenes to balance the ray tracing loss and point ratio loss. For the adaptive erosion module, for outdoor scenes, if the maximum width of the instance mask exceeds 10 pixels, image erosion is performed for 4 iterations; otherwise, 2 iterations. For indoor scenes, the values are 12 and 2 iterations, respectively. As we aim to evaluate the benchmarks, we utilize the label lists from the datasets to query the open-vocabulary 2D model for generating pseudo labels of the classes of interest. Following Cube R-CNN, we utilize the DLA34 backbone [69]-FPN [27], which is pretrained on ImageNet [49]. We employ the SGD optimizer, with the learning rate decaying by a factor of 10 at 60% and 80% of the training process. During training, we apply random data augmentation techniques such as horizontal flipping and scaling within the range of [0.50, 1.25]. The model is trained for 29,000 iterations with a batch size of 32 and a learning rate of 0.02 on both SUN RGB-D [52] and ARKitScenes [2], a batch size of 16 and a learning rate of 0.01 on KITTI [14], and a batch size of 32 and a learning rate of 0.01 on nuScenes [5]. We train on the KITTI dataset for ∼ 8 hours with 2 A100 GPUs, on nuScenes for ∼ 6 hours with 4 A100 GPUs, on SUN RGB-D for ∼ 12 hours with 2 A100 GPUs, and on ARKitScenes for ∼ 20 hours with 2 A100 GPUs.

F Failure Case

(b)

(a)

(d)

(c)

We present some failure cases of OVM3D-Det in Fig. 8. (a), (b) and (c) are instances of distant objects being missed by detection. In (d), the reflection of a car on the window is mistakenly detected as a vehicle. As our framework relies on foundational depth estimation models for generating pseudo-LiDAR for auto-labeling, the accuracy of depth estimation directly affects label precision. Particularly, errors increase quadratically with distance, resulting in lower-quality pseudo-labels for distant objects and thus limiting our model's performance at longer ranges. This issue merits further investigation in our future research endeavors.

G Licenses

H Limitations

Our proposed OVM3D-Det framework relies on advanced depth estimation models to generate pseudoLiDAR. Although state-of-the-art depth estimation models have demonstrated strong generalization capabilities, their predictions for distant objects are not yet accurate enough, which may lead to errors when generating pseudo labels for these distant objects. Additionally, the performance of monocular systems may degrade under adverse lighting or weather conditions, as these factors directly affect the quality and clarity of the input images. These limitations underscore the need for future research to enhance the robustness and versatility of open vocabulary monocular 3D detection systems, particularly in challenging and dynamic environments.

I Broader Impact

The end results of this research are a novel approach to 3D detection that can be applied to autonomous driving, augmented reality, and other robotics applications. By enabling improved 3D object detection using monocular images, our framework has the potential to enhance the safety and accuracy of this system, contributing to the overall advancement of autonomous technologies. Before deployment, it is crucial to establish appropriate safety thresholds and conduct thorough testing to ensure reliability. While our approach does not explicitly exploit dataset biases, it is subject to the same limitations and potential biases as other machine learning techniques.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have included our claims and contributions in the introduction/abstract. Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: We have discussed the limitations throughout the paper when necessary. Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our paper does not include theoretical results. Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have disclosed the implementation information necessary for reproducibility in the main paper and the appendix as thoroughly as possible.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code will be released.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/ CodeSubmissionPolicy) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/ CodeSubmissionPolicy) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have provided experimental details in the experiments section of our paper. More details can be found in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The computing resources we have limit the possibility of obtaining the experiment statistical significance.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have indicated the compute resources required for this paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification: We understand and conform to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discussed the detailed broader impact in the appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for the responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We believe no such risk is involved. Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have credited and referenced the papers of the codes and models used in producing this work. We have read and follow the licenses of the works.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We will release well-documented assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing nor research with human subjects involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No crowdsourcing nor research with human subjects involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper introduces OVM3D-Det, an open-vocabulary monocular 3D object detection framework that utilizes only RGB images for training. It leverages open-vocabulary 2D models and pseudo-LiDAR to automatically label 3D objects in these images. Additionally, it incorporates pseudo-LiDAR erosion and bounding box refinement modules, enhanced by prior knowledge from large language models. Experimental results demonstrate the effectiveness of the proposed method.

STRENGTHS:
1. This paper is well-written. It’s innovative to introduce the LLM for assisting in the generation of Pseudo 3D Bounding Boxes.
2. The Box Orientation Estimation and Bounding Box Search each have their unique designs.

WEAKNESSES:
1. Although the experiments indicate improvements with the proposed method, the overall AP remains low, and the final performance heavily depends on the Depth Estimator used. Moreover, the experiments seem to lack performance metrics for Unidepth.

2. The effect of GPT-4 in this method is minimal, as it only provides object scaling. The costs and benefits of using it are not well-delineated.

QUESTIONS:
1. Does this framework heavily depend on the performance of Unidepth? What is the detection performance of Unidepth? How significant is the impact of missed objects, such as those at a distant range, on the final training performance?

2. What are the costs and benefits of using GPT-4?  Is the improvement from using LLM minimal in the ablation study? The performance is very close to using an average vehicle scaling size in the ablation study?

3. The ablation studies are confusing. What are the improvements of each component compared to the baseline?

LIMITATIONS:
According to the ablation study, the proposed modules show improvements.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes OVM3D-Det, an open-vocabulary monocular 3D object detection framework that trains detectors using only RGB images. It introduces two key designs: adaptive pseudo-LiDAR erosion and bounding box refinement, addressing challenges arising from the absence of LiDAR point clouds.\n\nSTRENGTHS:\n1.\tThe paper introduces an innovative open-vocabulary monocular 3D object detection framework that utilizes only RGB images for training.\n2.\tThe proposed method demonstrates superior performance compared to existing state-of-the-art approaches in both indoor and outdoor scenarios.\n\nWEAKNESSES:\n1.\tThe KITTI dataset has a limited number of classes (5 classes). It would be beneficial to provide results on the NuScenes dataset (23 classes) to better demonstrate the open-vocabulary performance.\n2.\tThe adaptive pseudo-LiDAR erosion module is not illustrated in sufficient detail. The method section discussing this module is vague; please provide more comprehensive details on its design.\n3.\tMore visualizations of the pseudo-LiDAR erosion module are needed. Currently, the visualization of this module is only included in the overall framework figure. Additional visualizations would better demonstrate its effectiveness.\n4.\tThe related work section on monocular 3D object detection is not up-to-date. It is recommended to cite recent papers accepted by CVPR 2024:\na)\tMonoCD: Monocular 3D Object Detection with Complementary Depths, CVPR24\nb)\tLearning Occupancy for Monocular 3D Object Detection, CVPR24\nc)\tWeakly Supervised Monocular 3D Detection with a Single-View Image, CVPR24\nd)\tDecoupled Pseudo-labeling for Semi-Supervised Monocular 3D Object Detection, CVPR24\n5.\tFor Table 1 and Table 2, it would be helpful to include a baseline trained without ground-truth annotations for base classes to provide a better comparison.\n\nQUESTIONS:\nPlease refer to the weakness section.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel open-vocabulary monocular 3D object detection framework named OVM3D-Det. This approach aims to train 3D object detectors using only RGB images, making it cost-effective and scalable. The framework utilizes pseudo-LiDAR and large language models to generate pseudo 3D labels, enabling training without high-precision LiDAR data. The authors propose adaptive pseudo-LiDAR erosion and bounding box refinement to improve label accuracy. Extensive experimentation demonstrates the superiority of OVM3D-Det over existing baselines in both indoor and outdoor scenarios.\n\nSTRENGTHS:\n- **Innovative Approach**: The paper introduces a novel method for open-vocabulary 3D object detection using only RGB images, which significantly reduces the cost and complexity of data acquisition.  \n &nbsp; \n- **Effective Techniques**: The adaptive pseudo-LiDAR erosion and bounding box refinement with prior knowledge from large language models are innovative solutions that address the challenges of noisy point clouds and occluded objects.  \n&nbsp;\n- **Comprehensive Experiments**: The authors provide extensive experimental results, demonstrating the effectiveness of their approach across different datasets and scenarios.  \n&nbsp;\n- **Broad Applicability**: The proposed method has broad applications in autonomous driving, robotics, and augmented reality, making it a valuable contribution to the field.\n\nWEAKNESSES:\n- **Dependence on Depth Estimation Quality**: The performance of the proposed method heavily relies on the quality of the depth estimation model, which may not always be accurate, especially for distant objects.  \n &nbsp; \n- **Limited Evaluation on Adverse Conditions**: The paper does not extensively compare the method with other indoor OV-3Det methods [1, 2]  \n &nbsp; \n- **Complexity of Implementation**: The framework involves several sophisticated components, including depth estimation, pseudo-LiDAR generation, and large language models, which might complicate implementation and reproducibility.  \n &nbsp;\n\n[1] Yuheng Lu, Chenfeng Xu, Xiaobao Wei, Xiaodong Xie, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. Open-vocabulary point-cloud object detection without 3d annotation. In CVPR, 2023. 1, 3  \n[2] Yang Cao, Zeng Yihan, Hang Xu, and Dan Xu. Coda: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection. In NeurIPS, 2023\n\nQUESTIONS:\n- How does the performance of the proposed method vary with the quality of the depth estimation model? Can the authors provide more insights or quantitative analysis on this aspect?  \n &nbsp; \n- Have the authors considered comparing the indoor OV-3Det methods[1, 2]?  \n &nbsp; \n- Can the authors provide more details on the computational requirements and efficiency of the proposed framework? How does it compare with point cloud-based methods in terms of computational costs?  \n &nbsp; \n- How generalizable is the proposed method to other types of data or domains beyond those tested in the experiments?  \n &nbsp; \n- Are there any specific challenges or limitations in integrating the proposed method into real-world applications such as autonomous driving systems?  \n &nbsp;\n\n[1] Yuheng Lu, Chenfeng Xu, Xiaobao Wei, Xiaodong Xie, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. Open-vocabulary point-cloud object detection without 3d annotation. In CVPR, 2023. 1, 3  \n[2] Yang Cao, Zeng Yihan, Hang Xu, and Dan Xu. Coda: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection. In NeurIPS, 2023\n\nLIMITATIONS:\nThe authors have provided discussions about the limitations and broader impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents OVM3D-Det, a framework for generating 3D bounding box pseudo-labels from monocular RGB images using foundational vision models like GroundingSAM and UniDepth. Using these pseudo-labels, authors train an open-vocabulary version of Cube-RCNN. Authors demonstrate that their proposed approach generalizes to novel categories unseen during training.\n\nSTRENGTHS:\nProblem Motivation. Open-vocabulary 3D detection is an important problem of interest for a wide variety of applications. This is a challenging open problem which requires further study. \n\nSimple Approach. Authors propose a simple and straightforward approach for generating 3D bounding box pseduo-labels and training an open-vocabulary 3D detector. \n\nWell Written. The paper is written clearly and the provided figures (particularly Figure 3) clearly explain the proposed approach.\n\nWEAKNESSES:\nLimited Evaluation. Authors claim that there are no prior methods that address this problem. However, since the proposed method trusts monocular depth estimates as if they were ground truth LiDAR, one can simply evaluate existing RGB + LiDAR open vocabulary detector [1] using pseudo-lidar. Notably, there has been significant prior work in open-vocabulary 3D detection which should be acknowledged in this work. \n\n\"Closed World\" Setup. Splitting datasets into base vs. novel splits goes against the setup of open-vocabulary detection. Notably, in 2D open vocabulary detection, methods pre-train on diverse datasets like V3Det, Objects365, ODinW, and report zero-shot results on all classes in COCO. A similar setup here would be to train on diverse 3D datasets and report zero-shot results on all classes in KITTI and SUNRGBD.\n\nPseudo-Labeler Heuristics. The design of the pseudo-labeler is extremely brittle, particularly because it relies on hand designed heuristics. I would not expect such a pseudo-labeler to work well on COCO (despite CubeRCNN doing a reasonable job). \n\nSmall Scale Evaluation. Both SUNRGBD and KITTI are very small 3D detection datasets by modern standards. These datasets alone are not enough to study open-vocabulary detection. I would encourage authors to train on and evaluate with more diverse datasets. Notably, Omni3D, the dataset the accompanies CubeRCNN, already provides large-scale training data. Why were the other datasets in Omni3D not used?\n\n[1] Open-Vocabulary Point Cloud Object Detection without 3D Annotation. Lu et. al. CVPR 2023.\n\nQUESTIONS:\nLimitations of Base and Novel Splits. Splitting individual datasets into base and novel splits no longer makes sense in the era of foundation models. Explicitly trying to avoid data leakage will prevent methods from using the latest and greatest models, artificially hampering progress. The best way to address this issue is by evaluating on a sufficiently large and diverse test set such that even if a small part of the data has already been seen by some model (e.g. like UniDepth), the aggregate performance is still a reliable indicator of overall test performance. \n\nRealistic Benchmarking on Out of Distribution Datasets. Similar to 2D open-vocabulary detectors, it would be interesting to benchmark the performance of this approach on unseen datasets (not just unseen classes). For example, how might the proposed model perform on nuScenes if it was trained on KITTI? Notably, nuScenes has many classes not included in KITTI like construction vehicle and motorcycle. \n\nDimension Prior vs. Real Prior. Using LLMs to generate 3D shape priors is an interesting idea. Intuitively, LLMs seen many descriptions of object shapes on the web, so it makes sense that they can provide reasonable shape priors. How do the LLM shape priors (e.g. L, W, H) actually compare with the real shape priors learned from dataset statistics?\n\nWhy Pseudo-Label 3D Datasets? Although the premise of the paper makes sense, the execution is a bit confusing. In particular, both KITTI and SUNRGBD both already have 3D bounding box annotations. It doesn't make sense to pseudo-label these datasets. Instead, it makes more sense to try to pseudo-label datasets like COCO that cannot currently be used by existing 3D detectors.\n\nLIMITATIONS:\nYes, authors highlight that their proposed approach is reliant on depth estimation models, and is susceptible to their failure modes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}