LabelAny3D: Label Any Object 3D in the Wild
Authors: Jin Yao
Radowan Mahmud Redoy
Sebastian Elbaum
Matthew B. Dwyer
Zezhou Cheng
Venue: NeurIPS 2025 poster ()
Keywords: Open Vocabulary
3D Detection
Object Recognition
Primary area: applications

Abstract: Detecting objects in 3D space from monocular input is crucial for applications ranging from robotics to scene understanding. 
Despite advanced performance in the indoor and autonomous driving domains, existing monocular 3D detection models struggle with in-the-wild images due to the lack of 3D in-the-wild datasets and the challenges of 3D annotation. We introduce LabelAny3D, an analysis-by-synthesis framework that reconstructs holistic 3D scenes from 2D images to efficiently produce high-quality 3D bounding box annotations.
Built on this pipeline, we present COCO3D, a new benchmark for open-vocabulary monocular 3D detection, derived from the MS-COCO dataset and covering a wide range of object categories absent from existing 3D datasets. Experiments show that annotations generated by LabelAny3D  improve monocular 3D detection performance across multiple benchmarks, outperforming prior auto-labeling approaches in quality. These results demonstrate the promise of foundation-model-driven annotation for scaling up 3D recognition in realistic, open-world settings.

LabelAny3D: Label Any Object 3D in the Wild
Jin Yao" Radowan Mahmud Redoy"
Sebastian Elbaum
Matthew B. Dwyer Zezhou Cheng
University of Virginia
https://uva- computer vision lab. github. io/ Lo/LabelAny3D/
schair person
trisbes
elephant
Repesore
skateboard
glast
toilet
ebook
biicycle
pench
indre
g Carr
ssi tcase
banana
potted
plant truck
dining teble
umbrell la
teddy bear
Cal
Carbar puniiaat
pizza
Escorts will donut tv 1 vase
smnwlch
keyboard d zebra
kite CUP motor carccyle
orange
aink handbag
renerc
giraffe broccol cll -
Suoab
flove
backpack airplane
Kranng
bottle
tennis racket cake
clock
tra traffic light horse refrigerazor
bird apple
Figure 1: Samples from our proposed COCO3D benchmark.
Abstract
Detecting objects in 3D space from monocular input is crucial for applications ranging from robotics to scene understanding. Despite advanced performance in the indoor and autonomous driving domains, existing monocular 3D detection models struggle with in-the-wild images due to the lack of 3D in-the-wild datasets and the challenges of 3D annotation. We introduce LabelAny3D, framework that reconstructs holistic 3D scenes from 2D images to efficiently produce high-quality 3D bounding box annotations. Built on this pipeline, we present COCO3D. new benchmark for opca-vocyyylary monocular 3D detection, derived from the MS-COCO dataset and covering a wide range of object categories absent from existing 3D datasets. Experiments show that annotations generated by LabelAny3D improve monocular 3D detection performance across multiple benchmarks, outperforming prior auto-labeling approaches in quality. These results demonstrate promise foundation-mode modeel-drieeen annotation for scaling up 3D recognition in realistic, open-world settings.
analysis-by-symthsii
Introduction
Monocular 3D object detection--reoognizzin and localizing objects in 3D space from single RGB image-is an emerging research direction with wide-ranging applications in robotics, autonomous driving, and embodied AI. Compared approaches that rely on specialized sensors such LiDAR
"Equal contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

CE00300 Nawe (Ista00)
Lab
16.05
wio LahelAny3D
10.92
with LabelAny3D
w/o LahelAn333
(a) Omni3D: indoor or self-driving
(b) 3D annotations from LabelAny3D
(c) Monocular 3D Detection
Figure 2: Overview. (a) Omni3D [7] offers large-scale 3D annotations but primarily covers indoor and self-driving scenarios. (b) The proposed LabelAny3D reconstructs 3D scenes (left) to annotate objects in 3D (right). (c) Leveraging LabelAny3D pseudo-labels to train monocular 3D detector raises its Ap (average precision; higher is better) on both Omni3D novel categories [75] and our new COCO3D benchmark.
or multi-view stereo [46, 47, 11, 12, 77, 84, 80, 79, 52, 20], monocular methods are lightweight, accessible, and energy-efficient, making them well-suited for many real-world deployments (e.g., AR/VR wearables). Recent advances such as Cube R-CNN [7] have achieved strong 3D detection performance by training on large, high-quality datasets, while OVMono3D [75] extends this task to the open-vocabulary setting, aiming to detect arbitrary object categories in 3D from monocular views.
Despite progress, critical bottleneck has been highlighted in recent works [7, 75, 22, 27]: the need for large-scale 3D datasets with high-quality 3D bounding box annotations for effective training and evaluation. This aligns with broader trend observed in vision and language foundation models [1, 44, 60]: performance improves significantly with more diverse, realistic training data and high-quality supervision. However, unlike 2D image recognition or language tasks, existing 3D datasets [8, 23, 7, 4, 15] are still constrained in scalability, scene diversity, and geometric complexity. As illustrated in Figure 2a, Omni3D [7]- c the largest public dataset-is dominated by indoor [15, 4, 63, 2, 58] and autonomous driving scenes [24, 9], with limited coverage of common objects (e.g., animals). These constraints hinder both the training and evaluation of generalizable monocular 3D detectors. While prior works [7. 75, 78] demonstrate qualitative success on in-the- wild images from MS-COCO [40], the lack of 3D annotations prevents systematic and quantitative benchmarking. This motivates the central question in this work: how can we produce high- high-qqualty 3D annotations on natural images with minimal human supervision?
Two major challenges limit scalable 3D dataset construction. First, collecting 3D data is expensive: in-the-wild images rarely include depth, and LiDAR or depth sensors require costly hardware and careful calibration, making them far less scalable than RGB capture. Second, annotating 3D data is labor-intensive: labeling 3D bounding boxes requires significantly more effort than 2D annotations. Several prior works have attempted to address these issues. For instance, OVM3D-Dec [27] lifts 2D images into 3D using off-the-shelf metric depth estimation models [53, 6] generate pseudo-LiDAR data, and infer 3D bounding boxes using object size priors from large language models (e.g. average height for pedestrian 1.7 meters). While effective for objects with consistent sizes (e.g. cars), this approach struggles with categories exhibiting high intra-class variations (e.g., baby vs. adult elephant). also relies heavily on accurate metric depth prediction a task that remains fundamentally ill- posed when relying solely on 2D input, object appearance is entangled with both focal length and actual distance to the camera. Another approach, 3D Copy-Paste [22], augments 3D annotations by inserting synthetic 3D models into images, which however induces the sim-to-real challenge.
To address these limitations, we propose LabelAny3D, an automatic 3D annotation pipeline that efficiently generates 3D bounding boxes for objects across arbitrary categories. Unlike prior works [22, 27], LabelAny3D adopts an wattsssii-----ytthste
paradigm: it reconstructs the holistic 3D scene from monocular images and uses the synthesized representation to infer spatially consistent 3D object
annotations (Figure 2b and 3). Our framework is motivated by three key observations: (1) although metric depth estimation remains ill- ill-posed, relative depth estimation [66] is significantly more reliable and consistent; (2) recent advances in object-centric 3D reconstruction, powered by large-scale 3D shape datasets [17] and generative modeling techniques [72], have enabled accurate shape recovery; and (3) 2D vision foundation models [33, 44] offer strong generalization capabilities across diverse, in-the-wild visual domains.

---

By integrating diverse vision foundation models, LabelAny3D produces high-quality 3D annotations suitable for training monocular 3D models and constructing benchmarks with minimal human supervision. Our experiments demonstrate that the 3D annotations from our pipeline lead to consistent improvements in monocular 3D detection across multiple benchmarks (Figure 2c), outperforming existing auto-labeling methods [27]. We further introduce COCO3D, new benchmark curated from MS-COCO [40] validation images using our pipeline with human refinement, covering wide variety of everyday object categories encountered in the wild, as presented in Figure 1.
In summary, our key contributions are as follows:
We introduce LnyelAny3I,, an efficient 3D annotation pipeline that generates high-quality 3D bounding boxes on in-the-wild images in an analysis-by-syutheii manner.
We demonstrate that 3D annotations from LabelAny3D consistently improve monocular 3D detection performance, surpassing existing auto-labeling approaches.
We curate COCO3D, new benchmark for open-vocabulary monocular 3D detection, featuring diverse range of object categories beyond those covered in existing 3D datasets.
2
Related work
3D Datasets, Many datasets have been developed to support 3D detection. KITTI [23] and
nuScenes [8] focus on autonomous driving, offering LiDAR and camera data for object detection and tracking in urban environments. SUN RGB-D [63], Hypersim [58] and ARKitScenes [4] target indoor settings, capturing room-scale layouts with depth and semantic annotations. Objectron [2] leverage mobile devices to collect real-world 3D object scans, enabling fine-grained object-centric
learning. Omni3D [7] unifies multiple datasets to create large-scale benchmark for general 3D object detection, yet existing benchmarks remain limited in their coverage of diverse, open-world scenarios. In contrast, we propose to extend 3D datasets beyond indoor and autonomous driving domains. This enables broader generalization across diverse, in-the-wild scenarios.
Monocular 3D Detection. Early studies on this task focused predominantly on specialized applica- tions within either outdoor [14, 82, 83, 68, 13, 81, 26, 67] or indoor environments [16, 28, 49, 61, 35], particularly targeting autonomous driving and room layout estimation. The Omni3D dataset fa- cilitated Cube R-CNN [7] in pioneering unified monocular 3D detection. Subsequently, Uni- MODE [39] extended these advances by proposing the first successful BEV-based detector applicable across diverse environments. Despite these achievements, most existing methods are constrained by closed vocabularies. Recent advancements in open-vocabulle vocabulary 3D detec- tion [46, 47, 11, 12, 77, 84, 80, 79, 52, 20, 69] primarily focus on utilizing 3D point clouds, while OVMono3D [75] first explores the open vocabulary 3D detection task with only monocular image as input. De [78] further boosts performance through superior 2D priors and extensive training data. However, these models still face challenges with generalization to in-the-wild imagery like MS-COCO [40], highlighting the ongoing issue of domain coverage limitations in training datasets.
Label-efficient 3D Detection. Due to the high cost of 3D labeling, previous studies have investigated approaches to reduce the reliance on 3D supervision in monocular 3D detection tasks [27, 71, 31, 10, 25]. For instance, Huang et al. [27] utilizes open-vocabulary 2D models and pseudo-LiDAR to automatically annotate 3D objects in RGB images, while 3D Copy-Paste [22] inserts synthetic 3D object shapes into 2D images to augment 3D annotations. Additionally, SKD-W W.D-WM3D [31] introduces
monocular 3D detection framework by distilling knowledge from pre-trained depth estimation models. While these supee-ss methods have proven effective, their application is often limited to indoor or autonomous driving contexts. In this work, we aim to develop novel techniques to broaden the domain coverage of 3D detection models.
Data Labeling. Model-in-the- enhance and accelerate data annotation. Prior works have explored this methodology across various tasks. Stereo4D [32] utilizes stereo depth estimation [42] and 2D tracking [19] models to construct 3D trajectory datasets. Cap3D [48] employs vasion-language models such as BLIP2 [38], CLIP [54], and GPT-4 [1] to develop a scalable pipeline for captioning 3D assets. DynPose [59] creates dynamic camera pose datasets by integrating advanced tracking [19] and masking [55, 29] models. Unlike these approaches, our work focuses on the task of 3D bounding box annotation from single-view images. By incorporating advanced models, such as monocular depth estimation [66, 6] and image-to-3D reconstruction [72], pipeline enables accurate and efficient 3D labeling.
data labeling leverages foundation models
3

---

(a) 2D Image Processing
Instance Segmentation
Amodal Completion
Inpost Image
Image super-rerolation
(b) Monocular 3D Scene Reconstruction
2D-3D Alignment
3D Reconstructio (n Canonical Space
3D Scene Geometry Estimation
Mulá- Views of Aligndd 3D objecty
(c) 3D Annotation Generation
Mult-VViwws. of 3D scone recorstruction
Figure 3: LabelAny3D. (a) Given an image, we first extract high-resolution object crops; (b) A holistic 3D scene is then built from robust depth estimation, 3D object reconstruction, and 2D-3D) alignment algorithms, (c) Lastly, 3D labels can be easily extracted from the reconstructed 3D scene.
3 LabelAny3D: Automatic 3D Labeling via 3D Reconstruction
This section details the proposed Any33 annotation pipeline. As shown in Figure 3, the pipeline generates pseudo annotations from an input image through the following steps.
Image Super-resolution. Many objects in MS-COCO [40] appear at low resolution due factors such as small object scale or compression artifacts, which poses challenges for downstream tasks like 3D reconstruction. To address this, we leverage InvSR [76], diffusion-based super-resolution (SR) model, to enhance the input image by a factor x4. This improves perceptual quality by recovering fine details and sharpening object boundaries. Given an input image I R #x the enhanced image is SS E B4 x4Wx3
2D Instance Segmentation. Prior studies have shown that the ground truth segmentation masks MS-COCO [40] often exhibit annotation errors. To mitigate this, we leverage the COCONut dataset [18]. which provides refined and high-quality segmentation masks built upon the MS-COCO [40] annotations. Given an image and its super-resolved version SR we compute the intersection of each object mask M with boundary mask and if the total intersection of object exceeds a threshold, the object is considered truncated and excluded. Next, each mask M is upscaled to MRR R x4W using nearest-neighbor interpolation to match the resolution of fSR We also remove any object whose post-processed mask area is below a threshold, as such objects are too small for reliable geometry. Using SR and SR extract the enhanced crop of the target object.
handle occluded objects, we adopt an amodal completion strategy inspired by prior work in Gen3DSR [3]. We leverage the learned amodal comple- tion diffusion model from Gen3DSR to inpaint the missing regions the object crop, generating a completed version Ocomp of the targeted object. With O Ocomp apply single-view 3D reconstruction methods (e.g., TRELLIS [72]) to recover the full 3D mesh in a canonical pose with normalized scale.
Amodal Completion 3D Reconstruction.
Scene Geometry Estimation. We utilize an affine- invariinv representation of the scene geometry from MoGe [66], and another from metric depth estimation model, such as Depth Pro [6]. To recover the 3D geometry, the MoGe depth map is aligned to the scale and perspective of the metric depth map which is considered to represent real-world geometry [5, 3]. This step ensures that depth values from MoGe are scaled and transformed to match the metric scale and viewpoint of the target scene. Next, the aligned depth map is unprojected into space using the camera intrinsic matrix provided by the MoGe model, recover the full 3D structure of the target scene.

---

Pose Estimation via 2D-3D Aligmment. After obtaining 2D object regions Comp and 3D reconstrue- tions, we localize the 3D objects within the scene by estimating its pose relative to the input image. This is achieved through dense correspondence matching between the real image and set of rendered views of the object mesh Mesh We adopt MASt3R [37] to compute 2D-2D correspondences between the super-resolver ved real image and the rendered views. Let xg E IR² denote pixel coordinates in the real image, and 21 E IR² the rendered view. These matched keypoints are filtered near image borders and invalid depth regions. Using the rendered depth map along with known intrinsics and extrinsics from rendering, we unproject æ to obtain corresponding 3D points X. E IR'S on the Meshg. Given the resulting 3D-2D correspondences (Xe, Zo) and camera intrinsics K inferred from MoGe [66], we apply a Perspective-ne n-Pint (PnP) solver [36] with RANSAC [21] to estimate the relative camera pose (). The object is then transformed into the input image's 's coordinate frame using this pose, yielding a reconstruction aligned with the relative layout of the original scene.
Scale Estimation via Depth Alignment. To recover metric scale, we align the rendered object to the real scene using depth-based scale estimation. Specifically, given the binary mask M from segmentation and the rendered mask Mrender obtained using the previously estimated pose, we compute an overlap region Q = M n Mrender Letting Dreal and Drender denote the real and rendered depth maps from the same pose, respectively, we estimate the scale factor as the median depth ratio:
(S2) Drender (Fende((12)
median
The estimated scale s is applied to the rotation and translation parameters to form the final transfor- mation matrix. This transformation places the reconstructed 3D object into the metric-scale scene point cloud, completing the 3D scene reconstruction.
3D Annotation Generation. In this step, we uniformly sample a point cloud from the mesh surface, capturing the object's 's geometry in its posed state. As TRELLIS [72] generates objects in a canonical pose with the upright direction aligned to gravity, following prior works [27, 51], we align the vertical axis of the bounding box with this canonical upward direction. To estimate orientation and size, we project the point cloud onto the plane orthogonal to the upward axis and apply PCA to determine the dominant yaw. The point cloud is then rotated to align with the canonical axes, and a tight 31) bounding box is fitted around its extent. This yields the object's 3D bounding box attributes, including center position, orientation, and dimensions.
4 Downstream Applications
Section 4.1 introduces a new monocular 3D detection benchmark curated from MS-COCO [40].
Section 4.2 presents our training details using the pseudo labels generated by LabelAny3D.
4.1 COCO3D Benchmark
Human refinement. We apply LabelAny3D to curate benchmark for evaluating (open-vocabulary) monocular 3D detection models. To ensure the high quality of our 3D annotations, we employ a human-in-the-loww -ntto-ooo refinement process. Five annotators with prior 3D vision experience refined labels by adjusting bounding box dimensions, rotation, and center position based on the generated point maps. They also had the option to remove any generated bounding box they deemed excessively noisy. In addition, the annotators performed filtering step to exclude samples containing undesirable characteristics, such as reflections of objects on surfaces like mirrors, windows, or screens, as well as 2D posters or symbols representing 3D objects. Figure 4c presents a selection of diverse examples that were removed during this process. This refinement requires minimal manual effort, supporting efficient large-scale annotation.
the supeline-generatie
Statisties. The COCO3D benchmark? comprises 2,039 human-refined images with total of 5,373 instances spanning all 80 categories of the MS-COCO dataset [40]. Figure illustrates samples of the final curated results, along with the categories represented in our COCO3D benchmark. We construct this benchmark using the validation set of the MS-COCO dataset. Figure 4a presents the distribution of the top 50 categories within the COCO3D benchmark. Notably, the person category in MS-COCO includes a wide range of examples featuring individuals in various poses, sizes, and
A larger version 2 will be released soon. For this camera-ready version, we present results version 1.
5

---

54
feraltere
244
131
3433
kitckee
513
food
(a) COCO3D: Top 50 category distribution
(b) COCO3D: Super category distribution
Reflections
Projections
Screens
Posters
Cartoons
Symbols
Collage
(c) COCO3D: Removed photo samples during buman refinement based on mentioned characteristics
Figure 4: COCO3D benchmark. (a) Distribution of the top 50 categories in the COCO3D benchmark. (b) Super-category-wiss distribution the COCO3D benchmark, based on MS-COCO [40]. (c) Examples of samples removed from COCO3D during the human refinement process.
age groups, contributing to a rich and diverse set of samples. Figure 4b displays the distribution of supercategories, further demonstrating the diversity captured by our benchmark.
4.2 Train Monocular 3D Detector with LabelAny3D
We build our model upon OVMono3D [75], a state-of-the-art open-vocabulary monocular 3D de- tection model. OVMono3D consists of two stages: (1) detecting and localizing objects in 2D using open-vocabulary detectors (e.g., Grounding DINO [45]); and (2) lifting 2D bounding boxes to 3D cuboids in a class-agnostic manner. Specifically, given image I, text prompt T. and 2D bounding boxes with category labels from 2D detector, OVMono3D extracts multi-scale feature maps from pretrained vision transformers (e.g., DINOv2 [50]) for each 2D bounding box. These features are then processed by a feed-forward network to predict 3D attributes.
Training objective. We closely follow prior work [7] to train our models. We train only the lifting head of OVMono3D [75] using ground-truth 2D bounding boxes. The training objective of is defined as:
- 22 exp(-p) LaD + μ. where CaD is the loss from the 3D cube head, and μ denotes the uncertainty score. The 3D loss C3D consists of disentangled losses for each 3D attribute [62]]
(1)
LND Σ c(e) + +
(2)
where a ((DD yzn), 2, (w, h 1), r} denotes groups of attributes: 2D center shift, depth, dimensions, and rotation. Each component loss C.p) isolates the error of specific attribute group by substituting all other predicted variables with their ground-truth counterparts when constructing the predicted 3D bounding box Ban. The holistic loss Call compares the predicted 3D bounding box with the ground truth using the Chamfer Loss:
Call &hammeer BaD, B
Training Data. We curate training set of 15,869 images from the MS-COCO [40] training split, annotated using our LabelAny3D pipeline without any human refinement. This pseudo-labeled dataset is used to either train the OVMono3D model from scratch or fine-tune it. For fine-tuning,

---

Table 1: Performance of OVMono3D [75] with different training settings. We report scores COCO3D benchmark, and the novel and base category splits of Omni3D in OVMono3D evaluation. The best results for each metric highlighted in bold. The second best is underlined. "Rel" denotes the relative layout metrics. "*" denotes the model is initialized with the pretrained OVMono3D.
COCO3D
Omni3D Novel
Omni3D Basc
Training dataset
APaot
ARwt
A
APæt
AR
APıD
AR
Bascline: Omni3D 75]
5.87
10.51
20.86
30.06
16.05
36.85
24.77
47.28
OVM3D-Det* [27]
2.69
5.25
7.98
12.25
5.30
15.71
7.32
26.34
Ommi3D [7] OVMDD-De [27]
6.82
11.94
20.76
27.69
15.55
37.18
22.35
42.68
LabelAny3D
7.78
15.41
24.66
34.54
8.47
23.34
3.92
19.66
Omm33 LabelAny3D
10.92
20.10
32.02
43.82
16.98
36.96
22.74
42.46
initialize from the OVMono3D model pretrained on Omni3D [7] and further train on the combined LabelAny3D and Omni3D datasets. For training from scratch, the model is trained solely on the LabelAny3D annotations without relying on any external ground truth 3D supervision.
Experiments
5.1 Experimental Setup
Benchmarks. LabelAny3D is evaluated on our COCO3D benchmark. In this benchmark, we exclude 10 of the 80 categories from evaluation due to either too few evaluation instances extreme aspect ratios (e.g., spoon, baseball bat). We also assess the voca-vocyy detection capabilities of our trained model on Omni3D [7], which primarily encompasses indoor datasets such SUN RGB-D [63], ARKitScenes [4], and Hypersim [58]; the object-centric dataset Objectron [2]; and autonomous driving datasets including nuScenes [8] and KITTI [24].
Baselines. We evaluate LabelAny.3 in terms of both pseudo annotation quality and its effectiveness for training open-vocabulary monocular 3D detectors. Specifically, we compare LabelAny 3D with OVM3D-Det [27] on annotation quality and downstream detection performance. Additionally, using curated COCO3D validation set, benchmark the performance of OVMono3D [75] and OVMono3D fine-tuned on our pseudo annotations derived from the MS-COCO [40] training set.
Evaluations. Following [7. 75], we report mean AP3d and AR computed using Intersection-over- Union (IoU) between predicted and ground-truth 3D bounding boxes across IoU thresholds ranging from 0.05 to 0.50 in increments of 0.05.
Since the metric depth in our COCO3D validation set is derived from model predictions, it may contain biases due to the inherent difficulty in accurately predicting and human-validating absolute depth. In contrast, the relative depth from MoGe [66] is rigorously verified through human refinement, providing greater reliability. To address this, we introduce a novel metric, Relative Layout AP within our COCO3D benchmark. This metric assesses the consistency of the relative spatial layout between predicted and ground-truth bounding boxes, in the same spirit with the relative depth evaluation [73, 74]. Specifically, we align predicted bounding boxes B and ground-truth bounding boxes B by optimizing global scale factor E R+:
1
Σ
IoUsD(s B,, B,),
arg max N
where N is the total number of boxes. Due to the non-differentiaabiity of IoUnD with respect s, perform a grid search within a bounded interval. AP3D and AR are then computed on the aligned, scale-normalized predictions, emphasizing relative 3D box layout rather than metric accuracy.
5.2 Main Results
Lny3lAny3D improves monocular 3D detection on COCO3D. Table benchmarks OV- Mono3D [75] trained with different datasets. When pretrained solely on Omni3D [7]. the model exhibits limited generalization COCO3D images, likely due to substantial domain gaps. To improve performance, generate pscudo labels the MS-COCO [40] training set using the auto-labeling
7

---

Momon
OVMomo3D
Ours
Figure 5: Qualitative open- vece-vocayulary 3D detection results on in- wo-the-wild-ai -mages:: OVMono3D [75] our fine-tuned OVMono3D. We display both the 3D predictions overlaid on the image and top-down view with a base grid of I m x m tiles.
Table 2: Pseudo annotation quality on COCO3D benchmark. The best results for each metric are highlighted in bold. "Rel" denotes the relative layout metrics, For fair comparison, use the same depth for OVM3D-De as for ours, denoted by ***
Methods
AP
AP
AP
AP
ARsut
APRdl
ARRel
OVM3D-Der* [27]
10.03
16.88
9.03
1.44
17.82
10.04
17.84
LabelAny3D
64.17
82.11
74.47
57.34
73.57
64.17
73.57
pipeline proposed in OVM3D-Det [27]. However, training OVMono3D from scratch with these labels fails to converge. Even when initialized from the pretrained OVMono3D, fine-tuning on OVM3D-Der labels alone leads poor performance. When fine-tuned on the combined dataset of Omni3D and OVM3D-Det, the model achieves only a marginal 0.95 AP3d gain on COCO3D and shows degraded performance on novel categories, suggesting that excessive label noise negatively impacts learning.
In contrast, training OVMono3D from scratch on pseudo-labels generated by our LabelAny exhibits better performance COCO3D, demonstrating the effectiveness of our pipeline in supporting model training. Notably, the model trained solely on COCO3D pseudo labels achieves 8.47 AP on OVMono3D's 's out-of-domain novel categories, highlighting its improved generalizability Further gains are observed when fine-tuning the pretrained MVMono3D model on the combined Omni3D and LabelAny3D pseudo-labeled datasets, resulting in a 5.05 APaD increase on COCO3D and improved performance across novel categories. These results validate the effectiveness of our LabelAny3D pipeline in producing high-quality, in-the-wild 3D annotations.
Despite these gains, all fine-tuned models show some degradation on OVMono3D's base categories. We attribute this two factors: (1) increased scene and category diversity without corresponding increase in model capacity, leading to catastrophic forgetting; and (2) label noise in the pseudo annotations, which may introduce harmful gradients during training.
Figure presents qualitative examples on COCO3D. Compared to the bascline model, our fine-tuned OVMono3D demonstrates stronger detection robustness in diverse scenes, particularly for novel object categories that are underrepresented existing datasets, such as animals, pizza, tie, boat.
LabelAny3D achieves better pseudo annotations. Table compares LabelAny:3D with OVM3D- Det [27] on the quality pseudo annotations. LabelAny3D consistently outperforms the baseline across all reported metrics. The APıd between automatically generated and human-refined an- notations 64.17, indicating minimal manual correction and demonstrating the efficiency of our pipeline. Figure 6 provides a visual comparison with OVM3D-Det [27], revealing the limitations of metric-based priors in in-the-wild settings, where object size and shape vary widely even within the same category (e.g. baby elephants, children, and boats). In such cases, metric depth estimates

---

Input Image
OVM3D-Det
LabelAny3D
Input Image
OVM3D-Det
LabelAny3D
Figure 6: Qualitative comparisons between OVM3D-Det [27] and LabelAny3D without any human refinement. These examples illustrate that OVM3D-Det often produces inaccurate metric dimensions for categories with high intra-class size variability, such as humans, animals, vehicles, and furniture.
prone to inaccuracy leading misaligned results. In contrast, LabelAny3D leverages relative depth and mesh reconstruction to produce appearance-consittee 3D bounding boxes, resulting in higher annotation fidelity. We further compare the quality pscudo annotations produced by LabelAny3D and OVM3D-Det [27] on the KITTI [24] benchmark. LabelAny3D achieves higher overall AP of 13.6, compared 12.39 from OVM3D-Det. Notably, on the truck category, LabelAny3D significantly outperforms OVM3D-DE Det with an AP3d of 32.74 vs. 13.46, highlighting its effectiveness.
Ablation of components in LabelAny3D. We conduct ablation studies on subset of the COCO3D dataset and re- port annotation quality in Table 3. The vanilla 3D scene construction model Gen3DSR [3] achieves an AD of only 1.95. In contrast, our full LabelAny3D pipeline achieves a higher APaD of 43.17. This results shows our pipeline provides better 3D reconstruction for 3D box labeling. Re- moving the image super-resolution module leads to sub- stantial drop to 28. AP3D, highlighting its importance enhancing detail for small and distant objects. Eliminating amodal completion also reduces performance, showing its role to alleviate the negative impact of occlusion.
Table 3: Ablation studies. Default set- tings are marked in gray
Framework
AP t
Gen3DSR [3]
LabelAny3D
43.17
w/o Super Resolution
28.13
w/o Amodal Completion
39.22
w/o MoGe [66]
22.77
w/ DrcamGaussian [64]
36,84
Replacing MoGe's [66] relative depth (scaled match Depth Pro [6]) with Depth Pro alone leads to a significant drop performance, showing that MoGe provides more accurate and reliable relative depth. For 3D reconstruction, using TRELLIS [72] improves AP3d by 6.33 points over DreamGaussian [64], suggesting TRELLIS produces more realistic and higher- fidelity reconstructions. To align reconstructed objects with the image point cloud, we compare iterative closest point (ICP) against our used 2D matching + Perspective-n-Point (PnP) method. The latter yields superior results, largely due to the robustness of the underlying 2D matching model.
w/ ICP
24.28
6 Discussion
In this work, we introduced LabelAny3D, an anulysis-hy-sym/heis pipeline for annotating 3D bounding boxes of arbitrary objects from monocular, in-the-wild images. Our pipeline integrates specialized vision foundation models to reconstruct 3D scenes and derive accurate 3D annotations. Leveraging LabelAny3D, curated COCO3D, new 3D detection benchmark encompassing

---

diverse object categories beyond existing datasets, with minimal human intervention, Our findings demonstrate that LabelAny3D effectively enhances open- vven-vocabulary 3D detection performance with minimal human effort and enabling the large-scale development of diverse 3D datasets. Our pipeline also has the potential to benefit other 3D scene understanding tasks, such as amodal 3D reconstruction, 6D pose estimation, and scene completion. See supplementary material for implementation details, more qualitative results and analysis.
7 Limitations
While our LabelAny3D pipeline leverages mature foundation models for depth estimation, camera intrinsic estimation, amodal completion, image-to-3D generation, and matching, these models can still fail in challenging scenarios involving heavy occlusion, textureless regions, or small objects, which introduce noise into the final 3D bounding box annotations.
The RGB-to-3D model TRELLIS [72] in our pipeline may generate meshes with ambiguous depth along the viewing direction, causing misalignment with RGBD point clouds and inaccurate bounding boxes. Future work could condition 3D generation on RGBD data, as in Hunyuan3D-One [65].
Additionally, future work could investigate robust training strategies for noisy pseudo souudo-annttations.
To ensure benchmark reliability, exclude objects with erroneous depth estimation, severe occlusion, or truncation. Consequently, our dataset is not exhaustively annotated but can evaluate 2D box- prompted methods like OV DVMono3D [75] and DetAny3D [78], or end-to-end 3D detection methods using proximity-based metrics.
Since our pipeline integrates depth estimation and amodal complction modular APIs, future improvements these components can be directly incorporated to enhance annotation quality. These limitations underscore the need for more robust auto-labeling frameworks and training strategies for open-vocabulary monocular 3D detection.
8
Acknowledgement
The authors acknowledge the University of Virginia Research Computing and Data Analytics Center, Advanced Micro Devices AI and HPC Cluster Program, Advanced Cyberinfrastructure Coordi- nation Ecosystem: Services & Support (ACCESS) program, and National Artificial Intelligence Research Resource (NAIRR) Pilot for computational resources, including the Anvil supercomputer (National Science Foundation award OAC 2005632) at Purdue University and the Delta and DeltaAI advanced computing resources (National Science Foundation award OAC 2005572). This work was supported by Adobe Research Gift, the National Science Foundation (awards 2129824, 2312487, and 2403060), United States Army Research Office (grant W911NF-24-1 2-------1--,,, and Lockheed Martin Advanced Technology Labs. The authors thank Hao Gu, Guangyi Xu, and Jiahui Zhang for annotation assistance.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023. 2,
[2] Adel Ahmadyan, Liangkai Zhang, Artsiom Ablavatski, Jianing Wei, and Matthias Grundmann. Objectron:
A large scale dataset of object-centric videos the wild with pose annotations. In CVPR, 2021. 2, 3, 7, 24
[3] Andreea Ardelean, Özer. and Bernhard Egger. Gen3dsr: Generalizable 3d scene reconstruction via divide and conquer from a single view, 2025. 4, 9, 26
[4] Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Yuri Feigin, Peter Fu, Thomas Gebauer, Daniel Kurz, Tal Dimry, Brandon Joffe, and Arik Schwartz. Arkitscenes: diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data. NewrlPS, 2021. 3, 7,
[5] Shariq Farooq Bhat, Reiner Birkl, Diana Wolk, Peter Wonka, and Matthias Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth, 2023, 4
10

---

[6] Aleksei Bochkovskii, Amaël Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan R Richter. and Vladlen Koltun. Depth pro: Sharp monocular metric depth in less than second. arXiv preprint urXiv:2410.00073. 2024. 2, 3, 4, 9, 23,
[7] Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, and Georgia Gkioxari. Omni3d: A large benchmark and model for 3d object detection in the wild. In CVPR, 2023. 2, 3, 6, 7, 24, 26
[8] Holger Cacsar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020. 2, 3,7,7 7.25
[9] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020. 2, 24
[10] Benjamin Caine, Rebecca Roclofs, Vijay Vasudevan, Jiquan Ngiam, Yuning Chai, Zhifeng Chen, and Jonathon Shlens. Pseudo-labeling for scalable 3d object detcction. arXiv preprint arXiv:2103.02093 2021.
[11] Yang Cao, Zeng Yihan, Hang Xu, and Dan Xu. Coda: Collaborative novel box discovery and cross modal alignment for open-vocabulary 3d object detection. NeurlPS, 2024. 2,
[12] Yang Cao, Yihan Zeng. Hang Xu. and Dan Xu. Collaborative novel object discovery and box-guided cross-modal alignment for open- vocabulayy 3d object detection. arXiv preprint wxXv:244060008300 2024.
2,
[13] Hansheng Chen, Yuyao Huang, Wei Tian, Zhong Gao, Lu Xiong. Monorun: Monocular 3d object detection by reconstruction and uncertainty propagation, 2021.
[14] Xiaozhi Chen, Kaustav Kundu, Ziyu Zhang. Huimin Ma. Sanja Fidler, and Raquel Urtasun. Monocular 3d object detection for autonomous driving. In CVPR, 2016. 3
[15] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017. 2
[16] Saumitro Dasgupta, Kuan Fang, Kevin Chen, and Silvio Savarese. for cluttered indoor scenes. CVPR, 2016.
Delay: Robust spatial layout estimation
[17] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13142 13153, 2023. 2
[18] Xueqing Deng. Qihang Yu, Peng Wang, Xiaohui Shen, and Liang-Chies Chen. Coconut: Modernizing coco segmentation, 2024, 4, 25, 26
[19] Carl Doersch, Yi Yang, Dilara Gokay, Pauline Luc, Skanda Koppula, Ankush Gupta, Joseph Heyward, Ross Goroshin, João Carreira, and Andrew Zisserman. Bootstap: Bootstrapped training for boakking-nn--pinnn arXiv preprint vxXi:2402.00087. 2024. 3
[20] Djamahl Etchegaray, Zi Huang. Tatsuya Harada. and Yadan Luo. Find n' propagate: Open-vocabulary 3d object detection in urban environments. In ECCV, 2024. 2,3
[21] Martin A Fischler and Robert C Bolles. Random sample consensus: paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 2416)338 1981. 5
[22] Yunhao Ge, Hong-Xing Yu, Cheng Zhao, Yuliang Guo, Xinyu Huang, Liu Ren, Laurent Itti, and Jiajun Wu. 3d copy-paste: Physically plausible object insertion for monocular 3d detection. Advances Neural Information Processing Systems. 36:17757-17071, 2023. 2.
[23] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 22(11):1231-123) 2013. 2,
[24] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012. 2. 7,
11

---

[25] Andreas Lau Hansen, Lukas Wanzeck, and Dim P. Papadopoulos. Weak cube r-can: Weakly supervised 3d detection using only 2d bounding boxes, 2025. 3
[26] Kuan-Chih Huang, Tsung-Han Wu, Hung-Ting Su, and Winston H Hsu, Monodtr: Monocular 3d object
detection with depth-aware transformer. In CVPR, pages 4012-4021, 2022. 3
[27] Rui Huang, Henry Zheng, Yan Wang, Zhuofan Xia, Marco Pavone, and Gao Huang. Training an open- vocabulary monocular 3d object detection model without 3d data. arXiv preprint arXiv:2111.55557 2024. 5, 7, 8, 9, 23, 24, 26
[28] Siyuan Huang, Siyuan Qi. Yinxue Xiao, Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu. Cooperative holistic scene understanding: Unifying 3d object, layout, and camera pose estimation, 2019, 3
[29] Jitesh Jain, Jiachen Li, Mang Tik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. Oneformer: One transformer to rule universal image segmentation. In CVPR, pages 2989- 2023. 3
[30] Hanwen Jiang, Qixing Huang, and Georgios Pavlakos. Real3d: Scaling up large reconstruction models with woal-world images. arXiv preprint arXiv22406.08479 2024. 23
[31] Xucying Jiang, Sheng Jin, Lewei Lu, Xiaogin Zhang, and Shijian Lu. Weakly supervised monocular 3d detection with a single-view image. In CVPR, pages 10508 10518, 2024. 3
[32] Linyi Jin, Richard Tucker, Zhengqi Li. David Fouhey, Noah Snavely, and Aleksander Holynski. Stereo4d: Learning how things move in 3d from internet stereo videos. arXiv preprint, 2024.
[33] Alexander Kirillov, Fric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, and Wan-Yen Lo. Segment anything. arXiv preprint N.Ki:22304.026433 2023. 2
[34] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, and Wan- Yen Lo. Segment anything. ICCV, 2023. 24, 26
[35] Justin Lazarow, David Griffiths, Gefen Kohavi, Francisco Crespo, and Afshin Dehghan. Cubify anything: Scaling indoor 3d object detection. arXiv preprint arX1::2412.04458, 2024.
[36] Vincent Lepetit, Francese Moreno-Noguer, and Pascal Fua. Ep p: An accurate o (n) solution to the problem. International journal of computer vision, 81:155-166, 2009. 5
[37] Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mastär. In European Conference on Computer Vision, pages 71-91. Springer, 2024. 5, 26
[38] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2 Bootstrapping language- -inagg pre- pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023. 3
[39] Zhuoling Li, Xiaogang Xu, SerNam Lim, and Hengshuang Zhao. Towards unified 3d object detection via algorithm and data unification, 2024.
[40] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014. Proceedings, Part V 13, pages 740-755. Springer, 2014. 3, 4. 5, 6. 7, 25
[41] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 26
[42] Lahav Lipson, Zachary Teed, and Jia Deng. Raft-stereo: Multilevel recurrent field transforms for stereo matching, In 2021 International Conference on 3D Vision (3DV), pages 218-221 IEEE, 2021. 3
[43] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma Zexiang Xu, and Hao Su. One-2- 3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems, 36:22226-223 222446, 2023. 23, 26
[44] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Vang, Chunyuan Li, Jianwei Yang, Hang Su, and Jun Zhu. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.
2023.
[45] Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang. Hang Su, and Jun Zhu. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, arXiv preprint arXiv:2303.05499 2023. 6, 24.20 26
12

---

[46] Yuheng Lu, Chenfeng Xu. Xiaobao Wei, Xiaodong Xie. Masayoshi Tomizuka, Kurt Keutzer, and Shang- hang Zhang. Ooca-lacyy 3d detection via image-level class and debiased cross-modal contrastive learning. arXiv preprint (XX.:22207.0996555 2022. 2.3
[47] Yuheng Lu, Chenfeng Xu, Xiaobao Wei, Xiaodong Xie, Masayoshi Tomizuka, Kurt Keutzer, and Shang- hang Zhang. Open-vocabulary point-cloud object detection without 3d annotation. In CVPR, 2023. 2,
[48] Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with pretrained models. Advances in Neural Information Processing Systems, 36:75307-75337, 2023.
[49] Yinyu Nie, Xiaoguang Han, Shihui Guo. Yujian Zheng, Jian Chang, and Jian Jun Zhang. To- tal3dunderstanding: Joint layout, object pose and mesh reconstruction for indoor scenes from a single image, 2020. 3
[50] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafranice, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, and Alaacldin El-Nouby. Dinov2: Learning robust visual features without supervision. 077903,3 2023. 6, 24
[51] Liang Peng, Senbo Yan, Boxi Wu, Zheng Yang. Xiaofei He, and Deng Cai. Weakm3d: Towards weakly supervised monocular 3d object detection. arXiv preprint arXiv:2203.08322 2022. 5
[52] Xingyu Peng. Yan Bai, Chen Gao, Lirong Yang. Fei Xia, Beipeng Mu, Xiaofei Wang, and Si Liu. Global- local collaborative inference with 11m for lidar-based open-vocabulary detection. In ECCV. Springer. 2025.
2, 3
[53] Luigi Piccinelli, Yung-Hsu Yang. Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 101161 2024. 2, 26
[54] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, and Jack Clark. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021." 3
[55] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, and Laura Gustafson. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408 00714, 2024. 3
[56] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. arXiv preprint urXiv:2007.08501, 2020.
24
[57] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiay Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXvv:2401. 14159, 2024. 25
[58] Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: A photorcalistic synthetic dataset for holistic indoor scene understanding. In ICCV, 2021. 2.
[59] Chris Rockwell, Joseph Tung, Tsung Yi Lin, Ming Yu Liu, David Fouhey, and Mhunn Lin. Dynamic camera poses and where to find them. arXiv preprint arXiv:2504. 17788, 2025.
[60] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2022. 2
[61] Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Imvoxelnet: Image to voxels projection for monocular and multi-view general-purpos 3d object detection, 2021. 3
[62] Andrea Simonelli, Samuel Rota Bulo, Lorenzo Porzi, Manuel I. López-Anteuurra, and Peter Kontschieder. Disentangling monocular 3D object detection. In ICCV, 2019. 6
[63] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In CVPR, 2015. 2, 3, 7, 24, 26
[64] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.1665 16653, 2023. 9
13

---

[65] Tencent Hunyuan3D Team. -onyuan3d-omni: A unified framework for controllable generation of 3d assets, 2025. 10
[66] Ruicheng Wang, Sicheng Xu, Cassie Dai, Jianfeng Xiang, Yu Deng, Xin Tong, and Jiaolong Yang. Moge: Unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision. arXiv preprint arXiv:2410.19115, 2024. 2, 3, 4, 5, 7, 9, 23,
[67] Tai Wang, ZHU Xinge, Jiangmiao Pang, and Dahua Lin. Probabilistic and geometric depth: Detecting objects in perspective. In Conference on Robot Learning. PMLR, 2022.
[68] Tai Wang, Xinge Zhu. Jiangmiao Pang. and Dahua Lin. Fcos3d: Fully convolutional one-stage monocular object detection, 2021.
[69] Zhenyu Wang, Yali Li, Taichi Liu, Hengshuang Zhao, and Shengjin Wang. Ov-uni3detr: Towards unified open- vocabulary 3d object detection via mycle-modality propagation. arXiv preprint XXii:24033199888 2024. 3
[70] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Yen Lo, and Ross Girshick, Detectron2. https: 2019. 24
// //github..o comm/faccbboorrseerccceeeeeeeeee
[71] Zhongyu Xia, Jishuo Li, Zhiwei Lin, Xinhao Wang. Yongtao Wang, and Ming-Hsuan Yang. Openad: Open-world autonomous driving benchmark for 3d object detection. arXiv preprint arXiv:2411.17761, 2024. 3
[72] Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3d latents for scalable and versatile 3d generation, 2025. 2, 3, 4, 5, 9, 10, 24, 26
[73] Lihe Yang. Bingyi Kang, Zilong Huang. Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, pages 10371- 10381, 2024. 7
[74] Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. 0944444006.09414 2024. 7
[75] Jin Yao, Hao Gu, Xuweiyi Chen, Jiayun Wang, and Zezhou Cheng. Open vocabulary monocular 3d object detection. arXiv preprint urXiv:2411.16833 2024. 3, 6, 7, 8, 10, 24, 25, 26, 28
[76] Zongsheng Yue. Kang Liao, and Chen Change Loy. Arbitrary-step image super-resolution via diffusion inversion, 2025. 4, 26
[77] Dongmei Zhang, Chang Li, Renrui Zhang, Shenghao Xie, Wei Xue, Xiandong Xie, and Shanghang Zhang. Fm-ov3d: Foundation model-based cross-modal knowledge blending for open- vocaaulayy 3d detection. In AAAL, 2024. 2,3
[78] Hanxue Zhang, Haoran Jiang, Qingsong Yao, Yanan Sun, Renrui Zhang, Hao Zhao, Hongyang Li, Hongzi Zhu, and Zekong Yang. Detect anything 3d the wild, arXiv prepeint arXiv:2504.00 07955, 2025. 2, 10
[79] Hu Zhang, Jianhua Xu. Tao Tang. Haiyang Sun, Xin Yu, Zi Huang, and Kaicheng Yu. Opensight: A simple open- open- vocabulary framework for lidar-based object detection. In ECCV. Springer, 2025. 2,
[80] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointelip: Point cloud understanding by clip. CVPR, 2022. 2,3
[81] Renrui Zhang, Han Qiu, Tai Wang, Ziyu Guo. Ziteng Cui, Yu Qiao, Hongsheng Li, and Peng Gao. Monodetr: Depth-guided transformer for monocular 3d object detection. ICCV, pages 9155-9166, 2023.
[82] Dingfu Zhou, Xibin Song, Yuchao Dai, Junho Yin, Feixiang Lu, Jin Fang, Miao I Liso, and Liangjun Zhang. Iafa: Instance-aware feature aggregation for 3d object detection from a single image, 2021. 3
[83] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points, 2019. 3
[84] Chenming Zhu, Wenwei Zhang, Tai Wang, Xihui Liu, and Kai Chen. Object2scene: Putting objects in context for open- vocabuacy 3d detection, arXiv preprint 2rX1vv2309.09455 2023. 2,
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly state the scope of our work in both the abstract and the introduction. Additionally, we highlight the main contributions of our paper at the end of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We mention the limitations of our work in the discussion section and include detailed separate section about the limitations in the Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification: The paper highlights on the performance of monocular 3D detection with a new proposed benchmark not including any theoretical assumptions and results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
In the experiments section, we mention all of our settings and data used for model evaluation, pseudo label training and ablation studies following which the mentioned results could be reproduced.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
16

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We publicly release code and data project page.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.c cc/
seliic/guudess htt///ee//oeeeuaiiiseseeeseeeeee SoodeeSubiissioooooooooo conolicy)) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //ni //nips. peips.cc/publl pppplcc/guidesssseppppppppbepaep httpg//ddde//eeuuumiiiiiaie SrodeeSumiissionPono
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have mentioned experimental setup, data and details used for all processes in experiments section and the implementation details the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to limited computing resources, have to limit the possibility of obtaining statistical significance in our experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: Yes]
Justification: We include the details about the computing resources used for our experiments in the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https http://eeeupppeeeeee https///ccccccttcceceeeeeeaaeea
Answer:
[Yes]
Justification:
We conform with the NeurIPS Code of Ethics in every aspects.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We include potential broader societal impact in the Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
To the best our knowledge, no such risk associated with the work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The papers of existing code, models and datasets are adequately credited and cited in all sections of the paper. We have reviewed and complied with the licenses of all referenced works. We include a section of licenses of assets used our supplementary material.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will release all new assets along side proper documentation. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing platforms or research with hu- man subjects defined by ethics review standards. All data annotations were performed internally by co-authors or lab members.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects, and therefore does not require IRB or equivalent ethical approval.
Guidelines:
20

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The study deals with 3D detection for in the wild monocular images leveraging state of the art foundation models, not involving LLMs for any critical operations or experiments.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: / /tteu/isss.////// cc/Conf /te/////eereecce/222555 /LLM) for what should or should be described.
21

---

Appendix
A Human Refinement Interface
Consr (pamm)
Right-clck Pan Whlae Zoors
Properttes
Shiw Selvcuu Crty
547692
Figure 7: Human refinement interface.
Figure illustrates our annotation interface for human refinement LabelAny3D's automatically generated pseudo labels COCO3D. The top left panel displays the point cloud and corresponding 3D bounding boxes, which annotators can rotate and translate for an optimal viewing angle. The top-right panel shows three 2D projection views of the point cloud along the local axes of the 3D bounding box. Annotators can adjust the 3D bounding box by manipulating the edges, corners, or center in each view. The lower-left panel shows the attributes of the selected bounding box. Annotators can modify specific parameters (e.g., width) using keyboard shortcuts. They also have the option delete a bounding box if it corresponds
invalid 3D object (e.g., a person The lower-right panel visualizes the 2D projections of the current 3D boxes to provide additional context. Annotators may also choose to discard the entire image the relative geometry is incorrect or no valid 3D objects are present.
poster).
B Annotation Efficiency
Table 4 presents category-wise overview of APıd, ARD. and average IoUnp for the pseudo annotations generated by LabelAny3D, evaluated against our refined COCO3D benchmark. The results demonstrate that our pipeline produces high-quality annotations with human-like accuracy. In addition achieving high precision, the average IoU exceeds 0.40 for the majority of categories, indicating strong alignment of spatial layout.
22

---

Table 2 compares the overall AD of our method against the baseline DVM3D-DET [27]. with our approach achieving higher AP of 64.17. Among the 5,373 annotations the COCO3D benchmark, 3, 146 were accepted by human annotators without modification, while 2,227 required only minor refinement. Just 466 were rejected due issues such as reflections, 2D object representations, or insufficient point cloud quality.
These results demonstrate that LabelAny3D produces high-quality pseudo annotations that can serve as effective initialization for human annotators. With minimal effort required for refinement, our pipeline significantly reduces manual workload and accelerates the overall annotation process.
Table 4: Per-category 3D annotation quality for the top 50 categories. Comparison between pseudo annotations from LnyelAny3D and human-refined annotations. Ranking is based IoU3D-
Category
A
ARaD
IoU
Category
AP3d
AR
IoUnD
sports ball
81.58
88.26
80.94
bed
88.84
93.33
76.32
fire hydrant
91.14
94.80
75.16
airplane
90.61
93.53
68.83
couch
70.05
88.57
67.89
snowboard
63.50
72.14
68.27
parking meter
92.56
94.29
63.94
mouse 68.98
78.70
63.57
vase
89.89
92.73
61.38
skateboard
69.87
80.73
60.70
fork
64.76
75.65
58.93
cat
72.94
80.49
57.66
bicycle
83.45
94.65
56.60
microwave
82.65
90.53
56.00
dog
79.54
90.36
54.73
tie
48.71
67.83
54.60
bus
84.28
93.61
52.20
dining table
86.38
92.22
51.59
bench
84.80
89.81
49.43
surfboard
50.10
65.00
48.24
sink
60.22
71.95
46.97
sandwich
65.13
77.94
47.38
refrigerator
72.87
86.52
45.04
kite
70.54
80.71
44.99
laptop
69.56
84.69
43.92
cake
80.60
90.40
43.38
oven
66.77
79.05
43.29
umbrella
92.28
95.21
41.30
tv
43.71
62.54
41.10
bowl
79.32
88.40
40.70
horse
76.75
89.84
40.38
bear
82.62
89.71
40.28
backpack
69.39
82.11
38.90
keyboard
47.08
61.82
38.36
pizza
70.90
81.82
37.22
skis
40.44
64.25
36.93
traffic light
62.11
77.12
36.80
cup
89.94
93.44
36.35
remote
59.63
71.60
35,49
tennis racket
38.10
59.31
31.62
clock
39.06
57.36
33.25
elephant
87.71
98.82
33.19
handbag
68.75
82.28
32.23
wine glass
97.11
98.51
31.76
chair
61.75
88.63
32.60
horse
76.75
89.84
40.38
boat
80.06
90.29
12.15
bird
70.48
85.53
16.68
motorcycle
92.65
96.77
13.99
book
65.94
76.75
13.57
C Implementation Details
During annotation, exclude objects whose masks contain fewer than 400 pixels. Following [30], we also discard objects whose masks overlap the image boundary by more than 10 pixels, treating them as truncated.
For depth estimation, we align the scale-invariant depth map from MoGe [66] with the metric-scale depth map predicted by Depth Pro [6] using global scale transformation. Specifically, first use MoGe to generate relative depth map and extract camera intrinsics. Then, we apply Depth Pro to predict a metric-scale depth map for the same scene. We perform RANSAC-based linear regression to fit a robust scale factor that maps MoGe's relative depths to Depth Pro's 's metric scale. This allows retain the fine geometric details from MoGe while calibrating the scale to real-world distances.
For 2D-3D matching, we first estimate the camera elevation angle using the elevation module from One-2-3-45 [43], based on the amodal-completed object crop. We then render 8 views of the object at the estimated elevation, with azimuths spaced at 45-degree intervals. After computing correspondences between the amodal crop and the 8 rendered views, obtain an initial camera pose.
23

---

Using this pose, we render the mesh again and perform an additional 2D-3D matching step to refine the camera pose. Generating pseudo annotations for single object takes approximately one minute.
Our implementation based on Py [56] and Detectron2 [70]. Following [75], we use DINOv2-Bass [50] as the image feature encoder and freeze its parameters during training. The model is initialized from the publicly released OVMono3D weights and fine-tuned for 58k steps with batch size of 64, We train the model using SGD with an initial learning rate of 0.0012, which decays by a factor of 10 at 60% and 80% of training. A linear warm-up is applied for the first 1.8k steps. Training takes approximately 48 hours on 4 NVIDIA A40 GPUs. We apply standard image augmentations during training, including random horizontal flipping and resizing. In addition, random positional perturbation applied the imput 2D bounding box, with maximum offiet ratio of 0.2. For evaluation on COCO3D. we use ground-trur -tunc 2D boxes as input. For Omni3D [7], we adopt the same 2D detections from Grounding DINO [45] as used OVMono3D [75].
D Labeling Performance on Additional Benchmarks
We compare LabelAny3D with baseline OVM3D-Det [27] on three established benchmarks: SUN- RGBD [63], nuScenes [9], and Objectron [2], covering indoor, self-driving, and object-centric domains, respectively. We randomly sample approximately 300 images from each dataset's 's test split in Omni3D [7]. Using ground truth 2D boxes as input, we query SAM [34] to obtain instance masks, then generate 3D boxes using both methods.
Since our pipeline filters out small or heavily occluded objects-as TRELLIS [72], the state-of-the-art 3D reconstruction model used in our pipeline, is sensitive occlusion and object size-we first evaluate performance on highly visible objects. LabelAny3I achieves AD of 37.71, 13.59, and 6.07 on SUN-RGBD [63], nuScenes [9], and Objectron [2], respectively, while baseline OVM3D-Det [27] achieves 37.58, 1 1.33,, and 3.84.
On low-visibilig objects. LabelAny3D achieves AP3D of 20.20, 6.69, and 2.54, while OVM3D-Det achieves 25.56, 8.50, and 3.85 on the same benchmarks. These results demonstrate that LnyelAny3D performs better low-occlusion objects, as our 3D scene reconstruction yields more accurate boxes. The baseline method excels on low-visibility objects due to its use of metric-scale object size priors. Note that both methods achieve lower performance Objectron, likely due to inaccurate metric depth estimation for this dataset.
We further ensemble the two methods, using the baseline for low-visibility objects and LnbelAny3I for visibiiiiy objects. The ensembled method achieves AP of 30.23, 9.12, and 4.74 on the three benchmarks, compared to 29.89, 8.22, and 3.73 for the baseline alone, demonstrating that the two methods are complementary.
Table 5: Comparison of COCO3D with Other Benchmarks. We report statistical comparison between COCO3D and existing 3D detection benchmarks their respective sets. We report the number of images, categories, covered domains, and the distribution of instances across MS-COCO super categories (percentages reported).
Benchmark
+ Img
Car
Domain
Person Food Animal Vehicle Kitchen Furniture Accessory Indoor Sports Electroicc Ourdoor Appliance
SUN RGB-D
5,050
82
Indocr
0.1
0.2
65.5
8.0
11.8
7.8
0.5
ARKiiSemes
7.610
15
Indocr
10.3
80.5
3.1
4.1
2.1
Hypersim
7,690
29
Indoce
B.
37.2
3.6
49.7
8.5
Objectron
Object
KITTI 3,769
8
Driving
12.
64.4
22.8
nuSeenes
6,019
9
Driving
15.8
63.1
21.1
COCO3D
2,039
80
In the Wikl
27.9
9.1
10.9
11.0
8.2
4.2
4.5
4.0
1.0
2.0
E
Comparison of COCO3D with Existing Benchmarks
Table 5 shows a statistical comparison between COCO3D and existing 3D detection benchmarks on their respective test sets. We report the number of images, categories, covered domains, and the distribution of instances across MS-COCO super categories.
Compared to prior benchmarks- which often focus on specific domains such as indoor environments (e.g., SUN RGB-D), object-centric scenes (e.g., Objectron), or self-driving datasets (e.g., KITTI, nuScenes) -CCCOD offers broader coverage across indoor and outdoor everyday scenes. From the
24

---

cuperryateoryy perspective. COCO3D uniquely includes categories fromfoodd animal, and sports, which are often absent in existing 3D datasets. conclusion, our COCO3D benchmark offers diverse, real-world, and scalable benchmark for advancing the open-vocabulary monocular 3D detection field.
COCO3D Benchmark Samples
Figure 8 presents additional samples from human-refined COCO3D benchmark. The results demonstrate that the relative geometry and spatial layout the scenes and 3D bounding boxes are highly accurate and align well with human perception.
G More Qualitative Comparisons
We report more qualitative comparisons of OVMono3D [75] with our finetuned variant in Figure 9. Trained with additional pseudo-labeled in-the-wild images, our model produces more accurate predictions for challenging object categories such as animals, athletes, and food.
H Failure Case
Figure 10 illustrates several failure cases of LabelAny3D. highly occluded scenes, such as the cow in Figure 10(a), the amodal completion model fails to reconstruct the full object, resulting in 3D bounding box that captures only partial region. As our method relies on ground-truth 2D instance segmentations, it may incorrectly generate 3D boxes for objects that are not physically present in the 3D scene for example, the person on a television screen in Figure 10(b). Additionally, for crowded scenes, such as in Figure 10(c), the COCO [40, 18] dataset often lacks per- cer-instance segmentation labels. In such cases, instance segmentation models like Grounded SAM [57] also struggle to separate individual objects accurately, causing method miss multiple instances.
I Broader Impact
Our work facilitates efficient 3D annotation of objects from any category in diverse, in in-the-wild scenes. By integrating the generated pseudo labels, existing open-vocabulary monocular 3D detectors become more robust to out-of-domain categories (c.g., animals), which can enhance the reliability of autonomous systems such robots and self-driving vehicles, particularly in safety-critical scenarios.
Our dataset is curated from publicly available sources, and therefore does not raise privacy concerns. While our algorithm is category-agnose agnostic and does not introduce explicit bias, the underlying datasets may reflect societal or geographic biases present in the source data. We encourage future work to investigate and mitigate such biases when deploying systems trained on annotations.
Licenses
25

---

Table 6: Licenses of assets used.
Asset
License
Cube R-CNN [7]
CC-BY-NC 4.0
Grounding DINO [45]
Apache License 2.0
Segment Anything [34]
Apache License 2.0
Unidepth [53]
CC-BY-NC 4.0
MoGe [66]
MIT License
Depth Pro [6]
Apple License (Link)
InvSR [76]
S-Lab License 1.0 (Link)
One-2-3-45 [43]
Apache License 2.0
TRELLIS [72]
MIT License
Gen3DSR [3]
CC-BY 4.0
MASt3 [37]
CC-BY-NC-SA 4.0
OVMono3D [75]
Apache License 2.0
OVM3D-Det [27]
Apache License 2.0
KITTI [24]
CC-BY-NC-SA 3.0 DEED
nuScenes [8]
CC-BY-NC 4.0
SUN RGB-D [63]
MIT License
ARKitScenes [4]
Apple License (Link)
COCO [41]
CC-BY 4.0
COCONut [18]
Apache License 2.0
26

---

Input Image
BBox 2D Projection
3D View 1
3D View
Figure 8: More COCO3D benchmark samples (after human refinement). For each example, we show: (1) the input image, (2) the projected 3D bounding boxes overlaid the image, and (3-4) two 3D views of the scene point map with the 3D bounding boxes. Please see our project page for rendered videos from 3D Scene and bounding boxes.
27

---

ON Do-ooec
Ours
OVNanad
XB04
Ours
Figure More qualitative open-vocabulayy 3D detection results on wethe-wlld-imaeee OV- Mono3D [75] vs. our finetuned OVMono3I. We display both the 3D predictions overlaid on the image and a top-down view with base grid of m x tiles.
28

---

(a) Highly occluded objects
(b) Objects on 2D plane
(c) Too crowded objects
Figure 10: Failure cases.
29


Reviews:
[{"review_id": "PUduT6VPFs", "reviewer": "Reviewer_huQ4", "summary": "This paper proposes an annotation pipeline to annotate the 3D bounding boxes given 2D images. Specifically, the authors propose an analysis-by-synthesis framework that first reconstructs holistic 3D scenes from 2D images and then recovers the 3D bounding boxes with absolute scale. Along with the proposed pipeline and benchmark, the authors also train a monocular 3D object detector based on the dataset, which shows superior performance on the designed COCO3D benchmark.", "questions": "Regarding object orientation, how should the proposed pipeline handle round objects that may lack a clear definition of orientation?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strength\n1. The writing is clear, and the paper is easy to follow.\n2. The proposed annotation pipeline is theoretically significant, as stated in the paper, it enables labeling of any 3D objects from images.\n3. According to the visualizations in the paper, the annotation quality is excellent, and the model trained on the datasets demonstrates strong 3D generalization capability on the COCO dataset.\n\n\n### Weakness\n1. The overall annotation pipeline lacks rigorous evaluation. The authors could validate the annotated 3D bounding boxes on datasets with 3D ground truth from LiDAR data (e.g., KITTI, nuScenes, or ScanNet).", "was_revised": true, "final_justification": "The author's response addresses most of my concerns. I will keep my positive rating.", "rebuttal": "We thank the reviewer for the constructive comments and for recognizing our “clear writing”, “theoretically significant”, and “strong 3D generalization capability\". \n\n**\\[W1\\] Validation of the pipeline on datasets with 3D ground truth.**\n\nWe thank the reviewer for raising this important point. In the original submission (Lines 280–283), we validated our pipeline on the **KITTI** benchmark, where LabelAny3D achieved **13.60 AP3D**, outperforming the baseline OVM3D-Det (**12.39 AP3D**). In this rebuttal, we provide additional validation on the **SUN RGB-D** benchmark. LabelAny3D surpasses OVM3D-Det by **\\+2.32 AP3D** (41.39 vs. 39.07). OVM3D-Det relies on object size priors from large language models to infer 3D bounding boxes. This strategy is particularly effective for objects with relatively consistent dimensions (e.g., cars), but struggles with categories exhibiting high intra-class variation (Lines 41-45). In contrast, LabelAny3D adopts an analysis-by-synthesis paradigm: it reconstructs the holistic 3D scene from monocular images and uses the synthesized representation to infer 3D annotations (Lines 52-54). We will include these results and extend our validation to more benchmarks in the final version of the manuscript.\n\n**\\[Q1\\] How should the proposed pipeline handle round objects that may lack a clear definition of orientation?**\n\nWe thank the reviewer for this insightful question\\! Orientation ambiguity in symmetric objects is a well-known challenge in 3D vision. For such objects, multiple orientations may yield indistinguishable appearances, making pose estimation inherently ambiguous.  \nOur pipeline does not explicitly resolve this ambiguity. Instead, it inherits the pose definition from TRELLIS, the 3D reconstruction model we used, which reconstructs objects in the canonical space defined in its training data, where the orientation of symmetric objects remains ill-defined. \n\nSome recent works (e.g., Orient Anything \\[1\\]) address this issue by using vision-language models (VLMs) to identify symmetric objects and label them as having no meaningful pose. However, our primary focus is on tight 3D bounding box prediction, which remains meaningful even for symmetric objects. We therefore retain symmetric objects in both training and evaluation to preserve dataset diversity. This approach aligns with standard practice—most large-scale 3D datasets (e.g., KITTI, ARKitScenes) do not explicitly handle symmetry during training or evaluation. \n\nAn exception is Objectron \\[2\\], which incorporates pose equivalence in its evaluation metrics to account for such ambiguity. Specifically, predicted bounding boxes for symmetric objects are rotated along the symmetry axis, and the rotation that yields the maximum IoU with the ground truth is selected. \n\nTo assess the impact of symmetry on our benchmark, we manually annotated symmetric instances in 300 randomly sampled images  from COCO3D. We found that symmetric objects comprise 13% of all instances, primarily in cylindrical shapes such as apples, bottles, and bowls.  Using the pose-equivalent metric from Objectron, our 3D detector achieves **12.55 AP3D** on symmetric objects, compared to **12.53 AP3D** without the metric—a negligible difference of **0.02 AP3D**.\n\nIn summary, our results suggest that our current results are robust to orientation ambiguity. We will include a detailed discussion of this limitation and our evaluation results in the final version of the paper.\n\n\\[1\\] Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models. Wang et al. ICML 2025\\.  \n\\[2\\] Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotations. Ahmadyan et al. CVPR 2021\\."}, {"review_id": "MykN740JNf", "reviewer": "Reviewer_oo5r", "summary": "This paper proposes LabelAny3D, a framework for efficiently generating high-quality 3D bounding box annotations from 2D images. Based on the LabelAny3D pipeline, the authors introduce COCO3D, a new benchmark for open-vocabulary monocular 3D detection, derived from the MS-COCO dataset. The COCO3D benchmark comprises 2,039 human-refined images with 5,773 instances spanning all 80 categories of MS-COCO. The authors conduct experiments by training monocular 3D detection models on a dataset annotated via the LabelAny3D pipeline, demonstrating that the performance of monocular 3D detectors can be significantly improved.", "questions": "1. The author should provide detailed comparision between the proposed dataset and existing open-vocabulary 3D detection.\n2. The author should conduct experiments with lareg model architectures to validate the explanations of performance degradation in OVMono3D Novel/base.\n3. The author should provide variance analysis across multiple runs on proposed COCO3D benchmark.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. This paper proposes a data processing pipeline to generate pseudo-3D annotations from monocular images. The pipeline leverages several state-of-the-art models (e.g., Depth Pro, Mast3R) to produce plausible 3D bounding boxes.\n2. The authors construct the COCO3D dataset, comprising 2,039 human-refined images with 5,373 total instances, establishing a new benchmark for monocular open-vocabulary 3D detection.\n3. Experimental results demonstrate that:\n(1) The proposed LabelAny3D pipeline achieves remarkable performance improvements compared to state-of-the-art detectors;\n(2) Fine-tuning on pseudo-labels generated by LabelAny3D enhances detector performance on the COCO3D benchmark.\n\nWeaknesses\n1. Missing detailed comparision between the proposed COCO3D dataset and other exist benchmark dataset for open-vocabulary 3D detection. While the authors claim existing benchmarks lack diversity for open-world scenarios, they fail to provide a detailed quantitative comparison between COCO3D and established datasets. A systematic comparison should include: (1) Dataset scale (number of images/instances); (2) Category coverage and diversity;  and (3) Scenario variety.\n2. Finetuning with the pseudo-label generated by LabelAny3D achieve not significant performance improvement even with degration in OVMono3D Novel and OVmono3D Base, the author explain that the eason can be attribute to (1) limited model capacity and (2) label noise. The author should provide a experiments with larger model capacity to validate. Without such evidence, the explanations remain speculative.\n3. The proposed COCO3D only consist of 5373 instances. The statistical significance for reliable evaluation is questionable.", "was_revised": true, "final_justification": "This paper proposes a pipeline for generate high-quality 3D bounding box annotations from 2D images and the authors conduct experiments to validate the effectiveness of the proposed method. The rebuttal addressed my concerns, I keep my rating as positive.", "rebuttal": "We appreciate the reviewer for the thoughtful feedback and praises, including \"remarkable performance improvements\" and \"enhances detector performance\".\n\n**\\[W1 & Q1\\] Systematic comparison of COCO3D benchmark with other benchmarks.**\n\nWe thank the reviewer for suggesting this important point\\! The table below presents a statistical comparison between COCO3D and existing 3D detection benchmarks on their respective test sets. We report the number of images, instances, categories, covered domains, and the distribution of instances across MS-COCO super categories. As an ongoing effort, we have completed annotation of all COCO validation images since the submission. We refer to this full set as COCO3D v2.\n\nCompared to prior benchmarks—which often focus on specific domains such as indoor environments (e.g., SUN RGB-D), object-centric scenes (e.g., Objectron), or self-driving datasets (e.g., KITTI, nuScenes)—COCO3D offers broader coverage across indoor and outdoor everyday scenes. From the super-category perspective, COCO3D uniquely includes categories from **food**, **animal**, and **sports**, which are often absent in existing 3D datasets. We believe COCO3D can play a similar role in open-vocabulary monocular 3D detection as MS-COCO does in 2D detection—offering a diverse, real-world, and scalable benchmark for advancing the field.\n\n| Dataset | # Images | # Instances | # Categories | Domain | Person | Food | Animal | Vehicle | Kitchen | Furniture | Accessory | Indoor | Sports | Electronic | Outdoor | Appliance |\n|:---:|:---:|:---:|:---:|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| SUN RGB-D | 5,050 | 37,826 | 82 | Indoor | 0.1 | - | - | 0.2 | 6 | 65.5 | 8.0 | 11.8 | - | 7.8 | - | 0.5 |\n| ARKitScenes | 7,610 | 56,404 | 15 | Indoor | - | - | - | - | 10.3 | 80.5 | - | 3.1 | - | 4.1 | - | 2.1 |\n| Hypersim | 7,690 | 239,781 | 29 | Indoor | - | - | - | - | 0.9 | 37.2 | 3.6 | 49.7 | - | 8.5 | - | - |\n| Objectron | 9,314 | 10,814 | 9 | Object  | - | - | - | 3.4 | 32.4 | 12.5 | 20.9 | 12.8 | - | 17.9 | - | - |\n| KITTI | 3,769 | 25,909 | 8 | Driving | 12.9 | - | - | 64.4 | - | - | - | - | - | - | 22.8 | - |\n| nuScenes | 6,019 | 57,735 | 9 | Driving | 15.8 | - | - | 63.1 | - | - | - | - | - | - | 21.1 | - |\n| COCO3D v1 | 2,039 | 5,373 | 80 | In the Wild | 27.9 | 9.1 | 10.9 | 11.0 | 8.2 | 4.2 | 4.6 | 4.1 | 4.5 | 4.0 | 1.0 | 2.0 |\n| COCO3D v2 | 4,025 | 18,395 | 80 | In the Wild | 35.6 | 11.6 | 8.8 | 8.2 | 7.4 | 6.6 | 5.2 | 5 | 4.7 | 3.4 | 2.3 | 1.3 |\n\n**\\[W2 & Q2\\] Analysis of larger model performance on COCO3D benchmark**\n\nWe thank the reviewer for suggesting experiments with larger model capacity. The results are provided in the following table. Both the baseline OVMono3D and our model were initially trained with a frozen DINOv2 backbone. To increase model capacity, we unfroze the DINOv2 backbone for both models, thereby introducing more learnable parameters.\n\nAfter enlarging the model capacity, both the baseline and our model demonstrated performance improvements. This indicates that the backbone produces more task-related features after unfreezing. More importantly, the performance gap between our model and OVMono3D on base categories narrowed from 2 to 1.3 AP points. This suggests that with larger model capacity, models can better handle more diverse scenarios and categories, thus alleviating the catastrophic forgetting issue.\n\nWe also attempted to enlarge model capacity by replacing the backbone from DINOv2-base with DINOv2-large. However, the required training time exceeds the rebuttal period given our computational resources. We will strive to include this analysis in our revised version.\n\n| Method | Model Capacity | $\\text{AP}_{3\\text{D}}$ COCO3D | $\\text{AP}_{3\\text{D}}^{\\text{Rel}}$ COCO3D | $\\text{AP}_{3\\text{D}}$ OVMono3D Novel | $\\text{AP}_{3\\text{D}}$ OVMono3D Base |\n|:---:|:---:|:---:|:---:|:---:|:---:|\n| Baseline: Omni3D | Standard | 5.87 | 20.86 | 16.05 | 24.77 |\n| Baseline: Omni3D | Large | 6.47 | 22.36 | 17.05 | 24.87|\n| Omni3D + LabelAny3D | Standard | 10.92 | 32.02 | 16.98 | 22.74 |\n| Omni3D + LabelAny3D | Large | 11.94 | 33.44 | 17.08 | 23.54 |\n\n\n**\\[W3 & Q3\\] Variance analysis across multiple runs on proposed COCO3D benchmark.**\n\nWe appreciate the reviewer's constructive suggestion\\! We conducted three training runs for each model in Table 1 (except for the baseline, for which we directly evaluated its publicly-released model checkpoint). The evaluation results on COCO3D are shown in the following table, with data presented in mean ± standard deviation format. These results validate that our benchmark provides consistent and reliable model rankings despite the inherent randomness in training procedures.\n\n\n|  | $\\text{AP}_{3\\text{D}}$ | $\\text{AP}_{3\\text{D}}^{\\text{Rel}}$ |\n|---|---|---|\n| Baseline: Omni3D |5.87  | 20.86 |\n| OVM3D-Det | 2.68 ± 0.14 |8.17 ± 0.14  |\n| Omni3D + OVM3D-Det | 6.68 ± 0.27 | 19.92 ± 0.55 |\n| LabelAny3D | 7.68 ± 0.13 | 24.64 ± 0.08 |\n| Omni3D + LabelAny3D | 10.92 ± 0.36 | 32.12 ± 0.26 |\n\n\nWe also conducted multiple evaluation runs for each model using different subsets of COCO3D. Specifically, we sampled three subsets from the COCO3D benchmark, each containing 1500 images. Each model in Table 1 was evaluated on each subset separately, and we report the mean ± standard deviation of AP points in the following table. The results across the three subsets are stable, which further demonstrates the robustness of our benchmark.\n\n|  | $\\text{AP}_{3\\text{D}}$ | $\\text{AP}_{3\\text{D}}^{\\text{Rel}}$ |\n|---|---|---|\n| Baseline: Omni3D |6.10 ± 0.18 | 21.51 ± 0.73 |\n| OVM3D-Det |2.81 ± 0.13  |8.20 ± 0.33  |\n| Omni3D + OVM3D-Det |7.03 ± 0.40  | 21.31 ± 0.42 |\n| LabelAny3D |8.21± 0.46  | 25.32 ± 0.37 |\n| Omni3D + LabelAny3D |11.28 ± 0.62 |32.59 ± 1.46 |\n\n\nWe agree that including more images and instances would enhance the reliability and robustness of the benchmark. Since the time of submission, we have completed annotations for the entire MS-COCO validation set, and we plan to incorporate additional datasets such as Objects365 to further expand the benchmark."}, {"review_id": "ANgwRb8J8H", "reviewer": "Reviewer_MxxK", "summary": "This paper proposes an auto labeling framework, called LabelAny3D, for generating 3D bounding box pseudo-labels for in-the-wild 2D images from the MS-COCO dataset. The produced labels are contributed as a new benchmark COCO3D for training and testing open-vocabulary monocular 3D detection systems. \n\nTo validate the proposed auto labeling method, the authors compare their approach with an earlier work, OVM3D-Det, in terms of bounding box quality against human labels. In addition, the authors compare the performance of a popular detector, when trained using either the proposed LabelAny3D or OVM3D-Det. Results show that the proposed LabelAny3D yields higher quality bounding boxes than its counterpart over the newly created COCO3D benchmark.", "questions": "Please my comments in the weakness section.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Pros\n1. The paper is well written and touches on an important topic with real-world impacts on camera-only autonomous driving and robotics. \n2. The proposed auto labeling framework leads to improved bounding box quality compared to a prior art.\n\n\nCons\n1. The proposed auto labeling pipeline is intuitive and has many hand-engineered modules as well as hyper-parameters. This leads to my concern about the generalization of the proposed approach and sensitivity of those hyper-parameters over a new dataset.\n2. Table 1 shows that adding LabelAny3D pseudo labels only leads to marginal performance improvement or degradation over the OVMNono3D Novel and OVMNono3D Base datasets, when comparing against the inclusion of pseudo labels produced by OVM3D-Det. \n3. Table 2 only evaluates the two auto labeling methods over the newly proposed COCO3D benchmark, which is susceptible to dataset bias and therefore insufficient. If the quality is indeed improved drastically as listed in the table compared to OVM3D-Det, why does LabelAny3D show mixed results in Table 1? I would suggest the authors run evaluation and compare the two methods over the same datasets employed in the OVM3D-Det paper.\n\n\nMinor\n1. The paper seems incomplete without a conclusion section. Please consider adding that section if accepted.", "was_revised": true, "final_justification": "After reading additional results and comments from other reviewers, I raise my rating to borderline accept, as my major concern about generalization has been addressed.", "rebuttal": "We thank the reviewer for the constructive feedback and for finding our \"topic important” and \"well written”.\n\n**\\[W1\\] Concern about the generalization of the proposed approach and sensitivity of the hyper-parameters over a new dataset.**\n\nWe thank the reviewer for raising this important concern about generalization and hyperparameter sensitivity. To address this, we have evaluated our pipeline across multiple datasets **without any hyperparameter adjustment**, demonstrating its robustness.  \n\nIn the original submission, we have validated our pipeline on the **KITTI** benchmark (Lines 280–283). LabelAny3D achieved **13.60 AP3D**, outperforming the baseline OVM3D-Det (**12.39 AP3D**).  In this rebuttal, we provide additional validation on the **SUN RGB-D** benchmark. LabelAny3D surpasses OVM3D-Det by **\\+2.32 AP3D** (41.39 vs. 39.07). We will include these results in the final version of the manuscript.\n\n**\\[W2\\] Adding LabelAny3D pseudo labels only leads to marginal performance improvement or degradation over the OVMNono3D Novel and Base datasets, when comparing against OVM3D-Det.**\n\nThank you for raising this point\\! Both our model and OVM3D-Det are trained on a mix of clean Omni3D labels and generated pseudo labels on COCO, which explains their similar performance on the OVMono3D Novel and Base categories—both derived from Omni3D.\n\nHowever, our model demonstrates superior performance on COCO3D,  demonstrating the strength of our high-quality pseudo labels in more diverse and open-world settings. Due to the underlying distribution differences, the benefit from our high-quality COCO pseudo annotations to Omni3D is naturally limited. Despite this, we still observe a **\\+0.93 AP3D improvement** on OVMono3D Novel categories, while OVM3D-Det shows a **\\-0.5 AP3D drop** on OVMono3D Novel categories. \n\nOVM3D-Det shows slightly higher AR3D, which we attribute to overconfident predictions leading to more false positives. We note that average recall (AR3D) captures detection rate but not the accuracy of 3D localization, which is better reflected in AP3D.\n\nWe will clarify this point in the revised version. \n\n**\\[W3\\] Why does LabelAny3D show mixed results in Table 1? … run evaluation and compare the two methods over the same datasets employed in the OVM3D-Det paper.**\n\nThank you for the constructive suggestion. As noted in our response to W2, while LabelAny3D consistently achieves higher AP3D (precision), OVM3D-Det shows slightly higher AR3D (recall) due to overconfident predictions, which increase false positives. In response to W1, we have also evaluated both methods on the KITTI and SUNRGBD datasets used in the OVM3D-Det paper. These results will be included in the revised version of the manuscript.\n\n**\\[Minor\\] The paper seems incomplete without a conclusion section. Please consider adding that section if accepted.**\n\nThank you for pointing this out. Our **Discussion** section (Sec. 5, Lines 304-316) is intended to serve as the conclusion by summarizing our main contributions and acknowledged limitations. For clarity, we will rename it to **Conclusion** in the revised version."}, {"review_id": "V0CCQn2FZb", "reviewer": "Reviewer_FwMH", "summary": "This paper presents a pipeline to annotate 3D objects from in the wild images. It is an analysis-by-synthesis framework that reconstructs holistic 3D scenes from 2D images to efficiently produce high-quality 3D bounding box annotations. Built on this pipeline, they present COCO3D, a new benchmark for open-vocabulary monocular 3D detection, derived from the MS-COCO dataset and covering a wide range of object categories absent from existing 3D datasets. Experiments show that annotations generated by LabelAny3D improve monocular 3D detection performance across multiple benchmarks, outperforming prior auto-labeling approaches in quality.", "questions": "1. I am confused how to combine MoGe with Depth Pro at Line 141? Why using metric depth estimator there (the paper claim only using relative depth estimator)?", "limitations": "1. The paper should test more models to show the effectivess of their new dataset. Now they only test one model, OVMono3D, which I think it is not enough.  \n2. Why reconstructs the whole 3D scene? I think only reconstrucing the 3D objects are enough to annotate the 3D object bbox.\n3. Can you compare the COCO3D benchmark with other open-vocabulary 3D detection benchmarks?", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "strengths:\n1. clear writing and figures.\n2. It provides a novel and effective pipeline to annotate 3D objects from any images.\n3. A new open-vocabulary 3D detection benchmark derived from MS-COCO.\n\nweakess:\n1. The paper should test more models to show the effectivess of their new dataset. Now they only test one model, OVMono3D, which I think it is not enough.  \n2. Why reconstructs the whole 3D scene? I think only reconstrucing the 3D objects are enough to annotate the 3D object bbox.\n3. Can you compare the COCO3D benchmark with other open-vocabulary 3D detection benchmarks?", "was_revised": true, "final_justification": "The rebuttal is good. I am willing to accept this paper.", "rebuttal": "We thank the reviewer for the constructive comments and for recognizing our “clear writing” and “novel and effective pipeline”.\n\n**\\[W1\\] The paper should test more models to show the effectiveness of their new dataset.** \n\nThank you for the constructive suggestion\\! Open-vocabulary monocular 3D object detection is still an emerging task. To our knowledge, only three works focus specifically on this problem: **OVMono3D** (which we use as our base model), **OVM3D-Det** (our baseline with a pseudo-annotation pipeline), and **DetAny3D** \\[1\\], recently accepted to ICCV 2025\\. DetAny3D’s model weights were released after the NeurIPS submission deadline (on June 25\\) and we have incorporated all its baseline comparisons. Due to time and resource  constraints, we will include results from DetAny3D finetuned on our pseudo labels in the revised version. Since the author of DetAny3D didn’t provide evaluation scripts, it needs some engineering effort to complete. We will strive to provide the evaluation results during the discussion period.  \n\n**\\[W2\\] Why reconstructs the whole 3D scene? Only reconstructing the 3D objects are enough to annotate the 3D object bbox.**\n\nGreat question\\! While we use TRELLIS for 3D object reconstruction, it generates objects in a normalized canonical space, without alignment to image pixels or the global scene. To obtain accurate 3D bounding box attributes—such as center, size, and pose—we need to map these objects back into their true positions within the scene. To achieve this, we first reconstruct a scene-level point map, and then perform 2D–3D alignment using MASt3R and a PnP solver. This alignment step is essential to recover each object’s 3D position and spatial context. More details are provided in Lines 148–159 of the manuscript.\n\n**\\[W3\\] Can you compare the COCO3D benchmark with other open-vocabulary 3D detection benchmarks?**\n\nThank you for the insightful suggestion! The table below presents a statistical comparison between COCO3D and existing 3D detection benchmarks on their respective test sets. We report the number of images, instances, categories, covered domains, and the distribution of instances across MS-COCO super categories. As an ongoing effort, we have completed annotation of all COCO validation images since the submission. We refer to this full set as COCO3D v2. \n\nCompared to prior benchmarks—which often focus on specific domains such as indoor environments (e.g., SUN RGB-D), object-centric scenes (e.g., Objectron), or self-driving datasets (e.g., KITTI, nuScenes)—COCO3D offers broader coverage across indoor and outdoor everyday scenes. From the super-category perspective, COCO3D uniquely includes categories from **food**, **animal**, and **sports**, which are often absent in existing 3D datasets. We believe COCO3D can play a similar role in open-vocabulary monocular 3D detection as MS-COCO does in 2D detection—offering a diverse, real-world, and scalable benchmark for advancing the field.\n\n| Dataset | # Images | # Instances | # Categories | Domain | Person | Food | Animal | Vehicle | Kitchen | Furniture | Accessory | Indoor | Sports | Electronic | Outdoor | Appliance |\n|:---:|:---:|:---:|:---:|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| SUN RGB-D | 5,050 | 37,826 | 82 | Indoor | 0.1 | - | - | 0.2 | 6.0 | 65.5 | 8.0 | 11.8 | - | 7.8 | - | 0.5 |\n| ARKitScenes | 7,610 | 56,404 | 15 | Indoor | - | - | - | - | 10.3 | 80.5 | - | 3.1 | - | 4.1 | - | 2.1 |\n| Hypersim | 7,690 | 239,781 | 29 | Indoor | - | - | - | - | 0.9 | 37.2 | 3.6 | 49.7 | - | 8.5 | - | - |\n| Objectron | 9,314 | 10,814 | 9 | Object  | - | - | - | 3.4 | 32.4 | 12.5 | 20.9 | 12.8 | - | 17.9 | - | - |\n| KITTI | 3,769 | 25,909 | 8 | Driving | 12.9 | - | - | 64.4 | - | - | - | - | - | - | 22.8 | - |\n| nuScenes | 6,019 | 57,735 | 9 | Driving | 15.8 | - | - | 63.1 | - | - | - | - | - | - | 21.1 | - |\n| COCO3D v1 | 2,039 | 5,373 | 80 | In the Wild | 27.9 | 9.1 | 10.9 | 11.0 | 8.2 | 4.2 | 4.6 | 4.1 | 4.5 | 4.0 | 1.0 | 2.0 |\n| COCO3D v2 | 4,025 | 18,395 | 80 | In the Wild | 35.6 | 11.6 | 8.8 | 8.2 | 7.4 | 6.6 | 5.2 | 5.0 | 4.7 | 3.4 | 2.3 | 1.3 |\n\n\n**\\[Q1\\] How to combine MoGe with Depth Pro at Line 141? Why using metric depth estimator there?**\n\nThank you for the great question\\! In our pseudo-annotation pipeline, we align the **relative depth map from MoGe** with the **metric-scale depth map predicted by Depth Pro** using a global affine transformation. Specifically, we first use **MoGe** to generate a relative depth map and extract camera intrinsics. Then, we apply **Depth Pro** to predict a metric-scale depth map for the same scene. We perform RANSAC-based linear regression to fit a robust scale and shift transformation that maps MoGe’s relative depths to Depth Pro’s metric scale. This allows us to retain the fine geometric details from MoGe while calibrating the scale to real-world distances. As a result, our final pseudo 3D annotations are based on **scale-calibrated relative depths**, which we found empirically to improve downstream 3D detection performance compared to using raw MoGe depths alone. This also enables a fairer comparison with baseline methods like **OVM3D-Det**, which rely exclusively on metric depth estimation.\n\nIt’s important to note, however, that we do **not** rely on metric depth at evaluation time on the COCO3D benchmark, due to the general difficulty of accurate metric depth prediction in unconstrained scenes. Instead, we introduce a new evaluation metric—**Relative Layout Average Precision** (Lines 235–245)—to assess how well the predicted 3D bounding boxes preserve the relative spatial layout compared to the ground truth.\n\nWe will clarify this explanation in the final version of the paper. Thank you again for pointing it out\\!\n\n\\[1\\] Detect Anything 3D in the Wild. Zhang et al. ICCV 2025\\."}]

Author rebuttal:

Meta-review: Based on a thorough review of the submission, the reviewer discussions, and the authors' rebuttal, AC recommends accepting this paper, mainly due to the contribution of the new open-vocabulary monocular 3D detection benchmark. The work introduces LabelAny3D, a novel and effective analysis-by-synthesis framework for generating high-quality 3D bounding box annotations from 2D images, and presents the COCO3D benchmark to advance open-vocabulary 3D detection. Reviewers unanimously praised the paper's clear writing, the significance of tackling 3D detection in the wild, and the strong potential impact of the proposed annotation pipeline and dataset. Initial concerns regarding the pipeline's generalization, the lack of comparison to existing benchmarks, and the breadth of experimental validation were comprehensively addressed by the authors during the rebuttal period. The authors provided extensive new experiments across four additional established benchmarks, demonstrating consistent performance gains without hyperparameter tuning and thus validating the method's robustness. They also provided detailed quantitative comparisons of their new benchmark and addressed questions about model capacity and evaluation on symmetric objects. The thorough rebuttal successfully resolved the reviewers' concerns, leading all of them to converge on a positive recommendation for acceptance.
Decision: Accept (poster)