Title: LabelAny3D: Label Any Object 3D in the Wild
Venue: NeurIPS 2025
Accepted: yes
Keywords: Open Vocabulary

3D Detection

Object Recognition
Abstract: Detecting objects in 3D space from monocular input is crucial for applications ranging from robotics to scene understanding. 
Despite advanced performance in the indoor and autonomous driving domains, existing monocular 3D detection models struggle with in-the-wild images due to the lack of 3D in-the-wild datasets and the challenges of 3D annotation. We introduce LabelAny3D, an analysis-by-synthesis framework that reconstructs holistic 3D scenes from 2D images to efficiently produce high-quality 3D bounding box annotations.
Built on this pipeline, we present COCO3D, a new benchmark for open-vocabulary monocular 3D detection, derived from the MS-COCO dataset and covering a wide range of object categories absent from existing 3D datasets. Experiments show that annotations generated by LabelAny3D  improve monocular 3D detection performance across multiple benchmarks, outperforming prior auto-labeling approaches in quality. These results demonstrate the promise of foundation-model-driven annotation for scaling up 3D recognition in realistic, open-world settings.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a pipeline to annotate 3D objects from in the wild images. It is an analysis-by-synthesis framework that reconstructs holistic 3D scenes from 2D images to efficiently produce high-quality 3D bounding box annotations. Built on this pipeline, they present COCO3D, a new benchmark for open-vocabulary monocular 3D detection, derived from the MS-COCO dataset and covering a wide range of object categories absent from existing 3D datasets. Experiments show that annotations generated by LabelAny3D improve monocular 3D detection performance across multiple benchmarks, outperforming prior auto-labeling approaches in quality.\n\nSTRENGTHS AND WEAKNESSES:\nstrengths:\n1. clear writing and figures.\n2. It provides a novel and effective pipeline to annotate 3D objects from any images.\n3. A new open-vocabulary 3D detection benchmark derived from MS-COCO.\n\nweakess:\n1. The paper should test more models to show the effectivess of their new dataset. Now they only test one model, OVMono3D, which I think it is not enough.  \n2. Why reconstructs the whole 3D scene? I think only reconstrucing the 3D objects are enough to annotate the 3D object bbox.\n3. Can you compare the COCO3D benchmark with other open-vocabulary 3D detection benchmarks?\n\nQUESTIONS:\n1. I am confused how to combine MoGe with Depth Pro at Line 141? Why using metric depth estimator there (the paper claim only using relative depth estimator)?\n\nLIMITATIONS:\n1. The paper should test more models to show the effectivess of their new dataset. Now they only test one model, OVMono3D, which I think it is not enough.  \n2. Why reconstructs the whole 3D scene? I think only reconstrucing the 3D objects are enough to annotate the 3D object bbox.\n3. Can you compare the COCO3D benchmark with other open-vocabulary 3D detection benchmarks?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an auto labeling framework, called LabelAny3D, for generating 3D bounding box pseudo-labels for in-the-wild 2D images from the MS-COCO dataset. The produced labels are contributed as a new benchmark COCO3D for training and testing open-vocabulary monocular 3D detection systems. \n\nTo validate the proposed auto labeling method, the authors compare their approach with an earlier work, OVM3D-Det, in terms of bounding box quality against human labels. In addition, the authors compare the performance of a popular detector, when trained using either the proposed LabelAny3D or OVM3D-Det. Results show that the proposed LabelAny3D yields higher quality bounding boxes than its counterpart over the newly created COCO3D benchmark.\n\nSTRENGTHS AND WEAKNESSES:\nPros\n1. The paper is well written and touches on an important topic with real-world impacts on camera-only autonomous driving and robotics. \n2. The proposed auto labeling framework leads to improved bounding box quality compared to a prior art.\n\n\nCons\n1. The proposed auto labeling pipeline is intuitive and has many hand-engineered modules as well as hyper-parameters. This leads to my concern about the generalization of the proposed approach and sensitivity of those hyper-parameters over a new dataset.\n2. Table 1 shows that adding LabelAny3D pseudo labels only leads to marginal performance improvement or degradation over the OVMNono3D Novel and OVMNono3D Base datasets, when comparing against the inclusion of pseudo labels produced by OVM3D-Det. \n3. Table 2 only evaluates the two auto labeling methods over the newly proposed COCO3D benchmark, which is susceptible to dataset bias and therefore insufficient. If the quality is indeed improved drastically as listed in the table compared to OVM3D-Det, why does LabelAny3D show mixed results in Table 1? I would suggest the authors run evaluation and compare the two methods over the same datasets employed in the OVM3D-Det paper.\n\n\nMinor\n1. The paper seems incomplete without a conclusion section. Please consider adding that section if accepted.\n\nQUESTIONS:\nPlease my comments in the weakness section.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes LabelAny3D, a framework for efficiently generating high-quality 3D bounding box annotations from 2D images. Based on the LabelAny3D pipeline, the authors introduce COCO3D, a new benchmark for open-vocabulary monocular 3D detection, derived from the MS-COCO dataset. The COCO3D benchmark comprises 2,039 human-refined images with 5,773 instances spanning all 80 categories of MS-COCO. The authors conduct experiments by training monocular 3D detection models on a dataset annotated via the LabelAny3D pipeline, demonstrating that the performance of monocular 3D detectors can be significantly improved.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. This paper proposes a data processing pipeline to generate pseudo-3D annotations from monocular images. The pipeline leverages several state-of-the-art models (e.g., Depth Pro, Mast3R) to produce plausible 3D bounding boxes.\n2. The authors construct the COCO3D dataset, comprising 2,039 human-refined images with 5,373 total instances, establishing a new benchmark for monocular open-vocabulary 3D detection.\n3. Experimental results demonstrate that:\n(1) The proposed LabelAny3D pipeline achieves remarkable performance improvements compared to state-of-the-art detectors;\n(2) Fine-tuning on pseudo-labels generated by LabelAny3D enhances detector performance on the COCO3D benchmark.\n\nWeaknesses\n1. Missing detailed comparision between the proposed COCO3D dataset and other exist benchmark dataset for open-vocabulary 3D detection. While the authors claim existing benchmarks lack diversity for open-world scenarios, they fail to provide a detailed quantitative comparison between COCO3D and established datasets. A systematic comparison should include: (1) Dataset scale (number of images/instances); (2) Category coverage and diversity;  and (3) Scenario variety.\n2. Finetuning with the pseudo-label generated by LabelAny3D achieve not significant performance improvement even with degration in OVMono3D Novel and OVmono3D Base, the author explain that the eason can be attribute to (1) limited model capacity and (2) label noise. The author should provide a experiments with larger model capacity to validate. Without such evidence, the explanations remain speculative.\n3. The proposed COCO3D only consist of 5373 instances. The statistical significance for reliable evaluation is questionable.\n\nQUESTIONS:\n1. The author should provide detailed comparision between the proposed dataset and existing open-vocabulary 3D detection.\n2. The author should conduct experiments with lareg model architectures to validate the explanations of performance degradation in OVMono3D Novel/base.\n3. The author should provide variance analysis across multiple runs on proposed COCO3D benchmark.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}

comments: SUMMARY:
This paper proposes an annotation pipeline to annotate the 3D bounding boxes given 2D images. Specifically, the authors propose an analysis-by-synthesis framework that first reconstructs holistic 3D scenes from 2D images and then recovers the 3D bounding boxes with absolute scale. Along with the proposed pipeline and benchmark, the authors also train a monocular 3D object detector based on the dataset, which shows superior performance on the designed COCO3D benchmark.

STRENGTHS AND WEAKNESSES:
### Strength
1. The writing is clear, and the paper is easy to follow.
2. The proposed annotation pipeline is theoretically significant, as stated in the paper, it enables labeling of any 3D objects from images.
3. According to the visualizations in the paper, the annotation quality is excellent, and the model trained on the datasets demonstrates strong 3D generalization capability on the COCO dataset.


### Weakness
1. The overall annotation pipeline lacks rigorous evaluation. The authors could validate the annotated 3D bounding boxes on datasets with 3D ground truth from LiDAR data (e.g., KITTI, nuScenes, or ScanNet).

QUESTIONS:
Regarding object orientation, how should the proposed pipeline handle round objects that may lack a clear definition of orientation?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-04