Title: 3D-Agent: A Tri-Modal Multi-Agent Responsive Framework for Comprehensive 3D Object Annotation
Venue: NeurIPS 2025
Accepted: yes
Keywords: 3D object annotation ，Multi-agent systems，Cross-view consistency
Abstract: Driven by the applications in autonomous driving, robotics, and augmented reality, 3D object annotation is a critical task compared to 2D annotation, such as spatial complexity, occlusion, and viewpoint inconsistency. The existing methods relying on single models often struggle with these issues. In this paper, we introduce Tri-MARF, a novel framework that integrates tri-modal inputs (i.e., 2D multi-view images, text descriptions, and 3D point clouds) with multi-agent collaboration to enhance the 3D annotation process. Our Tri-MARF consists of three specialized agents: a vision-language model agent that generates multi-view descriptions, an information aggregation agent that selects optimal descriptions, and a gating agent that aligns text descriptions with 3D geometries for more refined captioning. Extensive experiments on the Objaverse-LVIS, Objaverse-XL, and ABO datasets demonstrate the superiority of our Tri-MARF, which achieves a CLIPScore of 88.7 (compared to 78.6–82.4 for other SOTA methods), retrieval accuracy of 45.2/43.8 (ViLT R@5), and an impressive throughput of 12,000 objects per hour on a single NVIDIA A100 GPU.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a multi-agent framework which take tri-modal inputs to enhance 3D annotation process. Extensive experiments on several datasets demonstrate that the proposed method provides annotations with higher quality and faster throughput.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed method is reasonable, which addresses the limitations of the current 3D object annotations. It may benefit other researchers in the community who need more comprehensive 3D object annotations.\n2. The authors provide thorough experiments on the design choices.\n\nWeakness:\n1. The submission is poorly written, with numerous typos in equations and bad paragraph formatting. For example, in section 3.2.2, the inline equation in line 227, 231, 235 includes additional semicolons and commas. The mutiple bolded sub-titles in the paragraph starting from line 238 should be divided into multiple paragraphs. The quality of the performance metrics figures in the Appendix could be significantly improved.\n2. The proposed method seems to achieve better result by offering a novel combination of existing technique with additional inputs. The reviewer acknowledge the better performance but consider the technical contribution is limited.\n3. The experiment settings is questionable, as the previous methods are not using the similar VLM (or comparable agents). It is not clear whether previous methods with comparable VLMs could achieve comparable results.\n4. Although the authors provide many evaluation metrics, only textual description is provided in the appendix. Concrete math representations would help the readers to get a better understanding on the experimental results.\n\nQUESTIONS:\n1. The authors state that lower Conf(C) (eqa. (1)) values indicate higher confidence (higher token probabilities). It is counterintuitive to define a confidence score that lower confidence score indicates higher confidence. Also, the explanation for eqa. (1) is not sufficient, i.e., why lower confidence score indicates higher confidence?\n2. Does the $S_{\\text{conf},i}$ in line 214 the same as the confidence in eqa.(1)? If so, please use the same notation. Also, is $w_i$ in line 216 the same as $w_{v,i}$ in eqa. (3)? It is confusing to use different subscripts for the same variable.\n3. For the balance score $\\alpha$ in line 216, does it need to change for different datasets?\n4. The core description extraction in line 250 is confusing, as no intuitive explanation on why the authors are doing so.\n5. To ensure the fair comparison, the baselines and the proposed method should use similar/comparable pre-trained backbones/VLM/LLMs.\n6. The authors should provide the concrete math representations for the metrics. For example, the reviewer could not understand why human annotations is worse than VLM generated for GPT-4o scores (line 313).\n7. Why the time reported in Table 3 is not consistent with the inference time reported in sec 4.1 (12k/hour)? What is the difference? How does it compare to baseline methods?\n\nThe reviewer may consider increasing the score if the authors could address the questions, or at least provide some clarification.\n\nLIMITATIONS:\nThe authors describe the limitation in the paper checklist section, including (1) lack of more efficient communication strategies to refine decision-making and reduce computational overhead. (2) Full scene point clouds presents challenges due to its object-centric optimization.\nThe authors does not provide potential negative social impact.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTri-MARF introduces a Multi-Agent Framework for Captioning 3D Models. By using prompting strategies, embedding clustering, score filtering, visual text-alignment, reinforcement learning and geometrical gating they obtain consequential gains over previous methods. They show strong preference over human generated captions and state-of-the-art results in retrieval and clip scores.\n\nSTRENGTHS AND WEAKNESSES:\nQuality: The quality of the paper is decent. Provided figures are understandable and written prose in most places is fairly composed. Overall, the motivation is mostly clear although the literature gaps in previous works is not directly addressed in the evaluations. For example “overlooking geometric information (from previous works)” is mentioned as a pain-point from previous methods but the **main paper** does not directly address this via ablations on the gating agent. Related work fairly covers seminal works but some recent works are skipped.\n\nClarity: Although the motivation is clear and the consequent design choices, the methodology is lacking clarity in terms. Some inline equations are not well setup, variables are inconsistent and some important details are only brushed over.\n\nSignificance: Using a novel reinforcement learning framework adds to the significance of this method which treats FM and LLM agents as part of a more elaborate captioning system which is an interesting approach previous methods have not truly utilized.\n\nOriginality: The methodological novelty as mentioned previously and some new evaluation schemes (Type Annotation Experiment) adds to the originality of the method.\n\nQUESTIONS:\n--Questions--\n1. Section 3.1 line 145 refers to prompting by explicitly orienting the model to the current view-point direction. I have two concerns here: 1) To be able to render a 3D object from the “front” one needs to assume standardized/canonical poses. Is this assumed for the 3D models? If not, how is it determined whether a view is “front, back…”? 2) Are their mechanisms to handle symmetric or cylindrical objects where views may be redundant or is this case considered an experimental outlier?\n    \n2. For Core Description Extraction seems like a lot of faith is put into the LLM here. Is there any experimentation which suggests that LLM are predisposed to producing core descriptions as the first sentence of an output sequence. Is this by any chance an induced behaviour? for example via in-context learning?\n\n3. Given that CLIP is used in Relevance Weighting (3.2.1) and hence the final score which dictates textual relevance, could you justify the use of CLIPScore to measure performance?\n\n4. My major concerns from the result section are regarding A/B Scores in Table 1. If my understanding is correct, an A/B score of 3 is a tie, anything higher indicates that the method being compared against was preferred. In all the datasets, CAP3D, ScoreAgg and 3D-LLM were preferred over the proposed method. Please clarify whether this is the case, if not, specify the win and lose rate (as in CAP3D) for more clarity. If this is expected please further elaborate why this could be happening?\n\n5. line 264-267 seem to imply that the gating agent simply filters out captions which do not align well with the provided geometry. Was there any consideration given to automatic self-refinement so that manual labeling could be limited?\n\n--Suggestion--\n1. The question: “how can we design a system that collaborates like a team of human experts” (line 35) remains to be answered. Ablation over each agent individually and showing A/B scores or VilT retrieval scores for each additional agent would significantly improve quality in my opinion.\n\n\n--Minor concerns--\n1. The experimental setup for measuring A/B testing quality is not clear. How many observations are made for each comparison across all objects? Is this the same setup as described in section 5.1 in CAP3D?\n\nOverall Comment: Although I am generally liking the the novel idea of using RL concepts in agentic frameworks, I do feel that overall the paper did not answer some of its own questions. The lack of ablations in the main-text weakens the paper. Some evaluation concerns like the use of CLIP is not directly addressed here and the A/B scores are not properly analysed. If the above concerns are addressed, possibly with further ablations and evaluations, I am happy to increase the rating.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Tri-MARF, a tri-modal framework for 3D object annotation that integrates 2D images, 3D point clouds, and text. The framework is structured into three components: VLM-based description generation, an information aggregation module, and a gating module to enhance the annotation quality. The paper demonstrates the effectiveness of the framework through experiments on multiple datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n* The framework’s architecture is easy to understand, and the paper is well-structured.\n* The system's decomposition into specialized agents for annotation, aggregation, and validation is an interesting way to address the complexity of 3D object inputs, potentially improving the system's adaptability across various tasks.\n* Tri-MARF shows good performance across datasets, with results surpassing SoTAs in some metrics. The throughput of 12,000 objects per hour demonstrates the system’s efficiency.\n\nWeaknesses:\n* Limited Contribution and Novelty: \n  1. The combination of 2D images, 3D point clouds, and text is not a novel contribution. Existing works like ULIP and 3D-LLM have explored similar modalities. The paper repackages existing techniques rather than introducing truly novel methods. This limits the overall innovation of the paper.\n  2. The core components of Tri-MARF, including the VLM-based description generation, aggregation module, and gating module, are based on widely used models in areas like 3D visual grounding and navigation. The reliance on these off-the-shelf components weakens the paper's contribution, as the paper does not propose novel modules specifically designed for 3D object annotation.\n\n* Insufficient Experiments: \n1. The multi-agent design, which is the paper’s main contribution, is not effectively isolated in the experimental setup. There is no comparison against a baseline that uses a single agent with the same inputs and architecture, which makes it unclear how much the multi-agent setup actually contributes to the overall performance.\n2. While quantitative metrics are strong, the paper lacks a detailed qualitative analysis, particularly regarding failure cases, hallucinations, or semantically inaccurate outputs. This oversight limits understanding of the system's limitations and real-world viability.\n\n* The use of large-scale pretrained models raises concerns about reproducibility. The paper does not address the computational and memory costs of these models, which could impact the scalability of the system. \n\n* The performance of the gating module is heavily dependent on the quality of the visual and textual encoders. If these encoders perform poorly on certain samples, the overall annotation quality may decline significantly.\n\n* The paper has several formatting issues, including small text in figures, unreadable symbols in equations, and inconsistent notation. These problems hinder the clarity of the presentation.\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is an application of large multi-modal models for generating captions for 3D models. \n\nThe method first uses VLMs to generate candidate captions and confidences for each view of the input model. Then, the candidates are clustered, where the candidate with the highest score represents each cluster. The candidate confidences are further weighted by the similarity of the semantic features and the image features. Finally, the candidates are sent to a multi-armed bandit-based agent for selection. The selected caption is accepted if it aligns well with the input point cloud embedding.\n\nThe approach outperforms previous studies on generating 3D model captions in the experiments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The experimental results show clear improvements over previous approaches on the selected datasets.\n2. The supplementary material is very comprehensive and provides an enormous analysis of the proposed framework.\n3. The paper is well-written and easy to follow.\n\nWeaknesses\n1. The proposed framework incorporates several off-the-shelf large models, and the key algorithmic contribution is the RL-based candidate selection (utilizing a multi-armed bandit-based agent). This, to some extent, limits the quality of this paper's technical contribution. In fact, I would encourage the authors to dive deeper into the core learning problem of the framework, i.e., candidate selection. Is there any other approach for candidate selection other than RL? Can the authors compare with other candidate-selection methods in other datasets? Answering these questions will strengthen this paper.\n\nQUESTIONS:\n1. In Line 174, the author mentions, \"Unlike traditional approaches that rely on limited perspectives.\" Could the authors include a few references for traditional approaches? In fact, multi-view images are actually a pretty popular input representation for point cloud classification.\n2. Are all large models used in this study frozen? Did the authors attempt to fine-tune some of the VLM/LLMs used?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}