3D-Agent: A Tri-Modal Multi-Agent Responsive Framework for Comprehensive 3D Object Annotation
Authors: Jusheng Zhang
Yijia Fan
Zimo Wen
Jian Wang
Keze Wang
Venue: NeurIPS 2025 poster ()
Keywords: 3D object annotation ，Multi-agent systems，Cross-view consistency
Primary area: applications
TL;DR: Tri-MARF, a novel tri-modal multi-agent framework, integrates 2D images, text, and 3D point clouds with specialized agents to enhance 3D object annotation, achieving superior accuracy, retrieval, and throughput.

Abstract: Driven by the applications in autonomous driving, robotics, and augmented reality, 3D object annotation is a critical task compared to 2D annotation, such as spatial complexity, occlusion, and viewpoint inconsistency. The existing methods relying on single models often struggle with these issues. In this paper, we introduce Tri-MARF, a novel framework that integrates tri-modal inputs (i.e., 2D multi-view images, text descriptions, and 3D point clouds) with multi-agent collaboration to enhance the 3D annotation process. Our Tri-MARF consists of three specialized agents: a vision-language model agent that generates multi-view descriptions, an information aggregation agent that selects optimal descriptions, and a gating agent that aligns text descriptions with 3D geometries for more refined captioning. Extensive experiments on the Objaverse-LVIS, Objaverse-XL, and ABO datasets demonstrate the superiority of our Tri-MARF, which achieves a CLIPScore of 88.7 (compared to 78.6–82.4 for other SOTA methods), retrieval accuracy of 45.2/43.8 (ViLT R@5), and an impressive throughput of 12,000 objects per hour on a single NVIDIA A100 GPU.

3D-Agent: A Tri-Modal Multi-Agent Responsive Framework for Comprehensive 3D Object Annotation
Jusheng Zhang', Yijia Fan', Zimo Wen', Jian Wang', Keze Wang!
Sun Yat-sen University
Shanghai Jiao Tong University
Snap Inc.
Corresponding author: kezewangggmail.com
Abstract
Driven by the applications autonomous driving, robotics, and augmented reality, 3D object annotation is critical task compared to 2D annotation, such as spatial complexity, occlusion, and viewpoint inconsistency. The existing methods relying single models often struggle with these issues. In this paper, introduce Tri-MARF, a novel framework that integrates tri-modal inputs (i.c., 2D multi-viww images, text descriptions, and 3D point clouds) with multi-agent collaboration enhance the 3D annotation process. Our Tri-MARF consists of three specialized agents: vision-language model agent that generates multi-view descriptions, an information aggregation agent that selects optimal descriptions, and a gating agent that aligns text descriptions with 3D geometries for more refined captioning. Extensive experiments on the Objaverse-LVIS. Objaverse-XL, and ABO datasets demonstrate the superiority of our Tri-MARE, which achieves CLIPScore of 88.7 (compared to 78.6 78.6-82.4 for other SOTA methods), retrieval accuracy of 45.2/43.8 (ViLT @ 5), and an impressive throughput of 12,000 objects per hour single NVIDIA A100 GPU.
Introduction
3D object annotation is crucial computer vision, providing semantic labels for 3D data across autonomous driving [30 5 28
61], robotics, and AR applications. The existing methods primarily use single large-scale models without leveraging multi-agent collaboration, creat- ing challenges with complex scenes 26
24
17
40
78
52 71]. Unlike 2D) annotation, 3D annotation faces unique difficulties: increased spatial re relationship complexity, occlusion issues [3] 32 70)
17
20
73
and viewpoint variations affecting cross- view consistency 45). Traditional 3D annotation methods face serious problems with multi-view data: single models struggle to handle viewpoint differences, geometric complexity, and semantic consistency simultaneously. When objects are partially occluded or only visible from specific angles, existing methods often generate incomplete or inconsistent annotations. Existing approaches, which typically rely on vision-language models, often encounter hallucination problems and description inconsistencies [75] 58 67]. The existing methods struggle to maintain perspective consistency when using multi-view information and often overlook inherent geometric information by relying solely on 2D images 42 [4]
After performing deep and comprehensive analysis of these challenges, find that it is vital to overcome the inherent difficulty a single decision-making system by optimizing multiple competing objectives simultaneously. i.e., accuracy, completeness, consistency [64] 46 41), and efficiency (18). In complex 3D annotation tasks, a single model often struggles to balance these goals effectively [62] akin to a solitary expert lacking proficiency across all domains. These formidable challenges inspire a key question: how can we design a system that collaborates like a team of human experts,
39th Conference on Neural Information Processing Systems (NcurIPS 2025)

---

Cap3d
A Aslivergray stroamiined sports car.
A black dragen, the whow is pizel style.
golden state worriors noa 2417
The strwatllieed sppr ba gray metal body. black lower spoller, cool scisior doors on the side and dark windows, and a low and wide rear, which highilights the futurisdie and whole
a silver-
black maciannical dragen maile metal, with sharp claws and purple eyes, The overall design is full of futurtstic technology and may be Tigare.
The image shows basket ball player in blue and yellow cribble the The
Scoreagg
the WWrrioo' number printed Iadicating that he is member of the Gollen State Warrfors.
fioe
This sleek, modern sports car, specifically Lamborghing rendered in gight
This 3D model is model an Ender
digital 3D model depicta Stophen Curry of the Golden State Warriors in action. He wears a blae and yellow Warrions uniform with the number 30. white amoe padi and turquolse shoes, holding
Dragon Minecralt aat with central
Tri-MARF
V
and carbon fiber. foaturing angles low profile, alloy wheels red Designed for high speed performance, is showcased against a plam whitee background, likely for promotional or lumury display purposes.
rectangular atructure and symmetrizal wing likee fins, Its industral destan.
black
calipers.
includes thrse vertical columns tapped with square elements, sugpesting stractural utiliiy Set
basketball with black Uses, artwork digital textures against
against a light background, it appears Intended for ochibition or themattcc choplays.
neutral background for promotional entertainment tsw..
Figure 1: Comparison example of our Tri-MARE captions with previous SOTA methods, Our Tri-MARF not only accurately recognizes the specific names of objects, but also provides rich and correct details. Some keywords in the annotations are shown in red, and the specific names of the objects are shown in orange. Please note that only our Tri-MARF can mark them out.
Motivated by this, multi-agent systems offer natural framework by decomposing complex tasks into specialized sub-tasks, allowing distinct agents to leverage their respective strengths. For 3D object annotation, this implies deploying dedicated agents to handle geometric feature recognition, semantic understanding, and cross-view consistency separately. However, central challenge in multi-agent systems lies in effectively coordinating the decisions of diverse agents [3 66), particularly under uncertainty and conflicting information. To address this coordination problem, reinforcement learning emerges as an ideal solution. By dynamically learning optimal policies, reinforcement learning techniques [43, 44] decision-making, enable the system to continuously refine and optimize collaborative decision making, surpassing the limitations of predefined rules. Integrating multi-agent systems with reinforcement learning offers numerous advantages [50 35 57 68), including robustness, adaptability, and enhanced performance in tackling complex problems 36 76). However, this introduces new difficulty, e.g., designing appropriate reward signals to evaluate annotation quality and embedding reinforcement learning seamlessly into the workflow.
addressing the task from multiple specialized perspectives 49 55
To address this issue, we propose novel annotation framework called Tri-MARF (Tri-Modal Multi- Agent Response Framework). The core idea to adopt a multi-stage pipeline where specialized agents handle each task, with reinforcement learning incorporated to enhance recision-making particularly in the critical text aggregation phase. As shown in Figure 2 our Tri-MARF consists of four stages to progressively refine and integrate annotation information: Data Preparation Stage: 3D objects are sourced from datasets like Objaverse [T). generating multi-view 2D images and point cloud features to capture structural details; Initial VLM Annotation Stage: A vision-language model (VLM) agent generates preliminary descriptions for each viewpoint. To ensure accuracy, employ multi-round Q&A with Qwen2.5 (39] For each view, five candidate responses are produced, and a ROBERT (3 model clusters these using DBSCAN to yield the text description for that perspective; Reinforcement Learning-Basse Based Information Aggregation Stage: We introduce an agent based on a multi-armed bandit [48] with an upper confidence bound algorithm to aggregate candidate descriptions from different views into a coherent, high-confidence global description.
This agent models multi-view annotation as a multi-armed bandit problem. where each description candidate serves as an "arm" and the system dynamically learns select optimal descriptions through balance. Unlike static rules or simple voting, our agent adapts to different object types and viewpoint scenarios. This agent learns balance visual consistency, geometric accuracy, and semantic richness through reward functions. This agent is trained using composite reward function incorporating VLM confidence scores and CLIP 40 similarity between images and generated captions to dynamically balance exploration and exploitation and optimizing cross-view description consistency through continuous learning; Gating Stage: Cosine similarity between aggregated text and 3D point cloud is computed via an encoder [53), with a threshold determining
exploration-exploiie

---

Apar C-CARR
9
Embeddiin Clustering
MMV
Phot Reaan
399999
Evt Sepaak
Visual--a zoant Aiggimmnt
2223 2 ?
mad
Candibere
Spetie
Simiham CSse
Madel
MVW
Prevrig
Eventppe
Surder)
Figure 2: The illustration of our Tri-MARF for 3D object annotation, featuring collaborative multi-agent mechanism. The process starts with Agent (VLM Annotation Agent), which uses visual language model (e.g., (wwe2..5-VL-72B IInstruct to generate 5 text descriptions for each view of 3D object from six standard viewpoints (front, back, left, right, top, bottom). These descriptions are then processed by Agent 2 (Information Aggregation Agent), which uses RoBERTa+DBSCAN for semantic embedding clustering, CLIP for visual-text alignment, and integrates multi muulti-armed bandit (MAB) model to optimize description selection and balance exploration and exploitation to obtain the final captions. Agent 3 (Point Cloud Gating Agent) uses threshold control to align text and 3D point clouds, further reducing the wrong results produced by VLM annotations. Please note that our point cloud is pre-rendered asset.
further annotation needs. Adaptive agent outperforms rule-based methods by dynamically adapting to scenes and learning from errors to optimize 3D annotation quality (Figure
This mainpaper introduces Tri-MARF, a novel multi-stage annotation framework that integrates specialized agents to tackle inconsistencies in 3D object annotation. By leveraging multi-agent col- laboration with reinforcement learning. Mri- enhances decision-mate maaiinn and ensures annotation consistency, offering robust and adaptive solution. Extensive experiments on benchmarks like Objaverse-I.VIS, Objaverse-XI, and ABO demonstrate Tri-MARF's superior performance, surpass- ing existing methods in annotation accuracy, description consistency, and linguistic quality. To date, it has annotated approximately million 3D models.
2 Related Works
Neural 3D Object Annotation has evolved from manual labeling to automated approaches. Chang
al. 7 established ShapeNet, Mo al. (2 developed PartNet, Yi et [63] focused on semantic
segmentation, and Savva et al. [38 60) contributed SHREC benchmarks. Recent vision-language
models transformed this landscape: ULIP (88 bridged 3D point clouds and text, PointCLIP (75)
adapted CLIP for point clouds, while (45) introduced cross-modal embeddings and Zeng et al. [51
explored prompt engineering. Cap3D 26 pioneered synthetic-to- transfer but struggled with
cross-view consistency. 3D-LLM [17 attempts to solve this via specialized objectives, while [32
developed a complex scene understanding method.
Multi-Agent Systems for Visual Understanding decompose complex visual tasks into manageable subtasks to enhance efficiency and accuracy in visual understanding. Maes et al. (27 established the foundational concept of collaborating agents, Zhong et al. [77] demonstrated that specialists outperform monolithic models, and Deng al. [13] introduced multi-agent approaches 3D scene understanding. Deng et al. [47] and Zhong et al. [77 showed improved robustness through viewpoint integration, while Aghasian al. M developed hierarchical protocols for agent collaboration, and Chafi et al. [4] implemented emergent communication between agents. For scene comprehension. Wei al. [54] proposed 3D scene graph generation frameworks, Johnson et al. [19]s 's DenseCap system demonstrats the importance of specialized roles for image annotation, and Cai et al. B and Liu et [22] explored dynamic agent routing to enhance system adaptability. Unlike these fixed-protocol systems, our Tri-MARF introduces tri-modal approach with multi-agent collaboration to optimize collaboration, adaptively weighting information across different modalities to maintain high performance across varying scene complexities.
3

---

Reinforcement Learning for Decision Making in Vision Systems has employed reinforcement learning to optimize viewpoint selection in 3D environments [30], complemented by advances in exploration strategies that improve training efficiency for visual perception tasks 65 Recently, multi- armed bandit algorithms [48], including upper confidence bound strategies that balance exploration and exploitation (2], have been well researched. These approaches have proven effective for content selection in visual applications [79] and have been extended to contextual settings where visual representations inform decision-making [8). Multi-modal fusion research has advanced through adaptive weighting mechanisms based on reinforcement learning principles [16], particularly in
lision-language tasks where modality importance varies contextually [15] 69]. Recent advances 72 74 demonstrate the ongoing evolution of these approaches for complex decision- decision-making tasks. In contrast, our Tri-MARF implements simple yet effective architecture with specialized agents for 3D annotation tasks, demonstrating superior adaptability across diverse object categories and viewpoint conditions.
3 Methodology
Our Tri-MARF framework addresses key challenges in 3D annotation through three spe- cialized agents in the following four-stage annotation pipeline. Formally, suppose V {front, back, left, right, top, ]ottom) represent the standardized viewpoints. Data Preparation Stage generates six corresponding images {Lo E V} for each object, which are encoded and passed to the vision-language model without manual feature engineering. For each viewpoint v, the Initial VLM Annotation Stage produces: $D, {C.,,) 2 1= $ set of M candidate descriptions (Tri- MARF employs M = 5 with temperature-contuen
sampling) each with an associated confidence score. Information Aggregation Stage transforms these texts into BERT embedding space for semantic clustering, while CLIP evaluates each description's visual alignment with image L, This dual-evaluation process produces scored response pairs (Co,à, Se,)). where s,, represents a composite score of semantic distinctiveness and vision-text correlation. In Information Aggregation Stage, we frames candidate descriptions as arms in multi-armes bandit problem, selecting one description Ō₂ per view through UCB (Upper Confidence Bound) pellortiionexpllitteene tion aggregation agent continuously updates reward estimates to favor higher-quality descriptions as it learns. Finally, we fuse the optimized view-specific descriptions C, : V into coherent global annotation, with weighted emphasis on informative perspectives. Gating Stage input the description obtained in the previous step into the pretrained text encoder for encoding. At the same time, we input the 3D point cloud of the model obtained in the preparation stage into the pre-trained point cloud encoder for encoding. Then, we calculate the cosine similarity between the encoded text and the point cloud to determine whether the cosine similarity is greater than the empirical threshold a 0.557 (Please refer to Supp. [1.6). If it is greater than this threshold, the corresponding sample is retained; if is less than this threshold, the sample is marked as a questionable sample for manual annotation.
balancing. The informa-
3.1 Initial VLM Annotation Stage
For each view image I,, Tri-MARF employs a sophisticated Aisionllaguaag model agent that uses an innovative multi-turn prompting strategy. Unlike conventional single-prompt approaches, we implement structured dialogue system with analysis. Our prompting protocol unfolds in three strategic phases: Viewpoint-aware identification. We orient the model to recognize its viewing perspective (e.g.. "This is the front view. What object do you see and what is its specific name?"), ensuring attention focuses on viewpoint-specific diagnostic cues. Systematic attribute elicitation. We use targeted follow-up prompts clicit key attributes such as color, material, and structural components, guaranteeing sufficient feature coverage even under complex viewpoints. Contextual integration. The extracted observations are integrated into consistent, coherent descriptions that preserve viewpoint alignment and emphasize distinguishing characteristics. This transforms annotation quality by decomposing complex visual reasoning into manageable sub-tasks, yielding significantly more detailed and accurate descriptions than conventional single-prompt methods. To maximize semantic coverage and reduce annotation bias, we empirically introduce stochastic diversity sampling at temperature = 0.7 with M = 5 descriptions per view, generating alternative interpretations that capture different object aspects. Each description Cp,a retains levenl log- bog-prbbabilities, enabling sophisticated confidence assessment.
1...--VLL72B-Insttrule that mirrors expert visual

---

Confidence Score Computation. We introduce a novel probabilistic confidence metric for each description, addressing the critical challenge of uncertainty quantification in 3D annotation. The con- fidence score Conf(C') quantifies semantic reliability through average token log-likelihood, providing a principled measure of model certainty. For tokens 11, 1,,12...,,4 with conditional probabilities P(t, context), we compute:
N
Conf(C) 1
Σ N
log P(t. context up 4:)l-
i=1
This formulation captures the model's internal uncertainty during generation-lower Conf(C)) val- ues indicate higher confidence (higher token probabílities), while elevated scores signal potential unreliability (perhaps from rare descriptors or uncertain attributions). This confidence metric serves dual purposes in our reinforeement learning pipeline: flagging potentially hallucinated content for rejection, and informing Sandit-based selection between semantically similar candidate descriptions. This probabilistic approach to confidence assessment represents significant advancement over deterministic methods, enabling more reliable annotation in ambiguous situations.
Importance of Multi-View Inputs. Our Tri-MARF innovatively processes six standard views of each 3D object, addressing the fundamental challenge of viewpoint inconsistency that plagues single-view view methods. Unlike traditional approaches that rely on limited perspectives, our Tri-MARF captures com- prehensive spatial relationships through front, back, left, right, top, and bottom views. This deliberate design tackles the inherent difficulty of 3D annotation objects often conceal critical features from any single viewpoint. For instance, a vehicle's diagnostic features distribute across multiple angles: brand identifiers the front, distinctive lighting arrays at the rear, profile silhouettes from sides, and functional components from top/bottom perspectives. Our Tri-MARF approach systematically mitigates occlusion problems by exploiting complementary information across perspectives, creat- ing an integrated understanding impossible with conventional methods. This redundancy provides crucial resilience: when noise or occlusion compromises one viewpoint, alternative angles maintain annotation integrity. Furthermore, our Tri-MARF supports cross-view verification, confirming the existence of features across multiple viewpoints and reducing the inconsistency problem prevalent in standard VLM methods. This comprehensive spatial coverage forms the foundation for Tri-MARF'ss exceptional accuracy and completeness.
3.2 Information Aggregation Stage
After obtaining multiple description candidates per view, our Tri-MARF employs the aggregation agent to perform semantic clustering to eliminate redundancy and then relevance weighting to evaluate each description Ce.,,..., Ce.M. These steps transform raw list of M descriptions into smaller set of unique, scored responses, setting the stage for the final selection. To identify when different generated sentences are essentially saying the same thing, we project each candidate description into a high-dimensional semantic space using a pre-trained language model (BERT). Let C..; and Ce. be two candidate descriptions for the same view u. We compute their embeddings E BERT(C,,,) F = BERT (C..g) as fixed-length vectors. The semantic similarity between the two descriptions cosine similarity of their embedding vectors: Ep, Ev,j Sij = cos (Ev,is Ev,)) = | Brsll
performing this step, Tri-MARF condenses the candidate descriptions, removing duplicative entries and preparing canonical description for each distinct idea. From each cluster, we select represen- tative deseription. Ideally, all descriptions in one cluster are paraphrases, so any could serve as the cluster's exemplar. Tri-MARF chooses the highest-scoring description in each cluster as the canoni- cal representative. At this stage, we have not yet defined the score that comes next with the CLIP weighting, but once scores are assigned, we compute: canonicl := arg max Co.i Cksu, i, which produces a set of unique descriptions for the view one per cluster. These are the candidate descriptions for competing in the final selection.
3.2.1 Relevance Weighting
Next, our Tri-MARF evaluates how well each candidate description is grounded in the actual image using CLIP CLIP provides a function that maps an image and text T into shared feature space such that related image-text pairs have high cosine similarity. We obtain an image embedding
5

---

I gig (I+) for the view v and a text embedding Te,: = SCLiP (Ce.). The relevance of description
C.., to image I, is: cos u |T We convert the raw similarity into a probabilistic weight via a softmax over the M descriptions of that view:
Iu,
exp(cos0e.)
W0,i 2a-1 exp(cos 0,.x)
(3)
The candidate descriptions that align better with the image are assigned higher weight. Then, our Tri-MARF combines the semantic clustering information and the CLIP visual alignment into a single score for each description. Let S denote confidence score for candidate description i, and let w; be the CLIP-based weight after softmax normalization. The final weighted score is: & := (1 - a) ,t + a wi, where a E [0, 1] controls the balance between text similarity and image-text similarity. In Tri-MARF, a smaller a prioritizes text-based confidence, while larger emphasizes visual-semantic alignment, combining signals for responses with both textual relevance and visual correctness.
3.2.2 Multi-Armed Bandit-Based Response Aggregation
Even after clustering and scoring, there may be multiple plausible descriptions for a given view. Rather than arbitrarily picking the top one, the information aggregation agent of our Tri-MARF uses a multi-armed bandit (MAB) model to adaptively select the best description over time, es- pecially when feedback signals are available. Define the set of arms sponding to the K canonical descriptions for the current view. When Tri-MARF chooses arm ag (i.e., uses description ((k)
a a2....... aK corre-
camomical as the annotation), it receives reward r% that reflects the an-
notation quality. Over many trials, the goal is to maximize:max,;; Et=1 Ta: .e assume
each arm O% has an underlying expected reward μa. The challenge is the classic exploration- exploitation trade-off: the algorithm should try different arms learn their rewards but also exploit the best one found so far. Tri-MARF employs the UCB1 variant, which calculates for each arm
a: 0;; arg A fa: +ic 22mm where fa is the empirical mean reward, na is how many times arm a has been chosen, is the current round, and is an exploration weight. This rule formalizes "optimism in the face of uncertainty," ensuring that arms with high potential or insufficient exploration are tried sufficiently. When a reward ra is observed, the empirical mean is updated by fa <- (n. (I. Over time, Tri-MARE converges to favoring the arm with the highest true reward. We chose UCB due to its simplicity, strong regret bounds, and easy interpretability.
Cross- View Processing and Global Descrip- tion Synthesis. Once the final descriptions of the individual views are selected, we ensure consistency and fuse them into global 3D object ummotation.FrontBBact. View Prioriti- zation. Front and back views receive priority, assuming they carry critical identifying infor- mation about category and appearance. We as- sign higher weight WEB to these descriptions: Priority(Cres) wFB Score(CrB). In prac- tice, front/back descriptions identify the object while other views (side, top, bottom) provide supplementary details. Core Description Ex- traction. From the front/bbcc combined description OFB, Tri-MARF extracts only the first sentence as the core identification sentence: Sere = Leass_SSeeeneeerrr description Cotber is formed by selecting the best or longest candidate from side/top/bottom views. Global Description Assembly. The final global description is then: Cglobal Scure + Cother- A
Point Cloud
VTT besed 30 encoder
Alphcs 0.557 GATE
The retrosstye sirorott much figiner jet from 1920s
NO
Condidate
Aligned CLIIP encader
Figure 3: Detailed demonstration of the gating agent of our Tri-MARF. The pre-trained Uni3d en- coder is used to handle point cloud and text match- ing on the open domain.
For other views, compiled
scoring formula like Scoreglobal = aligns with both identity and detailed attributes.
Score evaluates how well the merged description

---

Table 1: The comparison of caption quality and efficiency for 3D object annotation. The highest value of each metric is bold. The two values of ViLT R@5 (e.g. 45.2/43.8) represent the retrieval performance of Image-to- Textt (I2T) and Text-to-Iman Image (T21) respectively.
ObjaerseLV
Objavene- CLiPSoone
Secce
CLiPScorc
Human Amnotation
2.3
52.4
40.01 38,3
$1.0
37,0733.55
2.9
73.9
23.8 32.5
0.12%
Our Ti-MARF
53.7
45.2/43..
36.1
40.5/38.7 38.9
$2.3
37.11 35.6
12L
Cbp
3.3
78.6
35.2/33.4
3,5
76.4
32.1 30.5
3.5
74..8
28.9:27.3
8k.
ScornAgz
3.9
30.1
37.8/ 36.00
37
78..
34,5/
4.2
76,2
31.21 30,0
9k.
MMLLIM
3.2
77.4
34.9/331
75.6
31.8/ 30.3
3.3
73.0
38.4/ 36.9
6.5t
PoinCCLPP
2.0
65.3
22.4/20.8
63.1
19.57
2.2
17.21 15.7
ULIP-2
33.1/31.5
29.7/26.2
GPT4PPiint
18.7/17.1
14.3/
3.3
Gating Stage
To mitigate limitations of traditional 2D image annotation in discriminating geometric properties, we introduce similarity gating agent based on point cloud-text alignment, as shown Figure We employ pre-trained encoders E, (3D point cloud) and E, (text) to extract geometric and semantic features respectively, both in IR dimension. Cross-modal matching is quantified using: E, Cosine Similarity | where denotes dot product and represents L.2 norm. Based on validation grid search, we set dynamic threshold a = 0.577 as confidence criterion. When similarity falls below threshold, the gating agent triggers dual--heck: critical category samples undergo manual review while redundant samples are filtered out. This geometric-semanti consistency gating effectively suppresses annotation hallucination in visual language models and better leverages intrinsic 3D object information.
3
4
Experiments
We rigorously evaluate our Tri-MARF across four experiments to validate its 3D understanding
capabilities: (1) Caption quality assessed on Objaverse-LVIS [12], Objaverse-XL. [T]. and ABO
[ via A/B testing against human annotations and automated metrics (CLIPScore, ViLT 21 retrieval).
(2) Type inference accuracy on Objaverse-LVV
is compared against CAP3D 25). ScoreAgg (20].
(3) The effect of selecting different numbers of viewpoints on the annotation quality is used to justify why we choose viewpoints instead of viewpoints in the previous work [78]. (4) We also conducted annotation experiments on clean 3D point cloud datasets and real-world noisy datasets, demonstrating that our Tri-MARF has high generalization performance. Please refer to Supp. 14 for more detailed experimental settings.
and Human Annotation using GPT-4o [33] scoring and human validation.
4.1 3D Captioning Test
Experimental Setup. We evaluate the caption quality of our Tri-MARF for 3D object annotation Objaverse-LVie (1k sampled), Objaverse-XL. (5k sampled), and ABO (6.4k objects). The captions of our Tri-MARF are compared with those from Cap3D, ScoreAgg, ULIP-2 (59). PointCLIP (75), 3D-LLM 17), GPT4Point (37), human annotations, and metadata in terms of quality and efficiency. Random sampling ensures representativeness. Quality is measured via A/B testing (1-5 scale), CLIPScore, and ViLT retrieval. Note that the speed here is estimated according to the overall rate. Baseline models follow official configurations. Tri-MARF's detailed settings in Supp. We also explored the model's s ability to understand scenes (See Supp. 99 Experimental Results and Analyses.
As shown in Table [ our Tri-MARF achieves state-of-the-art performance across all semantic alignment metrics while maintaining the highest annotation throughput (12k objects/hour) on single NVIDIA 1000 GPU. By design, Tri-MARF serves as the reference baseline for human preference evaluation (A/B scores), implicitly outperforming all methods through pairwise comparisons. Tri- MARF dominates CLIPScore (88.7 vs. 78.6 82.4 on Objaverse-LVIS and ViLT R@ 5 (45.2/43.8 vs. 35.2-40.. 40.0), demonstrating superior cross-moda. aligmment. Notably. Tri-MARF-gee --AR--geeeetedd captions surpass human annotations in semantic precision (CLIPScore +6.3 on ABO) while avoiding human annotators' preference bias. Please refer to Supp. 12 for more details. Tri-MARF excels in 3D object captioning, with higher CLIPScore and ViLT retrieval accuracy showing effective feature capture by the multi-agent approach. High A/B test scores confirm the reinforeement learning- based aggregated
7

---

Table 2: Cross-dataset generalization experimental results comparison. The highest values highlighted in bold.
ShapeNet-Core
Sea
ModelNet40
Method
CLIP
VILT R@S1
GPT-47
CLIP
VILT R@st
GPT-4T
CLIP
VILT R@st
GPT-4t
Human Annotation
81.7
37.8/ 36.0
4.2
79.5
34.8/33.2
4,3
80.2
36.0/ 34,5
4,0
Tri-MARF
83.2
38.6 36.8
4.3
80.3
35.7/33.7
4.0
36.7/35.2
Cap3D
76.5
33.1/3155
3.6
73.2
29.8/28.1
3.2
74.3
31.2 29.8
3.4
ScoreAgg
33.9/
75.6
3.5
3D-LLM
75.8
32.5/30.9
3.5
72.5
29.1/27.6
3.1
73.6
30.7 29.2
3.3
PoinCCIIP
63.4
20.2
2.3
60.8
19.3/179
2.1
62.1
20.5/19.1
ULIP-2
73.7
31,4/29.8
3.3
70.6
26.8/26.3
2.9
72.3
29.4/ 28.0
3.1
GPT4Poinl
61.2
19.5/
58.7
1744 16.0
1.9
60.3
18.9 17.5
2.0
agent aligns with human description habits. Faster runtime underscores the lightweight MAB strategy, ensuring efficiency for large-scale 3D dataset annotation.
4.2 Type Annotation Experiment
To evaluate the performance of various classification methods the Objaverse-LVIS dataset, compare Tri-MARF and ScoreAgg against CAP3D and manual annotation. The comparison focuses on two primary evaluation metries: (1) the accuracy string matching between predicted and ground- truth labels, and (2) the semantic accuracy assessed by GPT-4o after comparing model-generated subtitles with standard answers account for potential synonym mismatches. This experimental design ensures a comprehensive evaluation of both syntactic and semantic alignment. We also compare caption results with a single-agent baseline using the same input and architecture(See 7 The first evaluation metric, string matching accuracy, measures the direct correspondence between the predicted labels and the ground truth. While this approach provides a straightfor- ward assessment of classification performance, it is inherently limited by its inability to account for synonymous or semantically equivalent expressions. For instance, "coffee mug" and "cup" may represent the same object but would be flagged incorrect under strict string matching. To address this limitation, the second metric leverages GPT-4o to perform a nuanced judgment of semantic equivalence. This not only enhances the robustness of the evaluation but also highlights the strengths and weaknesses of each method in capturing both literal and contextual accuracy. As shown in the Figure 4 in the semantic accu- racy score of GPT-4o, Tn-MARF achieved the highest accuracy (98.32%), which is about 2.6 percentage points higher than the accuracy of manual annotation (95.72%). This result shows that Tri-MARE can more accurately identify 3D categories and effectively integrate multi- view information. In the string matching score, Tri-MARF obtained the highest score (47. (7.28%) except for manual annotation, which has a nat- ural advantage due the special nature of the "multiple choice question" format (see supple- mentary materials for details). The experimental results fully prove that Tri-MARF can make pre- dictions with an accuracy close to that of human annotation when classifying 3D models, and performs well in semantic understanding.
Annotation Method Performance Comparison
100 91.72% 55.72%
72%L
Ramerual
ScortAp
Antotal Methods
Shing Seate
WFT Si
Figure 4: Classification accuracy of the four anno- tation methods on Objaverse-LVis by using string matching and GPT-4o scoring.
4.3 Number of Perspectives
To comprehensively evaluate the impact of the number of views on the performance of Tri-MARF in the 3D object description task, we conducted detailed comparison with existing multi-view rendering methods on the Objaverse-LVIS (optional 1k) dataset. We specifically selected two representative multi-view methods, Cap3D and ScoreAgg, as comparison benchmarks, and systematically tested the impact of different numbers of views (1, 2, 4, 6, 8) performance. The evaluation uses a variety of complementary indicators, including CLIPScore, ViLT retrieval rate (R@5), BLEU-4 [d and A/B test scores, to comprehensively measure the semantic consistency, retrieval ability, text fluency,

---

and human preference of the generated descriptions. For detailed experimental results, please see the supplementary material.
Figure 5 shows that when the number of input views is 6, all multi-view methods achieve the best performance, indicating that the 6 standard views (front, back, left, right, top, and bottom) provide the most comprchensive geometric and appearance information for 3D objects. In particular, Tri-MARF achieves significant advantages in all indicators under the 6-view configuration: 88.7 CLIPScore, 46.2/44.3 ViLT R@S, and 26.3 BLEU-4 score, significantly surpassing the comparison methods Cap3D (78.1 CLIPScore, 34.2/32.7 ViLT R@5, 22.6 BLEU-4) and ScoreAgg (79.3 CLIPScore, 35,9/34.3 ViLT R@5, 23.5 BLEU-4).
All methods peak at 6 viewpoints, then de-
cline due to redundant information affecting ef-
CLIPScore Trends with Varying Number
25
ficiency and consistency. Tri-MARF's multi- index evaluation demonstrates its superior per- formance in multi-view 3D object description, confirming the effectiveness and robustness of its multi-agent collaborative architecture across
Ca:33
Scorolog Tii HARS
CUPscoe
20-
various evaluation dimensions.
65
4.4 Generalization Ability Across Datasets
To comprehensively evaluate the generaliza- tion ability of Tri-MARF on data with dif- ferent distributions, we designed systematic cross-dataset experiment. Specifically, we se- lect three datasets with different characteristics for cross- dooss- testing, ie... ShapeNet-Coree IIO, and ModeINet40[561, and randomly selected 500 samples from each dataset to form test set with balanced category distribution. In the experimental process, we use the Tri-MARF model that is pre-trained on the Objaverse series of datasets (without fine-tuning), and generate six-view rendering images, sample point clouds, and complete the full annotation pipeline processing for each 3D object according to the standard process. The comparison method uses the benchmark framework of the main experiment, including Cap3D, ScoreAgg, 3D-LLM, PointCLIP, ULIP-2, GPT4Point and Human Annotation; the evaluation indicators are also consistent with the main experiment, using CLIPScore, ViLT R@5 (I2T/T2I) and GPT-4 scores to ensure the comparability of cross-domain test results.Table sholas Net40, second only to manual annotation. Compared to the original test set, Tri-MARF's CLIPScore drops by 7.2% (least), Cap3D by 11.5%, ScoreAgg by 9.8%, and others (3D-LLM, PointCLIP, etc.) by 10-15%. Tri-MARF's strong generalization stems from its reinforcement carning-based infor- mation aggregation and point cloud threshold agent, which mitigates inconsistencies by aggregating valid visual language model responses.
Number Views
Figure 5: Comparison of CLIPScore trends with varying view number on Objaverse-LVIS (Ik).
shows Tri-MARF outperforms other methods on Shape
ScanNet, and Model-
4.5 Ablation Studies
To provide detailed analysis of the sensitivity of our Tri-MARF to hyperparameters, conduct
variety of ablation studies in Supp. [1 e.g... different VLMMs in Supp. 11.1 reinforcement learning
strategy selection in Supp. 11.2 multi-view comparison in Supp. 11.3 object categories in Supp.
11.4. hyperparameters more details of human evaluation listed in Supp. [2] We also conducted experiments prove the
Supp. 11.5) and gating threshold for 3D point cloud in Supp. [1.6]. Besides,
marginal benefits of multi-armed bandit compared to traditional methods in Supp. g
5 Conclusion
In this paper, we presented novel multi-stage annotation framework. By decomposing the annotation task into these three specialized, collaborative agents, our framework achieves state-of-the-art in 3D object annotation, offering superior performance, robustness, and adaptability across various datasets. In the future, plan to focus communication strategies among agents to refine pocision-making

---

and reduce computational overhead. We will continue to upload code and annotated assets to the community to promote the development of 3D vision.
Acknowledgements
This work supported in part by the National Natural Science Foundation of China (NSFC) under Grant 62276283, in part by the China Meteorological Administration' 's Science and Technology Project under Grant CMAJBGS2025177, in part by Guangdong Basic and Applied Basic Research Foundation under Grant 8023A15150122 2985, in part by Guangdong Hong Kong- Mang-Macao Greater Bay Area Meteorological Technology Collaborative Research Project under Grant GHMA2024Z04, in part by Fundamental Research Funds for the Central Universities. Sun Yat-sen University under Grant 23hytd006, and in part by Guangdong Provincial High-Level Young Talent Program under Grant 1112224151---11
References
[1] Erfan Aghasian, Shai Avidan, Piotr Dollar, and Justin Johnson. Hierarchical protocols for multi-agent 3d scene understanding. CVPR, pages 7664-7673, 2021.
[2] Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47(2):235-25 256,, 2002.
[3] Shaofeng Cai, Yao Shu, Wei Wang, and Beng Chin Ooi. Dynamic routing networks, 2020.
[4] Marwa Chafii, Salmane Naoumi, Reda Alami, Ebtesam Almazrouei, Mehdi Bennis, Merouane Debbah. Emergent communication in multi-agent reinforcement learning for future wireless networks, 2023.
[5] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich 3d model repository, 2015.
[6] Angel X. Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich 3d model repository, 2015.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang3, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, Jianxiong Xiao, and Fisher Yu. Shapenet: An information-rich 3d model repository, 2023.
[8] Xinlei Chen, Li-Jia Li, Li Fei-Fei, and Abhinay Gupta. Iterative visual reasoning beyond convolutions. In CVPR, pages 7239-7248, 2018.
[9] Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Zhang, Tomas F. Yago Vicente, Thomas Dideriksen, Himanshu Arora, Matthieu Guillaumin, and Jitendra Malik. Abo: Dataset and benchmarks for real-world 3d object understanding, 2022.
[10] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nicßner. Scannet: Richly-annotacee 3d reconstructions of indoor scenes, 2017.
[11] Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, Eli VanderBilt, Aniruddha Kembhavi, Carl Vondrick, Georgia Gkioxari, Kiana Ehsani, Ludwig Schmidt, and Ali Farhadi. Objaverse-XL: A universe of 10m+ 3d objects. In Thirty-seventh Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[12] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsanit, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In 2023 IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pages 13142-13153, 2023.
[13] Jian Deng and Krzysztof Czamecki. Mlod: multi-vier 3d object detection based on robust feature fusion method. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 279 279-284, 2019.
[14] Yasutaka Furukawa and Carlos Hernández. 2015.
10

---

[15] Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven C. H. Hoi, Xiaogang Wang. and Hongsheng Li. Dynamic fusion with intra- and inter-modality attention flow for visual question answering. In CVPR, pages 6632-6641, 2019.
[16] Judy Hoffman, Mehryar Mohri, and Ningshan Zhang. Algorithms and theory for multiple-source adaptation, 2018.
[17] Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models, 2023.
[18] S. S. Hotegni, M. Berkemeier, and S. Peitz. Multi-objective optimization for sparse deep multi-task learning. 2024.
[19] Justin Johnson, Andrej Karpathy, and Li Fei-Fei, DenseCap: Fully convolutional localization networks for dense captioning. In CVPR, pages 4565-4574, 2016.
[20] Rishabh Kabra, Loic Matthey, Alexander Lerchner, and Niloy J. Mitra. Leveraging vlm-based pipelines to annotate 3d objects, 2024.
[21] Wonjac Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and and-language transformer without convolution or region supervision, 2021.
[22] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search, 2019.
[23] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoeer and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach, 2019.
[24] Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with pretrained models, 2023.
[25] Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with pretrained models, 2023.
[26] Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with pretrained models, 2023.
[27] Pattie Maes. Agents that reduce work and information overload. Communications of the ACM, 37(7): 30-40, 1994.
[28] Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su. Partnet: A large-scalc benchmark for fine-grained and hierarchical part-level 3d object understanding, 2018.
[29] Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In CVPR, pages 909-918, 2019.
[30] Arsalan Mousavian, Alexander Toshev, Marek Fišer, Jana Košecká, Ayzaan Wahid, and James Davidson. Visual representations for semantic target driven navigation. In ICRA, pages 8846 8852, 2019.
[31] Nurul Ai'zah Musa, Mohd Zaliman Mohd Yusoff, Roslan Ismail, and Yunus Yusoff. Issues and challenges of forensics analysis of agents' behavior in multi-agent systems: A critical review. In 2015 International Symposium on Agents, Multi-Agent Systems and Robotics (ISAMSR), pages 122-125, 2015.
[32] Gaku Narita, Takashi Seno, Tomoya Ishikawa. and Yohsuke Kaji. Panopticfusion: Online volumetric semantic mapping at the level of stuff and things, 2019.
[33] OpenAI. Gpt-4 technical report, 2024.
[34] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu, Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311-318, Philadelphia, Pennsylvania, USA, 2002. Association for Computational Linguistics.
[35] Damjan Pecioski, Viktor Gavriloski, Simona Domazetovska, and Anastasija Ignjatovska. An overview of reinforcement learning techniques. In 2023 /2th Mediterranean Conference on Embedded Computing (MECO). pages 1-4, 2023.
[36] Damjan Pecioski, Viktor Gavriloski. Simona Domazetovska, and Anastasija Ignjatovska. An overview of reinforcement learning techniques. In 2023 12th Mediterranean Conference on Embedded Computing (MECO). pages 1-4, 2023.

---

[37] Zhangyang Qi. Ye Fang. Zeyi Sun, Xiaoyang Wu, Tong Wu, Jiaqi Wang. Dahua Lin, and Hengshuang Zhao. Gpt4point: A unified framework for point-language understanding and generation, 2023.
[38] Jie Qin, Shuaihang Yuan, Jiaxin Chen, Boulbaba Ben Amor, Yi Fang, Nhat Hoang-Xura Chi-Bien Chu, Khoi-Nguyen Nguyen-Ngoe, Thien-Tri Cao, Nhat-Khang Ngo, Tuan-Lue Huynh, Hai-Dang Nguyen, Minh-Triet Tran, Haoyang Luo, Jianning Wang, Zheng Zhang, Zihao Xin, Yang Wang, Feng Wang, Ying Tang, Haiqin Chen, Yan Wang, Qunying Zhou, Ji Zhang, and Hongyuan Wang. Shrec'22 track: Sketch-based 3d shape retrieval in the wild, 2022.
[39] Qwen. Qwen2.5 technical report, 2025.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[41] Pradeep Ravikumar, Martin J. Wainwright. and John D. Lafferty. High- High-dimensional ising model selection using I1-regularized logistic regression. The Annals of Statistics, 38(3), 2010.
[42] Damien Robert, Bruno Vallet, and Loic Landrieu. Learning multi-viww aggregation in the wild for large-scale semantic segmentation. In CVPR, pages 5565-5574, 2022.
[43] Arup Kumar Sadhu and Amit Konar. Improve Convergence Speed of Multi-Agent Q- Q-Learning for Coopera- live Task Planning. pages 111-166. 2021.
[44] Arup Kumar Sadhu and Amit Konar. Consensus Q-Learning for Multi-agent Cooperative Plamning, pages 167-182. 2021.
[45] Sayan Deb Sarkar, Ondrej Miksik, Mare Pollefeys, Daniel Barath, and Iro Armeni. Crossover: 3d scene cross-modal alignment, 2025.
[46] Lennart Schneider, Bernd Bischl, and Janek Thomas. Multi-objective optimization of performance and interpretability of tabular supervised machine learning models, 2023.
[47] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Determin- istic policy gradient algorithms. In JCML, pages 387-395, 2014.
[48] Aleksandrs Slivkins. 2019.
[49] S. Talukdar. Multi-agent systems. IEEE Power Engineering Society General Meeting, 2004., pages 59-60 1, 2004.
[50] Jin Tanda, Ahmed Moustafa, and Takayuki Ito. Cooperative behavior by multi agent reinforcement learning with abstractive communication. In 2019 IEEE International Conference Agents (ICA), pages 8-13, 2019.
[51] Guoqin Tang. Qingxuan Jia, Zeyuan Huang. Gang Chen, Ning and Zhipeng Yao. 3d-grounded vision- language framework for robotic task planning: Automated prompt synthesis and supervised reasoning, 2025.
[52] Jinzhou Tang, Jusheng Zhang, Qinhan Lv, Sidi Liu, Jing Yang, Chengpei Tang, and Keze Wang. Hiva: Self Self-organized hierarchical variable agent via goal- goal-driven cemantic-topologeee evolution, 2025.
[53] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2017.
[54] Wenwen Wei, Ping Wei, Jialu Qin, Zhimin Liao, Shuaijie Wang, Xiang Cheng, Meiqin Liu, and Nanning Zheng. 3d scene graph generation from point clouds. IEEE TMM, 26:5358 5368, 2024.
[55] Michael Wooldridge and Nicholas R. Jennings. Intelligent agents: theory and practice. The Knowledge Engineering Review, 10(2):115-15) 1995.
[56] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes, 2015.
[57] Chi Xu, Hui Zhang, and Ya Zhang. Multi-agent reinforcement learning with distributed targeted multi communication. In 2023 35th Chinese Control and Decision Conference (CCDC), pages 2915-2920, 2023.
12

---

[58] Le Xue, Mingfei Gao, Chen Xing, Roberto Martín- Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning a unified representation of language, images, and point clouds for understanding. In CVPR, pages 1179-1189, 2023.
[59] Le Xue, Ning Yu, Shu Zhang, Artemis Panagopoulou, Junnan Li, Roberto Martin-Martin, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Nichles, and Silvio Savarese. Ulip-2: Towards scalable multimodal pre-training for 3d understanding. 2024.
[60] Jiawei Yao, Jusheng Zhang. Xiaochao Pan, Tong Wu, and Canran Xiao. Depthssc: Monocular 3d semantic scene completion via depth- alignment and voxel adaptation, 2024.
[61] Jiawei Yao, Jusheng Zhang, Xiaochao Pan, Tong Wu, and Canran Xiao. Depthssc: Monocular 3d semantic scene completion via depth-spatial alignment and voxel adaptation, 2024.
[62] Mao Ye. Gregory P. Meyer, Yuning Chai, and Qiang Liu. Efficient transformer- based 3d object detection with dynamic token halting. 2023.
[63] Li Yi, Vladimir G. Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang. Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3D shape collections. ACM TOG, 35(6):210, 2016.
[64] Da Zhang, Junwei Han, Long Zhao, Deyu Meng, and Yi Yang. Deep reinforcement learning for visual object tracking. IEEE TNNLS, 29 29(11):5119-5111, 2018.
[65] Jingwei Zhang, Lei Tai, Ming Liu, Joschka Boedecker, and Wolfram Burgard. Neural slam: Learning to explore with external memory, 2020.
[66] Jusheng Zhang, Kaitong Cai, Yijia Fan, Ningyuan Liu, and Keze Wang. Mat-agent: Adaptive multi-agent training optimization, 2025.
[67] Jusheng Zhang, Kaitong Cai, Yijia Fan, Jian Wang, and Keze Wang. CE-VIm:counterfactuall vision-language fine finentuning 2025.
[68] Jusheng Zhang, Kaitong Cai, Jing Yang, and Keze Wang. Learning dynamics of vim finetuming. 2025.
[69] Jusheng Zhang, Kaitong Cai, Qinglin Zeng, Ningyuan Liu, Stephen Fan, Ziliang Chen, and Keze Wang. Faivure-driven workflow refinement, 2025.
[70] Jusheng Zhang, Yijia Fan, Kaitong Cai, Zimeng Huang, Xiaofci Sun, Jian Wang, Chengpei Tang. and Keze Wang. Drdiff: Dynamic routing diffusion with hierarchical attention for breaking the efficiency-qualiee trade-off, 2025.
[71] Jusheng Zhang, Yijia Fan, Kaitong Cai, Xiaofei Sun, and Keze Wang. Ose: Cognitive orchestration through dynamic knowledge alignment multi-agent IIm collaboration, 2025.
[72] Jusheng Zhang, Yijia Fan, Kaitong Cai, and Keze Wang. Kolmogorov- arnold fourier networks, 2025.
[73] Jusheng Zhang, Yijia Fan, Wenjun Lin, Ruiqi Chen, Haoyi Jiang, Wenhao Chai, Jian Wang, and Keze Wang. Gam-agent: Game--heoretic and uncertainty- aware collaboration for complex visual reasoning. 2025.
[74] Jusheng Zhang, Zimeng Huang, Yijia Fan, Ningyuan Liu, Mingyan Li, Zhuojie Yang, Jiawei Yao, Jian Wang, and Keze Wang. Kabb: Knowledge-aware bayesian bandits for dynamic expert coordination in multi-agent systems, 2025.
[75] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointelip: Point cloud understanding by clip, 2021.
[76] Changgang Zheng, Shufan Yang, Juan Marcelo Parra-Ullauri, Antonio Garcia-Dominguez, and Nelly Bencomo.
generative adversarial networks for multi-agent systems. IEEE Transactions on Emerging Topics in Computational Intelligence, 6(3):479-45) 0187794488222 2022.
[77] Huasong Zhong, Jingyuan Chen, Chen Shen, Hanwang Zhang. Jiangiang Huang, and Sian-Sheng Hua. Self adaptiee neural module transformer for visual question answering. IEEE TMM, 23:1264 1277,, 2021.
[78] Junsheng Zhou, Jinsheng Wang, Baorui Ma. Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3d: Exploring unified 3d representation at scale, 2023.
[79] Luowei Zhou, Chenliang Xu, and Jason J. Corso. Towards automatic learning of procedures from web instructional videos, 2017.
13

---

6 Analysis of GPU Memory Usage and Computing Efficiency
Our Tri-MARF integrates multiple compute-intenstee components from visual language model (VLM) inference to BERT/CLIP embedding to multi-armed bandit (MAB) optimization, thus requiring detailed analysis of GPU memory usage and processing speed. This section quantifies the resource requirements and runtime performance of each module, tested on a single NVIDIA A100 GPU, common hardware choice for large-scale AI tasks. All measurements assume batch size of (single object annotation), reflecting typical real-time annotation scenario. Table summarizes the GPU memory usage and processing time of each module, with detailed breakdown provided below.
Data Preparation. The data preparation module renders six multi-view 2D images from the 31) mesh (1hx VJ. where V {front, back, left, right, top, bottom)). The input point cloud is downsampled to 10,000 points using Poisson sampling (for the point cloud encoder below), and the output image resolution is 512 512 (RGB). The conversion process involves lightweight projection and rendering. This step uses Open3D's rendering tool and takes up about 500 MB of GPU video memory for temporary buffers and intermediate representations. The average processing time per object is 0.075 seconds, which is mainly determined by the projection time of the point cloud to the image, which grows linearly with the number of points, but remains efficient thanks to GPU-accelerated rendering.
6.1 VLM Annotation Agent
The VLM annotation agent uses an API call to generate M=5 candidate descriptions for each view using Qween.5-VVL-2BB-
Unlike traditional deployment methods, this system is implemented with video memory usage of GB and no consumption of local GPU resources. In terms of time overhead, the API call for each view takes about 1-3 seconds (including network latency), and the total processing time for six views is about 6- 88 seconds. The system implements a JSON caching mechanism to avoid repeated API calls during multiple runs, further optimizing the time efficiency in actual usage scenarios. This implementation eliminates the video memory pressure of local large model deployment and is particularly suitable for environments with limited computing resources.
through a remote API call chatQ Veven2.55VL-222 W---2---nstruuttt---------------
6.2 Information Aggregation Agent
The module uses RoBERTa-large 355M parameters) for semantic clustering and CLIP (ViT- Large-patch14, 300M parameters) for visual-text alignment. RoBERTa generates embeddings for M x 6 = 30 descriptions (50 words on average), requiring 1.4 GB for model weights and 500 MB to I GB for embeddings (depending on batch size), computed
single forward pass. CLIP processes six images and 30 text candidates, adding 1.2 GB for weights and 500 MB to GB for embeddings. Peak GPU memory usage 3.1 GB to 4.6 GB (depending on specific use of temporary GPU memory), with negligible overhead for clustering (cosine similarity of 30 768 vectors) and softmax weighting. The total runtime is about 0.8 seconds, with RoBERTa accounting for 0.3 seconds and CLIP for 0.5 seconds, thanks to batch inference.
The response aggregation module implements a UCBI multi-armed bandit (MAB) for < 30 normalized descriptions (post-clustering). GPU memory usage is minimal 100 MB), involving
only scalar reward tracking and lightweight per-arm computations (fa + ev 21n ). Runtime depends on the number of trials, but single pass (I 1) takes approximately 0.01 seconds, making it nearly instantaneous compared to other stages.
Merging view-specific descriptions into global annotation involves text processing and scoring (Scoregloba). Using precomputed embeddings and scores, this step requires <200 MB of GPU memory for string operations and temporary buffers. Processing time is approximately 0.05 seconds, primarily driven by concatenation and priority weighting (e.g., wrB), achieving high efficiency.
6.3
Gating Agent
The gating agent mitigates the limitations of traditional 2D image annotation through point cloud-text alignment (a 3D point cloud of 10,000 points with descriptions). Using Uni3D-L. (306.7M parameters) to encode the point cloud and Uni3d-OpenAld processing (a 0.577) ineur minimal overhead. The

---

GPU Memory (GB)
Time
Stage
(s)
Data Preparation
0.5
0.075
Initial Annotation
0.0
6-18
VLM Agent
3. 3.1-4.6
0.8
Information Aggregation
<0.3
0.06
Gating Agent
2.8
0.15
Total (Single Pass)
R: 7.0
6.935-18.935
Table 3: The GPU usage and efficiency of different stages. Note that, denotes the Qwen2.5-V4 VL- 72B-Instruct API.
total peak GPU memory usage is around 2.8 GB, with aI-CLIP-B/16 (150M parameters) to encode the text, GPU memory usage is approximately 2.2 GB for weights (about 1.2 GB for the point cloud encoder and 0.6 GB for the text encoder). Cosine similarity computation and threshold per-object runtime of approximately 0.15 seconds, primarily driven by point cloud encoding.
6.4 Overall Analysis
Combining all modules, the peak memory usage of our Tri-MARF occurs in the information aggrega- tion stage (about 3. 3.1.6.6 GB when using RoBERTa-large and CLIP ViT-Large- 1 4), while the memory usage is about 2.8 GB when using full Uni3D-L (306.7M 7M parameters) and CLIP-B/16 (150M parameters) for point cloud gating. The initial annotation stage calls Qwen2.55VLL-72B-Inss through a remote API and does not occupy the local GPU memory, but the total processing time is 6-18 seconds due to network latency. The memory usage of other stages is kept low, such as about 0.5 GB for data preparation, less than 0.1 GB for response aggregation, and less than 0.2 GB for cress-view processing. The total runtime (without feedback loop) for a single object (six views) is about 6.935-18. 18.955 seconds, depending reduced using a multi-GGPL setup or model optimizations such as quantization. Memory and speed analysis show that Tri-MARE supports hear-real-time annotation of small batches, but multi-GPU expansion may be required in high-load scenarios. In actual large-scale annotation, we choose to use multiple GPUS and multiple machines parallelize annotation.
the latency of the VLM API call. Latency can be further
Summary:
The data based on a single NVIDIA A100 GPU. Initial annotation uses remote 50ee2..5VL772-B1BB
API, consuming no local GPU memory, with total time varying due to network latency. The peak GPU memory usage is determined by the maximum value 4.6 GB in the response clustering and weighting stage.
7 Isolating the Benefits of Multi-Agent Collaboration
7.1 Experimental Setup
further isolate and explicitly quantify the advantages derived from the multi-agent design, conducted additional ablation study. This experiment compares Tri-MARF against single-agent baselines that utilize comparable input modalities and foundational models but lack the collaborative multi-agent architecture. Specifically, we aim to demonstrate the performance gains achieved by Tri-MARF's specialized agents for VLM annotation, information aggregation, and gating, opposed simpler, non-collaborative approaches. We conducted this experiment on the Objaverse-LVIS dataset (Ik sampled objects), consistent with one of the primary benchmarks used in our main paper. Performance was evaluated using standard 3D captioning metrics: CLIPScore and ViLT Retrieval R @5 (Image-to-Tedl Text and Text-to- Text-to-lmage). Higher scores indicate better performance for all metries.
Dataset and Metrics:To
Baselines:
Q.en-2.5 VL (2D Single-Viens This baseline utilizes the Qwen-2.5 VL model, which is also component of Tri-MARF's Initial VL.M Annotation Agent. However, in this single- agent setup, Qwen-2.5 VI. generates descriptions based on only a single 2D view of the object (the front view was used for consistency). It does not benefit from the multi-view

---

information fusion or the multi-agent reinforcement learning based aggregation present in Tri-MARF.
UNi3D (Single Point Cloud): This baseline leverages a UNi3D-based architecture, inspired by its use point cloud encoder in Tri-MARF's Gating Agent, to generate descriptions solely from the 3D point cloud input. This represents a single-modality. single-agent approach, omitting the integration of 2D visual information and textual descriptions from multiple views and the collaborative refinement process of Tri-MARF.
Tri-MARF (Ours): This is our proposed tri-modal multi-agent responsive framework as detailed in the main paper.
All methods were evaluated under the same conditions for fair comparison.
7.2 Results
The results of this comparative analysis are presented in Table 4
Table 4: Comparison against Single-Agent Baselines on Objaverse-LVis. Performance is measured
by CLIPScore (%) and ViLT R@5 (%) for Image-to-Text (12T) and Text-to-Image (T2I) retrieval. Higher is better for all metrics. Our method, Tri-MARE, demonstrates superior performance.
Method
CLIPScore t
ViLT R@S (12T)
ViLT R@5 (T21) f
Qwen-2.5 VL (2D Single-View) UNi3D (Single Point Cloud) Tri-MARF (Ours)
81.4
38.5
36.7
58.3
20.9
19.1
88.7
45.2
43.8
The results presented in Table 4 clearly demonstrate the significant benefits of the multi-agent collaborative framework in Tri-MARF. Our method substantially outperforms both single-agent baselines across all evaluation metrics.
Compared to the Qwen-2.5 VL (2D Single-View View) baseline, Tri-MARF achieves a +7.3 point increase in CLIPScore and an improvement +7.3 in ViLT R@5 scores. While Qwen-2.5 VL is a powerful vision- lssoonlanguage model, its performance when restricted to a single view is inherently limited in capturing the comprehensive details of a 3D object. Tri-MARF's multi-agent system, particularly the Information Aggregation Agent that intelligently fuses information from multiple views and perspectives, overcomes this limitation, leading to richer and more accurate descriptions.
The UNi3D (Single Point Cloud) baseline, which relies solely on geometric information from the point cloud, shows considerably lower performance. Tri-MARE surpasses this baseline by substantial margin: +30.2 in CLIPScore and +24.7/+24.8 in ViLT R @55. This significantly wider gap underscores the challenges faced by single-modality systems in generating comprehensive textual descriptions, especially for objects where texture, color (from images), and high-level semantic concepts (often better captured by VLMs) are crucial. Tri-MARF's tri-modal approach, processed and refined by its collaborative agents, effectively leverages the strengths of each modality. The Gating Agent further ensures alignment between textual descriptions and 3D geometry, mitigating hallucinations that might arise from relying
single information source.
This experiment underscores the efficacy of our multi-agent design. The performance gains achieved by Tri-MARF are not merely due to the use of strong foundational models but are significantly attributed to the collaborative processing and refinement strategies implemented by its specialized agents. This clearly isolates the benefit of the multi-agent architecture in achieving more holistic and accurate understanding and annotation of 3D objects.
8 Justification for MAB-based Aggregation in Tri-MARF
8.1 Experimental Setup
Dataset and Metrics: This experiment aims to address this by directly comparing the MAB (UCB) strategy against several deterministic and heuristic-based aggregation methods. The goal is to evaluate
3

---

the marginal benefits of using MAB and provide a clearer justification for its inclusion in the Tri- MARF framework. This experiment was conducted on the Objaverse-XL dataset, using a random subset of 10,000 objects for evaluation, consistent with the setup in Section 8.2 of our main paper. Performance was assessed using a comprehensive set of metrics:
Likert Score (1-10): Human evaluation assessing accuracy, completeness, and fluency of the generated annotations.
CLIPScore (%) 1: Semantic alignment between generated captions and 3D objects.
ViLT R@5 (Image-to-Text, 12T) (%) t: Retrieval accuracy.
ViLT R @55 (Text-to-lmage, T2T) (%) f: Retrieval accuracy.
Inference Time (ms) L: The average time taken by the aggregation module to process single object on an NVIDIA A100 GPU.
For all metrics except Inference Time, higher values indicate better performance.
Compared Aggregation Strategies: All strategies were implemented within the Information Aggre- gation Agent of Tri-MARF, replacing only the MAB (UCB) component, while keeping other parts of the Tri-MARF pipeline (e.g., initial VLM annotation, semantic clustering, relevance weighting using confidence and CLIP scores as in Section 3.2.1, and final global description synthesis logic) consistent. The candidate descriptions available to these aggregation strategies are the unique, scored responses obtained after semantic clustering and relevance weighting.
Max VLM Confidence:
This heuristic selects the description candidate for each view that has the highest raw confidence score (Con (C') from Section 3.1) as produced by the VLM agent. The global description is then synthesized based these view-specific selections.
Max Combined Score (Heuristic): This strategy selects the description candidate for each view based the highest composite score s; (1 a)- Seon + w; (detailed in Section 3.2.1), which combines VLM confidence and CLIP- image-text alignment. This represents a strong, informed heuristic.
Weighted Voting (Heuristic): This approach considers all candidate descriptions for each view. The final description for view chosen based on hypothetical voting scheme where votes are weighted by the combined scores (s;). The global description is then assembled. For this experiment, we simulate this by selecting the description with the maximum combined score, which is similar to "Max Combined Score (Heuristic)" but framed as a proxy for more complex voting outcome.
Simple Concatenation (Prioritized): This method uses fixed rule for selecting deserip- tions from each view highest VLM confidence per view) and then applics the prioritized concatenation logic described in Section 3.2.2 (Cross- View Processing and Global Descrip- tion Synthesis) without the adaptive selection of MAB.
MAB (UCB) (Ours): This is the standard Tri-MARF approach using the Aamti-Armed Bandit (UCB1 algorithm) for adaptive selection of descriptions from each view, as detailed in Section 3.2.2 and validated in Section 8.2.
Results
The performance of these different aggregation strategies is summarized Table 5
Table 5: Performance Comparison of Different Aggregation Strategies within Tri-MARF on XL. Best results are in bold.
Aggregation Strategy
Likert (1-10)
CLIPPSoore (*))
VIET R@5 ($))
VILT R (T21) (T2) (%)
Inference Time (ms)
Mox VLM Confidence
8.6
81.37
37.51
35.28
6.3
Max Combined Score (Heuristic)
8.9
82.03
38.15
35.92
8.1
Weightd Nocing (Heuristic)
37.93
35.76
Simple Concatenation (Prioritized)
34,81
MAB (UCB) (Ours)
9.3
82.72
38.82
36.72
9.8
The results in Table 5 indicate that while simpler aggregation heuristics achieve commendable performance, the MAB (UCB) strategy employed in Tri-MARF provides a distinct advantage across the primary quality metrics.

---

The Max VLM Confidence and Simple Concatenation (Prioritized) strategies, being the simplest, yield the lowest scores terms of Likert, CLIPScore, and ViLT retrieval, although they offer the fastest inference times (6.3ms and 6.8ms, respectively). This suggests that relying solely on initial VLM confidence or fixed concatenation rules suboptimal for capturing the nuances required for high-quality 3D annotations.
The Max Combined Score (Heuristic) and Weighted Voting (Heuristic) strategies, which leverage both VLM confidence and CLIP-based image-fext alignment scores (as computed in Section 3.2. 1), perform significantly better. The "Max Combined Score (Heuristic)" achieves CLIPScore of 82.03% and a Likert score of 8.9. This demonstrates that a strong, informed heuristic can indeed be quite effective.
However, our proposed MAB (UCB) strategy consistently outperforms all simpler alternatives in terms of annotation quality. It achieves the highest Likert score (9.3), CLIPScore (82.72%), and ViLT R@5 scores (38.82% 12T, 36.72% T2I). The improvement in CLIPScore is approximately +0.7 points over the best heuristic ("Max Combined Score"), and the Likert score also shows a notable improvement, suggesting that the MAB'S s adaptive selection process leads to descriptions that are perceived as more accurate, complete, and fluent by human evaluators.
While the MAB (UCB) approach has a slightly higher inference time (9.8ms) compared to the simplest heuristics, this is marginal increase (e.g., +1.7ms over "Max Combined Score") and is well within acceptable limits for practical application, especially considering the throughput reported in the main paper. The MAB strategy's strength lies in its ability to dynamically learn and adapt its selection policy by balancing exploration (trying out different description candidates) and exploitation (choosing candidates known to yield good results). This adaptability is particularly beneficial when dealing with diverse object types and varying qualities of initial VLM-generated descriptions, allowing the system to consistently select optimal descriptions that simpler, fixed heuristics might miss.
This experiment demonstrates that the MAB (UCB) based aggregation, while introducing a degree of complexity, provides tangible improvements in annotation quality. The observed marginal benefits in key metrics are crucial for achieving state-of- ofte-oof-thh-at the-art performance. The MAB's ' adaptive nature justifies its use over static heuristics, particularly in framework designed for robust and high-quality annotation across large-scale and diverse 3D datasets.
Exploring Tri-MARE for 3D Scene Annotation
9.1 Motivation
The Tri-MARF framework, as presented in the main paper, is specifically designed for comprehensive 3D object annotation. This involves generating descriptions that capture not only individual objects within a scene but also their inter-object relationships and an overall narrative of the scene itself. Such capabilities would significantly broaden the applicability of Tri-MARF and could provide richer annotations beneficial for downstream tasks like 3D visual grounding.
Due to the current architecture of the Gating Agent (Agent 3), which leverages Uni3D-baser encoder primarily optimized for object-centric point clouds, its direct application to full scene point clouds presents challenges. Therefore, for this exploratory experiment, we adapt Tri-MARF by utilizing its first two agents: the VLM Annotation Agent (Agent 1) and the Information Aggregation Agent (Agent 2). This allows to assess the core descriptive and aggregative capabilities of Tri-MARF in a scene context, even without the final point cloud- basdd gating.
9.2 Experimental Setup
Dataset: We selected the ScanNet dataset for this experiment. ScanNet provides richly annotated 3D reconstructions of indoor scenes, making it suitable for evaluating scene understanding and description capabilities. A subset of 100 diverse scenes was randomly chosen for evaluation.
Method Adaptation (Tri-MARF for Scenes):
Input: For each scene, multiple 2D views were rendered from different camera poses within the reconstructed 3D scene.
5

---

Agent 1 (VLM Annotation Agent); The Qwen2.5-VV VL model was prompted with scene- level queries (e.g., "Describe this indoor scene. What are the main objects and how are they arranged? What is happening in this scene?"). This generated multiple descriptive candidates for each scene view.
Agent (Information Aggregation Agent): The MAB (UCB) based aggregation strategy was used to fuse the multi-vies view scene descriptions into single, coherent global description for the entire scene.
Agent 3 (Gating Agent); This agent was omitted in this experiment due the aforemen- tioned challenges of applying the object-centric Uni3D encoder to full scene point clouds. Future work will explore coene-compatibl gating mechanisms.
Baselines: Cap3D and ScoreAgg, which are primarily 3D object captioning models, were adapted for scene description as follows:
For each scene, prominent objects were assumed be detected (e.g., using off-the-shelf object detectors ground truth bounding boxes from ScanNet for a best-case scenario for the baselines).
Cap3D and ScoreAgg were then applied generate descriptions for these individual objects.
The resulting object deseriptions concatenated to form pseudo-scene description. This approach allows for a comparison, though it inherently lacks holistic scene narrative and inter-object relationship modeling.
Metrics: To evaluate the quality of scene annotations, we used the following metrics:
CIDEr (Consensus-bawee based Image Description Evaluation) t: A standard metric for captioning quality that measures consensus with reference human descriptions (for this experiment, we assume set of reference scene descriptions or use a reference-free variant if applicable, aiming for higher semantic quality).
Relationship Accuracy (%) t: We manually evaluated subset of generated descriptions for the correct identification simple spatial relationships between key objects (e.g.. "monitor the desk". "chair next to the table"). This scored based predefined list of expected relationships per scene.
Scene Element Coverage (%) f: Assesses the percentage of key objects and distinct scene elements (e.g., furniture types, room features) mentioned in the generated description compared to ground-truth list for each scene.
Higher scores are better for all metries.
9.3 Results
The comparative performance of the adapted Tri-MARF (Agents 1+2) and the baseline methods on the ScanNet scene annotation task is presented Table
Table 6: Performance Comparison for 3D Scene Annotation on ScanNet. Our adapted Tri-MARF (Agents 1+2) demonstrates superior capability describing scenes compared to adapted object- centric baselines.
Method
CIDEr
Relationship Accuracy (%)
Scene Element Coverage (%)
Cap3D (adapted for scenes) ScoreAgg (adapted for scenes) Tri-MARF (Agents 1+2 for Scenes)
0.627
45.3
65.9
0.684
50.
70.5
0.953
75.8
88.2
The results in Table T indicate that the adapted Tri-MARF framework, when utilizing only its first two agents, exhibits strong potential for 3D scene annotation, significantly outperforming the adapted object-centric baselines.
Our Tri-MARF (Agents 1+2 for Scenes) achieved CIDEr score 0.953, substantially higher than Cap3D (0.627) and ScoreAgg (0.684). This suggests that Tri-MARF's approach of generating

---

scene-aware descriptions from multiple views and then intelligently aggregating them leads to more human-like and semantically rich scene narratives. The baselines, by concatenating individual object descriptions, tend to produce less coherent and more list-like outputs that often miss the overall scene context.
In terms of Relationship Accuracy, Tri-MARF (75.8%) again shows a clear advantage over Cap3D (45.3%) and ScoreAgg (50.1%). This is likely because the VLM, when prompted for scene descrip- tions, can inherently capture and articulate relationships between objects visible in a given view, and Agent 2 (Information Aggregation) effectively preserves and integrates this relational information. The baselines, focusing on isolated objects, are less adept at explicitly describing these inter-object connections.
Similarly, for Scene Element Coverage, Tri-MARF (88.2%) surpasses Cap3D (65.9%) and ScoreAgg (70.5%), indicating its ability generate more comprehensive descriptions that cover wider array of objects and notable features within the scene. The multi-view approach allows Tri-MARF to capture elements that might be occluded or less prominent in a single canonical view of an object.
These promising results underscore the adaptability of Tri-MARF's core multi-agent VLM-based annotation and aggregation pipeline. While the omission of the point cloud Gating Agent (Agent 3) is a current limitation for full scene understanding (which ideally would leverage global scene geometry), this experiment demonstrates that the first two agents already provide powerful foundation for scene-level descriptive tasks.
10
Cost Calculation and Analysis
Total Cost Estimation. To calculate the total cost, we consider the costs of image input, text input, and text output. Let the cost per image input be C, the cost per thousand text input tokens be Cijin= and the cost per thousand text output tokens be Cr_out-
The total cost is derived follows:
Total Cost
Total Image Cost + Total Text Input Cost Total Text Output Cost
(4)
where:
The total image cost is calculated based on 30 images (6 views x 5 repetitions):
Total Image Cost 30 C₁
(5)
The total text input cost is calculated based on 4500 tokens (30 calls x 150 tokens per call):
4500
Total Text Input Cost
x Ct_in = 5 x Gr
1000
The total text output cost is calculated based on 21000 tokens (30 calls x 700 tokens per call):
21000
Total Text Output Cost
X Cr_out = 21 x Cr
(7)
1000
Thus, the total cost estimation formula is:
Total Cost = 30 x C, +5 x Cain + 21 X Czout
(8)
11 Detailed Ablation Studies
11.1 Analysis of Different VLMs
In our experiments, we call different visual language models (VLMs) for annotation through the API provided by OpenRouter. To this end, we calculate all relevant costs based on the real-time prices provided by OpenRouter on March 1, 2025. We randomly selected 1,000 samples from the Objaverse-XX XL dataset as a test set to evaluate the performance of generating subtitles after replacing different VLMs in the Tri-MARF framework. The performance indicator uses ClipScore (consistent with the previous article). The comparative experiments involve the following models: GPT-4.5- Preview, OpenAl-O1, Claude-3,7 B.addde-,7-Somme Sonneet, Claude-3.7-Somae (Thinking Mode), Gemini-Flast -lash-2.0,
7

---

Cost vs Performance of VLM Models
swer2 20l.23---22222oo iomuei
48-
Pumman Ammotaton
Opende D1
Caude333 thinking
(per2 pernuct
Scorvego
gver2. (rarartrtti
76
flech-
Storbups
Cap3s
Cost (USD/Run)
Figure 6: Comparison results of annotation using different VLM models.
Table 7: Quantitative Results the Objaverse-XX XL Dataset. The training includes 10,000 objects. and the test set includes 1,000 objects. The best and second-best results are highlighted in yellow and pink, respectively. The highest value is bolded, the second highest is underlined
Strategy
Metrics
Likertt (1-10)
CLIPScoret (%)
I-to-Tf (acc %)
T-to-I* (ace %)
Training time). (h)
Inference time.) (ms)
MAB (UCB)
9.3
82.72
38.82
36.72
2h 36min
9.82
MAB (Thompson Sampling)
9.0
82.51
39.01
36.21
2h 54min
11.21
PPO
8.2
81.02
38.60
35.72
4h 18min
32.12
A3C
8.5
80.51
37.59
35.23
3h 32min
23.24
SAC
8.5
80,91
37.32
34.91
4h 53min
37.87
MAB (Epsilon-Greedy)
8.7
81.84
38.57
36.08
2h 24min
9.91
MCTS
8.5
82.12
37.98
35.37
16h 27min
55.47
Qwen2.5- VLL--VI--II-sss--- Instructt,, GPT-4o, and D0wen2-VL-72BB-Inmatee We also estimate the cost of Cap3d, ScoreAgg, and manual annotation for comparison.
As shown in Figure 6 chose -e--n.---WL-22-Instrrue to achieve better performance (88.6) at lower price (0.0054$7RUN), which is the best choice of cost and performance compared to traditional methods. At the same time, we also noticed that some models based reinforcement learning for reasoning (such as ol, Claude3.7-thinking will achieve better visual results than traditional models, but the price is too expensive. Therefore, we choose Qwen2.5-VL-72B. Instrucct the default VLM agent.
11.2 Reinforcement Learning Strategy Selection
Experimental Setup. This study uses the Objaverse- XI. dataset to evaluate the impact of using various reinforcement learning (RL) strategies for the aggregation agent, including MAB (UCB) as baseline, MAB (Thompson Sampling), PPO, A3C, SAC, MAB (Epsilon-Greedds and MCTS, on the 3D object annotation quality and the overall training time and space consumption.
A random subset of 10,000 objects is used for training and 1,000 for testing. Performance is assessed via Likert scale human evaluation (1-10, across accuracy, completeness, and fluency), automated metrics (CLIPScore, ViLT Image-to- torgee-t-Tees and Text-to-Image Retrieval Recall @5), and efficiency metrics (training time hours, inference time milliseconds), conducted on a standardized NVIDIA A100 GPU environment, All strategies are trained for 100 epochs with tuned hyperparameters and three random seeds, with results averaged to ensure fairness and reproducibility, aiming to identify the best strategy for scalable annotation within Objaverse-XL XL.

---

Experimental Results. Table shows that on the Objaverse-XI dataset, the MAB (UCB) strategy performs best in core indicators, with a Likert score of 9.3 (1-10) and CLIPScore 82.72%. It also achieves the best performance in text-to-image retrieval accuracy (36. (66.72%) and inference efficiency (9. 19.82ms), and its training time (2h36min) is only slightly higher than the fastest MAB (Epsilon-Greedy) (2h24min). MAB (Thompson Sampling) ranks first with an image-to-text retrieval accuracy 39.01% %, but its training time (2h54min) and inference latency (11.21ms) are slightly inferior to the UCB variant. Among deep reinforcement learning methods, PPO, A3C, and SAC are inferior to the MAB series in terms of training efficiency (4h (4hl8min to 4h53min) and annotation quality (Likert 8.2-8.5), 5), and although MCTS performs moderately in text-to-image retrieval (35.37%) and CLIPScore (82. (22), its 16h27min training time and 55.47ms inference latency significantly reduce its practicality. Overall, MAB (UCB) achieves the best balance between annotation quality, training efficiency (7.7% time efficiency improvement over the suboptimal strategy) and inference speed, choose MAB (UCB) the baseline strategy for the aggregation agent.
11.3 Multi-view Comparisons
Table presents the performance comparison of three multi-view 3D object description meth- ods Cap3D, ScoreAgg, and on the Objaverse-L.VSS (1k) dataset, evaluated across varying numbers of views (1, 2, 4, 6, and 8). The metrics include CL.IPScore, ViLT R @ (for both Image-to-Text and Texl-to-Image retrieval), and BLEU-4, all of which are reported with higher values indicating better performance. Tri-MARE consistently outperforms the other methods across all metrics and view configurations, achieving the highest scores with views: CLIPScore of 88.7, 7, ViLT R @ 5 of 46. 2/44,, and BLEU-4 of 26.3. Cap3D and ScoreAgg show moderate improvements as the number views increases, peaking at views with CLIPScores of 78.1 and 79.3, respectively, but their performance declines slightly at 8 views. Notably, Tri-MARF demonstrates significant advantage even with a single view (CLIPScore of 77.2), surpassing the multi-view results of Cap3I) and ScoreAgg in most cases. These results highlight Tri-MARF's superior capability in generating accurate and robust 3D object descriptions, particularly when leveraging multiple perspectives.
Table 8: Performance Comparison of Multi-View 3D Object Description Methods Objaverse-LVIS (Ik)
Method
Number of Views
CLIPScoret
ViLT R@5 (12T/T201
BLEU-41
66.8
25.9/24.5
17.3
70.2
28.4/27.0
19.1
Cap3D
4
74.6
31.5/30.0
21.2
6
78.1
34.2/32.7
22.6
8
75.7
32.7/31.2
21.8
1
68.3
27.2/25.8
18.4
72.0
30.1/28.6
20.3
ScoreAgg
75.8
33.0/31.5
22.0
6
79.3
35.9/34.3
23.5
8
76.9
34.2/32.7
22.7
77.2
38.1/36.4
21.4
2
80.5
40.7/38.9
23.2
THi-MARE
4
84.3
43.5/41.7
25.0
88.7
46,2/44.3
26.3
8
85.8
44.6/42.8
25.4
11.4 Labeling Analysis of Object Categories
Table presents the CLIPScore performance of various methods across five major categories of the ShapeNet-Core dataset: Furniture, Vehicles, Electronic, Daily Necessities, and Animals. The results demonstrate that TriMMARF achieves the highest average CLIPScore, ranging from 81.9 (Daily Necessities) to 85.2 (Vehicles), with an overall peak of 84.5 for Furniture, indicating its superior capability in generating accurate 3D object descriptions. Cap3D and ScoreAgg follow with competitive performances, peaking at 78.5 (Vehicles) and 81.4 (Vehicles), respectively, while 3D-LLM, ULIP-2, PointCLIP, and GPT4Point trail behind, with the lowest scores recorded by

---

Table CLIPScore Performance of Different Methods on Major Categories of ShapeNet-Core
Method
Furniture
Vehicles
Electronic
Daily Necessities
Animals
Tri-MARF
84.5
85.2
82.7
81.9
83.6
Cap3D
77.3
78.5
75.6
74.8
76.9
ScoreAgg
80.2
81.4
78.3
77.5
79.8
3D-LLM
76.5
77.3
74.9
73.6
75.7
PointCLIP
64.2
65.8
62.5
61.7
63.9
ULIP-2
74.3
75.6
72.8
71.5
73.9
GPT4Point
62.3
63.5
60.1
59.4
61.8
GPT4Point (59.4-63.5) and PointCLIP (61.7-65.8). The data suggests that Tri-MARE consistently outperforms other methods across all categories, with a notable advantage in handling diverse object types.
11.5 Hyperparameter Sensitivity
We systematically evaluate the parameter sensitivity of the key modules in Tri-MARF: BERT deduplication, CLIP weighting, MAB response aggregation, and VLM initial annotation. Each module's critical parameters are analyzed over wide ranges to identify optimal configurations, with results visualized through performance metrics such CLIPScore, IZT @@5, and 12T R@5. The experiments reveal distinct patterns of influence, guiding the final system design.
The BERT deduplication module employs semantic clustering via DBSCAN identify and merge similar descriptions. We varied the neighhorhood radius (eps) parameter across a broad range, with results summarized in Figure The performance metries indicate trade-off between clustering granularity and deduplication accuracy, with an intermediate eps value yielding balanced results across all metrics.
Performance Metrics by Epsilon Value
90
87.8
872
8
80
70
Metrics
CLISScore R@s
60
T21 Res
50
412
43.0
165
44.5
46.0
40
0.05
0.15
0.25
0.30
0.40
0.60
0.80
Epsilon Values
Figure 7: Performance Metrics by Epsilon Value. The plot shows the sensitivity of the BERT deduplication module the eps parameter, with a moderate value optimizing CLIPScore and recall metrics.
10

---

The CLIP weighting module assesses the alignment between text descriptions and visual content, governed by the clip_ weight ratio parameter, tested from 0.0 to 1.0. As depicted in Figure X this parameter exhibits nonlinear impact, peaking optimal performance across all three metrics. Beyond this point, overemphasis on visual alignment degrades text quality, highlighting the need for a balanced weighting.
hrsp__eeght__rtiieee ratio=0.2, where the system achieves
Performance Metrics by CLIP Weight Ratio
90
Deaauit R0L
80
70
Metrics
60
50
63.8
40
40
107
51-7
385
0.00
0.05 0.10
0.20
0.30
0.50
0,70
1.00
CLIP Weight Ratio
CLIPScore
12T R@5
T21 R@S
Figure 8: Performance Metrics by CLIP Weight Ratio. The curve peaks at 0.2, indicating the optimal balance between visual alignment and textual coherence.
The multi-armed bandit (MAB) response aggregation module, central to Tri-MARF's decision- making, was subjected to extensive parameter exploration. The exploration weighwteghttttt parameter, controlling the exploration-exploitaiio trade-off, was tested from 0.01 to 5.0. Figure reveals an inverted U-shaped curve, with cecioratton_wiighteete delivering the best performance, balancing novel option discovery with reliance known high-quality responses. Similarly, the alpha parameter, defining the MAB's s prior distribution, was evaluated from 0.01 to 1.0 (Figure [O). The optimal value of 1lpha=0.1 maximizes performance by providing robust initial belief without overfitting early observations. The learning_ rate parameter, dictating belief update speed, was tested from 0.01 to 0.5, with learning 1 emerging the best performer (Figure 11). ensuring adaptive yet stable updates.
For the VLM initial annotation module, we analyzed the temperature parameter's impact on descrip- tion quality, alongside the number of candidate responses caai___adddattssFiiiie [2 illustrates that temperature=0.7 optimizes CLIPScore, around 86-87, reflecting a sweet spot for creative yet coherent outputs. Higher temperatures introduce noise, while lower values overly constrain diversity. The combined analysis of alpha and explo- ration ration_weight (Figure [3) further confirms their optimal pairing at 0. and 0.5, respectively, with CLIPScore stabilizing around 44.5 in the heatmap, underscoring their synergistic effect. Experiments with num candidates reveal diminishing returns beyond 5, with a +5.7 CLIPScore gain from to 5, but only +0.6 from 5 to 20, justifying num cattn_aaddddteSS the cost-effcctive optimum.
seen in the 3D surface and heatmap data peaking
In summary, the ablation study identifies eps (moderate), ration_ w0ight_-eeght0..5, alpha=0.1, learninggraateeeeee rate=0.1, temperature=0.7, and num num_ cum_candidates=5 as the optimal parameter set, maximizing performance across all evaluated metrics while maintaining computational efficiency.
hisp_weight_raiiin
explo-

---

Performance Metrics Exploration Weight
90
80
70
Metrics
1M
0.01
6.05
6.5
1.0
2.0
3.0
Exploration Weight
Figure 9: Performance Metries by Exploration Weight. An inverted U-shape peaks at 0.5, optimizing the exploration-exploiatiio trade-off.
Alpha Sensitivity Analysis
90
665
16.1
Performance Metrics
60
50
565
180
380
Alpha Value
Riss
T21R
Figure 10: Alpha Value Sensitivity. Alpha-0.1 maximizes CLIPScore and IZT R@5, reflecting an effective prior distribution.
11.6 Gating Threshold Derivation and Validation in Tri-MARF (a = 0.557)
In our Tri-MARF, we propose gating mechanism using cosine similarity between 3D point clouds and text embeddings filter annotations effectively. This section derives optimal threshold 0.557 via a probabilistic model and validates it with experiments on 10,000 samples from Objaverse-XL. Our Tri-MARF minimizes misclassification errors, achieving CLIPScore of 88.7 and ViLT R@5 of 45.2/43.8, demonstrating both theoretical rigor and practical utility.
Problem Formulation. We aim to minimize the misclassification error:
P(Spoe < a) + P(Sneg 2 a) min,
(9)
12

---

Learning Rate Sensitivity Analysis
90
Performance Metrics
60
50
442
685
Learning Rate
Figure 11: Learning Rate Sensitivity Analysis. Searningrrate-ll provides the best performance
across metrics, balancing adaptation and stability.
where Spos and Speg similarity scores for positive (correct) and negative (incorrect) point cloud- text pairs, respectively.
Probabilistic Modeling. Using pretrained encoders Ep (point cloud) and E, (text), we assume:
Positive pairs: Spos ~ Nerune (11, of < s 1),
Negative pairs: Sneg ~ Nerune o²: < 1).
Validation data yields the following parameters: μ4 0.65, μ1 0.35, 01 0.1, 02 0.15.
Optimal Threshold. The optimal a satisfies:
(a) [neg(a).
(10)
Substituting Gaussian PDFs:
(o-K)] )2
(m-
20
(11)
T 2T 2T
T2 V 2m
Taking the natural logarithm:
(a 221
(a [12
In
In
(12)
σ1
20
02
20
Rearranging:
(a - 22)
(a - 41) )2
2
(13)
02
of
Expanding into a quadratic form:
2
o
+
of
oz
+
2
=
(14)
Define 2 Ba C
= 0, where:
1
1
= 44.44
100 = 55.56.
0.132
0.12
B = 2
0.65
0.35
2(65
15.56)
98.89,
0.17
0.15"
C'
0.35
0.15%
0.05
0.14
2l
0.15
0.1
5.4444
42.25 2(0.4055) 33.66166.
13

---

Impact Temperature on VLM Performance
CLIPScore Surface
CLiPScore
57
80
16
$1.
$3.5
06.7
BC.32
07.2
64.3
02.1
02
S16
-84
53
1.0
Binpwatbry
15
2.0
6.1
0.3
0.5
0.7
1.0
1.5
2.0
Tempeeaturee
12T R@S Heatmap
T21 R@5 Heatmap
-44
-12
41
39.7
41.6
43.7
63.9
41.7
39.2
$7.1
40.0
42.1
42.5
40.1
37,6
-40
-18
63
6.3
0.5
0.7
1.0
1.5
0.3
0.5
0.7
1.0
1.5
rabure
Temppratuu
tomperature of 0.7 acheves optinal performance across all airee metric, showinng liverted U-shape reladorshiop
Figure 12: Impact of Temperature on VL.M Performance. The 3D surface and heatmap peak at temperature=0.7, with CLIPScore reaching 86-87.
Solving:
B + V B - 4AC
(15)
2A
98.892
4(-55.50) (04556)((33361)) -37.6166) = 1425.4625,
(16)
VĀ 37.75,
01 0.557,
0g 1.224.
(17)
Since 02 > 1 is invalid, select 0.557.
Experimental Validation. We use 10,000 point cloud-text pairs from Objaverse-X0 with 5,000 positive and 5,000 negative pairs (randomly mismatched). Cosine similarities are computed via E, (PointNet++-based and F,, (BERT-hased) on an NVIDIA RTX 3090 using PyTorch.
Distribution Verification We fit truncated Gaussians to Spos and Szeg. estimating parameters and performing KS tests. The results are shown Figure 14 indicating that the estimated parameters closely match our theoretical assumptions, with high KS p-values confirming consistency.
Threshold Optimization. We compute FNR, FPR, and total error for [0.4, 0.7], with results detailed in Figure [5] The AUC from ROC analysis is 0.91, and a = 0.557 achieves the lowest total of 0.25, outperforming other thresholds.
System Performance. evaluate Tri-MARF performance across values, focusing on CLIPScore and ViLT R @55, shown in Figure 16
At a 0.557, the system achieves CLIPScore of 88.7
14

---

Com bined Perameter Analysis: Alpha Exploration Weight
CLiPScore Surface Rot
CLIPsooee Histmap
003
165
67.6
82.0
97.
865
874
96.5
573
670 63
0.107
06.5
C.00
Dyperativn Hiig
2,075
00.5
Explcaanin Weett
High thealmep
121 W Healmmp
0.00
000
43.5
41.
42.
Difkal
000
431
Exp oopertiion Eaperstionnedg)) Welght
Oploratiin Waight
Figure 13: Combined Parameter Analysis: Alpha and Exploration Weight, The surface plot and heatmap confirm alpha=0.1 and around 44.5.
as the optimal configuration, with CLIPScore
Theoretical
Estimated
KS p-value
a 0.05
Distribution Parameters: Theoretical vs Estimated
0.647
0.650
0.6
p-0.5
0-7.99
p=0.81
0.8
0.5
p-value
Value
0.4
0.6
0.350
0.3
0.4
0.2
0.148
0.105
0.1
0.0
Figure 14: Distribution Parameters: Theoretical vs. Estimated with KS Test valuue
and ViLT R@5 of 45.2/43.8, Sensitivity analysis around a 0.557 + 0.02, presented in Figure shows fluctuations below 2%, confirming robustness.
KL Divergence. We calculate DKL(Ppos pos Pneg) assess discriminative power, with results in Figure 18 The peak value 2.30 at a 0.557 supports its optimality.
Robustness. We test 0.557 under varied distributions, as shown in Figure 19 Performance remains strong, with CL.IPSooee dropping only slightly to 87.5 under a more overlapping distribution, aided by Tri-MARF's multi-agent design.
15

---

2.6
vir comeans with
matemurr 13 2hyie giif
02
Optöial
Figure 15: Error Rates Across a
Performance Metrics Across Differe nt Values
60.
Beal CLIPSore (ox1.557)
CLiP
803
67.
873
Part2
42-
R25 (wil)
VILT Score
46.5
465
RgB
412
Peefomance analyde scrzee thme o (hrmahadte:
purfemence highest -Buh 551 Dis Shon computll tioouiiio aigiryy beeot po formenco in mobil ceciell mode ellccienusss
mot
astimu miseaross d manoo rekatuus
Figure 16: System Performance Across
Baseline Comparison. We compare a = 0.557 against baselines in Figure 20 consistently outperforms alternatives, achieving CLIPScore of 88.7 versus 86.3 for 0.5 and 85.7 for no gating.
Theoretically, a = 0.557 balances partially overlapping distributions (μ μ4 = 0.3 <
2/02 + of ~ 0.36). Experiments, as detailed in Figures 14 to 20 confirm its efficacy, with minor deviations (e.g., a = 0.551 in exact computation) resolved through practical tuning. The archi- tecture of Tri-MARF enhances the robustness of annotation. We derive and validate optimal gating threshold Tri-MARF, supported by rigorous theory and comprehensive experiments across Figures
0.557 as an
4 to 20 Future work may explore adaptive thresholds for varying distributions.
16

---

Sensitiity Analysis = 0.557 & 0.02
Senslivity
Netrics Senstivity
00.4
Vie Part
454
887
64
44.0
456
425
0.00
-52
155
0-7777
0- 5337
C5ST
0577
(ond
CJrbies
WCT Parl"
130
1,000
0.900
on 2557
C55T
2- 0577
The aptipii Binoholdd and 567 conniceerity delivars the resslte across cunkuatid martic.
Figure 17: Sensitivity Analysis at 0.557 0.02
KL Divergence (DKL) Across Different a Values
2.30
1:29
23
Divergence (DKL) (DKL)
2.
Optimal a= 0.557
811
28
KL Devoggnnc measures the informaioon gain when using the model.
1.0
Maaimumn 0.587 (DK). 2.30)
DXL formot comizee curve with opiimal sunge belivee 0.55-0.5
0.40
0.45
0,50
0.55
0:00
065
070
Value
1L. Riwsgsso Anipol 2000
Figure 18: KL Divergence Across
12 Details of Human Evaluation
Human evaluations are conducted to validate Tri-MARF's performance in caption quality assessment, type annotation validation, and reinforcement learning strategy selection. All annotators were hired through a crowdsourcing platform and required basic English proficiency and least one year of experience in image text annotation. Below are the details of each experiment's methodology, participant recruitment, and evaluation protocols. We obtained local Institutional review board (IRB) approvals before conducting the experiment.
17

---

Robustness Comparison Distributions
0.200
0.275
3.15
-9
5.24
Error Rate (FNA. FPR)
0.125
9.22
0.300
0.075
886
0.000
Naranters e2 42 opreee. mHers
Figure 19: Robustness Across Distributions
Baseline Comparison of Different Methods
46
01.7
45
CLiPSrore
-48
-43
-42
411
0.5
Gating
- 0.557
Rg3
Figure 20: Baseline Comparison
12.1 Human Evaluation 3D Captioning Test
Compare Tri-MARF generated captions against baselines (e.g., Cap3D, ScoreAgg, Human Annota- tion) via A/B testing. Five annotators were recruited from the crowdsourcing platform.
Two hundred objects were randomly sampled from each dataset- Obbsset---bavvv---LVV. (Ik), Objaverse- XL. (5k), and ABO totalingg 600 objects. Annotators evaluated pairs of captions (Tri-MARF vs. baseline, randomly ordered) on 1-5 Likert scale for accuracy (object description match), completeness (key feature coverage), and linguistic quality (clarity and grammar). Each annotator assessed 40 pairs per dataset (120 pairs total), with tasks evenly distributed. Scores were averaged across annotators and objects, with Tri-MARR as the reference baseline. The task was completed in five days, with each annotator working 5 hours per day.
12.2 Human Verification in Type Annotation Experiments
To verify the semantic accuracy object type classification by Tri-MARF and baselines, as well as automated metrics. Three annotators were hired from crowdsoureing platform.
300 objects were randomly selected from Objaverse-LVIS. Annotators received 3D models and renderings of their 6 viewpoints. They observed the 3D objects and selected the category (e.g., "mug" vs. "'cup") in six-choice question, simply selecting the most appropriate option. Each object was reviewed by two annotators, and the third annotator resolved disagreements by majority voting. The
18

---

results established a human annotation baseline. The task was completed in three days, with each annotator working for 2 hours.
12.3 Human Evaluation of Reinforcement Learning Strategies
Assess annotation quality of RL strategies (c.g.. MAB UCB, PPO, MCTS) using Likert Four annotators were recruited from the crowdsoureing platform.
Twenty-five annotations RL strategy (7 strategies, 175 total) were sampled from the Objaverse- XL test set (1,000 objects). Annotators rated each annotation on a 1-10 Likert scale for accuracy (description correctness), completeness (detail inclusion), and fluency (readability). Each annotator evaluated 43-44 annotations, with strategy origins blinded. Scores were averaged to yield the final Likert score. The task was completed in four days, with each annotator working 2.5 hours.
12.4 General Protocol and Quality Control
Training: Annotators completed 15-minute online training module via the platform, using sample objects and annotations to understand criteria.
Inter-rater reliability was tracked with Cohen's Kappa, achieving 0.76 (substantial agreement). Ratings differing by more than 2 points were reviewed by a platform supervisor, with 5% of responses rechecked for consistency.
Quality Control:
Compensation: Annotators were paid $ $15/hour.At the same time, ensure that no personnel replaced during the marking period
13 Robustness Evaluation Under Occlusion
In this section, we investigate the robustness of the Tri-MARF framework when 3D objects are partially occluded, common challenge in real-world scenarios such as autonomous driving and robotics. To evaluate this, we randomly selected 500 objects from the Objaverse-XL dataset and introduced artificial occlusion by overlaying random black planes their 3D assets, simulating varying degrees of obstruction. Both the unoccluded and occluded 3D models were processed using our Tri-MARF, with experimental parameters consistent with the main experiments, including the for initial annotation, ROBERTa-DBSCAN for clustering, MAB
use of Qwen2.5-VL-722--nssee (UCB) with exploration _erigghhhwwwwwhhhh...... 0.5, alpha 0.1, and learning_rate and a point cloud gating threshold alpha = 0.557. The CLIPScore was recorded to compare the quality of generated captions under occluded versus unoccluded conditions.
0.1 for aggregation,
The results indicate that Tri-MARF maintains robust performance under occlusion. For unoccluded objects, the average CLIPScore was 1. aligning with the main experiment (Table 1). For occluded objects, the CLIPScore dropped to an average of 82.3 (a 4.2% decrease), with variations depending occlusion severity suggests that the multi-agent collaboration, particularly the reinforcement learning- based aggregation and point cloud gating, effectively mitigates the impact of missing visual data by leveraging complementary views and geometric consistency. The figure 21 shows of test examples and the output results
The slight degradation in CLIPScore highlights the challenge of occlusion but demonstrates Tri- MARF's ability to infer missing features, supported by the VLM's multi-turn prompting and the MAB's dynamic sclection. Tri-MARF's robustness is evident, suggesting its generalization to occluded scenarios.
14 Additional Details of all the experiments
This section outlines the comparison models, datasets, and evaluation metrics utilized to evaluate the performance of Tri-MARF in 3D object annotation tasks. These components are selected to provide robust and comprchensive assessment of our proposed method against existing approaches.
19

---

Figure 21: Occlusion experiment demonstration: The object is likely a chair, viewed from behind, with 3D model showing upward perspective. It combines wood and metal, featuring rounded backrest, vertical supports, and a circular base suggesting swivel or rocking mechanism. The smooth, polished surface indicates it's well-maintained new. Designed for comfort, it could be rocking chair suited for indoor use in living rooms, bedrooms, or similar settings.
14.1 Comparison Models
Cap3D: Cap3D is leading model for 3D object captioning that uses multi-view rendering to produce descriptions, serving baseline to compare against Tri-MARF's multi-agent collaborative framework. excels generating captions from multiple perspectives but lacks the reinforcement learning and point cloud processing capabilities that enhance Tri- MARF's robustness and accuracy.
ScoreAgg:
This model improves captioning accuracy by aggregating scores from multiple views, though it falls short of Tri-MARF's performance due to its inability to handle noisy data effectively. provides useful benchmark for evaluating the benefits of Tri-MARF's advanced aggregation strategy.
ULIP-2: ULIP-2 integrates language and 3D point clouds for enhanced understanding but relies single-vic view processing, limiting its generalization compared to Tri-MARP multi-view, multi-agent approach. It highlights the advantage of our method achieving superior cross- modal alignment.
PointCLIP employs CLIP for feature extraction from point clouds, yet its sim- plistic aggregation struggles with complex 3D structures, unlike Tri-MARF's sophisticated framework. It serves to demonstrate Tri-MARF's improvement in handling intricate object details.
PIIn
Combining large language models with 3D data, 3D-LLM offers high-quality captions but is computationally heavy, contrasting with Tri-MARF's efficient, lightweight design. This comparison underscores our method's s balance quality and speed.
3D-LLM:
GPT4Point merges point cloud data with GPT-4 for captioning, but its high latency and weaker cross-modal alignment make it less competitive than Tri-MARE. illustrates the efficiency gains from reinforcement learning-basse based aggregation.
GPT4Point:
Human annotations provide gold-standard reference for caption qual- ity, though they are slow and costly compared to Tri-MARF's automated, tigh-throughput approach. Tri-MARF aims to rival exceed this standard efficiently.
Human Annotation:
20

---

Dataset metadata offers basic benchmark for annotation, often lacking the semantic depth Tri-MARF achieves with its contextually rich descriptions. helps quantify our method's improvement over rudimentary annotations.
Metadata:
14.2 Datasets
Objaverse-LViS:
Objaverse-LVIS is a large-scale dataset with richly annotated 3D objects across diverse categories, ideal for testing TriMMAR''s caption quality and type inference accuracy. It challenges models with its variety, ensuring robust evaluation of generalization.
An expanded version of Objaverse, Objaverse- XL includes vast array 3D objects, with a 5k-object subset used to assess Tri-MARF's scalability and perfor- mance on large-scale data. Its breadth tests the model's ability to handle extensive datasets efficiently.
Objaverse-XL. XL:
ABO: Focused on furniture and household items, ABO's 6.4k real-world objects evaluate Tri-MARF's precision in annotating detailed, specifie 3D models. It provides practical testbed for real-world application scenarios.
Containing 51,300 synthetic 3D models across 55 categories, ShapeNet- Core is used to test Tri-MARR's adaptability to different data distributions in cross-dataset experiments. Its structured nature contrasts with noisier real-world datasets.
ShapeNet-Core:
ScanNet: ScanNet's 1,513 scanned point clouds of indoor scenes introduce noise and incompleteness, assessing Tri-MARF's robustness in real-world conditions. It challenges the model to perform reliably despite imperfect data.
ModelNet40: With 12,311 CAD models across 40 categories, ModelNet40 tests Tri-MARF on clean, well-structured 3D data, evaluating performance consistency. Its standardized format complements the diversity of other datasets.
14.3 Evaluation Metrics
Human evaluators score captions on 1-5 scale to gauge quality and preference, offering subjective measure of Tri-MARF's alignment with human expectations. It directly assesses user satisfaction with generated annotations.
A/B Testing:
CLIPScore measures semantic alignment between captions and 3D objects using text-image embedding similarity, providing automated metric for Tri-MARF's accuracy. It ensures objective evaluation of cross-modal consistency.
CLIPScore:
ViLT Retrieval (R) This metric evaluates Tri-MARF's retrieval accuracy (recall at rank 5) for image-to-text and text-to-image tasks, testing its ability to match queries with correct annotations. highlights the model's ''s retrieval effectiveness.
GPT-4o Scoring: Used for type inference, GPT-4o compares predicted labels to ground truth, accounting for synonyms to assess Tri-MARF's semantic accuracy. It offers nuanced evaluation beyond strict string matching.
String Matching Accuracy:
This metric calculates exact matches between predicted and ground-truth labels, providing a simple yet strict measure of Tri-MARF's type inference precision. It may undervalue semantically correct but lexically different terms.
BLEU-4: BLEU-4 assesses caption fluency and grammatical correctness by comparing n- gram overlap with reference texts, used here evaluate Tri-MARF's viewpoint experiment outcomes. It ensures the generated text is linguistically sound.
21

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction (Section 1) clearly state the proposed Tri-MARF framework, its three specialized agents, and the experimental results Objaverse-LVIS. Objaverse- and ABO datasets.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
22

---

Limitations are discussed in Section 6.4 (computing resourees and need for multi- in high-load), Section 9.1 (current limitations of the Gating Agent for full scenes). and Section 13 (performance drop under occlusion).
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiscless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Section 3.2.2 describes the Multi-Armed Bandit (UCBI) formulation. Section 11.6 provides the full derivation, probabilistic assumptions, and validation for the Gating Threshold (a = 0.557).
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
23

---

Detailed experimental setups, including datasets and metrics, are provided in Section 4 and Section 14. Hyperparameters and model configurations are detailed in Section 11. 1.5 and Section 6.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Section 5 states that code and annotated assets will be uploaded to the commu- nity.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
/nips.cc//pul http:cccppppcccceetteetec ptgpuudeee/odeeibbiisssooooooooo
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
24

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 14 details datasets and metrics. Section 11.2 and 11.5 detail the RL strategy and hyperparameter selection (e.g., learning rate, exploration weight).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Section 11.2 mentions experiments are conducted with three random seeds and results are averaged. Section 116 includes sensitivity analysis with performance fluctuations reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed reproduce the experiments?
25

---

Answer: [Yes]
Section 6 provides comprchensive analysis of GPU memory usage and processing efficiency on single NVIDIA A100 GPU.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp::/neuripps httppccbcc.//iiiicctuiudddooeddd
Answer: [Yes]
Justification: The paper adheres to ethical guidelines, including IRB approval and fair compensation for human annotators (Section 12).
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The Introduction discusses positive impacts in autonomous driving and robotics. Section 13 discusses robustness under occlusion, which relates to safety in real-world applications.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
26

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The paper focuses on 3D object annotation and does not release high-risk generative models. However, Section 11.6 describes gating mechanism mitigate halluci- nations/errors.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper cites and uses standard datasets like Objaverse, ShapeNet, and ScanNet (Section 14).
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pawerssipppsiiiiie... thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper introduces 2 million annotated 3D models. The annotation process is well documented in Section 3, and Section states assets will be uploaded.
27

---

Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Section 12 provides detailed protocols for human evaluation, including task descriptions, quality control, and compensation ($1 1//ourr)).
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Section 12 explicitly states: "We obtained local Institutional review board (IRB) approvals before conducting the experiment."
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
28

---

Justification:
The core methodology relies on Vision-Language Models (e.g., Qwen2.5-VL) as described in Section 3 and Section 11.1.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non- standstdd components.
Please refer to our LLM policy hattps://neurips... 220p://ee////o.feeeeee/2//2///// for what should or should not be described.
29


Reviews:
[{"review_id": "hiPEnOEWKY", "reviewer": "Reviewer_2YC9", "summary": "This paper is an application of large multi-modal models for generating captions for 3D models. \n\nThe method first uses VLMs to generate candidate captions and confidences for each view of the input model. Then, the candidates are clustered, where the candidate with the highest score represents each cluster. The candidate confidences are further weighted by the similarity of the semantic features and the image features. Finally, the candidates are sent to a multi-armed bandit-based agent for selection. The selected caption is accepted if it aligns well with the input point cloud embedding.\n\nThe approach outperforms previous studies on generating 3D model captions in the experiments.", "questions": "1. In Line 174, the author mentions, \"Unlike traditional approaches that rely on limited perspectives.\" Could the authors include a few references for traditional approaches? In fact, multi-view images are actually a pretty popular input representation for point cloud classification.\n2. Are all large models used in this study frozen? Did the authors attempt to fine-tune some of the VLM/LLMs used?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The experimental results show clear improvements over previous approaches on the selected datasets.\n2. The supplementary material is very comprehensive and provides an enormous analysis of the proposed framework.\n3. The paper is well-written and easy to follow.\n\nWeaknesses\n1. The proposed framework incorporates several off-the-shelf large models, and the key algorithmic contribution is the RL-based candidate selection (utilizing a multi-armed bandit-based agent). This, to some extent, limits the quality of this paper's technical contribution. In fact, I would encourage the authors to dive deeper into the core learning problem of the framework, i.e., candidate selection. Is there any other approach for candidate selection other than RL? Can the authors compare with other candidate-selection methods in other datasets? Answering these questions will strengthen this paper.", "was_revised": true, "final_justification": "The authors' rebuttal has addressed my initial concerns, and I recommend accepting this manuscript.", "rebuttal": "Thank you very much for your insightful and valuable comments! We have carefully prepared the following responses to address your concerns in detail. It is our sincere hope that our response could provide you with a clearer understanding of our work. If you have any further questions about our work, please feel free to contact us during the discussion period.\n\n**Q1. Is there any other approach for candidate selection other than RL? Can the authors compare with other candidate-selection methods in other datasets?**\n\n**A1:** Yes. To address your concerns regarding technical contribution and demonstrate the superiority of the Multi-Armed Bandit (MAB)-based Reinforcement Learning aggregation strategy over traditional methods, we have designed and conducted an additional ablation study, which is conducted on the Objaverse-XL dataset, comprising 10,000 randomly sampled objects, with the goal of quantifying the marginal benefits introduced by the MAB strategy. To ensure a fair comparison, we have strictly controlled the experimental variables. Specifically, we keep all other modules of our Tri-MARF framework identical (such as the initial VLM annotation, semantic clustering, comprehensive scoring mechanism, and final description synthesis logic), and replace the aggregation strategy being evaluated within the Information Aggregation Agent (Agent 2). The compared strategies include our MAB (UCB) method and four heuristic baselines: Maximum VLM Confidence, Maximum Combined Score, Weighted Voting, and Simple Prioritized Concatenation. The performance of all strategies is assessed using a comprehensive suite of metrics covering both quality and efficiency, including a human-evaluated Likert scale (1-10), automated CLIPScore and ViLT R@5 retrieval accuracy, and the inference time of the aggregation module measured on a single NVIDIA A100 GPU.\n\n**Table S1: Performance Comparison of Different Aggregation Strategies on the Objaverse-XL Dataset**\n\n| Aggregation Strategy            | Likert (1-10) | CLIPScore (%)↑ | ViLT R@5 (I2T) (%)↑ | ViLT R@5 (T2I) (%)↑ | Inference Time (ms)↓ |\n| ------------------------------- | ------------- | -------------- | ------------------- | ------------------- | -------------------- |\n| Maximum VLM Confidence          | 8.6           | 81.37          | 37.31               | 35.28               | 6.3                  |\n| Maximum Combined Score (Heuristic) | 8.9           | 82.03          | 38.15               | 35.92               | 8.1                  |\n| Weighted Voting (Heuristic)     | 8.8           | 81.85          | 37.93               | 35.76               | 8.5                  |\n| Simple Concatenation (Prioritized) | 8.5           | 80.74          | 37.08               | 34.81               | 6.8                  |\n| **MAB (UCB) (Ours)** | **9.3** | **82.72** | **38.82** | **36.72** | **9.8** |\n\nTable S1 indicates that while simple heuristic aggregation strategies can achieve decent performance, our MAB (UCB) strategy employed in Tri-MARF demonstrates a distinct advantage across all core quality metrics. The simplest strategies, \"Maximum VLM Confidence\" and \"Simple Concatenation,\" though fastest in inference, perform the worst on the Likert scale, CLIPScore, and ViLT retrieval accuracy, proving that relying solely on the VLM's initial judgment or fixed concatenation rules is sub-optimal. The better-performing \"Maximum Combined Score\" heuristic, which combines VLM confidence with CLIP visual alignment, achieves a Likert score of 8.9 and a CLIPScore of 82.03%, demonstrating the effectiveness of a robust, information-rich heuristic rule. In contrast, our MAB (UCB) strategy consistently outperforms all other alternatives in annotation quality, achieving the highest Likert score (9.3), CLIPScore (82.72%), and ViLT R@5 scores (I2T 38.82%, T2I 36.72%). Compared to the best-performing heuristic baseline, MAB improves the CLIPScore by approximately 0.7 percentage points and also shows a significant increase in the Likert human evaluation, indicating that its generated descriptions are perceived as more accurate, complete, and fluent by human assessors.\n\nThe advantage of the MAB strategy lies in its ability to dynamically learn and adapt its selection policy by balancing exploration and exploitation. This adaptive capability is particularly crucial when dealing with diverse object categories and initial VLM descriptions of varying quality, enabling it to consistently select optimal descriptions that fixed heuristic methods might miss. Although the MAB (UCB) has a slightly higher inference time (9.8ms), this minor increase (e.g., only 1.7ms slower than \"Maximum Combined Score\") is perfectly acceptable in practical applications, considering its throughput performance. In summary, this experiment proves that the MAB (UCB) aggregation strategy, despite adding a degree of complexity, delivers tangible and quantifiable improvements in annotation quality. This enhancement is vital for achieving state-of-the-art performance, thereby justifying its necessity and rationale as a core component of the framework.\n\n\n**Q2. References to traditional approaches that rely on a limited perspective?**\n\n**A2:** We have added related references in the revised paper on traditional methods for 3D understanding that rely on a limited perspective, such as Sparse Multi-View [s1], Voxel-Based [s2], and Monocular [s3] approaches.\n\n[s1] H. Su, S. Maji, E. Kalogerakis, and E. Learned-Miller, ‘Multi-view Convolutional Neural Networks for 3D Shape Recognition’, in *Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV)*, 2015, pp. 945–953.\n\n[s2] B. Zhang, J. Tang, M. Nießner, and P. Wonka, ‘3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models’, *ACM Trans. Graph.*, vol. 42, no. 4, Jul. 2023.\n\n[s3] X. Zhao, Z. Liu, R. Hu, and K. Huang, ‘3D object detection using scale invariant and feature reweighting networks’, in *Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence*, 2019.\n\n\n**Q3. Are all large models used in this study frozen? Did the authors attempt to fine-tune some of the VLM/LLMs used?**\n\n**A3:** Yes, all large models used in our study (e.g., Qwen2.5-VL, BERT, CLIP) are kept **frozen**, and we do not perform any fine-tuning. This decision is based on two core design considerations:\n\n+  **Generality and Scalability:** Our objective is to create a universal 3D annotation framework. By leveraging the powerful zero-shot capabilities of pre-trained models, our Tri-MARF can be directly applied to different domains without the need for expensive and time-consuming fine-tuning on new datasets. This point is substantiated by our excellent cross-dataset generalization results (Section 4.4, Table 2).\n\n+  **Efficiency and Cost-Effectiveness:** Fine-tuning large models requires immense computational resources, which contradicts our goals of achieving high throughput (12,000 objects/hour on a single A100) and low cost. Our Tri-MARF framework ensures high efficiency and cost-effectiveness by using API calls for the VLM and integrating lightweight local aggregation and gating modules, all while maintaining high-quality outputs.\n\nTherefore, keeping the models frozen is a key design decision made to strike an optimal balance between performance, efficiency, generality, and cost."}, {"review_id": "kZ6a831Eyj", "reviewer": "Reviewer_aDoZ", "summary": "This paper presents Tri-MARF, a tri-modal framework for 3D object annotation that integrates 2D images, 3D point clouds, and text. The framework is structured into three components: VLM-based description generation, an information aggregation module, and a gating module to enhance the annotation quality. The paper demonstrates the effectiveness of the framework through experiments on multiple datasets.", "questions": "See Weaknesses", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n* The framework’s architecture is easy to understand, and the paper is well-structured.\n* The system's decomposition into specialized agents for annotation, aggregation, and validation is an interesting way to address the complexity of 3D object inputs, potentially improving the system's adaptability across various tasks.\n* Tri-MARF shows good performance across datasets, with results surpassing SoTAs in some metrics. The throughput of 12,000 objects per hour demonstrates the system’s efficiency.\n\nWeaknesses:\n* Limited Contribution and Novelty: \n  1. The combination of 2D images, 3D point clouds, and text is not a novel contribution. Existing works like ULIP and 3D-LLM have explored similar modalities. The paper repackages existing techniques rather than introducing truly novel methods. This limits the overall innovation of the paper.\n  2. The core components of Tri-MARF, including the VLM-based description generation, aggregation module, and gating module, are based on widely used models in areas like 3D visual grounding and navigation. The reliance on these off-the-shelf components weakens the paper's contribution, as the paper does not propose novel modules specifically designed for 3D object annotation.\n\n* Insufficient Experiments: \n1. The multi-agent design, which is the paper’s main contribution, is not effectively isolated in the experimental setup. There is no comparison against a baseline that uses a single agent with the same inputs and architecture, which makes it unclear how much the multi-agent setup actually contributes to the overall performance.\n2. While quantitative metrics are strong, the paper lacks a detailed qualitative analysis, particularly regarding failure cases, hallucinations, or semantically inaccurate outputs. This oversight limits understanding of the system's limitations and real-world viability.\n\n* The use of large-scale pretrained models raises concerns about reproducibility. The paper does not address the computational and memory costs of these models, which could impact the scalability of the system. \n\n* The performance of the gating module is heavily dependent on the quality of the visual and textual encoders. If these encoders perform poorly on certain samples, the overall annotation quality may decline significantly.\n\n* The paper has several formatting issues, including small text in figures, unreadable symbols in equations, and inconsistent notation. These problems hinder the clarity of the presentation.", "was_revised": true, "final_justification": "Thank you to the authors for the rebuttal. It has helped clarify some of my concerns. However, I remain unconvinced regarding one of my main concerns about the novelty and technical contribution, which was also raised by Reviewer FNrm. While the system-level architectural design presents a different combination of components, it appears to reflect solid engineering practice rather than a substantial academic or technical innovation. In my view, this makes it difficult to meet the bar.\n\nIn addition, the formatting and presentation issues, also noted by the other two reviewers, suggest that the paper may not yet be fully ready. For these reasons, I have decided to maintain my original score.", "rebuttal": "Thank you very much for your insightful and valuable comments! We have carefully prepared the following responses to address your concerns in detail. It is our sincere hope that our response could provide you with a clearer understanding of our work. If you have any further questions about our work, please feel free to contact us during the discussion period.\n\n**Q1. Limited Contribution and Novelty.**\n\n**A1:** While our Tri-MARF framework indeed utilizes established pre-trained models, which is standard practice in the current field, we emphasize that the core innovation lies in the **system-level architectural design**, i.e., a collaborative network composed of three agents for VLM Annotation, Information Aggregation, and Gating Verification. This \"team of experts\" architecture is specifically designed to address key challenges that single models struggle with, such as cross-view inconsistency and missing geometric information. Its design philosophy is fundamentally different from existing methods that aggregate multimodal inputs.\n\nThe value of our architecture is not merely conceptual but has been experimentally validated. For instance, the Information Aggregation Agent employs a Multi-Armed Bandit (MAB) dynamic strategy customized for this task, rather than a simple off-the-shelf rule. The ablation study in Section 8 (Table 5) confirms that our MAB approach significantly outperforms four heuristic baseline methods across all metrics. Therefore, our contribution is not a mere stacking of modules, but a breakthrough in performance and efficiency achieved through sophisticated agent \"orchestration\" and synergy, which is in itself a significant system-level innovation.\n\n\n**Q2. Insufficient Experiments. Comparison against a baseline that uses a single agent with the same inputs and architecture.**\n\n**A2:** To more precisely dissect the contribution of our multi-agent framework, we have designed and added a step-wise ablation study to the revised paper. This study aims to demonstrate the performance evolution from a single agent to our complete three-agent collaborative architecture. The experiment is conducted on the Objaverse-LVIS dataset (with a random sample of 1k objects, the same subset used in the original paper), using CLIPScore and ViLT R@5 as the core evaluation metrics. We compare four key configurations: (1) **Single VLM Annotation Agent**, using only a single front view for description; (2) **VLM Annotation Agent + Information Aggregation Agent (IAA)**, a combination of the first two agents to evaluate multi-view fusion capabilities; (3) **Single Uni3D Gating Agent**, to separately assess the ability to describe based solely on 3D geometric information; and (4) **Tri-MARF (Ours)**, our proposed complete three-agent collaborative framework.\n\n| Method                      | CLIPScore↑ | ViLT R@5 (I2T)↑ | ViLT R@5 (T2I)↑ |\n| --------------------------- | ---------- | --------------- | --------------- |\n| Single VLM Annotation Agent | 81.4       | 38.5            | 36.7            |\n| VLM Annotation Agent + IAA  | 63.2       | 23.7            | 22.9            |\n| Single Uni3D Gating Agent   | 58.3       | 20.9            | 19.1            |\n| **Tri-MARF (Ours)** | **88.7** | **45.2** | **43.8** |\n\nThe results above demonstrate the necessity of the holistic and synergistic design of our three-agent framework. The strong performance of the single VLM agent (CLIPScore 81.4) proves that we have selected an effective base model, which significantly outperforms the geometry-only Uni3D agent (58.3). However, a key and counterintuitive phenomenon is that combining the first two agents leads to a performance drop to 63.2. We attribute this to the fact that, without final geometric verification, the blind aggregation of multiple (and potentially noisy) views \"pollutes\" the high-quality description from the best single view. This phenomenon precisely highlights the indispensable role of the Gating Agent (Agent 3).\n\nAfter introducing the Gating Agent, the complete Tri-MARF framework's performance leaps from a CLIPScore of 63.2 to 88.7, a stunning 25.5-point increase. This decisively proves that the Gating Agent acts as a critical \"arbiter,\" leveraging 3D point cloud information to resolve multi-view conflicts and VLM hallucinations, thereby unlocking and amplifying the full potential of the collaborative framework. Therefore, this experiment strongly indicates that our Tri-MARF framework's superior performance stems from the tightly-coupled, synergistic design where all three agents are indispensable, rather than from a simple stacking of any single component.\n\n\n**Q3. Lack of detailed qualitative analysis, especially discussion of failure cases or illusions.**\n\n**A3:** In fact, we have already provided several examples of qualitative analysis in our paper. Figure 1 offers a direct qualitative comparison of our Tri-MARF against SOTA methods like Cap3D and ScoreAgg. Figure 1 shows that our Tri-MARF can generate richer and more accurate details (e.g., \"scissor doors,\" \"carbon fiber\") and even recognize proper nouns (e.g., \"Golden State Warriors\"), which other methods fail to do. Figure 21 (Page 33) demonstrates our Tri-MARF's robustness in handling partially occluded objects, a typical scenario prone to failure. The descriptions generated in the figure remain accurate and detailed, proving our Tri-MARF's effectiveness.\n\nWe agree that the discussion of failure cases can be further strengthened. In the revised paper, we have added a dedicated subsection to discuss typical failure cases, such as challenges that may arise when processing objects with extreme geometric ambiguity, objects with distracting textures, or scene-level 3D models, to provide a more comprehensive perspective.\n\n\n**Q4. Reproducibility and computational cost brought by pre-trained models.**\n\n**A4:** In fact, we provide a very detailed analysis of this in Section 6 of the supplementary materials, titled \"Analysis of GPU Memory Usage and Computing Efficiency\" (Pages: 14-15). Table 3 in that section (Page 15) lists the GPU memory usage (GB) and processing time (seconds) for each module in our framework (Data Preparation, VLM Annotation, Information Aggregation, Gating) on a single NVIDIA A100 GPU. We explicitly state that the VLM Annotation Agent is called via an API and therefore consumes no local GPU resources. We also report the framework's peak memory usage (approximately 4.6GB) and the total time for a single run. This detailed data provides a transparent and concrete reference for the reproducibility of our method, while the affordable computational resource requirements and high throughput simultaneously demonstrate our architecture's scalability.\n\n\n**Q5. The performance of the gating module is heavily dependent on the quality of the visual and textual encoders.**\n\n**A5:** We argue that while the performance of any multi-modal system is indeed correlated with encoder quality, this is precisely why we design our three-stage, progressive multi-agent framework to distribute and mitigate over-reliance on any single module. The Gating Module (Agent 3) is the last line of defense in our pipeline, not the sole guarantor of quality. Before it, the VLM Annotation Agent (Agent 1) and the MAB-based Information Aggregation Agent (Agent 2) have already performed initial generation and multiple rounds of optimization on the content.\n\nThe comparison experiment in Figure 20 of the supplementary materials (Page 32) also substantiates this. Our Tri-MARF without the Gating Module achieves a CLIPScore of 85.7. Although lower than the full Tri-MARF's 88.7, this is still a very strong performance, proving the robustness of the first two agents. The Gating Module provides a further performance boost on top of this foundation. Its role is more of a final filtering function rather than providing the core annotation content, so it is unfair to discuss its performance separately from that of the first two agents.\n\n\n**Q6. The paper has several formatting issues, including small text in figures, unreadable symbols in equations, and inconsistent notation. These problems hinder the clarity of the presentation.**\n\n**A6:** We are very grateful to the reviewer for the valuable feedback on the paper's formatting and clarity. We fully agree that a clear and professional presentation is crucial for communicating our research findings. Following your suggestions, we have conducted a comprehensive proofreading and revision of the formatting issues in the revised paper to enhance overall readability. Specifically, we have made the following changes:\n\n+ **Enhanced Figure Clarity:** We have redrawn all core figures, including the architecture diagram (Fig. 2), performance comparison charts (Fig. 4), and the VLM cost-benefit analysis graph (Fig. 6). We have significantly increased the font size of legends, axis labels, and internal annotations to ensure all details are clearly legible.\n+  **Standardized Mathematical Formulas:** We have carefully checked and corrected the typesetting of all equations. For example, we have fixed rendering errors in the original paper caused by extraneous symbols (like semicolons) and ensured that mathematical symbols such as $\\text{Conf}(C)$ and $\\hat{r}_{a}$ are standard and consistent throughout the text.\n+  **Unified Key Notation:** We have performed a consistency check on the use of symbols throughout the paper. For instance, we found that the symbol \"$\\alpha$\" is used to represent two different concepts in the original paper (a weighting coefficient on Page 6 and a gating threshold on Page 7), which could cause confusion. In the revised paper, we have renamed the gating threshold to a new symbol, $\\tau$, to eliminate this ambiguity.\n\nWe believe these detailed revisions have effectively addressed the issues you raised and significantly enhanced the professionalism and readability of the paper."}, {"review_id": "8YpbWklKFq", "reviewer": "Reviewer_JB3j", "summary": "Tri-MARF introduces a Multi-Agent Framework for Captioning 3D Models. By using prompting strategies, embedding clustering, score filtering, visual text-alignment, reinforcement learning and geometrical gating they obtain consequential gains over previous methods. They show strong preference over human generated captions and state-of-the-art results in retrieval and clip scores.", "questions": "--Questions--\n1. Section 3.1 line 145 refers to prompting by explicitly orienting the model to the current view-point direction. I have two concerns here: 1) To be able to render a 3D object from the “front” one needs to assume standardized/canonical poses. Is this assumed for the 3D models? If not, how is it determined whether a view is “front, back…”? 2) Are their mechanisms to handle symmetric or cylindrical objects where views may be redundant or is this case considered an experimental outlier?\n    \n2. For Core Description Extraction seems like a lot of faith is put into the LLM here. Is there any experimentation which suggests that LLM are predisposed to producing core descriptions as the first sentence of an output sequence. Is this by any chance an induced behaviour? for example via in-context learning?\n\n3. Given that CLIP is used in Relevance Weighting (3.2.1) and hence the final score which dictates textual relevance, could you justify the use of CLIPScore to measure performance?\n\n4. My major concerns from the result section are regarding A/B Scores in Table 1. If my understanding is correct, an A/B score of 3 is a tie, anything higher indicates that the method being compared against was preferred. In all the datasets, CAP3D, ScoreAgg and 3D-LLM were preferred over the proposed method. Please clarify whether this is the case, if not, specify the win and lose rate (as in CAP3D) for more clarity. If this is expected please further elaborate why this could be happening?\n\n5. line 264-267 seem to imply that the gating agent simply filters out captions which do not align well with the provided geometry. Was there any consideration given to automatic self-refinement so that manual labeling could be limited?\n\n--Suggestion--\n1. The question: “how can we design a system that collaborates like a team of human experts” (line 35) remains to be answered. Ablation over each agent individually and showing A/B scores or VilT retrieval scores for each additional agent would significantly improve quality in my opinion.\n\n\n--Minor concerns--\n1. The experimental setup for measuring A/B testing quality is not clear. How many observations are made for each comparison across all objects? Is this the same setup as described in section 5.1 in CAP3D?\n\nOverall Comment: Although I am generally liking the the novel idea of using RL concepts in agentic frameworks, I do feel that overall the paper did not answer some of its own questions. The lack of ablations in the main-text weakens the paper. Some evaluation concerns like the use of CLIP is not directly addressed here and the A/B scores are not properly analysed. If the above concerns are addressed, possibly with further ablations and evaluations, I am happy to increase the rating.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality: The quality of the paper is decent. Provided figures are understandable and written prose in most places is fairly composed. Overall, the motivation is mostly clear although the literature gaps in previous works is not directly addressed in the evaluations. For example “overlooking geometric information (from previous works)” is mentioned as a pain-point from previous methods but the **main paper** does not directly address this via ablations on the gating agent. Related work fairly covers seminal works but some recent works are skipped.\n\nClarity: Although the motivation is clear and the consequent design choices, the methodology is lacking clarity in terms. Some inline equations are not well setup, variables are inconsistent and some important details are only brushed over.\n\nSignificance: Using a novel reinforcement learning framework adds to the significance of this method which treats FM and LLM agents as part of a more elaborate captioning system which is an interesting approach previous methods have not truly utilized.\n\nOriginality: The methodological novelty as mentioned previously and some new evaluation schemes (Type Annotation Experiment) adds to the originality of the method.", "was_revised": true, "final_justification": "Successfully defend their novelties, completely ablate the individual modules (agents) in their framework, improve the quality of writing and figures. I am encouraged by their results compared to the baselines. Aside from some concerns regarding metrics I find that the evaluation suite is quite robust, which gives me confidence in this work. The use of a reinforcement learning paradigm in weighting caption-significance carries some significance; however, the RL algorithm is not a contribution itself (hinting at what other reviewers have also shown concern for). Having said that, the rigorous empirical and experimental evidence does elicit a rating leaning towards acceptance in my opinion. With this, I recommend a \"Borderline Accept\".", "rebuttal": "**Q1: 1) Are standardized/canonical poses assumed for the 3D models? 2) Are there mechanisms to handle symmetric or cylindrical objects?**\n\n**A1:** Regarding viewpoint standardization, our processing pipeline assumes a standardized object pose. During the data preparation stage, we normalize the 3D models by aligning their principal axes with the coordinate system. This is a standard preprocessing step to ensure we consistently render from six standardized orthogonal viewpoints. \n\nThe issue of potential information redundancy from symmetric objects highlights the advantage of our Information Aggregation Agent (Agent 2), particularly its adaptive selection mechanism based on the Multi-Armed Bandit (MAB). Unlike existing works use fixed rules (like simple concatenation or averaging), our MAB agent learns to dynamically evaluate the information value provided by each viewpoint. For a symmetric object, if multiple views generate similar or redundant descriptions, our MAB learns during its training process to lower the weight of these redundant \"arms\" (i.e., descriptions) and prioritize those that offer unique information. Thus, our Tri-MARF has an intrinsic robustness to handle such cases rather than treating them as experimental outliers. We have made the preprocessing step more explicit with more details in the revised paper. \n\n\n**Q2. Are LLMs predisposed to producing the first sentence of an output sequence? Is this an induced behaviour? via in-context learning?**\n\n**A2:** This is indeed a heuristic design, but it is an effective choice based on observations of LLM's behavior and our unique prompting strategy. Our VLM Annotation Agent (Agent 1) employs a multi-turn, structured dialogue strategy. We first ask the model to identify the object (\"What is this?\") and then follow up with questions to obtain specific attributes like color and material (Refer to Figure 2 for more details). This general-to-specific questioning approach, guided by prompt engineering, naturally leads the LLM to summarize the object's core identity in the first sentence of its response, with subsequent sentences providing detailed elaborations. This is a stable behavior of the LLM that we have observed and guided through prompt engineering, not a blind trust in the model.\n\nIn our Tri-MARF, this \"core sentence\" is primarily used to construct the beginning of the final description, ensuring it has a clear topic sentence. The rich details provided by other viewpoints are then appended to form a complete and comprehensive global description. \n\n**Q3. Justify the use of CLIPScore to measure performance**\n\n**A3:** Our Tri-MARF does not rely solely on CLIPScore, neither in its internal mechanism nor in its external evaluation. Firstly, in the core relevance weighting module of our Tri-MARF, we use a hyperparameter α to balance the VLM's own confidence with CLIP's visual alignment, which jointly form the final description score $s_i$. This ensures the model is not designed merely to optimize for CLIP similarity. More importantly, we employ a diverse set of complementary metrics in the final performance evaluation. As shown in Table 1, we also include ViLT R@5 retrieval accuracy and human A/B test scores. These results strongly prove that its performance advantage is comprehensive and genuine, not a case of \"overfitting\" to a single metric.\n\nSecondly, we chose to include CLIPScore since it has become the de facto standard in the field for evaluating image-text semantic alignment, as used by Cap3D, ScoreAgg, etc. \n\n**Q4. The doubts about the A/B test scores (Table 1).**\n\n**A4:** This is a misunderstanding of how the results are reported; the correct interpretation is the opposite. In our A/B tests, Tri-MARF is set as the fixed reference baseline. The A/B score listed under another method (e.g., for Cap3D) represents the average preference score given *to our method* by human evaluators in a pairwise comparison against that method (based on a 1-5 scale, where 3 is a tie).\n\nThe correct way to read the score is as follows: the 3.3 score under Cap3D means that in a direct comparison with Cap3D, human evaluators preferred our Tri-MARF, giving it an average score of 3.3. Since all baseline methods have an A/B score greater than 3, this indicates that our Tri-MARF is preferred in all pairwise comparisons.\n\nTo completely eliminate any ambiguity, we have followed your suggestion and redesigned the A/B test results table in the revised paper to explicitly show the win/loss/tie ratios, and we have rewritten the caption in detail to make the results clear at a glance.\n\n\n**Q5. Why does the gated agent only perform filtering and not try to self-correct to reduce manual labeling?**\n\n**A5:** Our current design is primarily based on considerations of efficiency and modularity. The Gating Agent (Agent 3) plays a lightweight final verification role in our Tri-MARF, whose main task is to quickly filter out hallucinated descriptions generated by the VLM that are clearly inconsistent with the 3D geometry, thereby forwarding only the most questionable samples to human reviewers. This ensures the high throughput of the entire process (12,000 objects/hour). Introducing an automatic self-correction loop would significantly increase system complexity and processing latency. This would be contrary to our design goal of achieving high efficiency.\n\n\n**Q6. Separate ablations for each agent and display the A/B score or VilT retrieval score for each newly added agent.**\n\n**A6:** To highlight the contribution of our Tri-MARF, we have designed and added a step-wise ablation study in the revised paper to demonstrate the performance evolution from a single agent to our Tri-MARF. This experiment is conducted on the Objaverse-LVIS dataset (with a random sample of 1k objects, the same subset used in the original paper). We compare four key configurations: (1) **Single VLM Annotation Agent**, using only a single front view for description; (2) **VLM Annotation Agent + IAA**, a combination of the first two agents to evaluate multi-view fusion capabilities; (3) **Single Uni3D Gating Agent**, to separately assess the ability to describe based solely on 3D geometric information; and (4) **Tri-MARF (Ours)**.\n\n| Method                      | A/B Score↑ | CLIPScore↑ | ViLT R@5 (I2T)↑ | ViLT R@5 (T2I)↑ |\n| --------------------------- | ---------- | ---------- | --------------- | --------------- |\n| Single VLM Annotation Agent | 8.6        | 81.4       | 38.5            | 36.7            |\n| VLM Annotation Agent + IAA  | 7.2        | 63.2       | 23.7            | 22.9            |\n| Single Uni3D Gating Agent   | 6.5        | 58.3       | 20.9            | 19.1            |\n| **Tri-MARF (Ours)** | **9.3** | **88.7** | **45.2** | **43.8** |\n\nThe results above demonstrate the necessity of the holistic and synergistic design of our Tri-MARF. The strong performance of the single VLM agent (CLIPScore 81.4), as an effective base model, significantly outperforms the geometry-only Uni3D agent (58.3). However, there is a counterintuitive phenomenon that combining the first two agents leads to a performance drop to 63.2. We attribute this to the fact that, without final geometric verification, the blind aggregation of multiple (and potentially noisy) views \"pollutes\" the high-quality description from the best single view. This phenomenon precisely highlights the indispensable role of the Gating Agent (Agent 3). After introducing the Gating Agent, our Tri-MARF's performance leaps from a CLIPScore of 63.2 to 88.7, a stunning 25.5-point increase. This decisively proves that the Gating Agent acts as a critical \"arbiter\", leveraging 3D point cloud information to resolve multi-view conflicts and VLM hallucinations. \n\n\n**Q7. The setup for measuring A/B testing quality is not clear.**\n\n**A7:** We have already provided a comprehensive and standardized explanation of the detailed settings for human evaluation in Section 12, which clarifies the recruitment criteria for evaluators, including their required annotation experience and language proficiency. It then details the specific procedures for each experiment (e.g., 3D caption quality evaluation, type annotation validation, etc.), covering task design, the number of samples and evaluators, and explicit scoring criteria (e.g., a 1-5 Likert scale). To ensure the reliability of the evaluation results, we have also implemented strict quality control measures, including a pre-task training process for the evaluators, and quantified the reliability by calculating inter-annotator agreement (Cohen's Kappa reached 0.76), thereby guaranteeing the fairness and scientific validity of the entire evaluation process.\n\n**Q8. Typos in formulas and grammar.**\n\n**A8**: We have corrected these and proofread the whole paper. Specifically, regarding the mathematical expression on Line 122, we have removed the extraneous dollar signs surrounding the formula to present it as a standard inline equation, $ D_{v}={C_{v,i}}{i=1}^{M} $. Regarding the grammatical error on Line 128, we have corrected the verb \"frames\" to \"frame\"; for the inline mathematical expression on Line 202, we have rewritten it using clearer, standard notation, changing the original format to $C{\\text{canonical}}^{(k)} := \\underset{C_{v,i} \\in \\mathcal{C}k}{\\arg\\max} s{v,i}$; regarding the cluttered layout in Section 3.2.2, we have thoroughly re-typeset the subsection for clarity, which includes correcting the jumbled symbols in formula (Line 227) to a standard reinforcement learning objective function, fixing the extraneous semicolon and using the standard '\\arg\\max' command in the UCB1 algorithm expression (Line 231) to be $\\underset{a \\in \\mathcal{A}}{\\arg\\max} \\left( \\hat{r}_a + c \\sqrt{\\frac{2 \\ln t}{n_a}} \\right)$, correcting the erroneous comma in the empirical mean update formula (Line 235), and adjusting the paragraph structure throughout the subsection to ensure a logical and coherent flow of the argument."}, {"review_id": "MF2zSsyhof", "reviewer": "Reviewer_FNrm", "summary": "The authors propose a multi-agent framework which take tri-modal inputs to enhance 3D annotation process. Extensive experiments on several datasets demonstrate that the proposed method provides annotations with higher quality and faster throughput.", "questions": "1. The authors state that lower Conf(C) (eqa. (1)) values indicate higher confidence (higher token probabilities). It is counterintuitive to define a confidence score that lower confidence score indicates higher confidence. Also, the explanation for eqa. (1) is not sufficient, i.e., why lower confidence score indicates higher confidence?\n2. Does the $S_{\\text{conf},i}$ in line 214 the same as the confidence in eqa.(1)? If so, please use the same notation. Also, is $w_i$ in line 216 the same as $w_{v,i}$ in eqa. (3)? It is confusing to use different subscripts for the same variable.\n3. For the balance score $\\alpha$ in line 216, does it need to change for different datasets?\n4. The core description extraction in line 250 is confusing, as no intuitive explanation on why the authors are doing so.\n5. To ensure the fair comparison, the baselines and the proposed method should use similar/comparable pre-trained backbones/VLM/LLMs.\n6. The authors should provide the concrete math representations for the metrics. For example, the reviewer could not understand why human annotations is worse than VLM generated for GPT-4o scores (line 313).\n7. Why the time reported in Table 3 is not consistent with the inference time reported in sec 4.1 (12k/hour)? What is the difference? How does it compare to baseline methods?\n\nThe reviewer may consider increasing the score if the authors could address the questions, or at least provide some clarification.", "limitations": "The authors describe the limitation in the paper checklist section, including (1) lack of more efficient communication strategies to refine decision-making and reduce computational overhead. (2) Full scene point clouds presents challenges due to its object-centric optimization.\nThe authors does not provide potential negative social impact.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The proposed method is reasonable, which addresses the limitations of the current 3D object annotations. It may benefit other researchers in the community who need more comprehensive 3D object annotations.\n2. The authors provide thorough experiments on the design choices.\n\nWeakness:\n1. The submission is poorly written, with numerous typos in equations and bad paragraph formatting. For example, in section 3.2.2, the inline equation in line 227, 231, 235 includes additional semicolons and commas. The mutiple bolded sub-titles in the paragraph starting from line 238 should be divided into multiple paragraphs. The quality of the performance metrics figures in the Appendix could be significantly improved.\n2. The proposed method seems to achieve better result by offering a novel combination of existing technique with additional inputs. The reviewer acknowledge the better performance but consider the technical contribution is limited.\n3. The experiment settings is questionable, as the previous methods are not using the similar VLM (or comparable agents). It is not clear whether previous methods with comparable VLMs could achieve comparable results.\n4. Although the authors provide many evaluation metrics, only textual description is provided in the appendix. Concrete math representations would help the readers to get a better understanding on the experimental results.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns in the rebuttal phase through experiments and clarifications. However, my major concern of this paper is the limited novelty and contributions (also identified by reviewer aDoZ), and poor presentation (also identified by reviewer JB3j and aDoZ). Overall, I would like to increase my rating from Reject to Borderline reject.", "rebuttal": "**W1. Poor presentation**\n\n**A1:** We have thoroughly revised the format and layout of the entire paper in the revised version. Specifically, we have proofread and corrected all mathematical equations, especially the formatting errors in Section 3.2.2, to ensure that they are clear and accurate. We have restructured the content starting from Page 6, splitting the original paragraphs that mixed multiple topics into independent sub-paragraphs with clearer logic. In addition, all performance charts in the Appendix (such as Figures 7-13, etc.) have been regenerated to provide higher resolution and better readability. To make evaluation indicators clear enough, we have added a dedicated section in the revised appendix to provide clear mathematical definitions and equations for all key evaluation indicators. We believe that these corrections have significantly improved the overall quality of our paper.\n\n\n**W2. Limited technical contribution**\n\n**A2:** The core innovation of our work is to design a multi-agent collaborative architecture. To highlight our contribution, we conduct a dedicated ablation study in Section 7 (Pages 15-16).\nTable 4 directly compares our Tri-MARF with a strong baseline that uses the exact same VLM (Qwen-2.5 VL) but relies on only a single agent. The results show that our multi-agent collaborative architecture brings a **+7.3 percentage point increase in CLIPScore**. This decisively proves that the superior performance of our Tri-MARF stems from the tight and effective synergy between the agents.\n\n**W3. Unfair experiment settings**\n\n**A3:** No, our experimental settings are fair.\n+  To compare against published SOTAs (like Cap3D, ScoreAgg), we follow the original configurations reported in their papers or use their released codes. Forcibly integrating the latest VLMs into these SOTAs is a non-trivial task and might introduce other variables.\n+  To demonstrate that our Tri-MARF's performance is not dependent on a specific VLM, we conduct a comprehensive performance and cost evaluation in Figure 6, Section 11.1 (Page 21), where we replace the VLM in our Tri-MARF with several industry-leading models (including GPT-4.5, Claude-3.7, Gemini-Flash-2.0, etc.). The results show that Qwen2.5-VL represents the optimal balance between performance and cost, not that its performance far exceeds other models.\n\n**W4: Concrete math representations.**\n\n**A4**: We have addressed this in the revised paper by adding a new subsection adjacent to Appendix C.1 dedicated to providing rigorous mathematical definitions for our core evaluation metrics. For **Uncertainty Accuracy ($UA$)**, we have introduced a classification-based equation, such as $UA = \\frac{TP_{\\text{uncertain}}}{TP_{\\text{uncertain}} + FP_{\\text{uncertain}}}$, to clearly define its calculation. For **Calibration Error ($CE$)**, we have explicitly included the standard mathematical equation for the Expected Calibration Error (ECE) variant, namely $\\sum_{m=1}^{M} \\frac{|B_m|}{n} |\\text{acc}(B_m) - \\text{conf}(B_m)|$, detailing the process of binning and the calculation of the difference between confidence and accuracy. Finally, for **Dynamic Adaptability ($DA$)**, we have elevated it from a descriptive proxy (\"correlation between... signals and... mechanisms\") to a formal statistical measure via the Pearson correlation coefficient, to precisely quantify the relationship between system uncertainty and the activation of deeper reasoning mechanisms like debate triggering. \n\n**Q5. In Conf(C) Eq. (1), why do lower values represent higher confidence?**\n\n**A5:** This score is mathematically defined as the Average Negative Log-Likelihood. Our paper explicitly states on Page 5, Line 160, that this confidence score \"quantifies the semantic reliability through the average token log-likelihood.\" Its calculation method is detailed in Eq. (1) (Page 5). Since the generation probability P(ti) of a token is between 0 and 1, its logarithm, log(P), is negative. Therefore, taking the absolute value is mathematically equivalent to calculating its Negative Log-Likelihood (NLL), i.e., -log(P). The higher the probability of an event (i.e., the more \"confident\" the model is), the lower its negative log-likelihood value. Thus, as our paper directly states on **Page 5, Line 166**: \"a lower Conf(C) value indicates higher confidence (i.e., higher token probabilities).\" To resolve this clarity issue, we have renamed it in the revised paper to \"Avg. NLL Score\"**. \n\n**Q2. Inconsistent symbols.**\n\n**A6:** We sincerely apologize for this confusion. S_conf,i (Line 214) does refer to the confidence score calculated for the i-th candidate description using Eq. (1). Similarly, w_i (Line 216) is indeed the same variable as w_v,i defined in Eq. (3), which is the weight calculated based on CLIP. In the revised paper, we have conducted a thorough proofreading of the entire paper to unify all mathematical symbols, e.g., by consistently using Conf(C_{v,i}) and w_{v,i}, to ensure the rigor and clarity of our paper.\n\n\n**Q3. Does the balanced score's alpha change across datasets?**\n\n**A7:** NO! We have conducted a detailed sensitivity analysis of this hyperparameter in Section 11.5 (Figure 8, Page 24) and show that our Tri-MARF performs optimally on the large and diverse Objaverse-XL dataset when alpha is set to 0.2. In all the experiments, we keep alpha fixed at 0.2 without any re-tuning. \n\n\n**Q4: Line 250 is confusing.**\n\n**A8:** This is indeed a heuristic design, but it is supported by our unique prompting strategy. Our VLM Annotation Agent (Agent 1) employs a structured, multi-turn dialogue strategy, as detailed in Section 3.1 (Page 4). Agent 1 does NOT generate a long description all at once. Instead, Agent 1 responds to \"What is this object? What is its specific name?\" and further follows up with questions about attributes like color and material. This general-to-specific questioning manner naturally guides the VLM to first summarize the object's core identity in the initial sentence of its response, with subsequent sentences providing detailed elaboration. Therefore, extracting the first sentence as the \"core description\" is to ensure that the final global description has a clear and concise topic sentence, which is then enriched with details provided by other viewpoints. \n\n**Q5. Use similar/comparable pre-trained backbones/VLM/LLMs.**\n\n**A9:** We must clarify that some of the core baseline methods do not use a VLM in their setup. Despite the inherent architectural differences with some baselines, our experimental design is rigorously justified from two perspectives to ensure fairness, and we have added comparisons with other VLM-based methods. Firstly, to demonstrate the independent value of our Tri-MARF, we conduct a crucial ablation study (Table 4, Supplementary) comparing the full framework against a single-agent baseline using the same VLM. The results show that our Tri-MARF provides a +7.3 percentage point increase in CLIPScore, proving that the performance gain comes primarily from the synergistic mechanism, not the VLM. Secondly, to demonstrate the fairness of our VLM selection, we have evaluated multiple industry-leading VLMs (Figure 6, Supplementary). \n\n**Q6. Why are human annotations worse?**\n\n**A10:** This is due to the two inputs being fundamentally different (Section 4.2), i.e., the human answers are \"multiple-choice,\" while our Tri-MARF's output is \"free-text.\" Specifically, this experiment is to evaluate the accuracy of type annotation, and annotaters choose from a pre-set, limited list of options. If the most accurate answer is not in the list, they have to select a semantically broader or not perfectly fitting approximate answer. In contrast, our Tri-MARF generates a rich and detailed natural language description. GPT-4o, acting as the judge, has powerful semantic understanding capabilities and can recognize synonyms or more precise hyponyms. For example, when the ground truth is \"cup,\" the human's options might only include the general term \"container,\" but our Tri-MARF might generate \"a white ceramic coffee mug.\" GPT-4o can accurately determine that \"coffee mug\" is semantically much closer to \"cup\" than \"container\" is, thus giving our Tri-MARF a higher score. Therefore, this result precisely validates the precision and richness of the descriptions generated by our Tri-MARF.\n\nWe have revised Section 4.2 to more clearly explain the evaluation mechanism of this scoring task, explicitly pointing out the difference between \"multiple-choice\" and \"free-text\" input formats and their impact on the final score. \n\n\n**Q7: Table 3 is not consistent.**\n\n**A11:** This discrepancy arises from measuring two different performance metrics: Latency versus Throughput. Specifically, Table 3 (Page 15) reports the end-to-end latency of 7-19 seconds for processing a single object on a single GPU. This time is predominantly driven by the serial network latency of remote API calls to the VLM. In contrast, the 12k objects/hour claimed in Section 4.1 (Page 7) is the aggregate throughput of our entire annotation process. As mentioned on Page 15, we achieve extremely high processing efficiency at a macro level by parallelizing thousands of independent API call requests across multiple machines and GPUs in our large-scale annotation process. This throughput metric is also the standard for efficiency comparison with baseline methods, as shown in the \"obj/PGCHour\" column of Table 1. Ours (12k/hour) is significantly superior to methods like Cap3D (8k/hour) and ScoreAgg (9k/hour).\n\n**Q12: Limitation Section.**\n\n**A12:** We have expanded it in the revised paper by adding a more in-depth discussion on potential negative social impacts. Specifically, we explicitly state that our Tri-MARF is designed to provide high-quality annotations for beneficial fields like autonomous driving and robotics. Thus, it could potentially be misused to generate misleading or biased descriptions of real-world scenes or objects."}]

Author rebuttal:

Meta-review: The paper proposes a method for generating captions from 3D models. The main contribution is to integrate multi-modal inputs using multi-agent collaboration. The reviewers all appreciated the motivation and the results; however, several reviewers raised concerns on the limited contributions, limited evaluations, and unsatisfactory writing. 

The rebuttal added several ablation and comparison results, and pitched the main contribution as the careful system-level integration of existing methods. These contributions have value, when they are thoroughly evaluated, as is the case with this paper (considering the additional results in the rebuttal). Two reviewers are still concerned with this nature of contributions, as well as with the quality of exposition. 

I am inclined to accept the paper, seeing that the results are state of the art, and that the different components of the system are evaluated well. However, it is very important to include the additional results, as well as to improve the writing in the final version.
Decision: Accept (poster)