Why Keep Your Doubts to Yourself? Trading Visual Uncertainties among Vision-Language Models
Authors: Jusheng Zhang
Yijia Fan
Kaitong Cai
Jing Yang
Jiawei Yao
Jian Wang
Guanlong Qu
Ziliang Chen
Keze Wang
Venue: ICLR 2026 Poster (2026)
Keywords: agent; Vision Language Model; Uncernity
Primary area: foundation or frontier models, including LLMs
TL;DR: We presents a framework that reframes coordination as a decentralized market for uncertainty.

Abstract: Vision-Language Models (VLMs) enable powerful multi-agent systems, but scaling them is economically unsustainable: coordinating heterogeneous agents under information asymmetry often spirals costs. Existing paradigms, such as Mixture-of-Agents and knowledge-based routers, rely on heuristic proxies that ignore costs and collapse uncertainty structure, leading to provably suboptimal coordination.
We introduce Agora, a framework that reframes coordination as a decentralized market for uncertainty. Agora formalizes epistemic uncertainty into a structured, tradable asset (perceptual, semantic, inferential), and enforces profitability-driven trading among agents based on rational economic rules. A market-aware broker, extending Thompson Sampling, initiates collaboration and guides the system toward cost-efficient equilibria. Experiments on five multimodal benchmarks (MMMU, MMBench, MathVision, InfoVQA, CC-OCR) show that Agora outperforms strong VLMs and heuristic multi-agent strategies, e.g., achieving +8.5% accuracy over the best baseline on MMMU while reducing cost by over 3×. These results establish market-based coordination as a principled and scalable paradigm for building economically viable multi-agent visual intelligence systems.

Published as a conference paper ICLR 2026
WHY KEEP YOUR DOUBTS TO YOURSELF? Trading VI- SUAL UNCERTAINTIES AMONG VISION-LANGUAGE MOD- ELS
Jusheng Zhang 2 Yijia Fan Kaitong C Yang', Jiawei Yao², Jian Wang", Guanlong Qu', Ziliang Chen SM Keze Wang
'Sun Yat-sen University University of Washington SSyracuse University.
Snap Inc.
5 Peng Cheng Laboratory.
ABSTRACT
Vision-Language Models (VLMs) enable powerful multi-agent systems, but scaling them is economically unsustainable: coordinating heterogeneous agents under informa- tion asymmetry often spirals costs. Existing paradigms, such Mixture-of-Agents and knowledge-baad based routers, rely on heuristic proxies that ignore costs and collapse uncer- tainty structure, leading to provably suboptimal coordination. We introduce Agora, framework that reframes coordination as decentralized market for uncertainty. Agora for- malizes epistemic uncertainty into a structured, tradable asset (perceptual, semantic, infer- ential), and enforces popftabbilttyddriven trading among agents based rational economic rules. A market-aware broker, extending Thompson Sampling, initiates collaboration and guides the system toward cost-efficient equilibria. Experiments on five multimodal bench- marks (MMMU, MMBench, Math Vision, Info CC-OCR) show that Agora outper- forms strong VLMs and heuristic multi-agent strategies, e.g., achieving +8.5% accuracy over the best baseline on MMMU while reducing cost by over 3x. These results establish market-based coordination as a principled and scalable paradigm for building economi- cally viable multi-agent visual intelligence systems.
INTRODUCTION
The rapid advancement of Language Models (VLMs) (Li et al. 2022 2023
Liu ct al. 2023a Bai
et al.
2023
Zhang et al. 2025f g b has propelled the development of multi-agent systems (MAS) (uoo
2024 Wang al.) 2024c), moving closer to the vision of powerful, collective intelligence. Yet, al.
as these systems scale, they inevitably collide with foundational challenges from economic theory: coordi- nating self-interested agents under information asymmetry and making globally optimal decisions under bounded rationality. We argue current paradigms fail to address these root problems, leading to a crisis of economic viability where operational costs spiral, precluding effective, large-scale deployment (Gandhi ett al.] (025). This inefficiency stems from a failure to treat intelligence not as a brute-force commodity, but as scarce economic resource requiring principled management.
Existing coordination strategies can be understood as heuristic patches, i.e., computationally cheap workarounds for these deep-seated barriers. Paradigms like Mixture-of-AAgets (MoA) (Guo et al.] (024) knowledge-based routers (e.g., KABB) (Zhang et al.) (025i) attempt to bypass the complexity of true optimization by relying on simplistic proxies for value, such as consensus or semantic similarity. As we for- mally prove in Section 2
these heuristics render the systems fundamentally agnostic to the core economic
* Equal contribution. "Corresponding authors.

---

Published as a conference paper ICLR 2026
variables of cost and the fine-grained structure of uncertainty. This agnostic nature not a minor flaw but a theoretical dead-end, leading to provably suboptimal performance and systemic waste.
To dismantle this economic bottleneck, we argue for a paradigm shift: from heuristic patches to mechanism that embraces the decentralized nature of the problem. Accordingly, we construct Agora, a framework that redesigns multi-agent coordina- tion a decentralized micro-economy, Agora does not attempt to approximate central planner; in- stead, it uses market-basee mechanisms to achieve efficient coordination despite information asymme- try and bounded rationality. Within this framework, cognitive uncertainty is no longer monolithic li- ability but "minted" into a quantifiable. tradable asset. Agents, guided by price signals and driven by economic incentives (Gale & and 1962 von Neu- mann & Morgenstern, 2004) AAkerlof 1970). trade this asset to reveal private information and drive the entire system towards cost-effective equilibrium.
Please describe the image.
KABB
Oheer
① ccs
Scuma
Could you describe this mage detall?
Output
MAB
Prysical
Selected Agent Generate
The scene deplets urbaa
There le clearty diroemciie leghte four voahie
Key cbjecte identiiled Hacley Derkison .....
appware antively
Our methodology, detailed in Section 3 provides a constructive, non-agnostic solution. We first es- tablish a multi-dimensional uncertainty quan- tification model, creating structured asset that makes the system structure-aware. Second, we introduce profitability that enforces economic rationality, making the sys- cost-aware. Finally, the entire market is or- chestrated by an intelligent market-aware Broker, which uses sophisticated utility function find economically sound initializations for the collabo- rative process.
Agora
Trading Center
Parceptial Uineettrrty
Suratic Uncerteinty
Uncertanty
trading protocol
Could you describe mage detai??
Trade
History
Trade
Output
Trade
Rewarrd
TNss scene depions riding...
Our comprehensive experiments on multiple vi- sual understanding benchmarks (e.g.. MMMU (Yue eet al. 2024). MMBench (Liu al.) (023b)) demonstrate that Agora not only achieves state- of-the-art performance but also dramatically im- proves cost-effectivenesss validating our market- based paradigm. This work lays theoretical and practical foundation for building truly scal- able and economically viable multi-agent intelli- gent systems.
Agentz
primary idorofluble product
motoroycle,
Agent3
MAB
The rider activdly operoting the
Figure Comparison of heuristic coordination and Agora. Unlike heuristics that rely on flawed proxies, Agora forms a dynamic market for uncertainty, where emergent prices enable coordination.
PROBLEM FORMULATION
The rise of multi-agent systems (MAS) promises powerful collective visual intelligence, yet this ambition faces crisis of economic viability: soaring operational costs preclude scalable deployment. The bottleneck lies not in hardware, but in a conceptual failure-tres treating intelligence as a brute-force commodity rather than scarce economic resource. When cognitive uncertainty, the primary cost driver, is handled without
2

---

Published as a conference paper at ICLR 2026
economic discipline, redundant computation ensues, making decisions prohibitively expensive. This section formalizes the problem and traces it to the heuristic-driven coordination paradigms dissected in Subsec-
tion 2.2
2.1 THE ECONOMIC OBJECTIVE OF MULTII-GRNN COORDINATION
To ground our analysis, we first establish the ideal of economic rationality that any advanced MAS should pursue. Setup. We consider system with set of N heterogeneous VLM agents, A =
Comparison of Uncartainty Distributions
Agore Pourt 0.15 <X50 V 0.20:
10
Agorn (Sarn)
aN). Each agent a; is defined by a unit processing cost > 0 and an expertise vector & [Sr,eree Ei.sem- Si.inr],, iT, where fi,k E [o, 1] quantifies its effi- ciency uncertainty type k. For any task t drawn from distribution T, the system confronts an initial epistemic uncer- tainty vector utt) = [Mpere: Uyem: UMar] ) For centralized ta- ble and detailed definitions of all core variables and functions, please refer to [ppendix[[ Objective. The system's 's goal is to learn an allocation policy ग that performs principled economic optimization. This policy must route uncertainty components to the most suitable agents to minimize total expected opera- tional cost, while ensuring the final uncertainty is resolved to an acceptable level This is the core constrained optimization problem:
Final Eplstanic Uncertainty (onaayad
Figure 2: Final epistemic uncertainty of Agora (blue, 0.16) vs. KABB-VLM (or- ange, 0.21).
min E [C(z, u(t), c, ]
s.t.
|Manal| <
(1)
I
where C(-) is the total cost function, is the vector of agent costs, and 3 the matrix of agent expertise.
2.2 THE FAILURE OF HEURISTIC PROXIES FOR ECONOMIC RATIONALITY
Existing coordination paradigms fail to solve Eq. | because they do not perform true optimization, but in- stead rely on heuristic proxies fundamentally misaligned with the economic objective. We highlight two dominant paradigms that exemplify this failure. 1. SnggegattionBBased Heuristics (e.g,, MoA): These equate statistical consensus with epistemic truth. Models like Mixture-of-Agemt (MoA) assume that aggre- gating multiple agent outputs converges on the correct answer, which only holds if errors are independent and identically distributed (i.i.d... In MAS with shared architectural biases, this assumption breaks down, leading to systemic irrationality. Proposition I (Correlated Error Amplification). Let Sprop C AA be a set of agents with a common perceptual bias. For ambiguous inputs, they will likely produce correlated hallucina- tions. An aggregator seeking consensus will then amplify this shared error. 2. Routing- Based Heuristies (e-g., KABB): These rely on proxies for value, derived from historical performance and semantic similarity, guide agent selection. State-of-the-an routers maximize a scoring function, e.g.:
(2) where Prist is historical performance and Simgem is semantic similarity. This surrogate conflates past performance with future cost-effectiveeness remaining Cost-Agnostic (the cost vector с is absent) and (the vector u(t) is collapsed into scalar proxy). This structural igno- rance results in higher residual uncertainty, as empirically demonstrated in Figure 2 where our structure- aware Agora yields markedly lower final epistemic uncertainty than the heuristic baseline.
S= a - Pustt + 8 -Simsem
2.3 THE CORE CHALLENGE: A CALL FOR A NEW PARADIGM
The specific flaws in aggregation and routing are manifestations of a deeper, shared theoretical limitation, which we formalize agnostic coordination. Definition 1 (Agnostic Coordination). A coordination mecha-
3

---

Published as conference paper ICLR 2026
Selected Agent
Selected Agent
Value Model
Selected Agent
Query Images
Output Text
Is this place crowded?
Trading Agent Selection with MAB
Query Text
Policy Selection Reward
Uncertainty Evaluation Center
Perceptrua (kicertaiity
Agent
Selected Agent
Semento Uncertainty Use
Agent
Uncertain
Agent 2
fefennce dncuelainty Ue
MAB
Painwese Historical Trading Uncertartty (U'')
Agent
Trade History
Kany??
Agent
Historical Transaction Ledger
Encoolig
Trade records
Uncortainty-Awwa Trading Reward
Touc
Figure 3: In Agora, query uncertainty is split into perceptual (eper)). semantic (Usem), and inferential (Unat) A market-aware broker trades these among agents for efficient resolution.
nism M is defined as agnostic if its agent selection process is (i) Cost-Agnostic (invariant agent processing costs) and (ii) Sinerrainty-Strructii StructuureAgnostii (collapses the uncertainty vector into a scalar proxy). Both MoA and KABB archetypes of agnostic coordinators, consensus heuristic disregards the cost of polling agents and the specific structure of the uncertainty it aims resolve. KABB's routing heuristic, shown in Eq. [ explicitly demonstrates both agnostic properties. This shared, fundamental flaw leads to provably suboptimal performance. Theorem 1 (The Inefficiency Theorem for Agnostic Coordination). Any coordination mechanism M that is agnostic (per Definition 1) is not guaranteed to solve the objective in Eq. [ and is provably suboptimal for any task where the heuris- tically superior agent is not the most cost-effective resolver. This impasse shows that incremental fixes heuristic coordinators are inadequate. A paradigm shift is needed-from heuristic proxies to non-agnostic mechanisms capable genuine economic reasoning. This crystallizes our central question, addressed Section 3 how to design coordination mechanism that explicitly integrates cost and uncertainty to navi- gate the optimization landscape of Eq. [
conceptually illustrated in Figure D MoA's
3 METHODOLOGY: THE AGORA MARKET FRAMEWORK
To address the theoretical deficiencies of agnostic coordination identified in Section 2 we introduce Agora: a framework that recasts multi-agent coordination as a decentralized micro-economy. Our methodology provides constructive, non-agnostic solution to the optimization problem in Eq. b designing system that is inherently cost-aware and structure-aware. At its core, this is achieved by turning uncertainty into a quantifiable, tradable asset and defining protocols for its efficient reallocation. As established in analysis related work (Appendix [B), prior heuristic-driver paradigms are fundamentally agnostic to these factors. In response, Agora introduces the principled economic mechanism for coordination illustrated in Figure
4

---

Published as a conference paper ICLR 2026
3.1 ESTABLISHING THE MARKET: FROM UNCERTAINTY TO TRADABLE ASSETS
A market cannot exist without well-defined asset. To counter the SerrrtaittySSuuuttre-ggaee Theorem 1, our first step is to "mint the currency" by formalizing cognitive uncertainty as a structured, quantifiable portfolio. We decompose total uncertainty u into two classes: tradable component, Epistemic Uncertainty (Ulpiss, which represents the reducible information gap from our problem formulation; and a non-tradable component, Aleatoric Uncertainty (Uatea), which represents irreducible systemic risk. The tradable asset, Uepics is a vector in a paree-dimensiona, state space, Uepss = |uperee Usem Utar]" corresponding the fundamental cognitive domains of perception, semantics, and inference. This vectorization transforms a monolithic problem into a portfolio of distinct assets that can be independently priced and traded. Each agent a, maintains an uncertainty portfolio U(a t), which is the linear superposition of its self-generated uncertainty and the net uncertainty acquired through market trades:
from
Uaa t) = base (ai, t) + > jtt
Uransfer (a, - a, t)
The value of transferred uncertainty, U Uransfer: is aggregated from historical transaction ledger, weighting past trades based on relevance and cost-effectiveness.
3.2 THE CORE MECHANISM: PROFITABILITY- DRIVEN TRADING PROTOCOL
With structured asset in place, Agnosticism. This mechanism is a Profitability-Driven Trading Protocol that governs all transactions based pure economic rationality. A trade is initiated when an arbitrage opportunity-4 potential for system-wide cost reduction-is identified. To evaluate this, we calculate the change in total cost, or cost delta (AC), that would result from reallocating an uncertainty packet. The derivation for packet of magni- tude (t) being transferred from agent a, to aj is:
introduce the core mechanism designed to overcome Cost-
(t) = [C((U.(t) - (t) + (t) -1- (t)] - [Q.UU(t)+ +CU,(t
Cost After Trade
Cost Before Trade
(tz 1e, - Ej) cc]
(4)
This leads to a simple, powerful admissibility rule. A trade is executed if and only if is profitable (AC < 0) and feasible, meaning the receiving agent a, possesses the required cognitive capacity Ct(t):
Execute trade(i - j. (t)) = (AC(T,)(t) < 0) A (U) + (t,(t) S Cy(t)
(5)
This protocol, by its very construction, is both cost-aware and structure-aware. thus violating both conditions for suboptimality from Theorem 1. Each admissible trade represents greedy step that descends the cost landscape of the global objective function in Eq.
3.3 MARKET EXECUTION: THE BROKER AND THE AGORA ALGORITHM
The market is set in motion by an intelligent Broker, an extension of Thompson Sampling (TS) that finds cconomically sound starting point for the decentralized optimization. It selects initial agent by maxi- mizing a market-aware expected utility function, 0
(6) where the terms account for expected reward minus cost, adjusted for task distance, strategic utility, agent synergy, and temporal decay (see Appendix details). The entire process is operationalized by the Agora Algorithm, presented in Algorithm []
g EE ( (t)
Cost
exp(-A-
Dist(S,t)) Ustratgic (S) Synergy(S)) 7
The algorithm proceeds in two phases: (1) miximiiia
5

---

Published as conference paper ICLR 2026
Table 1: Comprehensive performance on visual benchmarks. Scores are percentages; best bold, second best underlined. Agora (Ours) routes tasks within the baseline pool, parentheses show gains over the top.
Model
MMMU(Val) MMBench. V11 V11_Test Test Math Vision InfoVQA(testt
CC-OCR
qwenn..5v-72b-iisstuuet
70.2%
88.4%
39.3%
87.3%
79.8%
gemini-2.0-flash
70.7%
83.0%
41.3%
83.2%
73.1%
(won2.5vl-76-instte
58.6%
82.6%
25.1%
82.6%
77.8%
gemma-3-276
64.9%
78.9%
27.5%
59.4%
72.6%
gpt-4o-mini
60.0%
76.3%
26.3%
68.7%
64.2%
gpt-40-2022-08-066
70.7%
74.3%
30.4%
68.7%
66.6%
gemini-2.5-pro--xp-02-222
81.7%
88.3%
63.5%
81.0%
73.0%
InterVLL3-8BB
72.2%
87.7%
43.1%
84.1%
80.3%
Agora(Ours)
79.2%)cs..9))
(9.5%))).199)
44.3%
88,9 (c..)))%
81.2 11.2%% (.1.44))
initialization by the Broker, followed by (2) an iterative market phase. In this phase, the system performs a deterministic greedy descent on the total cost function by repeatedly applying the trading protocol from Eq. 5 This continues until no further profitable trades are possible, at which point the market has converged to a locally optimal and cost-efficient equilibrium.
Algorithm 1 Agora: A Distributed Economic Optimization Algorithm
1: Input: Agent set A. costs c, expertise E. initial uncertainty Ujnitial Broker MAB. Output: Final allocation 11.
3:
b Phase 1: Maxlity-Maximizinn Initialization
4: Ahandler < Proker.select iinitalliiialllle.. Lagent(AA, ingttiia.aeen((44444 Uinitial
5: Initialize allocation II: Uhandler <- Dinirial' <- O for i + handler.
6: while true
> Phase Herattvv Greedy Cost Descent via Trading Protocol
7: best_trade +- FindMessProfiiableennded...
8: if best_trade # null then
9: Let (i,j. ks amt) be the components of best trade
10:
b. Execute trade based on the protocol from Eq.
11: ny [+]] u, [A]] + [M]; u +-
12: else
13: break
Market converged to locally optimal equilibrium
14:
end if
15:
end while
16:
return II
4 EXPERIMENTS
We conduct experiments to validate Agora, using NVIDIA A100 GPUs. The agent pool consists of five representative VLMs: qwen2. 5v1-722binsttuct gemi 1imini-22 2200-fassh qwen2.
In Agora, 'expert or 'agent' is an active configuration base model with specific prompt and role. The number concurrent ex- perts, denoted by N, varies by setup. We evaluate five aspects: (1) comprehensive visual understanding across benchmarks; (2) the role of our MAB strategy in uncertainty trading; (3) comparison with alternative routing and MAS strategies; (4) cost performeree trade-offs across N; (5) module/strategy ablations.
gemma-3-27b, and gpt-
6

---

Published as a conference paper at ICLR 2026
Table 2:
MAB strategy on MMMU (Val). Except "No Trading," all use multi-agent trading (COI > D).
Lower is better for U final episs COI; higher for Accuracy, UAPS. Best scores bold, second best underlined.
Method
MMMU Acc. (%)
Usinal epis + COI 4
UAPS (%) f
Agora (Ours)
79.0
0.15
1.2
70.5
Agora (No Trading)
75.5
0.22
1.0
65.0
KABB Selector + Trading
76.0
0.25
1.5
65.5
PPO Selector Trading
74.0
0.28
1.6
62.0
MCTS Selector + Trading
74.5
0.26
1.4
63.0
A2C Selector Trading
73.5
0.29
1.7
61.0
DQN Selector + Trading
73.0
0.30
1.7
60.0
Accunscy
sarity
0.05
59-
0.03-
AAXI
1/6
144-
0.25
12
Apw
Ap
Figure 4: Comparison with alternative routing and multi-agent strategies on MMBench. VI1_Test (N=6). Lower is better for Cost, Time, COI, and U/ inaalepisi higher is better for Accuracy.
Comprehensive Visual Understanding Performance Experiment Setup. Agora's 's performance is bench- marked against its constituent models (individually) and external SOTA VLMs, including gpt-4o-2024-08-00
(OpenAl 2024), cemini-2.5-pro-exp-00-----
(Google) (025), and Intern VLB3-78B (Chen et al.] 2024b). Eval-
uation covers diverse benchmarks: MMMU (Val) (Yue al. 2024). MMBench. VVI_ch..Vi.Testt (Liu et al. [023b),
Vision (Wang et al., 2024b)) Info (test) (Mathew et al. 2021), and CC-OCR (Yang et al. 2024). All models, including baselines and SOTA comparators, are accessed via the OpenRouter API. We use greedy decoding ('do_sample "toossmlcc==aase) for determinism and comparability. Additional details are in the ap- pendix. Experimental Results and Analyses. Table shows that Agora delivers robust performance across challenging benchmarks. It achieves new SOTA on MMBench V11 Test, InfoVQA, and CC-OCR, owing to its architecture that explicitly manages and trades uncertainties among heterogeneous agents. On reasoning- heavy tasks like MMMU and Vision, "thinking model," but Agora still secures second place. Overall, these results highlight Agora's 's effective col- laborative capability for complex dision-languae tasks. Agora achieves consistent gains (+1.1-8.55% across benchmarks), confirming effectiveness in collaborative ment.
--r------------0--222-- performs strongly as specialized
via dynamic uncertainty manage-
4.1 COMPARISON WITH ALTERNATIVE ROUTING AND MULTI-AGENT STRATEGIES
Experiment Setup. We benchmark Agora (Ours) against representative router models and multi- strategies, including FrugalGPT (Chen al.) 2024a), RouteLLM (Ong et al., 2024), EmbedLLMM Zhuang
et al.) 2025), HybridLLM (Ding et al., 2024). KABB (Zhang et al. 2025i), and MOA (Wang et al. 2024a). All methods are adapted to the VLM multi-agent setting and operate on the same agent pool (N=6). Eval- uation is performed on MMBench. lath_i11_Tell Test (Liu et al. 2023b). reporting Accuracy (%), Relative Cost (normalized to Agora=1.00), Average Inference Time (s), Collaboration Overhead Index (COI), and Final Epistemic Uncertainty (Uninalepis). Details of model adaptation and hyperparameters
in Appendix K Experimental Results and Analyses. As shown in Figure 4 Agora attains the highest accuracy (89.50%) while remaining cost-efficient. KABB-VLM and MOA achieve competitive accuracy (87.12%, 86.65%
7

---

Published as a conference paper ICLR 2026
but much higher cost (1.24x and 3.11x), COI (1.53, 1.82), and residual uncertainty (0.21, 0.25). In contrast, FrugaIGPT, RouteLLM, EmbedLLM, and HybridLLM reduce cost (0.73-0.91) but suffer notable accuracy drops (-8 to -9.6 points) and higher uncertainty (0.27-0.33). These results underline Agora's s superior eecuracy-effcienne trade-off. Experiment Setup. To evaluate economic efficiency, we analyze Agora's cost-performance on MMBench. TEST V11, varying agent pool size (N = 1-9) and comparing with baselines, external SOTA VLMs, and KABB-VLM. The Cost-Performance Ratio is defined as relative cost 1ept-4o-mini-1.00 over accuracy, using OpenRouter prices; lower is better.
4.2 THE ROLE OF THE MARKET-AWARE MAB STRATEGY
To validate the central role of our market-aware Multi-Armed Bandit (MAB) broker, we conducted com- parative experiment on the MMMU (Val) benchmark. In this controlled setup, we replaced our selector with prominent heuristic (KABB) and reinforcement learning (PPO, MCTS, A2C, DQN) alternativss, while the underlying uncertainty trading protocol remained constant. The results, presented in Table 2 are de- cisive. Our MAB-based approach outperforms all baselines, achieving the highest accuracy (79.0%) and Uncertainty-Awan Performance Score (UAPS) of 70.5%. Notably, it surpasses the next-best heuristic se- lector (KABB) by a margin of 3.0% in accuracy and 5.0 UAPS points. While the RL agents demonstrate learning capabílities, they struggle match the efficiency of our method within this economic coordination task, consistently yielding lower scores and higher final epistemic uncertainty (Uinal epis). These findings underscore that the specific design of our market-aware Broker, which leverages an economically-inforee utility function, is a critical contributor to the Agora framework's 's superior performance and efficiency.
4.3 COST AND PERFORMANCE BALANCE ANALYSIS
Experimental Results and Analyses. Figure 5 plots the accuracy-cost trade-off, where the Cost- Performance Ratio (lower is better) reveals the eco- nomic efficiency of different strategies. The results demonstrate Agora's 's ability to establish a superior Pareto frontier.
90
88
Qwwn2.5vk- Soent..w--D--mtraad
gemink-2.5p
IntemVL3-780
MMBench Accuracy (56
86
VLM (VLM) varsion)
84
Even with a single agent type (N = 1), Agora achieves competitive 87.5% accuracy at ex- ceptionally low cost ratio (0.02057), outperforming even costly SOTA models like 'gemini-2.5-pre and 'Intern VL3-78B'. As the agent pool diversifies, ac- curacy steadily climbs a peak of 89.6% at N 8,
82
80
78
76-
Spl-Tomiie
with only marginal changes at N = 9. Crucially, every Agora configuration (N 2 1) maintains a sig- nificantly better cost ratio than strong baselines like 'qwen-72b' (0.05656) and alternative multi-agent systems like KABB- VLM (0.05191).
74
got-4o 20-2022--8-00
0.02
d:04
0.06
0.08
0.10
Cost-Perf. Ratio (Lower Better)
Figure 5: Cost Perffrerffrm vs. Accuracy on MM- Bench TEST V11.The curve illustrates Agora's 's abil- ity to achieve superior Pareto frontier. As the agent pool grows, the system improves accuracy at a sub- linear cost increase, highlighting the efficiency of its market-aware MAB.
These trends reveal key insights into Agora's 's eco- nomic design. First, the remarkable efficiency of the N' case is not merely about using a cheap model; it highlights the intelligence of our market- aware broker, which selects the most suitable con- figuration for a given task, avoiding unnecessary costs. Second, the graceful scaling from N = 2 to N = validates our core thesis: as the market gains access to more specialized agents (a more heterogeneous pool), the uncertainty-tratee mechanism more
8

---

Published as a conference paper ICLR 2026
effectively allocates cognitive labor the cheapest specialist. This allows the system to push the accuracy boundary without a proportional surge in cost. Finally, the performance plateau around N 8 indicates a point of diminishing returns, a classic economic principle. It suggests that Agora does not require an ever- expanding, costly pool of agents to maintain its edge. Instead, it efficiently leverages a finite set of resources to approach an optimal performance ceiling. This economically rational behavior stands in stark contrast to the brute-force strategy of monolithically applying single, expensive SOTA model to all problems, a tactic that Figure 3 shows is fundamentally inefficient.
4.4 MODULE AND STRATEGY ABLATION STUDIES
Experimental Setup. To dissect the contribution of each component in Agora, we conduct an ablation study on the key multiplicative factors within our market-aware Thompson Sampling utility function: 0
[E]Reward ( exp(-À- Dist(S, t)) - Synergy(S)" Usuaaeii (5)". Specifically, we create variants by individually ablating the Strategic Uncertainty (Ustratgic), Synergy, Task Match (Dist), and Time Decay (At) factors. These variants are benchmarked against a baseline, "Only Net Return," which relies solely on the expected net return for agent selection. All experiments in this study are performed on the MMBench. Baachh.Vi___ee V11_Test dataset, utilizing a consistent agent pool of N = 6 experts. For each ablated variant, its corresponding term in the utility function neutralized by setting Experimental Results and Analyses.
one. Further ablation on key
hyperparameters is detailed in Appendix G
As presented in Table 3 the results validate the ef- ficacy of the complete Agora strategy and quantify the contribution of each component. The full model outperforms all variants, achieving the highest ac- curacy (89.50%), lowest final epistemic uncertainty (0.16), lowest Collaboration Overhead Index (COI, 1.25), and highest Encertainty-Awar Performance Score (UAPS, 78.33%). Ablating any strategic fac- tor impairs performance. Critically, removing the novel Strategic Uncertainty (Ustmeeiic factor causes the most substantial performance degradation (Accu- racy -3.08% Unialepis +0.07, UAPS -6.75 points, Cost
Table 3: Ablation of the Agora strategy on MM- Bench. Bench_Vi1.1 West.... Uncertainty trading is enabled for all variants. Rel. Cost is normalized to the full model. Best results are in bold.
Variant
Ace. (%)
Unat
COI,
UAPS (%)
Rel. Cost
Agora (Full)
w/o Unazee
$6.42
0.23
1.45
71.58
1.06
wlo Synergy
87.91
who Dist
88.53
0,18
1.27
76.21
1.01
who
89.05
0.17
1.26
77.14
1.00
Only Net Return
82.15
0.31
1.08
60.72
0.92
+6%), underscoring its pivotal role in guiding agent selection toward profitable uncertainty trades. The re- moval of the Synergy, Task Match, and Time Decay factors also leads to measurable performance drops, confirming their positive contributions. In stark contrast, the "Only Net Return" baseline, which ignores all strategic heuristics, performs substantially worse than any other variant (e.g.. -7.35% accuracy and -17.61 UAPS points vs. the full model). This confirms that all strategic components are integral to achieving the high-accuracy, cost-efficient coordination that defines the Agora framework.
5 CONCLUSION
We propose Agora, a market-based framework for coordinating multi-agent Vision-Language Models (VLMs). Unlike heuristic methods such Mixture-of-Agent, or knowledge-basse routers, which collapse uncertainty and ignore costs, Agora casts epistemic uncertainty as a structured, tradable asset across per- ceptual, semantic, and inferential dimensions. Guided by market-aware Thompson Sampling broker, proftability-drivee protocol enables agents to trade uncertainty rationally and reach cost-efficient equilibria. Experiments on five benchmarks show consistent gains- up +8.5 % accuracy at over 3x lower cost.
9

---

Published as conference paper ICLR 2026
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. Our study does not involve human-subjects research, the collection of personally identifiable information, or the annotation of sensitive attributes. We do not create distribute any human data. All experiments language benchmarks strictly under their respective licenses and terms of use. The methods developed in this paper are designed for advancing academic understanding of multi-agent coordination and not intended for deployment in sensitive or high-stakes applications without additional safeguards.
conducted on publicly available, widely used vision-
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we provide detailed descriptions of our experimental settings, including datasets, baselines, and evaluation metries, in the main text and appendix. Hyperparameters, training protocols, and ablation results are comprehensively documented to allow replication of results. All models are implemented using standard open-source frameworks, and benchmark datasets are accessed through their official releases. Pseudocode, algorithmic steps, and theoretical proofs are included to enable faithful reproduction of both the methodology and results. Additional runtime and configuration details are provided the supplementary material for transparency.
REFERENCES
George A. Akerlof. The market for "lemons": Quality uncertainty and the market mechanism The Quar- terly Jourmal of Economics, 8483),488-500, 08 1970. ISSN 0033-5533. doi: 10.2307/1879431. URL
https://dois http://..oorg/....222.2 2ttp://1..207787792555
Farzad Azizi, Benyamin Teymuri, Rojin Aslani, Mehdi Rasti, Jesse Tolvanen, and Pedro H. J. Nardelli. Mix-mab: Reinforcement learning- baase based resource allocation algorithm for lorawan, 2022. URL https:
// //arxiv.orq/////oa..a......o org/bbs////////// harrg//bb/220055 2206. 03401
Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023. URL httpt::tt////aat www.vorrgaaaaeeee abs / / 2308 12966
Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. Transactions on Machine Learning Research, 2024a.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.052771 2024b.
Yan Dai, Ruosong Wang, and Simon S. Du. Variance-aware sparse linear bandits, 2023. URL https:
//arxiv. htaxxxv.rrg/abs/2055111 13450
david cortes. Github david-contes/cnneetuullaaaa Python implementations of contextual bandits algo- 2025.
rithms -
[Accessed 16-05-2025].
J. de Curtò, de Zarzà, Gemma Roig, Juan Carlos Cano, Pictro Manzoni, and Carlos T. Calafate. Llm- informed multi-armed bandit strategies for non-stationary environments. Electronics, 12(13), 2023. ISSN 2079-9292, doi: 10.3390/electroniic/ 12 11328144 URL https: / //www mdpi. 13/2814
10

---

Published as a conference paper ICLR 2026
Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks V. S. Lakshmanan, and Ahmed Hassan Awadallah. Hybrid Ilm: Cost-efficient and quality-aware query routing, 2024. URL /tttt::://axxxioorr//a/bto
a0tp//4...22222..../ 14618
D. Gale and L. S. Shapley and. College admissions and the stability of marriage. The American Mathemat- ical Monthly, 69(1):9-15, 1962. doi: 10. 10.1080000028900.11 190.0022999..99..111
URL/ttps:: httph::/tcoo. http//ooooooooooo/ooooooo.ooooo.
10.
196.. 11 989827
Shubham Gandhi, Manasi Patwardhan, Lovekesh Vig, and Gautam Shroff. Budgetmlagent: A cost- effective IIm multi-agent system for automating machine learning tasks. In Proceedings of the 4th International Conference AI-ML Systems, AlMLSystems '24, New York. NY. USA, 2025. Asso- ciation for Computing Machinery. ISBN 9798400711619. doi: 10.144577034122.7034116 URL 3703416
https: / doii wwo.oooggic....
Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxi- ang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, Muhammad Shahzad, Wen Yang, Richard Bamler, and Xiao Xiang Zhu. survey of uncertainty in deep neural networks, 2022. URL
https://arxiss org/ ats//bbs/2211 2107. 200./21100333222
Google. Gemini 2.5 pro Scepmind.goope gemini/pro/ 2025. [Accessed 16-05-2025).
Jason M. Gregory, Sarah Al-Hussaini, and Satyandra K. Gupta. Heuristics-based multi-agent task allocation for resilient operations. In 20/9 JEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), pp. 1-8. IEEE Press, 2019. doi: 10. 10.1109/SSRR.20198888888 28898999,,,,, URL https: pettps://do:: http://oo.ooo/ org/
10. 15.1109//5RR..O 2019 8848939
Taicheng Guo. Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. In Kate Larson (ed.), Proceedings of the Thirty-Thrrd International Joint Conference Artificial Intelli- gence, IJCAI-24, pp. 8048 8057. International Joint Conferences on Artificial Intelligence Organization, 8 2024. doi: 00.24963/jccai.22448333 Survey Track.
Nikita Gupta, Shikhar Anand, Tanuja Joshi, Deepak Kumar, Manojkumar Ramteke, and Hariprasad Kodamana. Process control of mab production using multi-actor proximal policy optimization. Digital Chemical Engineering. 8:100108, 2023. ISSN 2772-5081. doi: https://bbi.org/10.00666 www.
j.dche 2.dche.2023.1m 100108. URL ottpss//ww...iiieeeeeeee....oo htt.p//eee..coeeccceeee///ttttat
0217250812300022
Jiuzhou Han, Wray Buntine, and Ehsan Shareghi. Towards umcertainty-awart language agent. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Lin- guistics: ACI 2024, pp. 6662-6685, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.1188561110000 1.05 findings--acl.398, Findings ac1.398/
Chen Hao, Guangkai Yang, Junge Zhang, Qiyue Yin, and Kaiqi Huang. Multi-agent uncertainty sharing for cooperative multi-agent reinforcement learning, pp. 1-8, 07 2022. doi: 10.1 N0.1109//CNN55544.0022... 9891948.
Jeonghwan Kim and Heng Ji. Finer: Investigating and enhancing fine-grained visual concept recognition in large vision language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Pro- ceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 6187-6207
11

---

Published as a conference paper ICLR 2026
Miami, Florida, USA, November 2024, Association for Computational Linguistics. doi: 18.13653/v1/ 1/ m024..3.lp-main3356. URL httpp::/aae..aataoooooooo/////// e www.1pema......mmm..mmmmmmmmmmm.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436644, 2015.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In /CMI,, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Confer- ence on Machine Learning, ICML'2 A.M.' JMLR.org. 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. 2023a.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06288 2023b.
Zeqian Long. Mingzhe Zheng, Kunyu Feng, Xinhua Zhang, Hongyu Liu, Harry Yang. Linfeng Zhang, Qifeng Chen, and Yue Ma. Follow-your sour-pppp- Shape-aware image editing via sereector--guidee region control. arXiv preprint arXiv:2508.088 08150000081444 34, 2025.
Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Siran Chen, Xiu Li, and Qifeng Chen. Follow your pose: Pose-guided text-to-video generation using pose-free videos. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 38, pp. 4117-4125, 2024a.
Yue Ma, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung- Yeung Shum, Wei Liu, et al. Follow-your-emopy Fine-controllable and expressive freestyle por- trait animation. In SIGGRAPH Asia 2024 Conference Pupers, pp. 1-12, 2024b.
Yue Ma, Yingqing He, Hongfa Wang, Andong Wang. Leqi Shen, Chenyang Qi, Jixuan Ying, Chengfei Cai, Zhifeng Li, Heung- Yeung Shum, et al. Follow-your-click: Open-domain regional image animation via motion prompts. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 6018-6026, 2025a.
Yue Ma. Yulong Liu, Qiyuan Zhu, Ayden Yang. Kunyu Feng, Xinhua Zhang, Zhifeng Li, Sirui Han, Chenyang Qi, and Qifeng Chen. decoupled finetuning. arXiv preprint arXiv:2506.00507, 2025b.
Video motion transfer via efficient spatial-temporal
Yue Ma, Zexuan Yan, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng. Chengfei Cai, Heung- Yeung Shum, al. gollo-yoou-emoji-eee faster: Towards efficient, fine-controllable, and expressive freestyle portrait animation. arXiv preprint arXiv:2509..66 16630, 2025c.
Minesh Mathew, Viraj Bagal, Rubèn Pérez Tito, Dimosthenis Karatzas, Emnsst Valveny, and C. V Jawahar. Infographicvga, 2021. //t//t/t////////.oorrrgg httpgg/ab/22100...
John F. Nash. Equilibrium points in n-person games, 1950. URL 1073/pnas. 36. 48
Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. Routellm: Learning to route 1lms with preference data, 2024. URL https:
//arxiv. wwwxivvoogg/bba wttp/aa/////oooooooe//../oeaaaer aaaaabbss...// 2406. 18665
OpenAl. Gpt-4o system card, 2024. URL http:p/:/aar....oo////////1..110
12

---

Published as a conference paper ICLR 2026
Fiorenzo Parascandolo, Nicholas Moratelli, Enver Sangineto. Lorenzo Baraldi, and Rita Cucchiara. Causal graphical models for vision-language compositional understanding, 2025. URL https://arxiv:
org/abs/2412..933
Wujian Peng. Sicheng Xie, Zuyao You, Shiyi Lan, and Zuxuan Wu. Synthesize, diagnose, and optimize: Towards fine-grained languag understanding. In 2024 IEEE/CVF Conference on Computer Vision doi: 10. 10.109CCYPR527332224.000000
and Pattern Recognition (CVPR), pp. 13279-13288 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https://arxiv.org
abs/2103.0C0B
Yutao Shen, Junkun Yuan, Toru Aonishi, Hideki Nakayama, and Yue Ma. Follow-your-preefrenne: Towards preference-aligned image inpainting. arXiv preprint arXiv:2509.23088) 2025.
Jiahang Sun, Zhiyong Wang, Runhan Yang, Chenjun Xiao, John C. S. Lui, and Zhongxiang Dai. Large lan- guage model-enhanced multi-armed bandits, 2025.
01118
John Neumann and Oskar Morgenstern. Theory of Games and Economic Behavior. Princeton University URL https://
Press, Princeton, 2004. ISBN 978 400829466 doi:
doi doi.org/10 1515/ 978177881447 0081978811889986
Junlin Wang. Jue Wang. Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabilities, 2024a. URL pttttps./aaxii..or//aaa..
04692
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. mathvision. In The Thirty-eight Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2024b.
Qineng Wang, Zihao Wang, Ying Su, Hanghang Tong, and Yangqiu Song, Rethinking the bounds of LLM reasoning: Are multi-agent discussions the key? In Lun-Wei Ku, Andre Martins, and Vivek Sriku- mar (eds.), Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 6106-6131, Bangkok, Thailand, August 2024c. Association for Compu- tational Linguistics. doi: 10. 118653/1122024.allloogg 2024. acl long.oo..3.
/org/ URL htttp:://aclatthhtt
Weihan Wang. Zhen Yang, Bin Xu, Juanzi Li, and Yankui Sun. Vilta: Enhancing vision-language pre- training through textual augmentation. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3135-3146, 2023. doi: D0.110///CV11000000230..0
Fanzeng Xia, Hao Liu, Yisong Yue, and Tongxin Li. Beyond numeric awards: In-context ducling bandits with Ilm agents, 2025. URL /ttttss/sttttiiteeeeeeeeee
/tt.b////244.. 008877
Zhenlin Xu, Yi Zhu, Siqi Deng, Abhay Mittal, Yanbei Chen, Manchen Wang, Paolo Favaro. Joseph Tighe, and Davide Modolo. Benchmarking zero-shot recognition with vision- models: Challenges on granularity and specificity. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1827-1836, 2024. doi: 10.1 N01199CCCPPNW3332222000001
Zhibo Yang, Jun Tang. Zhaohai Li. Pengfei Wang, Jianqiang Wan, Humen Zhong, Xuejing Liu, Mingkun Yang, Peng Wang. Shuai Bai, Lian Wen Jin, and Junyang Lin. Co-oer: A comprehensive and challenging ocr benchmark for evaluating large multimodal models in literacy, 2024. URL htR.stts:: http:::://axxx..oroo
abs/ / 2412.4222220 02210
13

---

Published as a conference paper at ICLR 2026
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang. Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi- discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024.
Jusheng Zhang, Kaitong Cai, Yijia Fan, Ningyuan Liu, and Keze Wang. MAT-agent: Adaptive multi-agent training optimization. In The Thirty-ninth Annual Conference Neural Information Processing Systems, 2025a. httthttp:///// http::///pppeeeeieeee htwp://wwwwwttttoooooooeo/erE/r.
Jusheng Zhang, Kaitong Cai, Yijia Fan, Jian Wang, and Keze Wang. CF-VLM: Counterfactual vision-
language fine-tuning, In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025b. httptsss: httpttts/s/pppeeeeiieeeeee//e http///pppeeeee.ccceee/ee net htt///orrmm/o/// wiw.mmiidi-oee 0gt Co
Jusheng Zhang, Kaitong Cai, Xiaoyang Guo, Sidi Liu, Qinhan Lv. Ruiqi Chen, Jing Yang, Yijia Fan, Xiaofei Sun, Jian Wang. Ziliang Chen, Liang Lin, and Keze Wang. Mm-cotaa benchmark for probing visual chain-of-thought reasoning in multimodal models, 2025c. URL.htttss//arxiiv org/ abs/ 2512. 08228
Jusheng Zhang, Kaitong Cai, Jing Yang, and Keze Wang. Learning dynamics of vlm finetuning, 2025d.
URL https: // wwwaaaavvvvooo org/ abs www.orgg//////oo//o//.../...eeee
Jusheng Zhang, Yijia Fan, Kaitong Cai, Xiaofei Sun, and Keze Wang. Ose: Cognitive orchestration through dynamic knowledge alignment in multi-agent IIm collaboration, 2025e. htRp/tttss://rxiv.oom
abs/2509 04876
Jusheng Zhang, Yijia Fan, Wenjun Lin, Ruiqi Chen, Haoyi Jiang, Wenhao Chai, Jian Wang, and Keze Wang. GAM-agent: Game-theoretic and uncertainty-aware collaboration for complex visual reasoning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. 2025F. httpst:::
//openreview. net. feee/fforrt for d=EXJhU5 1 o5o
Jusheng Zhang, Yijia Fan, Zimo Wen, Jian Wang, and Keze Wang. Tri-MARF: A tri-modal multi-agent responsive framework for comprehensive 3d object annotation. In The Thirty-nimth Annual Conference on Neural Information Processing Systems, 2025g. URL YGIbwfNNot
net wtt.:/wwwww.eeeettnm....... Forum?id-
Jusheng Zhang, Xiaoyang Guo, Kaitong Cai, Qinhan Lv, Yijia Fan, Wenhao Chai, Jian Wang, and Keze
Wang. Hybridtoken-vim: Hybrid token compression for vision-language models, 2025h. URL https:
//arxiv.org/abstt /btp///aa/22/2.... 08240
Jusheng Zhang, Zimeng Huang, Yijia Fan, Ningyuan Liu, Mingyan Li, Zhuojie Yang, Jiawei Yao, Jian Wang, and Keze Wang. Kabb: Knowledge-aware bayesian bandits for dynamic expert coordination in multi-agent systems, 2025i. URL/tttps://axxiv org/ abs 2502 07350
Jusheng Zhang, Yijia Fan, Kaitong Cai, Jing Yang, Yongsen Zheng, Kwok- Yan Lam, Liang Lin, and Keze Wang. Spectral gating networks, 2026.
07679.
Yinglun Zhu, Dylan J. Foster, John Langford, and Paul Minciro. Contextual bandits with large action spaces: Made practical, 2022. URL httpt::/aaaxx...oom http://orgg/ab/a//aaeaae abs/2207 05836
Richard Zhuang, Tianhao Wu, Zhaojin Wen, Andrew Li, Jiantao Jiao, and Kannan Ramchandran. Em bedILLM: Learning compact representations of large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL. https:://ppentt ew. net./ forum?id= F
14

---

Published as conference paper ICLR 2026
A
APPENDIX
SUPPLEMENTARY MATERIAL OVERVIEW
This Supplementary Material provides a detailed expansion of the Agora framework presented in the main paper. begins with foundational Preliminary concepts in Appendix Aand review of Related Work in C offers in-depth Theoretical Proofs and Supplements for the core mechanisms discussed. Further empirical validation is provided through an analysis the Impact Agent Pool Con- figuration in Appendix D a FLOPs Comparison and Computational Efficiency study in Appendix B and an extensive Supplementary Core Component Ablation Discussion in Appendix [F] To ensure repro- ducibility and transparency, detail the Hyperparameter A Allatoon Experiments in Appendix G list all
Appendix [ Appendix
Hyperparameters Used the Experiments in Appendix K| present a Runtime Analysis in Appendix [I] and include the Prompt Setting Statement for our VLM agents in Appendix ① Finally, Appendix M offers a qualitative Case Analysis with examples of successful and unsuccessful expert collaborations.
A PRELIMINARY
Vision-Languaal Models (VLMs) and Decision Uncertainty: VLMs systems that process multimodal inputs, such as visual data I, and generate textual responses R based on task descriptions T. At their core, they rely Large Vision-Language Model. Formally, a VLM agent a acts function fa: R fa T), where / E is the visual input space, T E T is the task description space, and R E R. is the response space. Fach agent incurs processing cost Cu. reflecting computational resource usage. In heterogeneous multi- agent setups, the agent set A = {a1, ..,..., .... an varies significantly in capabilities and costs.
Uncertainty plays crucial role in agent decision-making. For an agent a, input 1, and task T,, uncertainty U measures the dispersion in the response probability distribution Pa (R) T), often quantified via Shannon entropy H:
(a, (((,((II,T) = Σ Palr)4,T))og Pa TR
Here, PI denotes the distribution over possible responses E R. Higher U indicates lower confi- dence, potentially increasing computational costs. In multi-agent systems, uncertainty can be decomposed into epistemic (reducible through collaboration) and aleatoric (irreducible) components, allowing for tar- geted trading to optimize resource allocation.
Multi-Armed Bandit Problem (MAB): MAB involves sequential decisions where a learner selects from actions (arms) to maximize cumulative rewards. Selecting arm a at time yields random reward (t) from an unknown distribution.
Thompson Sampling (TS) addresses MAB by balancing exploration and exploitation via Bayesian methods. For each arm a, it maintains posterior on reward probability Oa, often a Beta distribution Beta(o.,,8 3a). At each step, sample 0g ~ Beta(Oa, (a) and choose:
a" arg max 0g.
Update after reward r:
(a. + 1, (a Ba 1)
(aas Ba)
r
This approach is particularly useful agent selection, as it adapts to performance over time, reducing regret in uncertain environments.
15

---

Published as a conference paper ICLR 2026
Cost-Benefit Modeling and Comparative Advantage: In multi-agent systems, agent a'ss efficiency in handling uncertainty U is modeled by cost function (,( a U Bi where a; is the marginal cost per unit uncertainty and 3, the fixed cost. Total system cost Cuotal ΣE= Ci(U,), with U, assigned to a,.
Drawing from comparative advantage theory, our framework reallocates uncertainty based on relative effi- ciencies: agents with absolute disadvantages can still improve system efficiency if strengths differ across dimensions. For agents a, dj and dimensions d1, d2, the comparative advantage index is:
Ci(d))/C((d)) <1,
CAI(a aj.d1, d2) = Cy(d1)/Cy(d2)
implying o, advantages in d1 relative to d2, and vice versa. Trading then reduces total cost:
ACioaal [C; AUA,) + C2(U, + AUa,)) (C,(U.) + C2(U)) 0.
Analysis and Improvements in Agora: Traditional multi-agent coordination often relies on heuristics like consensus in Mixture-of-Agents (MoA) or semantic routing in KABB, which cost-agnostic and col- lapse uncertainty into scalars, leading to suboptimal performance as proven by the Inefficiency Theorem. These approaches fail to address information asymmetry and bounded rationality, resulting high costs and inefficiencies.
Agora improves upon this by framing coordination a decentralized market for uncertainty, minting into tradable assets (perceptual, semantic, inferential). This structure-awareenss enables profitability-drivee trades, ensuring cost-aware optimization. The market-aware Broker, extending TS, initializes collabora- tions efficiently, while the trading protocol greedily descends the cost landscape. Experiments show Agora achieves up to +8.5% accuracy on MMMU with 3x cost reduction, demonstrating scalable, economically viable intelligence.
B RELATED WORK
B.1 VISION-LANGUAGE MODELS IN MULTI-AGENT SYSTEMS
The integration of Vision-Languaae Models (VLMs) Parascandolo al. 2025 Peng et al. 2024 Kim Ji. 2024 Xu et al. 2024 Wang et al. 2023 Radford et al. 2021 into multi-agen systems (MAS) has unlocked new capabilities for collaborative multimodal tasks (Nash] 1950 Ma et al. 2024a b 2025c b). However, prevailing coordination paradigms, such centralized controllers or heuristic- based task allocators (Gregory et al. 2019 Han et al. 2024), often struggle with the economic realities scaling these systems. They tend overlook the steep computational costs inherent in large VLMs (Wang al. (002a) and rely on static uncertainty-handling mechanisms, which fundamentally limits their efficiency and scalability. In contrast, Agora introduces market-driven framework that directly addresses these shortcomings. It enables agents to dynamically trade uncertainty as a resource, optimizing for both performance and cost by leveraging decentralized economic principles to resolve information asymmetry- key limitation of prior heuristic- based approaches.
B.2 UNCERTAINTY QUANTIFICATION AND MANAGEMENT
While uncertainty quantification a recognized field in deep learning, particularly within Bayesian meth- ods (LeCun al. 2015) and active learning, its application in multi-agent VLM systems remains underde- veloped. Existing research is often limited in scope: many methods decompose uncertainty into epistemic and aleatoric types but focus primarily
single-agent settings (Gawlikowski et al. 2022 Zhang al. 2025e]d] Ma et al., 2025a] Long et al. 2025) [Shen et al. (2023). Other studies investigate uncertainty sharing for perceptual tasks (Hao al. 2022) but lack formal economic model for efficient resource allocation.
16

---

Published as conference paper ICLR 2026
Agora uniquely bridges this gap. formalizes multi-dimensional uncertainty (perceptual, semantic, and inferential) as a structured, tradable asset. This enables novel, profitability-rrive trading protocol that reduces system-wide costs and enhances collaborative efficiency, moving beyond the static and heuristic methods found in existing literature.
B.3 MULTI-ARMED BANDITS AND DECISION-MAKING
Multi-armed bandit (MAB) frameworks (Dai et al.
2023 Sun et al. 2025
Zhang et al. 2026 2025a are
cornerstone of sequential makknig in MAS (de Curtó al. 2023 Xia et al. 2025). Advanced
methods like contextual bandits (Zhu et al. 2022 david cortes 2025) and reinforcement learning-
MABs (Azizi al. 2022 Gupta al. 2023) incorporate state information to refine action selection. How- ever, their direct application often falls short in the complex economic landscape of large-scale MAS, as traditional MABs are typically engineered to maximize abstract reward signal. They rarely situate the decision- decision-making process within formal economic framework that explicitly models the trade-offs between performance, computational cost, and the fine-grained structure of uncertainty.
In contrast, the Agora framework makes several novel contributions that extend the MAB paradigm from a simple decision decision-making tool market-awar aware economic broker: Uncertainty as Tradable Asset: We are the first to formalize multi-diment dimensional cognitive uncertainty (perceptual, semantic, and inferential) as a quantifiable and tradable economic asset. This moves beyond merely using uncertainty as a feature for exploration. Trofitability-Drivvee Driven Coordination: We introduce trading protocol governed by economic rationality, where agent collaboration based on explicit cost-benefit analysis (AC) rather than heuristic rules. Market-Aware Utility Function: The Broker in Agora utilizes a novel, market- utility func- tion (Eq. 6) that integrates not only expected reward but also explicit costs, task similarity, team synergy, and unique Strategic Uncertainty Index. This design aligns the MAB's selection policy directly with the economic efficiency of the entire multi-agent system. This economically grounded approach yields superior cost-performance trade-off's compared to traditional MAB applications, marking significant advancement in building truly viable decision-making frameworks for MAS.
C THEORETICAL PROOFS AND SUPPLEMENTS IN THE MAIN TEXT
C.I MULTI-DIMENSIONAL VISUAL UNCERTAINTY QUANTIFICATION MODEL (3.1)
To achieve fine-grained management and efficient trading of visual uncertainty, Agora proposes multi- dimensional uncertainty quantification model. This model decomposes the overall uncertainty faced by an agent into three fundamental dimensions: perceptual uncertainty (Uperc), semantic uncertainty (Usem), and inferential uncertainty (Uint), as formalized in Section
C.I.1 FORMAL DEFINITION AND EXPANSION OF CORE UNCERTAINTY DIMENSIONS
a. Perceptual Uncertainty (Upere) Perceptual uncertainty (Upere) quantifies the lack of confidence in iden- tifying raw visual signals (e.g., object categories, basic features) due to factors such as image quality and visual ambiguity. is defined as follows:
Upene (I, R)
fpese Percepiual Uncertainty Aggregation Function
Stat Statistical Amalysis Function
fvisual (T)
((R)
Visual Festurrs Extracted From Image
ContectullMohiaaiigg Factors Relited to Response
Detailed Expansion and Explanation: Input Visual Signal. fvisua (T): Visual feature extraction module. For example, fvisual (7) V, where V is set of feature vectors extracted by Convolutional Neural
17

---

Published as a conference paper at ICLR 2026
Network (CNN) or Vision Transformer (ViT), (taa((visua()); Statistical evaluation of the extracted vi- sual features to quantify their clarity, consistency, or the model's raw confidence in these features. For example: If (T) yields a probability distribution P(O]T) = {p(orlT)..... p(ox (T)} over K pos- sible visual categories OR, then Stat(fvisml (T) can be: The entropy of this distribution: H(P(O[T)) = _kzz ppox//)))))))))))))( [) log p(ox\T), Or the complement of the highest probability: 1 max; p(ox|T). (R):: A function that adjusts or focuses the assessment of perceptual uncertainty based on the current agent's response R (or task context). For example: (it) might selectively weight certain types of perceptual un- certainty based on the content of R, or adjust the overall scale of uncertainty according to task importance. fpere( (:-)): The final aggregation function that combines the quantified visual feature information from Stat()) :) and contextual
b. Semantic Uncertainty (Vsem) Semantic uncertainty (Msem) reflects the ambiguity or multiple possi- bilities in understanding the meaning of a scene, interactions between objects, or symbolic interpretations, assuming the visual signals have been perceived. It is defined as follows:
Weight of Semunic Type
Ambiguity Reqonse R regarding Semantic Type
SSenTypes
Wi
(
Usem (R)
(8)
N
Normolization Complexity
Smooothing
Number Semantic Elanems
Constant
The response generated by the agent or its internal semantic rep- resentation. SemTypes: A predefined set of semantic types, e.g.. fobject attributes, spatial relationships, be- havioral intentions, ...}. wy: Importance weight assigned to semantic type i, typically w; 2 0 and Σ Wl = 1 (or other normalization methods). (R): A function that quantifies the ambiguity or complexity related to semantic type in response R. For example: If semantic type focuses on "inter-object relationships," and there are M AB possible valid relationships between object A and object B mentioned in response R. then CR) could be a function of MAR (e.g., log MA), or the entropy of the probability distribution of these relationships. (R):: A measure of the overall complexity of response R (e.g., number of entities, propo- sitions, or words contained), used as a normalization term in the denominator to obtain an average per-unit semantic ambiguity. A small positive constant (A 0) to prevent division by zero
Detailed Expansion and Explanation: R:
c. Inferential Uncertainty (uint) Inferential uncertainty (Wiet) measures the agent's confidence in its pre- dictions about future events, unknown states, or decision outcomes based on current information. It is defined as follows: (mr(R, S) 7- (1 P(S)) + (1 H(S) Detailed Expansion and Explana- tion: R: The current agent response or contextual information extracted from it. S = {s) $2,, S SM }: set of M mutually exclusive potential outcomes of future events, states, or decisions to be predicted. P(S) R, T): The predicted probability distribution over the outcomes in S, given current information. P(S) = ma P(s,1 R. I): The probability value of the most likely predicted outcome in this distri- bution. The first term 7 (1 11-77 P(S)) thus quantifies the uncertainty arising from a lack of confidence in the "best guess." " H(() = -==ı E, P(s,) RR I) log Rs, The Shannon entropy of this predictive probabil- ity distribution. The second term (1-7)7(()) thus quantifies the uncertainty due the dispersion or disorder of the overall prediction outcomes. 7 E [0, 1]: A hyperparameter that balances the relative importance of these two sources of uncertainty.
C.1.2 MANAGEABILITY DIMENSIONS: EPISTEMIC UNCERTAINTY AND ALEATORIC UNCERTAINTY
Uncertainty is further divided into manageable epistemic uncertainty (Uepis) and inherent aleatoric uncer- tainty (Uale)), as introduced in Section
18

---

Published as conference paper ICLR 2026
a. Epistemic Uncertainty (Uepis)
Degis
Stuse.epis (peerc Oxem, Minf
Seiis (A(R))
(9)
Mapping from Base Dimensions to Epistemic Uncertainty (Reducible Purt)
Epistemic Uncertainty Directly Contributtd by Knowledge Gap Cues A(R)
function that aggregates those parts of perceptual, semantic, and inferential uncertainty considered "knowable" or "reducible" (through more information or better models). A AR): Represents "Explicit Cues of Knowledge Gaps" related to response R. For ex- ample: AR) could quantify the deviation of the current query from the training data distribution (Out- of- df-Distribution detection), or the model's familiarity score with specific concepts in the query. feps (-): A function that converts these knowledge gap cues into an additional amount of epistemic uncertainty.
Detailed Expansion and Explanation: (...
Aleatoric Uncertainty (Ualea)
Malea
Jnase,alea Uperee Usem
(S((R)
(10)
Uncertainly (Inherent Random Par, Primarily from Perception and Semantics)
Mapping Dimensions
Alcatoric Uncertuinty Directly Contributed by Environmental Randomness Signals (2(R)
alsee (., :): A function that aggregates those parts of perceptual and semantic uncertainty considered "inherent" or "irreducible" (stemming from the randomness of the data itself or the intrinsic ambiguity of the task). 52 (): Represents "Explicit Signals of Environmental Ran- domness" related to response R. For example: 52(R)) could be inherent randomness explicitly stated in the task description (e.g.. "result of dice roll"), unpredictable disturbances perceived from the environ- ment. files (-): A function that converts these environmental randomness signals into an additional amount of aleatoric uncertainty.
Detailed Expansion and Explanation:
C.1.3 TOTAL UNCERTAINTY (UTOTAL.)
Finally, the uncertainties from the three base dimensions are weighted and fused to obtain the total uncer- tainty:
total
"prr
Uprr +
Wsem
Usem
+
Winf
Vinf
(11)
Perceptual Weight
Semantic Weight
Inferential Weight
Detailed Expansion and Explanation: Wperes Wsem Winf Weights for the perceptual, semantic, and inferential uncertainty dimensions, respectively. These weights typically satisfy wa 2 0 and > W4 1 (or other normalization methods), reflecting the relative importance of different uncertainty dimensions in specific task or system objective.
C.2 DYNAMIC UNCERTAINTY TRANSFER MECHANISM (3.2)
To achieve active management of uncertainty and optimize system operational costs, Agora introduces a dynamic uncertainty transfer mechanism that explicitly tracks the flow of uncertainty among agents.
C.2.1 UNCERTAINTY FLOW EQUATION
At any task I, the total uncertainty U(a;,) () borne by agent a, consists of its self-generated base uncer- tainty and the transferred uncertainty received from other agents. This dynamic process is described by the following core equation (see E).
U(a;
U Ubse (ai,
U Utranster (ag - as
(12)
Base Uncertainty
j
(Generated by o, for task itself)
Transferred Uncertainty
(Sum received from other agents o.)
19

---

Published as a conference paper ICLR 2026
Detailed Expansion and Explanation: U(a,;)); The total uncertainty vector-borne by agent aj task t. This is a multi-dimensional vector where each dimension corresponds to a specific type of uncertainty (e.g., Uperes Usem: Und defined earlier, or more fine-grained subtypes). U(a,,) t)
[upere(a,, t), Usem (a,, t), Manr(a,, t)....] U ta; The base uncertainty vector generated by agent a, due to its direct interaction with task t. Its calculation can depend on historical information, agent profiles, or default values. U Urannfer (az -> 0z t): The uncertainty vector successfully transferred from agent aj and re- ceived by agent o, in task t. jag Unanster(a) - ag t): Summation of uncertainty vectors transferred from all other agents 0j (j + i) to a, yielding the total uncertainty received by a; via the transfer mechanism task L.
C.2.2 TREND OF CHANGE IN SYSTEM-TOTAL UNCERTAINTY: CONERVATION/CONVEREEN ANALYSIS
Definition of System-Total Uncertainty Let A= (01: 02 .........0...................... .N. be the set of agents in the system. At time I, the System-Total Uncertainty (t) borne by agents in the system can be defined as the sum of the total uncertainties of individual agents: Usys(t) 2 E U(a,,) t Substituting the uncertainty flow equation:
U (t)
Ubase (ai, +)+ + Σ
U Utaanser (aj ai,
(13)
N
U (1) =
U huse (as,
U Urranser (a; - die 1)
(14)
(tyshar()): System total base uncertainty
Uppuramlerpaaiiee (t): System total reseived uncertainty
b. Impact of Uncertainty Transfer on Total Uncertainty Consider specific transfer event: at some stage of task I, agent dg successfully transfers an amount uncertainty Thx (d) (in dimension d) to agent dg- As per Eq. 4 the change in uncertainty for the sender and receiver after the transfer: Sender aj 's uncertainty change: U = U2- - kTig, receiver aj's 's uncertainty change: U/ = (1- Where: Ty: The amount of uncertainty declared for transfer from ag 01- [0. 1]: Transfer Efficiency factor. 1 means the declared amount is fully removed from the sender. & E [o, 1]: Receiver aj 's Expertise/Resslltion Factor. & 0 means the receiver, due to its expertise, effectively bears or perceives an incremental uncertainty less than the declared transfer amount, i.e., part of the uncertainty is "resolved" or "absorbed." A successful transfer T& from dg to aj leads change in system total uncertainty A Assss Assume this transfer is the only change in the system, and other agents' uncertainties remain constant.
AU = (U) + U/+
Σ
U (U + U
Σ Um) mykk
(15)
AU sys (Uk- Up) -U, UU)
(16)
AU sys (-K(Tia) + (1-E:)TTm))
(17)
A sys
(1-
Tkt
(18)
Net change factor in system uncertainty due to single trade
Analysis: Conservation: Strict Conservation: AUsys sys O when & + 1. If (fully removed), then E = (fully borne) is needed uncertainty merely redistributes. Generally Non-conserved: Typically A and & > 0, AU,, - TH. When uncertainty transfers to agents with expertise (E) > 0), total system uncertainty decreases, not physical disappearance but effective resolution by more suitable agents. 2. Convergence: Total Uncertainty Amount: With continuous U base (a,, t) and transfers to skilled agents
20

---

Published as a conference paper ICLR 2026
(E) >> 0), the system reaches a dynamic equilibrium where new uncertainty balances resolved uncertainty. If U Uusee (a, + 0 and transfers continue, (t) decreases, potentially to zero if all uncertainty is resolvable. Specific State: System state convergence depends on trading protocols and cost optimization. If each trade reduces cost, the system reaches the local optimum with stable uncertainty distribution an "equilibrium" state where uncertainty continues being processed dynamically. Conclusion: The dynamic uncertainty transfer mechanism, especially when considering the receiver's s expertise factor & > O, has the potential to reduce the system's effective total uncertainty. The absolute convergence of the system's 's total uncertainty depends on the rate of base uncertainty generation and the continued effectiveness of the trading mechanism. The convergence of uncertainty distribution among agents is closely related to the trading equilibrium state driven by cost optimization.
C.2.3 DEEPENING THE TRANSFER COST-BENEFIT ANALYSIS: CONSIDERING TOTAL TRANSFER AMOUNT AND EXPERT KNOWLEDGE
In the dynamic uncertainty transfer mechanism, key decision criterion is whether trade can reduce the cost of handling uncertainty at the system level. This depends not only on the comparison of unit costs but also on the actual total amount of uncertainty transferred and the receiver's expertise in handling that uncertainty.
a. Variable Processing Cost Change of a Trade Consider a transfer of uncertainty in a specific dimension d from agent a; (sender) to agent aj (receiver). Let (d(d) and (d) be the uncertainty stock of a; and aj in dimension d before the trade, respectively. * Let c; and Cj be the marginal processing costs for a; and aj handle unit of uncertainty in dimension d, respectively. * Let (ij(d) be the total amount of uncertainty declared for transfer from og to aj in dimension d. (d) > 0. Let E E [J be the expertise factor of receiver aj when processing uncertainty of dimension d. (1- (1-E,) (i)
represents the effective increase in uncertainty borne by aj. If E, > 0, part of the uncertainty is "resolved" or efficiently processed by a, 's expertise. Before the trade, the total processing cost related to (d) and (d(d) (considering only these stock parts) is:
Chefore
C,U,(d)
+ (dU
(19)
Cost of Agent i Cost of Agent
After the trade, agent a,'s uncertainty stock becomes U.(d) - (dy(d). Agent aj's uncertainty stock effec-
tively increases by (1 - (d),)((d), becoming (d) (1 Ej T.y (d). The new total processing cost related to this is:
Cater = (U.((d)) - (c)) + + (d) + + (1-
(20)
New cost of Agent
New cost of Agont
b. Deriving the Cost-Benefit Condition for a Trade A trade is beneficial in terms of processing costs if and only if the total processing cost after the trade strictly less than the total processing cost before the trade, i.e., Catter < Crefore. (This is equivalent to Eq. g which states AC < 0).
QUi(d) + (d)
(U.((d)) (d) + es (d) + (1-
(21)
Total processing before trade (LHS)
Total processing after trade (RHS)
Formal Expansion and Proof:
1. Subtract common terms (d) and (dU,(d) from both sides of the inequality:
(T)(()) + cy(1 (d) 2. Rearrange terms to centralize those containing (d)::):
(22)
cTT.(dd) c,(1 (d)
(23)
21

---

Published as a conference paper ICLR 2026
3. Factor out (d) (by definition, the actual transferred amount (i)(d) 0): (d) (c. cj(1 E2)) > 0
(24)
4. Since (d) > O, the necessary and sufficient condition for the above inequality to hold is:
G c Eg)
(25)
c
(26)
Sender unit
Receiver dy effective cost (considering expertise Eÿ)
Theoretical Significance: This condition explicitly states that only when the sender's unit processing cost is higher than the receiver's effective unit processing cost can the trade yield benefits at the variable processing cost level.
c. Connection with CE Ratio and Broader Cost Considerations The naive CE ratio defined as CE(a; aj, u) c:(w) suggests that if (w) > (u), the transfer is beneficial. Now, incorporating the receiver's expertise factor Ey. we can define Effective cost-Effeetiveness Ratio (CE)
((a) (here using as the sender, j as receiver, consistent with current notation)
Sender's unit
(d)+ & c-1- Ej
(27)
Receiver's effective unit cost
Then, the derived cost-benefit condition > cy(1 Ej) is equivalent to:
(d) >
(28)
This indicates that a transfer is beneficial in terms of direct processing costs only when the sender's unit processing cost is higher relative to the receiver's "effective" unit cost. Notes on Fixed Costs and Transac- tion Costs: The above derivation primarily focuses on the reduction of variable costs directly related to the amount of uncertainty processed. A complete trading decision also needs to consider more comprehensive cost-benefits: Fixed Costs (3,, 8y): a trade causes an agent to change from inactive to active (incurring new By), from active inactive (saving 3,), these changes in fixed costs need be included in the calculation of the total benefit. 2. Transaction Costs: Communication and computation overheads that may exist for executing the trade itself. Therefore, the condition & ey (1 & is core element for judging whether a trade can potentially reduce variable processing costs, but the final decision to execute the trade must be made through a more comprehensive benefit evaluation (which should internalize all relevant cost and benefit items).
C.3 UNCERTAINTY TRADING PROTOCOL
This protocol defines the rules and conditions for agents trade uncertainty, aiming to transform uncertainty into a manageable and optimizable resource to reduce total system operating costs. The core of trading is transferable epistemic uncertainty (Uepis), conducted based on principles of comparative advantage and cost- effectiveness, per Eq. 5
C.3.1 PREREQUISITES FOR A TRADE
A potential trade to transfer uncertainty of dimension d E Diradable (set of tradable uncertainty dimensions) from agent a; (sender) to aj (receiver) must first satisfy the following conditions:
22

---

Published as a conference paper ICLR 2026
Trade Trigger Condition
To ensure the necessity of trade and avoid ineffective fluctuations, an uncer-
tainty differential threshold is set:
Ed D Praddble
S.I.
Uild)
U
Tirade
(29)
Receiver 0, 's current uncertainty level dim d'
Minimum uncertainty differential threshold trigger trade
unceriainty level in dim df
Theoretical Significance: Tirade 0 ensures that trade intention is initiated only when there is significant imbalance uncertainty distribution, sufficient to overcome potential transaction friction costs and form an effective comparative advantage.
b. Receiver Capacity Constraint The planned amount of uncertainty to be transferred Tiy(d) must not exceed the processing capacity of the receiver aj. Considering the receiver's expertise factor Ey. the effective increase is (1 (d):
Vd E Pradable:
(g(d)
(1 (1-
Cj
(30)
Receiver o, pre-trade uncertainty dim df
Effective uncertainty increment in dim d
Receiver d, 's uncertainty cupacity lanit in dim of
Theoretical Significance: This constraint prevents the receiver from being overloaded by taking on too much uncertainty, ensuring own task-processing capability and system stability.
C.3.2 COST-BENEFIT ANALYSIS OF A TRADE
Condition for Reducing Variable Processing Costs A trade must at least show advantage in directly related variable processing costs. Consider the transfer of uncertainty (d) in dimension d from aj to ay: Before the trade, the local processing cost related (d( and Uy(d) is:
Cproe, pre
CdU,(d)
(UU
(31)
Cost: processes U,(d)
Cost: ay processes Uy(d)
After the trade, the relevant new local processing cost:
Oprae, post - +
(32)
Cost: a; processes remainder
Cost: processes total (inel. effective new)
where Cj are the marginal costs for a; aj to process unit uncertainty in dimension d, and &, is a, expertise factor. The condition for the trade to reduce variable processing costs Cproc, pre Cpric, post:
GU.(d) + cdU > (.(U(dd) (d)( + (U, (1) -xTTis(dd))
(33)
LHS
RHS (expanded)
Formal Expansion and Proof:
1. Subtract common terms (dU(dd) and (dU;(d) from both sides of the inequality:
0 > (d) + cy(1 (d)
(34)
2. Rearrange terms to centralize those containing (d):
(TTi c, T. (d)
(35)
3. Factor out (d) (by definition, the actual transferred amount Tú, (d) 0):
T3y (c: (1 E,)
(36)
23

---

Published as conference paper ICLR 2026
4. Since Tly > the necessary and sufficient condition for the above inequality hold is:
G cj(1 Ex)
(37)
(38)
Sender unit
Receiver dy effective unit cost (considering expertise {i)
Theoretical Significance: This condition explicitly states that only when the sender's unit processing cost is higher than the receiver's effective unit processing cost can the trade yield benefits at the variable processing cost level.
b. Overall Expected Benefit Condition The final decision to execute a trade depends on whether its overall expected benefit exceeds threshold Thenefit (as in Eq. 5
AC(Ti)(t) < OA <ON(U,() (c(()
(39)
Formal Expansion and Explanation: Upee (1. Gpre (i. Measure of uncertainty and costs for agents dg aj before the trade. ji could be 1(U, Ug) agg" norm or weighted sum. (i might be unit or total cost estimate. Upoal (f, j, Tiy); Opon (i. Tiy): Mcasures after trade. Post-trade vectors: U/ = U, kTij. where & is transfer efficiency. (i, Tiy) could be [w;;U; wzU3) agg" with weights W): w/. Thenelit 2 0: Ensures significant trade henefit, covering implicit transaction costs/risks. Theoreti- cal Significance: Comprehensive trade evaluation ensuring not just marginal cost benefits system-wide favorability after considering total uncertainty changes, fixed cost impacts, and return requirements.
U' = (1-0((((E)(1000
C.3.3 MARKET EQUILIBRIUM ANALYSIS BRIEF THEORETICAL PERSPECTIVE
Market equilibrium refers to state where potential trades satisfying all trading conditions (trigger, ca- pacity, cost-benefit, overall expected benefit) exist the system, leading to a relatively stable distribution of uncertainty. Definition 3.3.1 (Local Equilibrium State). The system reaches local equilibrium if, for any pair of agents (a;, a;) and any tradable dimension E Dindables at least one of the following does not hold:
1. Uy(d) U,(d) > Trade
2. U,(d) (1 (T) S (d(d (for some permissible (iy(d) > 0)
3. q cj(1 Ei)
4. AC(T.)) < 0 (for some Tiy determined by 1-3)
Proposition 3.3.1 (Convergence of Trading Process to Local Equilibrium). If: (A1) The total tradable uncertainty in the system is finite, the volume of single trade (dy has positive lower bound. (A2) Each successful trade strictly reduces the global cost function Cays by an amount greater than &min > 0. (A3) Cays is bounded below. Then the sequence of trades is finite, and the system will converge to local
equilibrium state defined above. Proof Outline: clk+1) < (k) 6min (each trade reduces cost). Since
Cays has lower bound Cmin- and the initial cost is c(f-, the maximum number trades Nmaxppt I (Cl Cuin) mimin hence the trade sequence is finite. When the sequence terminates, more trades satisfy all conditions, and the system reaches local equilibrium. This equilibrium state represents a point where, under the current protocol and information, system costs cannot be further optimized through bilateral trades.
24

---

Published as a conference paper ICLR 2026
C.3.4 APPLICATION COMPARATIVE ADVANTAGE THEORY
The theory of comparative advantage provides a theoretical basis for uncertainty trading: even if some agents do possess absolute cost advantage in processing all uncertainty dimensions, long as there are differences in the relative processing efficiencies (opportunity costs) of various agents across different dimensions, specialization, and trade can still enhance overall system efficiency and reduce total costs, Definition 3.4.1 (Comparative Advantage). For agents aR; dj and uncertainty dimensions d1, d2, if their unit processing costs ez (dy):
(x(11) cc(di) (x
(40)
(d22)
then ag has comparative advantage over do processing d1 (relative to d2).
a. Comparative Advantage and Cost Optimization Trades based on comparative advantage aim to allocate specific types of uncertainty to the agent with the lowest opportunity cost for that type.
Aotal [U; AUA,) + cy(U, + AUA)) ) [V,( +
(41)
Expanding this:
ACuotal (aj- - az)) AUd;
(42)
For ACmal < O, it is required that aj < ag That is, uncertainty di should flow from an agent with a higher unit processing cost (a,) one with lower unit processing cost (az).
b. Implicit Implementation of Comparative Advantage by Agora Protocol The core trading condition of Agora, > c2(1 - Ej), is based on effective absolute cost advantage. However, if the cost parameters C₁ and expertise & of agents dynamically reflect their true efficiency and specialization handling different dimensions of uncertainty (which may stem from their comparative advantages), then a series of local trades based on effective absolute cost advantage will, at a macro level, guide the system's 's uncertainty distribution towards configuration that aligns with the principles of comparative advantage. For instance, an agent with comparative advantage in dimension di might develop a low c(di) and high E(di) ) for processing d1, thereby becoming natural "sink" for uncertainty in that dimension.
C.4 UNCERTAINTY-AWA. MAB SELECTION STRATEGY
a. Beta Posterior Parameter Update For all S E A (set of agents), at decision round f, the Beta distri- bution parameters (a , are updated: Let rs 1) E {0, 1} be the observed binary reward for agent S in round 1.
(
T(g = S) -)
(43)
Prior
1-1 T(( =S) -1--55)
80
(44)
Prioe 9p
where I(((()) = S) is an indicator function, indicating whether agent S was selected in round r. Typically, a (0) 1, (0) 1.
Baseline Expected Reward E
E
(4)
ag
+
25

---

Published as a conference paper at ICLR 2026
e, Comprehensive Scoring Function 0
(E) R.Rewaddd"" |
Cou
frar Asa))
(Sya(S:9)
(S
(45)
Eiperrad (1
Tad Masting Paatir (lste (1ME3")
Thw maaaa thausrr
Tour Syneggy Facter (1ST
Striegio Unceriorty Indes Focke (SUFFa)
Expected Net Reward (ENR): (E[Rewardg)) C This term represents the fundamental utility of selecting agent 8 for task t, balancing its expected rewards against its operational costs; [[Rewardg))
B Remux (t): This is the anticipated raw reward from agent S for task t; B[og' The posterior mean of the success probability for agent S. typically derived from a Beta distribution Beta(as (e) . calculated
(
Roma (t): The maximum possible reward achievable for task t; Cost (!) (US.2
Seess: TaskFeatt)
+.80 U.)) O S,est + Bena,s + Cussk (4, TaskFeat,): The estimated cost for agent to handle task 1; Ocoos, The marginal cost for agent S to process one unit of uncertainty; S,est || 8,buse + U
Sttrasfer U S,trasterr.oul The estimated total uncertainty agent S handles for task t (using the L1 norm). This con- siders its self gelfraeneetted base uncertainty (U% S,buse uncertainty received from other agents (U(+) Stmamserr.i and uncertainty offloaded to others (U%+) S.mansfer.omu) Peas,s: The fixed base operational cost for agent S: Cuus (t, TaskFeat,): ): Additional costs incurred due to specific features of task & (TaskFeat,). Task Matching
Factor (TM): (S, t; Adia) = exp(-Adis: ds.,) This factor quantifies the compatibility or relevance of agent S to the current task A higher match (smaller distance dg,z) results in a factor closer 1; Aaia: A hyperparameter that weights the influence of the distance ds.; ds.:: The distance or dissimilarity be- tween the feature vector of agent S (vs) and that of task (ve). Two alternative calculations are suggested: Normalized Euclidean distance: ds,; max l[vs villzte Normalization is done by dividing by the maximum distance found among all agents for that task, with being a small constant to prevent division by zero; Cosine dissimilarity: 1 Vy
lIva-vell
This measures the difference in orientation between the two
[
vectors. Time Decay Factor (TDF): (TD(Ats; (
Ats This factor prioritizes more recent infor- 7deeay
mation regarding agent S's performance state, diminishing the impact of older data; 'eeay: The decay base hyperparameter, where 0 < 'leeay S 1. If "deaay < 1, older information receives a lower weight; Als = - last (S): The time elapsed since agent S's parameters (e.g., Beta distribution parameters) were last updated. Team Synergy Factor (TSF): (Sy = (1 SynValg)) This factor assesses the
potential for agent S to collaborate effectively with other agents in the current team or context for task t; Syn [e1 E) J+s Comp(S,) (S.j) Pot(j, Task The synergy value for agent S at time t. It's an average of compatibility scores (Comp(5,))) between agent S and its teammates j. weighted by each teammate's potential (Pot(j, ())) for the current task. [Team!")] is the number of agents in the current team; 1y: A hyperparameter exponent that controls the degree of influence of the team synergy value. Strategic Uncertainty Index Factor (SUIF): fsua(
(1 + (tt). This novel factor incorporates
the strategic value of agent S's uncertainty within the uncertainty trading market. Agents that can contribute more to system-level cost savings via uncertainty trading are favored; U'A The strategic uncertainty value of agent S at time t. It quantifies the expected net cost saving that agent S can bring to the system by participating in the uncertainty market (as seller or buyer). This is detailed further in Appendix C.4.2; wt A hyperparameter exponent that modulates the importance of this strategic uncertainty value in the overall score.
26

---

Published as a conference paper ICLR 2026
C.4.1 THEORETICAL GUARANTEES: REGRET CONVERGENCE
( a. Redefining Regret Let S.true (C(0) be the true expected comprehensive score of agent S at time given
context (+)) (including task characteristics, market state, etc.). Let (t (C = arg maxsEA (
S.rrue (C(t)).
The context-cumulative regret RG over time steps is:
R2
E cto
@(t) Sug (Cto) (ctO), truwe
(C g(o) 800) true (c())
(46)
t=1
where ((t) is the agent actually selected at time (context ((
b. Assumptions for Convergence Analysis (A1) Boundedness: For all S.t, the values of 05 (and its components) are within bounded interval, e.g., [0, max]] (A2) Lipschitz Continuity (some factors): For changes in some contextual variables E (()), the change in 03 is Lipschitz continuous, i.e.,
10 (c1)
8 (c2) & L|c1 - 02|- (A3) Learning and Adaptation: The agent's estimate of E converges, and its estimates of dynamically changing contextual factors (like Cost Ustrategic (S)) are also progressively adapting.
c. Direction of Convergence Although proving classic Olog T) or O(VT) regret bounds is very difficult, the strategy is designed such that the selection probability P(t S History c(t)) gradually biases towards agents with higher true expected
@!0)) 05, (C()). E (c(+)) itself converges to a stationary value 0z strru (C*) (under stationary context C*), then the selection will converge to the optimal arm Sipt arg max 0% Sture (C*). If the context is non-stationary, the strategy attempts to track the optimal arm, similar to a multi-armed bandit problem non-stationary environment. Its performance depends on the speed and predictability of contextual changes, as well as the accuracy and adaptation speed of the factor estimates.
C.4.2 MATHEMATICAL DECONSTRUCTION OF STRATEGIC UNCERTAINTY INDEX (STRATEGGI(S))
Core Objective Function of Uttrategie(S) (S) Let M be the uncertainty market state at time Ustratgic (S. (())) represents the expected net cost saving (S, M(99) that agent S can bring the entire system by participating in the market defined by (
Ustrategic (8, (0) Eyrados involving
(Cays(pre-tr) - (posttr))) (tr occurs)
(47)
(0)
where M()) is the set of all potential trades involving S (as buyer or seller) that satisfy the trading conditions.
b. Expansion of System Cost Change from Trade AGss(tradde Consider trade tr (s, r, k: Tork)
transferring an amount Tark of uncertainty in dimension dy from ag to ay-
M
(ore(pr-)))
Σ
Σ
ane + ((((T))))
(48)
1EA
(post--))
Σ
(((((r)))) C. (U.pre)) e Tant) + (U((re))) +ex(1 (.n
(49)
27

---

Published as conference paper ICLR 2026
(trys
(pre(tr)) Csys (post-tr)
Ulire) sk _[(U!!or) + (antui + By (U(ppe))
Oikk (U Tskk) + +3, (U(ur)) TT sk
ark (U(pee) + (1 - Erk (ork) + (U( +ex Ere (orr))
-ackTerk arkk (1 Erk)Tork
Variable cost saving ACvar
(BL(U((ro)) 8 (U(()))
TT Tark)
Sender fixed cost change A
(U((r))))) (U( ex Erk) (Trk)
(50)
Receiver fixed cost change 2,3
Aays (tr)
T (rsk(Ossk a,r(1 Erk) + AA8 + +A8
(51)
This is related (t) in Eq. Ustrategle (S) as an Expected Sum
U strattge (S, M())
ΣΣ
B([(Cond (. (S as seller)
dED $
Σ
B[u(Cond AC, ( (S as buyer)
(52)
d'ED i
where the expectation EE|]] is taken over the probability distribution of future market states, other agents' behaviors, and trade volumes T. The introduction of Usrategic (S) extends the MAB's decision-makinn from focusing solely on single-agent, single-task "local" utility to considering system-level "global" economic benefits. It guides the exploration//cpllotttionn mechanism by altering the "effective value" of each arm to favor agents that can maximize the efficiency the entire uncertainty trading network.
D IMPACT OF AGENT POOL CONFIGURATION ON AGORA
This appendix provides a systematic evaluation of the Agora framework's performance and opera- tional characteristics under varying agent pool configurations. The strategic composition of the agent ppol-specifically its heterogeneity, the degree of agent specialization, and its overall size-represent crit- ical degrees of freedom in deploying Agora. Understanding the framework's sensitivity to these factors essential for tailoring deployments to specific operational constraints and performance objectives, thereby maximizing resource utilization and system effectiveness. The experiments herein quantify these impacts precisely, offering empirical guidance for optimal pool design.
D.1 EXPERIMENTAL SETUP
All experiments in this appendix conducted using the MMMU (Val) and MMBench V11 Test datasets, the main paper. Computational resources comprised NVIDIA A100 GPUs, and Vision-Language Model (VLM) access was facilitated via the OpenRouter API. To ensure statistical robustness, all reported results
28

---

Published as a conference paper ICLR 2026
Table 4: Impact of agent pool configuration on Agora performance. All metrics are mean + std. dev. over 5
runs.
Experiment
Configuraatin
MMMU Acc. (16)
AMMBescc Acc. (%)
Inf. Time (shask)
Trade Freq. (tndes/taski
Uncert. Red. (%)
Nomn Cost
Fail Kato (16
Heterogencity
Heterngencous
T9.2(5)
80.0:4)
2.5(1]
(18(1))
26. 26.4(12)
1.10[S]
6.2(8)
Homoggeneous
86,315)
2.7(3]
20.1(15)
1.15/6]
8.7(11)
Specialization
80.044]
2.5(1]
25.4(12)
1.10(5]
High
80.8(4)
90.2/3)
2,6[1]
30.1(15)
1.12(3]
72.3(7)
2.011]
10.5/12)
N=3
75.8(8)
80.815)
2:2:1]
(.5(1)
21,3(10)
1.003
7.8:10)
N-5
78.2(5)
80.0:4)
2.611]
(1811)
25.4(12)
1.30|5]
6.2(8)
N=10
80.114)
90.0:3)
3.0:2]
1.0(1)
(117(15)
1.2016]
4.9(7)
N=15
80.5(4)
90.3/3)
3.522)
1.2(2)
29:2(14)
1,307]]
4.7(7)
are averaged over independent runs, presented mean + standard deviation. The codebase for these experiments is available in the project's open-source repository, as referenced in the Introduction.
The core experimental variables were agent pool heterogeneity, specialization, and size, configured fol- lows: For Heterogeneity, two primary configurations were compared: a " pool, representing the default diverse agent set Sowee2.5v1-72b-instree 27b, gpt-40-mini); and "Homogeneous" pool, comprising five instances of the gwen2.5yl177bbinstmnn model, differentiated only by varied initialization seeds to account for stochasticity in their otherwise iden- tical capabilities.
gemini-2.0-flan 2.0-flas,,,,, Siven..v1-77b-instee
gemma-3-
For Specialization, pools were configured for 'Low' specialization, using the default general-purpost VLMs, versus 'High' specialization, where agents were restricted via prompt engineering to focus primarily on one dimension of uncertainty (c.g., perceptual, semantic, or inferential).
For Pool Size, the number of active agents (N) varied: N = 2 1eeee22.51-72biiie = 3 (adding gemini-2.0-flash to the N = pool); = 5 (the default heterogeneous pool); = 10 (default pool augmented with duplicates of its constituent models); and N 15 (further augmented with duplicates and additional distinct models such as InternVL3-788 and gemini 2.5 2.t.--p-------00-2255555
gpt-40-mini);
Performance was quantified using comprehensive suite of Metries: Accuracy (%) both datasets; average inference time per task (s/task); trading frequency (average trades executed per task); uncertainty reduction (%, defined the relative decrease in relevant uncertainty metrie from initial final state); normalized operational cost (relative to baseline gpt-40-mini agent); and failure rate (% on a predefined subset of complex tasks, similar those in Tables 10 and 11 of the main paper, which exhibit high ambiguity or reasoning demands).
D.2 RESULTS AND ANALYSIS
The empirical outcomes of the agent pool configuration experiments are presented in Table These results highlight the distinct effects of heterogeneity, specialization, and pool size on Agora's operational efficacy.
The data reveals several key insights. Regarding Heterogeneity, heterogeneous pools outperform homoge- neous ones, with higher accuracy on MMMU (79.2% vs. 74.5%) and MMBench (89.5% vs. 86.3%). This stems from increased trading frequency (0.8 vs. 0.3 trades/task) and greater uncertainty reduction (25.4% vs. 20.1%), validating Agora's 's ability leverage diverse capabilities for uncertainty resolution (as in Sec- tion 3 The lower failure rate on complex tasks (5.2% vs. 8.7%) highlights the benefits of varied expertise in challenging scenarios.
For Specialization, high specialization boosts accuracy (MMMU: 80.8% vs. 79.2%; MMBench: 90.2% vs. 89.5%) and uncertainty reduction (30.1% 30.1% vs. 25.4%) thanks to precise uncertainty routing. The slight
29

---

Published as a conference paper ICLR 2026
increases in inference time (2.6s vs. 2.5s) and normalized cost (1.12 vs. 1.10) reflect minor overhead from managing specialized agents.
The Pool Size analysis shows non- non-linear scaling: accuracy and uncertainty reduction improve up to N 10 (MMMU: 80.1 9%, Trade Freq: 1.0), but gains plateau at N = 15 (+0.4% on MMMU), with steeper rises cost (1.30) and time (3.5s). This indicates diminishing returns beyond a threshold, due to heightened selection and communication complexity. Smaller pools (N 2,3) suffer from limited trading options, leading to lower accuracy and higher failure rates (10.5% for N 2).
FLOPS COMPARISON AND COMPUTATIONAL EFFICIENCY
To validate the computational efficiency of the Agora architecture, we designed a simulation- FLOPs comparison experiment. The objective was to quantify the reduction in system-level Floating Point Opera- tions (FLOPs) achieved by our uncertainty-drives agent selection and task trading mechanisms.
E.1 EXPERIMENT SETUP
We built system composed of heterogeneous Vision-I Langugge computational costs. The agent characteristics were defined as follows:
Vision-Language Agents (VLAs) with varying scales and
Small Agent (e.g., Qwen2.5-VL-7B based): A 7-billion parameter model, estimated to consume approximately 1.4 TFLOPs per generated token. This estimation based on the premise that FLOPs roughly proportional to parameter count, similar to models like Llama 7B which ap- proximately 14N FLOPs for N parameters during prefill and 2N for generation; here, we aggregate these into a per-token value.
Medium Agent (e.g,, Intern based): A 14-billion parameter model, estimated at approx-
imately 2.8 TFLOPs per generated token.
Large Agent (e.g., InternVL3-78B based): A 78-billion parameter model, estimated approxi-
mately 15.6 TFLOPs per generated token.
Each task was assumed generate average of 20 output tokens. The simulation covered 100 visual- language tasks, and we tracked the total FLOPs incurred by the system under different strategies.
Several baseline strategies were included for comparison:
Small-only: All tasks are processed exclusively by the Small Agent.
Medium-only: All tasks are processed exclusively by the Medium Agent.
Large-only: All tasks are processed exclusively by the Large Agent. This serves as performance upper bound.
Random Assignment: Tasks are randomly allocated to one of the three agent types.
Top-2 Routing: Tasks are alternated routed based simple heuristics between the Small and
Large models (simulating common mixture of experts or high-performance focused routing).
Tiered Cascade: Tasks are first attempted by the Small Agent; fails (or similar heuristic
applies), the task is escalated to the Medium Agent, and then to the Large Agent if necessary.
Our proposed Uncertainty-Aware (Agora) strategy operates as follows: All tasks are initially attempted by the Small Agent. If the estimated uncertainty (or associated cost of resolving it) for task exceeds predefined threshold, the task is escalated to a more capable (and computationally expensive) agent. In this simulation:
30

---

Published as a conference paper ICLR 2026
88 tasks were completed by the Small Agent.
tasks were escalated to and completed by the Medium Agent.
8 tasks were escalated to and completed by the Large Agent.
E.2 EXPERIMENTAL RESULTS AND DISCUSSION
The computational efficiency and performance trade-offs of various agent dispatching strategies are illustrated in Figure 6 which plots opera- tional accuracy against the cost-performance ratio (PFLOPs per percentage point of accuracy). Our analysis, based on FLOPs (e.g., Small-only strat- egy at 2.8 PFLOPs, Large-only at 31.2 PFLOPs), reveals that our Uncertainty-Awant (consuming approximately 5.54 PFLOPs) achieves a remarkable balance. It delivers an accuracy of 88.7%. closely approaching the Large-only strat- egy's 89.2%, yet it slashes the computational load by approximately 82.2% more than 5.6-fold re- duction from the 31.2 PFLOPs required by the Large-only approach.
arge-only
Uncertainity Awars
87.5
Tlared Concade
Routing
Random Assignment
Accurscy (8)
42.5
Agora strategy
heuding Ststejies
10.0
Adaptive (hor))
Mediumnnly
37.5
75.0
mall--nnl
0,3070
0.05
0.25
0.35
As depicted Figure this efficiency translates to a superior cost-performance ratio of 0.0625 for the Agora strategy. This is significantly more favorable than the Large-only strategy (0.3496) and strikes effective balance compared to the Small-only strat- egy, which, despite lower ratio of 0.0389, suffers from substantially reduced accuracy (72.0%). Fur- thermore, when compared against other dynamic approaches such as Top-2 Routing (cost-perf. ratio 0.0789, accuracy 86. 1%) and Tiered Cascade (cost-perf. ratio 0.0910, accuracy 86.8%), the Agora frame- work, as visualized in the figure, consistently demonstrates more advantageous position by maintaining higher accuracy for a competitive or superior cost-performance metric.
Cost-Pert. Ratio (TPLOPs per Accuracy!
Figure 6: Performance comparison of routing strate- gies: Accuracy (%) versus Cost-Performance Ratio (TFLOPs per % Accuracy; lower is better). Our pro- posed Uncertainyy-Aaar strategy (red star marker) achieves an excellent balance between high accuracy and cost efficiency.
Overall, this FLOPs comparison underscores the efficacy of the Agora architecture's uncertainty-driven multi-agent dispatching mechanism. By intelligently allocating resources based on quantified uncertainty, it effectively balances high performance with minimized computational overhead. This capability, clearly visualized in Figure 6 shows its superiority over traditional static allocation and simpler dynamic strategies, rendering it particularly well-suited for large- scale, multi- modal deployments where both accuracy and cost- efficiency are critical.
F SUPPLEMENTARY CORE COMPONENT ABLATION DISCUSSION
To rigorously validate the individual contributions of the Agora core architectural components and their interactions, we supplemented them with series of detailed ablation studies. These investigations de- signed to dissect the framework, isolating the impact of specific design choices regarding multi-dimensional uncertainty quantification, the strategic handling of epistemic versus aleatoric uncertainty in the trading pro- tocol, and the sensitivity of market dynamics to its key operational parameters. The objective to provide empirical evidence substantiating the necessity and efficacy of each component, thereby ensuring that the
31

---

Published as a conference paper at ICLR 2026
a. Uncertainty Dimensions Impact
b. Aleatoric Trading Strategy
Trading Protocol Parameters
75.0
11.8
10
62.
07.5
Jevt
dey 9oor
t0
only Land
Low 10s ano
87.0
t.0k 0,5
Caperiy Brikned
Cplcit Serd
oeyy
Row 300
Delaam
Figure 7: Supplementary Ablation Study Results for Agora Core Components and Uncertainty Trading on MMBench VII Test. Summary: Removing perceptual (U pere), semantic (Usem). ), or inferential (Uint) uncertainty lowers accuracy and raises costs, with perceptual removal causing the largest accuracy drop (to 85.37%). Single-dimension setups underperform, showing all dimensions are vital. Trading only epistemic uncertainty (Uepis) optimizes performance, while including aleatoric uncertainty (Ualea) ) increases errors and residual uncertainty, confirming Uatea is non-tradable.
overall framework's ' observed performance and cost-efficiency gains are directly attributable to its principled design, rather than emergent coincidental factors. Each study systematically deactivates or varies tar- geted element while holding others constant, allowing for precise assessment of its marginal contribution the system's objectives the MMBench, Mireench_V11_11 V11 y_stt benchmark. All experimental results are shown in the Figure| 7
F.1 IMPACT OF UNCERTAINTY DIMENSIONS
The Agora framework posits that a multi-dimennional representation of uncertainty, encompassing percep- tual (Upere,, semantic (Usem), and inferential (Unat) aspects, is critical for nuanced agent selection and effective uncertainty trading. To verify this, experiments were configured where the influence of each dimension was systematically nullified, or where the system was restricted to operating
single dimension. The experi- mental setup was as follows; the baseline Agora configuration utilizes optimized weights for all uncertainty dimensions (Wperc. Wsems War) as detailed in Appendix F (e.g., Wpere 0.4, Wsem 0.3, Wint 0.3); di- mensional ablation variants involved setting the weight of the target dimension to zero (e.g., Wpere 0 for perceptual ablation), with other weights proportionally adjusted kept per a defined strategy maintain normalization if necessary; single-dimension variants restricted the system to one dimension (e.g.. Wpere = 1, W'sem 0, Wint 0); performance was evaluated using MMBench Accuracy (%), final epistemic uncertainty (U )p,),, Collaboration Overhead Index (COI), Sincetainty-Adjusse Score (UAPS, %), and Relative Operational Cost (Rel. Cost); sensitivity across task types was qualitatively assessed by considering performance on benchmarks like MMMU and InfoVQA during analysis.
Performance
The results presented in Figure 7 unequivocally demonstrate the criticality of the multi-ddimensional uncer- (Upere), semantic (Use)), or inferential
tainty framework. Removal of any single dimension-percenee Precipppeepps a notable degradation in overall performance (MMBench Accuracy and UAPS) and an increase in residual epistemic uncertainty (na)s and collaboration overhead (COI). The absence of per- ceptual uncertainty (w/oUpee perc) incurs the most substantial performance penalty (Accuracy drop to 85.37%, UAPS 70.59%) underscoring its foundational role in visual understanding tasks. Semantic uncertainty ablation (w/ oU also significantly impacts performance, confirming its importance for higher-level com- prehension. While the removal of inferential uncertainty (w/oU,s) shows comparatively smaller, yet still
32

---

Published as a conference paper at ICLR 2026
significant, decline, its contribution to refining decision confidence and strategic agent selection is evident. Furthermore, configurations relying solely on a single uncertainty dimension (e.g., "Only Uper"") exhibit markedly inferior performance across all metrics, highlighting the synergistic benefit derived from the holis- tic, multi-faceted uncertainty assessment integral to Agora. This empirically validates that each quantified dimension provides unique, non-redundant signals essential for optimal system operation and cost-efficient uncertainty management.
F.2 VALIDATION OF EPISTEMIC-ALEATORE DISTINCTION IN UNCERTAINTY TRADING
A foundational principle of the Agora trading protocol is the explicit distinction between tradable epistemic uncertainty (Uepis), which is presumed reducible through further processing or information, and typically non-tradable aleatoric uncertainty (Untea), stemming from inherent randomness ambiguity. This set of experiments investigates the ramifications of deviating from this principle. The experimental setup was as follows; the bascline Agora strictly adheres trading only Uepis: variant configurations involved introducing Ustes into the trading pool, either in a controlled manner (e.g., allowing predefined percentage, such as 10% or 30%, of the total uncertainty offered for trade to be Uatea: particularly if Uepis low or if Ustea components are heuristically deemed partially resolvable by specialist agents) or indiscriminately (treating Uepis and Uslea as a single, undifferentiated pool for trading decisions); key performance indicators included standard metrics, with a specific focus on any increase in decision error rates (proxied by accuracy drops) and adverse trends in Lnnal.ppss as trading Uakea is hypothesized to not lead to its actual reduction but rather its potentially detrimental reallocation; outputs were also qualitatively compared against failure cases (e.g., from Appendix H) to assess if improper handling of Uatea could exacerbate known system limitations.
The empirical results furnished in Figure 7 affirm the strategic imperative of selectively trading epistemic uncertainty. The baseline Agora, which exclusively trades Ueps.. maintains superior performance across all metries. Introducing even a controlled portion of aleatoric uncertainty (Ustea) into the trading mechanism (e.g., "Trade Uepis + 10% Usten") leads to a discernible decrease in accuracy (to 88. 17%) and UAPS (to 76.21%), coupled with an increase in final epistemic uncertainty (Uimal-epis 0.19) and relative cost. This detrimental effect is amplified when larger fraction of Ustaa is made tradable (30% Ustea), and becomes most pronounced under an indiseriminate trading policy where Uapts and Uatea are not differentiated, resulting in a significant accuracy drop to 84.66% and a UAPS of 67.92%. This degradation is consistent with the theo- retical premise that aleatoric uncertainty, being inherent to the task or data, cannot be effectively "resolved" reduced by redirecting to another agent; attempting to do so merely misallocates resources, potentially increases collaboration overhead for no tangible benefit, and can lead to suboptimal agent selection if the MAB believes an agent can reduce irreducible uncertainty. These findings strongly support Agora's 'ss design choice to focus uncertainty trading on the remediable epistemic component.
F.3 ROBUSTNESS AND BOUNDARY ANALYSIS OF TRADING PROTOCOT. PARAMETERS
The efficiency and stability of the Agora uncertainty market are critically dependent on the precise cali- bration of its trading protocol parameters, notably the trade trigger threshold (Tirade), the expected benefit threshold (7benet)) and receiver capacity constraints (d)( This subsection details experiments designed probe the system's 's sensitivity to variations in these parameters. The experimental methodology was as follows; the Agora system was initialized with default parameter values as specified in Appendix F (e.g.. Tinad = 0.15, Thenefit = 0.08); subsequently, each parameter was individually varied across a predefined range while others were held at their default values (c.g., Tuade scanned through values like 0.05, 0010, 0.15, 0.20, 0.25; Tbenefi through 0.02, 0.05, 0.08, 0.12, 0.16); for receiver capacity (d). distinct scenarios representing relaxed, moderate, and strict capacity limits were simulated; in addition to standard perfor- mance metrics, data was collected on trade frequency (average trades per task), average trade volume, and metries indicative of market equilibrium, such uncertainty distribution entropy among agents.
33

---

Published as conference paper ICLR 2026
The parameter sensitivity analysis, summarized in Figure 7 reveals that Agora's performance exhibits a degree of robustness around the empirically chosen default parameters, yet extremes can degrade efficacy. For the trade trigger threshold (Trade), a very low value (0.05) increases trade frequency (5.8 trades) and COI (1.36), leading to slightly higher costs and a marginal dip in UAPS, likely due to excessive, low-value transactions. Conversely, a high Timade (0.25) curtails trading activity (1.9 trades), reducing COI and cost but also slightly diminishing accuracy and UAPS, suggesting missed opportunities for beneficial uncertainty re- allocation. Similarly, the expected benefit threshold (Teenefit demonstrates a trade-off: a low threshold (0.02) encourages more trades (4.7) but may permit less impactful exchanges, increasing overhead; high thresh- old (0.16) is more conservative, reducing trade frequency (2.3) and costs but potentially forgoing cumulative gains from smaller, individually beneficial trades. Receiver capacity constraints also play a significant role: relaxed capacity allows for slightly improved peak performance (Accuracy 89.63%, UAPS 78.71% 1%) by fa- cilitating more optimal uncertainty flow, albeit with a minor increase in COI and reducing COI, marginally constrains performance, indicating that sufficient receiver bandwidth is necessary for the market function effectively. These findings confirm that the default parameters strike reasonable balance, but also suggest that adaptive or context-awe parameter tuning could offer further optimization pathways.
Strict capacity, while
G HYPERPARAMETER ABLATION EXPERIMENT
In this section, we present series of ablation studies to investigate the sensitivity of our Agora model to its key hyperparameters. These experiments were conducted on the MMBench_ Win_h__ii__ess dataset. Our goal demonstrate the rationale behind chosen default hyperparameter settings (as used in the Agora Full Strategy in the main paper) and to show their robustness. For each study, we vary one hyperparameter while keeping all others at their default optimal values.
G.1 ABLATION ON UCB1 EXPLORATION CONSTANT C'
The UCB1 (Upper Confidence Bound 1) algorithm, utilized by our MAB, employs an exploration constant C' to manage the exploration-explottaiiee arms with higher uncertainty. We evaluated several values for C, and the results are detailed in Table Our selected default value of C 1.0 (this value is hypothetical; please use your actual default) demonstrates a robust balance. Performance tends to degrade if C' is set too low (insufficient exploration) or too high (excessive exploration), as reflected in metries such MMBench Accuracy and UAPS.
dilemma. larger C value biases the MAB towards exploring
Table 5: Ablation study for the UCB1 exploration constant C on MMBench. V11 Test.. The default value used in our Agora (Full Strategy) is highlighted in bold.
UCB1 Constant C'
MMBench Ace. (%)
Upinatepis
CO1 |
CAPS (%) f
Rel. Cost |
0.1
88.23
0.19
1.35
75.12
1.02
0.5
89.15
0.17
1.28
77.58
1.01
1.0 (Default)
89,50
0.16
1.25
78.33
1.00
2.0
89.32
0.17
1.26
77.91
1.00
5.0
87.98
0.20
1.40
74.65
1.03
G.2 ABLATION ON MAB LEARNING RATE
The learning rate a is a critical parameter in many MAB algorithms, determining the step size for updating value estimations (e.g., Q-values) based new observations. An appropriate ensures efficient learn-
34

---

Published as conference paper ICLR 2026
ing and convergence. Table the results of varying a. Our default setting of 0.1 (hypothetical) appears optimal. Lower values can impede the learning process, making the MAB slow to adapt, whereas higher values might cause instability and prevent convergence to the best strategy due to oversensitivity immediate rewards.
Table 6: Ablation study for the MAB learning rate on MMBench.VIl VeT. The default value used in our Agora (Full Strategy) highlighted bold.
Learning Rate a
MMBench Acc. 1
U Finnllopis
COI +
UAPS (%) f
Rel. Cost +
0.01
88.65
0.18
1.30
76.05
1.01
0.05
89.21
0.17
1.27
77.82
1.00
0.1 (Default)
89.50
0.16
1.25
78.33
1.00
0.3
88.93
0.19
1.32
76.88
1.02
0.5
87.54
0,22
1.42
73.45
1.04
G.3 ABLATION ON TIME DECAY FACTOR Ase
The Time Decay (At) component within our Agora selection strategy allows the system to weigh recent observations more heavily than older ones, adapting potential drifts in data VLM performance. This mechanism is often governed by a decay factor, denoted here as AAr- value Aat closer to 1.0 indicates slower decay of influence from past data. We investigate the impact of varying A in Table The results suggest that our default value of As = 0.99 (hypothetical) is effective. If there is no decay (Ast = 1.0), corresponding to the "w/o Time Decay" scenario having the component active but static from our main paper's is ablation, performance is slightly reduced compared to slow decay. Conversely, very rapid decay (e.g., much lower Aai) could also be suboptimal by prematurely discarding valuable historical information.
Table 7: Ablation study for the Time Decay factor Asi on MMBench VI1 Test. The default value used in our Agora (Full Strategy) is highlighted bold.
Decay Factor Ase
MMBench Aec. (%)
Ufiualepii
CO1
UAPS (%)
Rel. Cost
0.90
88.78
0.18
1.29
76.50
1.01
0.95
89.12
0.17
1.27
77.43
1.00
0.99 (Default)
89,50
0.16
1.25
78.33
1.00
0.995
89.4 19
0.16
1.26
78.02
1.00
1.00 (Effectively w/o At)
89.05
0.17
1.26
77.14
1.00
These hyperparameter ablation studies underscore the robustness of our selected default parameters for the Agora model on the MMBench V11 Test dataset. While the model cxhibits graceful degradation with slight deviations from these optimal values, significant variations negatively impact performance, emphasizing the importance of careful hyperparameter configuration. The chosen defaults consistently yield strong results across the evaluated metrics.
35

---

Published as a conference paper ICLR 2026
H HYPERPARAMETERS USED IN THE EXPERIMENTS
This section outlines the hyperparameter configurations employed for the Agora framework, comparative models, and general model inference across the experiments detailed in this paper. Unless otherwise noted, these settings were applied consistently throughout.
H.1 AGORA FRAMEWORK PARAMETERS
The Agora framework's 's agent selection strategy (Section 3.5) leverages an extended Thompson Sampling (TS) mechanism. The score for selecting agent time is defined as:
[Rewardd - ) exp(- - Dist(S, ()) - Synergy(S)" Ustaacgic (S)
The hyperparameters for the "Agora (Full Strategy)" configuration, validated in Table 3 and Appendix G are:
Time Decay Base (7): 0.99, reported in ( there Aat)
Task Match Weight (A): 0.2, empirically optimized to balance task relevance and exploration, with its impact evidenced by the ablation "w/o Task Match (Dist)" in Table 3
Synergy Exponent (n)): 0.8, tuned modulate the influence of agent synergies, as demonstrated by the ablation "w/o Synergy (Synergy)) in Table
Strategic Uncertainty Exponent (w): 1.2, adjusted to emphasize strategic uncertainty, with its role highlighted by the ablation "w/o Strategic Uncertainty (Ustategice" in Table
Thompson Sampling Priors: For each agent S, the Beta posterior parameters (as, 3s) were initialized to (1,1), reflecting a uniform prior over success and failure.
Additional parameters for the Agora framework include:
manti-dimensional Uncertainty Weights (Wpere, Wssem Wir)): Set to 0.4, 0.3, and 0.3, respectively, reflecting slight emphasis on perceptual uncertainty. determined through cross-validation.
Task Similarity Threshold (Tsi)): 0.75, based normalized cosine similarity scale (0-1), opti- mized for task clustering efficiency.
Uncertainty Trading Trigger Threshold (Firade): 0.15, calibrated initiate trading when uncer- tainty differences exceed this normalized bound.
Trade Benefit Threshold (Theneit): 0.08, set to ensure trades yield meaningful cost reductions, validated via simulation.
Note: Ablation studies in Appendix G evaluated alternative MAB strategies, including UCBI with an ex- ploration constant C 1.0 (Table and learning rate = 0.1 (Table (6). These pertain exploratory variants, whereas the primary Agora configuration relies on Thompson Sampling.
H.2 HYPERPARAMETERI FOR COMPARATIVE MODELS AND STRATEGIES
For comparative experiments in Sections 4.2 and 4.3, alternative strategics were adapted to the VLM context, utilizing the same base VL.M agent pool Agora where applicable. Hyperparameters were derived from original formulations, standard practices, or task-specific tuning.
Agora (No Trading) (Section 4.2): Adopts the same hyperparameters as Agora (Full Strategy), with the uncertainty trading mechanism disabled.
36

---

Published as conference paper ICLR 2026
KABB Selector + Trading (Section 4.2) KABB-VLM Adapter (Sections 4.3, 4.4): Utilizes a knowledge graph with depth 3 and branching factor 2, paired with UCBI where the exploration constant 1.0.
RL-based Selectors + Trading (Section 4.2, Appendix Y):
PPO: Learning rate 3e-4, clipping 0.2, GAE À 0.95, mini-batch size 64, epochs 10.
MCTS: Simulation count = 100, exploration constant Cp = v2.
A2C: Learning rate 7e-4, discount "RRL 0.99, entropy coefficient = 0.01, n-steps 5. DQN: Learning rate lc-4, discount YRr. 0.99, eDxx from 1.0 to 0.01 over 10,000 steps, target update every 1,000 steps, replay buffer size = 10,000.
Alternative Routing Strategies (Section 4.3, Appendix Z):
FrugalGPT-VLM: Cost threshold 0.5, accuracy estimator with smoothing factor 0.1 based on historical performance.
RouteLLM-V VLM:: Employs fine-tuned BERT (12 layers), trained for 5 epochs with learning rate 2e-5.
EmbedLLLMVLLM: Uses pre-trained ResNet-50 (images) and BERT (text), similarity thresh- old 0.7.
HybridLLLM-VLM: Switches based on task complexity, with a lightweight VLM (e.g..
MobileNeet-bssed) for simple tasks and dense VLM for complex ones.
MOA-VLM: Engages experts per query, aggregated via confidence-weiggtte voting.
H.3 MODEL INFERENCE PARAMETERS
For all Vision-Language Models (VLMs) within Agora's pool and external baselines or SOTA comparators:
AI Access: Models were interfaced via the OpenRouter API.
Decoding Strategy: Greedy decoding was enforced by setting do.sample-Fales or tempera- ture to 0.001 for consistency across models.
Maximum Tokens: 2048, chosen accommodate complex visual-linguistic outputs.
Other API Parameters: Default OpenRouter API settings were retained unless specified.
I RUNTIME ANALYSIS
this section, we investigate the computational efficiency of our proposed Agora framework, specifically focusing on the average inference time per question under varying configurations processing rounds. The experiments are conducted on the MMBench. Cectench.V111111 V11.Test dataset. The number of "rounds" can be concep- tualized as the depth of iterative refinement or the extent of collaborative exchange among agents within the Agora system for given query. A higher number of rounds typically implies more thorough pro- cessing, potentially leading to more accurate or robust responses, but the cost of increased computation time. Our objective is identify practical operational range that balances performance with acceptable latency, adhering a general guideline of keeping the average inference time per question below approxi- mately 30 seconds for interactive or time-sensitive applications. Experimental Setup The runtime analysis was performed on a system equipped with an NVIDIA A100 GPU. accessed via the OpenRouter API. For each configuration of rounds (1, 3, 3, 5, and 10 rounds), processed representative subset of 500 ques- tions from the MMBench. V11 Test dataset. The inference time for each question was measured from the moment the query dispatched to the Agora system until the final aggregated response generated.
37

---

Published as conference paper ICLR 2026
We report the average inference time per question. All VLM agents were called with greedy decoding ('do (to.sample=Faase).
1.1 RESULTS AND DISCUSSION
The average inference times per question for different numbers of processing rounds are presented in Table 8
Table 8:
Average inference time per question on MMBench_ Vi1_Th_ for varying numbers of processing rounds within the Agora framework. The aim to keep the average inference time below 30 seconds.
Number of Rounds
Average Inference Time per Question (s)
8.73
3
14.29
5
22.86
28.51
10
36.17
As illustrated in Table g there is a clear positive correlation between the number of processing rounds and the average inference time per question. With single round, the system achieves a rapid average time 8.73 seconds, suitable for highly time-critical scenarios where minimal processing is acceptable. As the number of rounds increases to and 5, the average inference time rises to 14.29 seconds and 22.86 seconds, respectively. These configurations represent good trade-off, allowing more sophisticated agent interaction and uncertainty trading while maintaining responsive performance.
When the system operates with rounds, the average inference time reaches 28.51 seconds, which is close to our desired maximum threshold of 30 seconds. This contiguration might be employed when higher accuracy is prioritized, and a slightly longer latency is permissible. However, increasing the rounds to 10 results an average inference time of 36.17 seconds, exceeding the 30-second guideline. This suggests that while more rounds can offer deeper processing, configurations beyond approximately 7-8 rounds may lead to latencies that are less suitable for real-time applications unless specific optimizations implemented or the task demands such intensive computation.
Based these results, the Agora framework demonstrates flexible approach managing computational resources. For most applications targeting balance between performance and efficiency, operating within 3 to 7 rounds appears optimal, ensuring that the average inference time per question remains largely within the 30-second target. Future work could explore adaptive mechanisms to dynamically adjust the number of rounds based on task complexity or specific latency requirements.
J PROMPT SETTING STATEMENT
This section outlines the prompt configurations for various agents within the Agora framework. Prompts are essential for guiding the behavior of Large Language Models (LLMs) serving as expert agents and aggregators. The examples provided here represent subset of the prompts used across all experiments in this paper. These prompts are designed to enhance task-specific reasoning, ensure structured outputs, and promote collaboration among agents. By incorporating Chain-of-Thought (CoT) reasoning, role definitions, and evidence-based responses, they improve interpretability, reduce hallucinations, and align outputs with multimodal benchmarks like MMBench, MVBench, and MMMU.
38

---

Published as conference paper ICLR 2026
To optimize effectiveness, prompts are modular: general persona prompts define agent roles, while task- specific templates incorporate dynamic placeholders (e.g., finstructiii ion] for queries and ( (image) for visual inputs). This modularity allows flexibility across datasets and models. Key design principles include:
Structured Reasoning: CoT steps encourage step-by-step analysis, reducing errors in complex visual tasks.
Evidence Requirement: Mandating citations from inputs promotes grounded, verifiable responses.
Role Specialization: Distinct roles prevent overlap and leverage agent strengths for comprehensive coverage.
Uncertainty Awareness: Implicitly guides agents to highlight ambiguities, aligning with Agora's
uncertainty trading.
These principles were refined through iterative testing, yielding improved accuracy (e.g., +1-8% bench- marks) and cost-efficiency by focusing agents on high-confidence domains.
J.1 GENERAL PROMPTS FOR EXPERT ROLES AND THE AGGREGATOR
This subsection provides examples of general persona prompts used to initialize experts and the aggregator. These define foundational behaviors and can be combined with task-specific instructions (c.g., from Sec- tion J.2. They emphasize adaptability, critical thinking, and relevance, enabling agents handle diverse queries while maintaining focus.
Illustrative Analysis Expert Persona
You an expert in problem analysis and logical reasoning, skilled in applying analytical frame- works and systematic thinking approaches. Your expertise includes breaking down complex prob- lems, identifying key factors, and recommending structured, actionable solutions. You familiar with various problem-solvine evaluation, and adapt your approach based on the unique context of each task. Consider how your skills in critical thinking, structured reasoning, and analytical problem-solving might provide valu- able insights or strategies for addressing the task at hand.
methods such root cause analysis, decision matrices, and scenario
This persona emphasizes decomposition and evidence-based methods, making it ideal for tasks requiring logical breakdown. It reduces ambiguity by encouraging adaptive strategies, which aligns with Agora's uncertainty quantification, leading to more reliable outputs in reasoning-hean benchmarks like MMMU.
Analysis:
Illustrative Strategy Expert Persona
You are a business strategy expert with a deep understanding of markets, business models, com- petitive landscapes, and strategic planning. Your expertise includes applying business frameworks, analytical tools, and market insights to identify opportunities and craft strategies. While capable of providing comprehensive strategic analysis, you adapt your input focus on what is most valu- able, practical, and relevant for the situation. Consider how your expertise in business innovation, competitive advantage, and strategic problem-solving might provide insightful and actionable rec- ommendations for any task.
Analysis: Focused practicality and innovation, this prompt suits Manning-orieente tasks. Its adaptive focus minimizes irrelevant details, enhancing efficiency in multi-agent setups and contributing to cost reduc- tions by prioritizing high-value insights.
39

---

Published as conference paper ICLR 2026
J.2 PROMPTS FOR VLM EXPERTS IN BENCHMARK EVALUATIONS
For experiments on MMBench, MVBench, and MMMU, we employed six Base Experts for initial anal- ysis, covering diverse multimodal aspects. Each uses a CoT prompt for structured reasoning, ensuring clarity and evidence- responses. Placeholders like (instruction) and {image) are filled dynam- ically. Experts are assigned models from the pool: gemini 2. 10--f,shh, qwen2. 5v1 7b 7b-insttuct, genma 3 27b, or opt-4o-mmt -4o-mini.
These prompts were optimized for visual-language tasks, incorporating evidence citation to mitigate biases and improve factual accuracy. Ablations showed that CoT elements boost performance by 2-5% on reasoning metries.
J.2.1 BASE EXPERT PROMPTS
The Base Experts generate detailed analyses via CoT, tailored to their roles for comprehensive coverage.
Object Recognition Expert
Role Definition:
You are an expert in object recognition, specializing identifying and describing objects within visual inputs.
Assigned Model: qwen2. Syv1-2bb-o 1111-22b-inszuee
Prompt Template (prompt.cm template);:
As an object recognition expert, your task is to identify and describe all significant objects in the provided image(s) in response to the question: (instruction] Follow this Chain-of-Thougeee process to ensure a thorough and accurate response:
1. Analyze the Image: Carefully examine the image(s) identify all visible objects. Consider their shapes, sizes, colors, and any distinguishing features. Note the number of objects if multiple instances are present. 2. List Objects: Create comprehensive list of all significant objects. For each object, specify: The object's 's name or category (c.g., "chair," "car","""arr",,. A brief description of its appearance (e.g., "red wooden chair with four legs"). Its approximate location in the image (e.g.. "center," "efp corner"). 3. Provide Evidence: For each object, cite specific visual evidence from the image that supports your identification (e.g., "The object has a rectangular shape and metallic texture, indicating it is a laptop"). 4. Address the Question: Ensure your response directly addresses the original question. answer. 5. Synthesize the Response: Combine your findings into clear, concise, and organized answer. Use bullet points or numbered list for clarity, ensuring all objects are covered.
the question specifies certain objects or details, prioritize those in your
Example Response Format: Object 1: [Name/Category] Description: [Appearance details] Lo- cation: [Position in image] Evidence: [Visual cues supporting identification] Object 2: [Name/- Category] Description: [Appearance details] Location: [Position in image] Evidence: [Visual cues supporting identification]
Provide your final answer based on the image(s) and the instruction: (instruction) Ensure your response is accurate, evidence-basse based, and directly relevant to the question.
Analysis: This prompt excels in perceptual tasks by enforeing detailed listings and evidence, reducing misidentifications.. It contributes to low uncertainty in object-heavy queries, improving overall system accu- racy by 3-4% on MMBench.
40

---

Published as conference paper ICLR 2026
Aggregator Prompt
You are the Wise Integrator a multi-agent system tasked with delivering accurate, coherent, and actionable responses user queries. Your role is
Understand the user's intent and main question(s) by carefully reviewing their query.
Evaluate expert inputs, preserving their quality opinions while ensuring relevance, accu- racy, and alignment with the user's s needs.
Resolve any contradictions or gaps logically, combining expert insights into single, unified response.
Synthesize the most appropriate information into a clear, actionable, and user-friendly an- swer.
Add your own insight if needed to enhance the final output.
Your response must prioritize clarity, accuracy, and usefulness, ensuring it directly addresses the user's needs while retaining the value of expert contributions. Avoid referencing the integration process or individual experts.
The aggregator resolves conflicts effectively, ensuring unified outputs. Its emphasis synthesis minimizes redundancy, enhancing efficiency in collaborative settings and reducing final epistemic uncer- tainty by up 10%.
Analysis:
Scene Description Expert
Role Definition: You are expert in scene description, specializing in providing comprehensive overviews visual environments.
Assigned Model: gemma 1ammma-3-11 --22bb
Prompt Template pttompptteeplateeee
As scene description expert, your task is to describe the overall scene depicted in the provided imago(s) in response to the question: (instruction) Follow this Chain-of- of-honof-Thhough process to ensure detailed and accurate response:
Analyze the Image: Observe the image(s) to understand the setting, including the location (e.g.. indoor, outdoor), environment (e.g., urban, natural), and overall atmosphere (e.g., calm, busy). 2. Identify Key Elements: Note the main components of the scene, such as: Physical setting (e.g., "a kitchen with white cabinets")). Lighting conditions (e.g., "bright daylight"). -Spttial relationships (e.g., "a table is centered with chairs around it"). Any notable objects people contributing to the scene's character. 3. Provide Evidence: For each key element, cite specific visual evidence from the image (e.g., "The presence of trees and grass suggests park setting").. 4. Address the Question: Ensure your description aligns with the original question. If the question asks for specific aspects (e.g., mood, setting), emphasize those in your response. 5. Synthesize the Response: Combine your observations into a cohesive narrative or structured description. Use clear, descriptive language paint vivid picture of the scene.
Example Response Format: Setting: [Description of location and environment] Evidence: [Vi- sual cues supporting the setting] Lighting and Atmosphere: [Description of lighting and mood] Evidence: [Visual cues supporting the atmosphere] Spatial Relationships: [Description of objec- t/person placement] Evidence: [Visual cues supporting spatial observations]
Provide your final answer based on the image(s) and the instruction: {instruct ion]. Ensure your response is comprehensive, wridence-based and directly relevant the question.
41

---

Published as conference paper ICLR 2026
Analysis:
This prompt provides holistic scene overviews, capturing atmosphere and relationships. It aids in contextual tasks, reducing semantic uncertainty and boosting performance on descriptive benchmarks like MVBench by integrating spatial evidence.
Logical Reasoning Expert
Role Definition: You are an expert logical reasoning, specializing in deriving conclusions from visual and textual inputs.
Assigned Model: gemini O-flash
Prompt Template (prompt.c (prompt.templatee template):
As logical reasoning expert, your task is to analyze the provided image(s) and associated text to derive logical conclusions or solve reasoning tasks in response the question: {instruction]. Follow this Chain-of- Chain-of-Thought process to ensure clear and logical response:
Analyze Inputs: Review the image(s) and any accompanying text to identify relevant information, such as objects, relationships, or textual cues. 2. Break Down the Question: Understand the specific reasoning task (c.g.. deduction, inference, comparison). Identify what the question asking and any constraints. 3. Reason Step-by-Step: List all relevant observations from the image(s) and text (e.g.. "The image shows a red ball on the left and blue ball on the right"). Formulate logical steps address the question (e.g.. "If the red ball is heavier, then... "). Cite visual textual evidence for each step (e.g.. "The text states 'the red ball is heavier," supporting this inference"). 4. Check for Errors: Verify that your reasoning is consistent and free of assumptions supported by the inputs. 5. Synthesize the Response: Present your conclusion clearly, summarizing the reasoning steps and final answer concise format.
Example Response Format: Observation: [Key visual/textual evidence] Step 1: [First reasoning
step with evidence] Step 2: [Second reasoning step with evidence] Conclusion: [Final answer the question]
Provide your final answer based on the image(s), text, the instruction: (instruction) En- sure your response is logical, siddence-based, and directly addresses the question.
By enforcing step-by-step logic and error-checking, this prompt excels in inference tasks, min- imizing inconsistencies. lowers inferential uncertainty, contributing to higher accuracy on logic-based datasets like MMMU.
Analysis:
Contextual Analysis Expert
You are an expert in contextual analysis, specializing in interpreting the broader context of visual scenes.
Role Definition:
Assigned Model: gemma-3-27b
Prompt Template (prompt template):
As contextual analysis expert, your task to interpret the broader context of the scene depicted in the provided image(s) s) in response to the question: { nss ruct i ion). Follow this Chain-of Chain-of-Thooghhe Thouuhh process to ensure an insightful and accurate response:
1. Analyze the Image: Examine the image(s) to identify elements that suggest cultural, situational, or historical context (e.g., clothing, architecture, activities). Identify Contextual Cues: Note specific features that indicate the scene's significance, such Cultural indicators (e.g.. traditional
attire suggesting a festival). Situational context (e.g., a crowded setting implying a public event).
Historical temporal clues (e.g., old-fashionel vehicles suggesting past era). 3. Provide Evi- dence: For each contextual insight, cite specific visual evidence from the image (c.g., "The presence of banner with text suggests a community event"). 4. Address the Question: Ensure your anal-
42

---

Published as conference paper ICLR 2026
ysis aligns with the original question. If the question specifies a particular context (e.g., cultural significance), focus that aspect. 5. Synthesize the Response: Combine your insights into clear, cohesive explanation of the scene's 's context, emphasizing its broader implications.
Example Response Format: Contextual Insight 1: [Cultural/sttuational observation] Evidence: [Visual cues supporting the insight] Contextual Insight 2: [Historical/temporal observation] Ey-
idence: [Visual cues supporting the insight] Summary: [Overall interpretation of the scene's text]
Provide your final answer based on the image(s) and the instruction: {instruction): Ensure
your response is insightful, evidence-based, and directly relevant to the question.
This prompt uncovers broader implications like cultural cues, enriching interpretations. ad- dresses semantic gaps, reducing overall uncertainty and enhancing performance context-dependent tasks.
Analysis:
Attribute Analysis Expert
You are an expert in analyzing visual attributes, specializing colors, textures, and shapes.
Role Definition:
Assigned Model: qwen2. 5v1-7b-instruet
Prompt Template (prompt template):
As an attribute analysis expert, your task is to describe the dominant colors, textures, and shapes in the provided image(s) in response to the question: {instruction) Follow this Chain-of-Thonn Thought process to ensure a detailed and accurate response:
1. Analyze the Image: Carefully examine the image(s) to identify prominent visual attributes, focusing on colors, textures, and shapes of objects and backgrounds. Catalog Attributes:
Colors: List the dominant colors (e.g., "bright red," "muted green") and their distribution (e.g., "red on the central object"). Textures: Describe textures (e.g., "smooth," "rough") and where they appear (e.g., "rough texture the tree bark"). Shapes: Identify shapes (e.g., "circular," "rectangular") and their context (e.g.. "circular table in the center"). 3. Provide Evidence: For each attribute, cite specific visual evidence (e.g., "The object's 'ss glossy finish reflects light, indicating a smooth texture"). 4. Address the Question: Ensure your analysis addresses the original question. the question focuses on specific attributes, prioritize those. Synthesize the Response: Combine your findings into clear, organized description, using lists paragraphs to highlight each attribute category.
Example Response Format: Colors: [Dominant colors and distribution] Evidence: [Visual cues
supporting color observations] Textures: [Dominant textures and locations] Evidence: [Visual
cues supporting texture observations] Shapes: [Dominant shapes and contexts] Evidence: [Visual cues supporting shape observations]
Provide your final answer based the image(s) and the instruction: (instruction) Ensure your response is detailed, evidence-based, and directly relevant to the question.
Analysis: Focusing on fine- ane-ggrined attributes, this prompt supports detailed visual breakdowns. minimizes perceptual errors, aiding in uncertainty reduction for serribute-based queries.
Action Inference Expert
Role Definition: You expert in inferring actions events from visual cues. Assigned Model: gpt-4o-mini
Prompt Template (prompt template):
43

---

Published as a conference paper ICLR 2026
As an action inference expert, your task to identify and describe any actions or events depicted in the provided image(s) in response to the question:
Follow this Chain-of- Thouught
process to ensure a clear and accurate response:
1. Analyze the Image: Examine the image(s) to identify dynamic elements suggesting actions events, such as moving objects, people's postures, or environmental changes. 2. Identify Action- s/Events: List the inferred actions events, considering: What is happening (e.g., "a person is running"). Who or what is involved (e.g., "a dog chasing a ball"). The context of the action (e.g., "in park during daytime"). 3. Provide Evidence: For each action event, cite specific visual evidence (c.g., "The person's bent knees and forward lean suggest running"), 4. Address the Question: Ensure your response aligns with the original question. If the question specifies certain actions or events, focus on those. 5. Synthesize the Response: Combine your findings into a clear, concise description of the actions or events, emphasizing the sequence and context.
Example Response Format: Action/Event I: [Description of the action/event]- Involved Entities: [Who/what is involved] Context: [Setting or circumstances] Evidence: [Visual cues supporting the inference]- Action/Event 2: [Description of the action/event] Involved Entities: [Who/what is involved] Context: [Setting or circumstances] - Evidence: [Visual cues supporting the inference] Provide your final answer based the image(s) and the instruction: finstruction). Ensure your response is accurate, evidence-basee and directly relevant to the question.
Analysis:
This prompt infers dynamics from static images, capturing events effectively. It handles inferential uncertainty well, improving reliability in action- criinnorieettd tasks.
It important to note that these textual prompts form the core instructions. The effectiveness of these prompts can also be influenced by the specific capabilities of the underlying base VLM, its training data, and any additional system-level instructions few-shot examples that might be used in a complete imple- mentation.
K HYPERPARAMETERS USED IN THE EXPERIMENTS
This section outlines the hyperparameter configurations employed for the Agora framework, comparative models, and general model inference across the experiments detailed in this paper. Unless otherwise noted, these settings were applied consistently throughout.
K.1 AGORA FRAMEWORK PARAMETERS
The Agora framework's agent selection strategy (Section 3.5) leverages extended Thompson Sampling (TS) mechanism. The score for selecting agent S at time is defined as:
g [Rewaard (0) Cost (2)
exp(- - Dist(S, A Synergy(S)" Usunaegic (S)
g
The hyperparameters for the "Agora (Full Strategy)" configuration, validated in Table 3 and Appendix G are:
Time Decay Base (7)): 0.99, as reported in Table 7 (denoted there as Aai).
Task Match Weight (A): 0.2, empirically optimized to balance task relevance and exploration, with its impact evidenced by the ablation "w/o Task Match (Dist)" in Table 3
Synergy Exponent (7)): 0.8, tuned to modulate the influence of agent synergies, as demonstrated by the ablation "w/o Synergy (Synergy)) in Table 3
44

---

Published as conference paper ICLR 2026
Strategic Uncertainty Exponent (w): 1.2, adjusted to emphasize strategic uncertainty, with its role highlighted by the ablation "w/o Strategic Uncertainty (U (Ustrategie"" in Table
Thompson Sampling Priors: For each agent S. the Beta posterior parameters (as; Bs) were
initialized to (1,1), reflecting uniform prior over success and failure.
Additional parameters for the Agora framework include:
Multi-dimensionn Uncertainty Weights (Wpere: Usem, Wint): Set to 0.4, 0.3, and 0.3, respectively, reflecting slight emphasis perceptual uncertainty. determined through cross-validation
Task Similarity Threshold (Tsim): 0.75, based on a normalized cosine similarity scale (0-1), opti-
mized for task clustering efficiency.
Uncertainty Trading Trigger Threshold (Ftrade): 0.15, calibrated to initiate trading when uncer- tainty differences exceed this normalized bound.
Trade Benefit Threshold (Teemttt:: 0.08, set to ensure trades yield meaningful cost reductions, validated via simulation.
Note: Ablation studies in Appendix G evaluated alternative MAB strategies, including UCB1 with an ex- ploration constant C' 1.0 (Table 5 and learning rate 0.1 (Table (6. These pertain exploratory variants, whereas the primary Agora configuration relies on Thompson Sampling.
K.2 HYPERPARAMETERS FOR COMPARATIVE MODELS AND STRATEGIES
For comparative experiments Sections 4.2 and 4.3, alternative strategies were adapted to the VLM context, utilizing the same base VLM agent pool as Agora where applicable. Hyperparameters were derived from original formulations, standard practices, or task- specific tuning.
Agora (No Trading) (Section 4.2): Adopts the same hyperparameters as Agora (Full Strategy), with the uncertainty trading mechanism disabled.
KABB Selector + Trading (Section 4.2) / KABB-VLM Adapter (Sections 4.3, 4.4): Utilizes a knowledge graph with depth 3 and branching factor 2, paired with UCBI where the exploration constant C = 1.0.
RL-based Selectors + Trading (Section 4.2, Appendix Y):
PPO: Learning rate 3e-4, clipping 0.2, GAE X 0.95, mini-batch size 64, epochs 10.
MCTS: Simulation count = 100, exploration constant C, = V2.
A2C: Learning rate = 7e-4, discount "RL = 0.99, entropy coefficient = 0.01, n-steps = 5. DQN: Learning rate = Ic-4, discount RRL = 0.99, EDQN from 1.0 to 0.01 over 10,000 steps, target update every 1,000 steps, replay buffer size 10,000.
Alternative Routing Strategies (Section 4.3, Appendix Z):
FrugalGPT-VLM.: Cost threshold 0.5, accuracy estimator with smoothing factor 0.1 based on historical performance.
MouteLLM-VLM. Employs fine-tuned BERT (12 layers), trained for 5 epochs with learning rate 2c-5.
LmmeeLLLM-VLM: Uses pre-trained ResNet-50 (images) and BERT (text), similarity thresh-
old = 0.7.
HybridI.LM-VIIM: Switches based on task complexity, with a lightweight VLM (e.g.
MobileNet-based for simple tasks and a dense VLM for complex ones.
MOAAVLMM Engages 3 experts per query, aggregated via confidence- confidence-weighted voting.
45

---

Sec. 2.1 Published as conference paper ICLR 2026
K.3
MODEL INFERENCE PARAMETERS
For all Vision-Language Models (VLMs) within Agora's pool and external baselines or SOTA comparators:
API Access: Models were interfaced via the OpenRouter API.
Decoding Strategy: Greedy decoding was enforced by setting do_sample=False or tempera- ture to 0.001 for consistency across models.
Maximum Tokens: 2048, chosen to accommodate complex visual-linguistic outputs.
Other API Parameters: Default OpenRouter API settings were retained unless specified.
L CLARIFICATIONS ON METHODOLOGICAL COMPONENTS
This appendix provides detailed clarifications on the core components of the Agora framework. is intended address feedback regarding the clarity of key definitions and mechanisms, ensuring that the foundational concepts of our work are presented transparently and rigorously. We systematically elaborate on the defini- tions of variables, the method for uncertainty estimation, and the interpretation of specific elements within our architectural diagrams.
L.I DEFINITIONS OF CORE VARIABLES AND FUNCTIONS
To provide centralized reference, the table below summarizes the core mathematical and conceptual vari- ables used throughout the paper.
Table 9: Definitions of core variables and functions.
Symbol
Definition
Reference
A,
The set of heterogeneous VLM agents, and the i-th agent within that set.
Sec. 2.1
The marginal processing cost for agent a, to handle one unit of uncertainty.
Sec. 2.1
Ei
The expertise vector of agent at quantifying its efficiency in resolving different types of uncertainty (perceptual, se- mantic, inferential).
u(t)
The initial multi-dimmessonnl epistemic uncertainty vector [upere: Usem: What] for given task t. The allocation policy that routes uncertainty components to different agents.
Sec. 2.1
Sec. 2.1
C(-)
The total system cost function, which is the objective to be minimized our core optimization problem.
Sec. 2.1, Eq. (1)
AC
The change in total system cost resulting from single un- certainty trade between two agents. The derivation is de- tailed in the main text.
Sec. 3.2, Eq. (4)
The market-aware expected utility function used by the
Sec. 3.3, Eq. (6)
Broker to select an initial agent for collaboration.
The full mathematical models and implementation details for these components are provided their respec- tive sections, with comprehensive derivations located in Appendix C.
46

---

Published as a conference paper ICLR 2026
L.2 UNCERTAINTY QUANTIFICATION AND ESTIMATION
A crucial aspect of our framework is the method by which cognitive uncertainty is estimated and quantified. We do not treat uncertainty as a monolithic scalar but decompose it into a structured, multi-dimensional asset. The estimation process for each dimension is as follows, with full mathematical formalizations available in Appendix C.I:
Perceptual Uncertainty (Upprre:: This dimension quantifies the model's confidence in recognizing raw visual signals (c.g., object categories, attributes). It is estimated by statistically analyzing the model's raw outputs. For instance, for object classification task, Vppen be calculated using the Shannon Entropy of the predicted probability distribution over possible object classes. A higher entropy signifies greater uncertainty about what is being perceived.
Semantic Uncertainty (Usem): This dimension measures ambiguity in understanding the deeper meaning of a scene, including the relationships between objects and their context. It is estimated by quantifying the multiplicity of plausible interpretations. For example, if model identifies several valid potential relationships between two objects in an image, the semantic uncertainty is consid- ered higher.
Inferential Uncertainty (Vint): This dimension assesses the model's confidence in making pre- dictions drawing conclusions based the available information. Its estimation combines two factors: (1) the confidence in the single most likely outcome (i.e., max(p)), and (2) the overall dispersion of the entire predictive probability distribution (i.e., its entropy). This captures both the model's 's conviction in its top guess and its certainty across all possibilities.
L.3 EXPLANATION OF KEY ELEMENTS IN FIGURE
We clarify two components from the architectural diagram in Figure 3 (page 4) that were previously am- biguous: the "Value Model" and the "Too Many?" label.
The "Value Model":
This component serves as the reward signal generator for our Multi-Armed Bandit (MAB) agent selection mechanism. After the selected agents produce a final "Output Text," the Value Model evaluates the quality of this output (e.g., by comparing it against ground-truth answer or using a pretrained reward model). The result of this evaluation is a quantitative reward signal (labeled "Policy Selection Reward") that is fed back to the MAB. This reward is essential for the MAB's ' learning process, allowing it to update its policy (per the Thompson Sampling update rule in Appendix A) and improve its ability select high-performing agents in the future.
This label in the "Uncertainty Evaluation Center" is visual repre- sentation of the crucial receiver capacity constraint check within our trading protocol. Before an uncertainty trade is executed, the system must verify that the receiving agent will not be overloaded. This corresponds directly to the feasibility condition in Equation (5): (t( + (i)(t) S (t). The "Too Many?" check ensures that proposed trade is rejected if accepting the new uncertainty packet Tip (t) would push the receiving agent's 's total uncertainty portfolio U₃ beyond its operational ca- pacity limit (t). is a fundamental admission control mechanism that maintains system stability and agent effectiveness.
The "Too Many?" Label:
M CASE ANALYSIS
In this section, we present a series of case studies, including two successful and two unsuccessful examples, demonstrate how multiple experts collaboratively analyze images in response corresponding questions.
47

---

Published as a conference paper ICLR 2026
The expert configuration comprises three analysis experts: an Object Recognition Expert, Scene Descrip- tion Expert, and Text/OCR Analysis Expert. For each expert, we report both their analytical response and the associated uncertainty score. To improve clarity and conciseness, especially given the length of the responses, key excerpts are highlighted using colored underlines.
M.1 SUCCESSFUL CASE
tables [0and] illustrate that our method, by assigning clearly defined roles to each expert-nnmell object enables comprehensive analysis across multiple modalities and semantic dimensions of the input. This structured task decomposition enhances both the depth and breadth of information processing, allowing each analysis expert to specialize in a distinct sub- task and generate high-quality outputs accompanied by uncertainty estimates. A principal advantage of this approach is its explicit quantification of uncertainty, which allows the system to weigh and prioritize expert contributions based on their reliability. This multi-expert architecture markedly improves the system's s performance with respect to factual accuracy, contextual completeness, and logical consistency, thereby enhancing robustness, interpretability, and overall stability across diverse question types and input formats.
recognition, scene understanding, and text/OCR analysis eealyles-enaales
M.2 UNSUCCESSFUL CASE
Although our multi-expert analytical framework performs effectively and robustly in the majority of cases, certain challenges and limitations remain.table [2] exemplifies a limitation of our multi-expert analytical framework when faced with ambiguous spatial and perceptual cues that demand deeper three-dimensional reasoning and contextual inference beyond straightforward visual and textual recognition. While the Object Recognition Expert identifies two square sliders labeled "A" and suggests visual similarity in size, and the Scene Description Expert emphasizes the schematic nature of the diagrams featuring differing inclined plane shapes, the Text/OCR Analysis Expert rightly notes that size equivalence cannot be confirmed solely based on the visual and textual evidence.
Despite this inherent uncertainty, the final system output incorrectly asserts that the sliders are the same size. This exposes a critical shortcoming: the current framework lacks an advanced spatial reasoning mod- ule capable of integrating geometric perspective and resolving scale ambiguities inherent in 2D schematic depictions of 3D objects. Furthermore, the system does not adequately leverage uncertainty quantification withhold or qualify conclusions in cases of inconclusive or conflicting evidence.
This failure highlights the framework's 's overreliance on superficial visual similarity and label matching with- out robust geometric or physical reasoning. Consequently, it underscores the necessity of integrating more sophisticated reasoning components such as 3D shape reconstruction, perspective analysis, or probabilis- tie inference over spatial configurations- to accurately assess relative object dimensions when explicit size information is unavailable.
summary, this case illustrates that while the multi-expert system effectively parses and analyzes mul- timodal inputs, it remains limited in resolving ambiguities that require complex spatial cognition. Future work should focus on incorporating specialized reasoning capabilities to enhance accuracy in tasks involv- ing comparative spatial judgments under uncertain visual conditions.
Table 13 presents another failure case that reveals a fundamental limitation of our multi-expert framework: the difficulty in performing reliable cross-modal reasoning when critical semantic associations cannot be directly inferred from visual cues. Although the Object Recognition, Scene Description, and Tex//OR Analysis Experts accurately identify visual features (e.g., yellow cylindrical containers), contextual settings (e.g.. industrial storage area), and textual labels (e.g., liquid ammonia), the final output mistakenly claims that the object "has boiling point of C3".
48

---

Published as conference paper ICLR 2026
This error reflects key deficiency: the current analysis experts lack the domain knowledge and reason- ing mechanisms necessary to associate recognized objects (e.g., liquid ammonia) with their scientifically accurate properties. While C33.3%C roughly corresponds the boiling point of gaseous ammonia, the pre- cise boiling point of liquid ammonia is -33,42%C, More importantly, this physical property is not visually inferable from the image nor present any text extracted by the OCR expert.
This case underscores broader design limitation: the system assumes all factually relevant attributes can be inferred solely from image and text imputs, without access to external scientific knowledge bases or ver- ification mechanisms. Consequently, it fails to differentiate between superficially plausible yet unsupported assertions and those grounded in the image evidence.
In conclusion, this example highlights the critical need to incorporate knowledge-grounded reasoning mod- ules or external factual verification layers to bridge the gap between perceptual analysis and fact-based inference. This is particularly vital for tasks involving domain-specific scientific knowledge, where even minor factual inaccuracies can substantially undermine the system's 's credibility and reliability.
STATEMENT ON THE USE OF AI ASSISTANCE
the preparation of this manuscript, we employed a Large Language Model (LLM) as research and writing assistant. The use of the LLM was restricted to specific areas: (1) aiding in the initial phase of academic research by helping to survey and summarize relevant literature, and (2) assisting in the post-writing phase by polishing the manuscript's language, grammar, and formatting improve clarity and readability.
49

---

Published as conference paper ICLR 2026
Table 10: Case: Successful Expert Collaboration Example
Image
Major routes of the Medieval Silk Road
EUROPE
CENTRAL ASLA
EAST ASIA
MIDDLE EAST
SOUTH ASIA
AFRICA
overland routes the Silk Road oversea routes the Silk Road
5500 kilometers 1500 miles
Question
Based on the map, what was true about the Silk Road around the year 1300 CE?
Expert
Uncertainty
Output
Object Recognition
0.1365
1.The image depicts solid blue lines and dashed pink lines, representing the overland Silk Road routes and maritime Silk Road routes, respectively.
2.A high concentration of solid blue lines is observed
Central Asia, indicating its role major hub for overland trade.
3.Dashed pink lines extend from eastern coast Africa through the Middle East to East Asia, clearly il- lustrating key marilime trade corridors.
Scene Description
0.1096
The map covers extensive geographical area, in- cluding Hurope, Central Asia, the Middle East, South Asia, East Asia, and parts of Africa,
2.Central Asia depicted as pivotal intersection
where multiple routes converge, highlighting its strategic position within the Silk Road network.
3.The Middle East functions as a critical junction
connecting overland and maritime routes, serving as key intermediary between East and West.
4.The map employs simple and coberent color scheme and layour, facilitating clear interpretation of regional locations and route distributions,
Tex/OCR Analysis
0.0832
1. The map includes clearly legible labels for major re- gions such as Europe, Central Asia, the Middle Bast, South Asia, and East Asia.
50
2.The legend explicitly defines the solid blue lines and dashed pink lines as distinct types trade routes. 3.The overall textual arrangement well-organized, enhancing comprehension and efficient extraction of information from the map.
Answer
The correct answer is: The Silk Road connected parts of East Asia, the Middle East, and Europe.

---

Published as conference paper ICLR 2026
Table 11: Case: Successful Expert Collaboration Example 2
Image
Question
What will happen next?
Expert
Uncertainty
Output
Object Recognition
0.8854
1.The image shows the child's expression suggesting sadness distress, characterized by a downtumed mouth and teary eyes, which strongly signals distress.
2. Although predicting future events from still image speculative, the child's emotional state significant indicator.
Scene Description
0.8945
1.The child's somber expression combined with the stark black backeround highlights emotional moment and accentuates the child's 's emotional vulnerability. 2.It possible that the child might he comforted, poten- tially leading to interaction or conversation.
Tex//CCR Analysis
0.2454
1 I.oo supplementary contextual text is provided, keeping the focus the child's emotional expression.
Answer
Based the child's visible sadness, the most likely immediate reaction is that the child might cry.
51

---

Published as conference paper ICLR 2026
Table 12: Case: Unsuccessful Expert Collaboration Example
Image
A
A
Question
There are two physical models in the picture, are the two square sliders the same size?
Expert
Uncertainty
Output
Object Recognition
0.1479
1. The image contains two physical models, each with a square slider labeled "A". 2.The visual evidence indicates the sliders are identical in size, occupying similar proportions their inclined planes.
Scene Description
0.1744
1. The image shows two models featuring square sliders placed near the top of their respective inclined planes cylindrical, triangular.
2.The drawing is schematic with clear geometric shapes and detailed environment lighting, focusing spa- tial relationships.
Text/OCR Analysis
0.5382
1. Each diagram has square labeled "A," indicating the same object type. However, due to differing shapes and perspective, it is possible confirm the sliders' sizes solely from the image.
2.The visual text evidence does conclusively verify identical sizes.
Answer
The most reasonable conclusion is that the sliders are the same size.
52

---

Published as a conference paper ICLR 2026
Table 13: Case: Unsuccessful Expert Collaboration Example 2
Image
Question
The object is shown in this figure:
Expert
Uncertainty
Output
Object Recognition
0.2325
The image depicts multiple yellow cylindrical gas cylinders arranged in row, exhibiting regular shapes with rounded ends.
2.The cylinders aligned horizontally with orderly spacing and consistent color and shape, indicating industrial gas containers.
Scene Description
0.5276
1.The cylinders are positioned on concrete floor, suggesting industrial storage transportation environment.
2.Additional similar containers are visible in the back- ground, emphasizing storage loading area. 3.Lighting is uniform and bright, accentuating yellow cylinders and black inscriptions.
4.The cylinders are arranged neatly with even spacing, reflecting organizational order and safety considerations, 5.The overall atmosphere utilitarian, highlighting the practical and safety-focused nature of the industrial setting.
Text/ORR Analysis
0.0832
1.The side of the cylinders clearly displays bold black lettering indicating "liquid ammonia" in Chinese, which stands out against the yellow background and ensures high legibility.
2.The text's placement and size precise, indicating the
53 contents liquid ammonia and conforming to industrial labeling standards,
Answer
The object shown this figure has boiling point -33.3%C.


Reviews:
[{"review_id": "aj5Eea9564", "reviewer": "Reviewer_oqqC", "summary": "This paper introduces Agora, a novel market-based framework for coordinating heterogeneous Vision-Language Models (VLMs) in multi-agent systems. The key idea is to treat epistemic uncertainty as a tradable asset across perceptual, semantic, and inferential dimensions. Agents trade uncertainty based on a profitability-driven protocol, guided by a market-aware Thompson Sampling broker, to minimize system-wide cost while maintaining task performance. The authors provide both theoretical and empirical evidence that Agora outperforms strong baselines (e.g., Mixture-of-Agents, KABB) in terms of accuracy (+8.5% on MMMU) and cost-efficiency (3× reduction).", "questions": "- How would Agora perform in the presence of adversarial or strategic agents that intentionally misreport uncertainty or cost to exploit the market? For instance, how does the framework handle strategic manipulation or collusion that could distort trading dynamics or market equilibrium?\n- Could you clarify the computational complexity of the broker and trading protocol, particularly their overhead within a more realistic analysis?\n- Would Agora still work if agents have overlapping expertise or non-stationary cost profiles?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The core concept of a decentralized market for uncertainty is a powerful and novel metaphor that drives the entire paper.\n- The work tackles a foundational challenge (economic scalability) with a comprehensive solution that includes theoretical formulation, algorithm design, and empirical validation.\n- The framework demonstrates consistent and sometimes substantial gains in accuracy and cost-efficiency across multiple benchmarks.", "weaknesses": "-  The framework introduces significant complexity with its multi-dimensional uncertainty quantification, trading ledger, and sophisticated broker. While the paper claims scalability, the experiments are limited to ≤15 agents. It’s unclear how the market mechanism would behave with hundreds of agents or high-frequency trading.\n- The paper overlooks key transaction costs (coordination latency, broker computation, and communication overhead), which are excluded from its cost metrics. A system that is 3x cheaper in API calls but 10x slower due to coordination may not be viable for real-time applications.\n-  While numerous baselines are included, the adaptation of language-model-focused routers (like FrugalGPT, RouteLLM) to the multi-modal, multi-agent setting is only briefly mentioned in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ykBTO2epun", "reviewer": "Reviewer_iVUH", "summary": "This paper introduces Agora, an framework for efficient coordination among VLM agents. It conceptualizes epistemic uncertainty as a tradable asset which is decomposed into perceptual, semantic, and inferential components. The authors also design a profitability-driven trading protocol where agents exchange uncertainty based on rational market rules. The system employs a market-aware broker extending Thompson Sampling method to initiate and manage trades, aiming to achieve cost-efficient equilibria. Experiments on 5 multimodal benchmarks show performance gains about 8.5% accuracy while reducing cost by over 3-times compared to SOTA multi-agent and routing strategies.", "questions": "1. How sensitive is Agora’s performance to the parameters in Eq. 6? \n2. Can Agora handle non-stationary uncertainty distributions, for example, what if agents adaptively reprice uncertainty in evolving tasks?\n3. Since the framework could converge to suboptimal equilibria, are we able to observe such phenomenons in experiments? Or are there any useful heuristics to help escape local minima?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I'm not an expert in the field of VLM, but I appreciate the idea of treating uncertainty as a tradable commodity, which is novel seems to me.  And the introduced Broker algorithm that operationalizes the market through Thompson Sampling looks solid. The empirical study is reasonably sufficient and supportive. And the writing is easy to follow in general.", "weaknesses": "1. While the authors prove the inefficiency of heuristic coordination, the optimality guarantees of Agora’s decentralized equilibrium are not clearly established. Is the equilibrium globally optimal or locally optimal? More formal convergence or even a broader discussion would be appreciated.\n2. The framework are limited to visual-linguistic benchmarks, and I would be curious to see if Agora scales to pure language, control, or multi-round dialogue tasks. Can you include some discussion in this regard? If it cannot be easily adapted to other domains, what could be the reason?\n3. What is exactly the theoretical result regarding the regret bound in Appendix C.4.1? I get lost there and did not identify any concrete result.\n\nMinor issues and typos:\n1. P2: “Gale & and, 1962”\n2. Duplicated references of the paper 'Are more llm calls all you need? towards scaling laws of compound inference systems'.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O0Ye3JQTh9", "reviewer": "Reviewer_d1Fk", "summary": "This paper proposes Agora, a market‑based way to coordinate multi‑agent VLM systems. Instead of heuristic routing or consensus, Agora uses epistemic uncertainty in a tradable asset with three dimensions—perceptual, semantic, and inferential—and lets agents “trade” packets of uncertainty to reduce total cost. A market‑aware broker (an extension of Thompson Sampling) initializes collaboration; trades are executed only if the cost delta ΔC is negative (Eq. 4), yielding a locally cost‑efficient equilibrium (Algorithm 1). Authors use extensive experiments to demonstrate the Agora's effectiveness in comparison with single LLMs, SOTA, and other benchmarks.", "questions": "1. How do you do cross‑model calibration or normalization for Entropy/dispersion‑based scores?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper formalizes why common coordinators are cost‑agnostic and uncertainty‑structure‑agnostic (Theorem 1), then builds a mechanism that is explicitly cost‑aware and structure‑aware (Eq. 4; uncertainty portfolio on p. 5). Fig. 2 (p. 3) empirically shows lower final epistemic uncertainty.\n1. The trade condition ΔC<0 with capacity constraints is intuitive and easy to implement\n1. The paper measures COI, Ufinal, and UAPS (uncertainty‑aware performance), offering a fuller view of coordination quality", "weaknesses": "1. The market phase performs greedy descent and is guaranteed to reach only a local equilibrium (Algorithm 1). The bandit section explicitly notes that classic regret bounds are hard and provides only directional convergence intuition (Appendix C.4.1), leaving global‑optimality and regret guarantees open.\n1. The broker’s utility (Eq. 6) multiplies several factors (distance, synergy, strategic uncertainty, decay). Ablations (Table 3) show sensitivity (e.g., removing Ustrategic costs ~3% accuracy), suggesting tuning burden and potential dataset‑specific overfitting.\n1. Entropy/dispersion‑based scores (and their mapping to Uperc/Usem/Uinf) can be model‑calibration–dependent; without shared calibration or normalization, one model’s probabilities may look more or less “uncertain” than another’s even on identical behavior. The paper formalizes the decomposition but does not detail cross‑model calibration procedures.\n2. Total uncertainty fuses dimensions with fixed weights (e.g., wperc=0.4, wsem=0.3, winf=0.3); if those weights were tuned on setups where the proposed method reduces the dominant dimension, Ufinal_epis could favor it. (Weights are reported as defaults; sensitivity within Agora is ablated, but cross‑method sensitivity is not shown.)\n1. In the main body, head‑to‑head comparisons with alternative routing/multi‑agent strategies are only shown on MMBench V11 Test (Figure 4). For the other datasets (MMMU, MathVision, InfoVQA, CC‑OCR), the main table contrasts Agora vs. single models / external SOTAs, not against routing/MAS baselines—so readers can’t verify cross‑dataset superiority of the coordination strategy from the main text alone.\n1. The writing should be enhanced, especially regarding some details and accuracy. For example, the color description in Figure 1 is wrong. The Mathvision gap in Table 1 is wrong. The readability of Figure 5 is poor (the font could be adjusted to make it easier to read).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hLvDOYr9hy", "reviewer": "Reviewer_VAy3", "summary": "This paper introduces \"Agora,\" a novel framework for multi-agent coordination designed to address the high cost and inefficiency of multi-agent systems (MAS) built with VLMs.\n\nThe authors' core argument is that existing coordination paradigms, such as Mixture-of-Agents (MoA) or knowledge-based routers like KABB, are \"Cost-Agnostic\" and \"Uncertainty-Structure-Agnostic.\" They rely on heuristic proxies (e.g., consensus, semantic similarity) for decision-making, which leads to provably suboptimal coordination and spiraling economic costs.\n\nTo solve this, the Agora framework reframes coordination as a \"decentralized market for uncertainty.\" Its core mechanisms include: Assetization, trading and brokerage.\n\nComprehensive experiments are conducted on five multimodal benchmarks. The results demonstrate that Agora significantly outperforms strong VLM baselines and heuristic multi-agent strategies in both accuracy and cost-effectiveness. For example, on MMMU, Agora (79.2%) achieves a +8.5% absolute accuracy gain over the best baseline in its agent pool, while reducing costs by over 3 times.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Highly Novel and Well-Grounded Problem Formulation:** The paper's main strength is its core idea: \"market-based uncertainty trading.\" This provides a principled, economic alternative to heuristic routing in MAS. The theoretical formulation, which defines \"Agnostic Coordination\" and proves the \"Inefficiency Theorem for Agnostic Coordination\", provides an exceptionally strong motivation and foundation for the work.\n\n**Sound and Constructive Methodology:** The Agora framework is well-designed. The decomposition of uncertainty into three types (perceptual, semantic, inferential)  is a key, intuitive design choice that enables fine-grained cost management. The \"profitability-driven\" trading protocol directly and elegantly optimizes the core economic objective.\n\n**Extremely Comprehensive and Rigorous Experiments:** The experimental validation is a significant strength.", "weaknesses": "**Performance Claims:** Thank authors for the comparison in Table 1 against gemini-2.5-pro (81.7%). Does this imply that Agora's primary value is \"cost-saving\" (achieving near-SOTA with a non-SOTA pool) rather than \"SOTA-beating\"? If gemini-2.5-pro were included in Agora's agent pool, would you expect the Agora score to surpass 81.7%?\n\n**Uncertainty Quantification:** I found the uncertainty vector interesting. \n\n1. Could you provide a concrete example of how the uncertainty vector $[u_{perc}, u_{sem}, u_{inf}]$ is calculated from a VLM response? For instance, for the query in Figure 359, \"Is this place crowded?\", if an agent replies \"It seems somewhat busy,\" how are the numerical values for these three uncertainty dimensions estimated? \n2. Could we change the dimension of the vector? What would happen if we design the uncertainty vector $[u_{observation}, u_{answer}]$ or $[u_{perc}, u_{sem}, u_{inf}, u_{answer}]$​? \n3. How could we determine the weight of each dimension?\n\n**Framework Overhead:** The ablation study (Table 3) shows the \"Strategic Uncertainty Index\" ($U_{strategic}$) is the most critical component. As defined in Appendix C, computing this index seems complex, requiring an estimation of expected system-wide cost savings for all potential trades. Is this computation tractable in a real-time system? What is the computational overhead (e.g., in latency or FLOPS) of the Agora broker and trading mechanism itself, separate from the VLM inference costs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)