From Debate to Equilibrium: Belief‑Driven Multi‑Agent LLM Reasoning via Bayesian Nash Equilibrium
Authors: Xie Yi
Zhanke Zhou
Chentao Cao
Qiyu Niu
Tongliang Liu
Bo Han
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
Reasoning
Multiagent Reasoning
Primary area: deep_learning->large_language_models

Abstract: Multi-agent frameworks can substantially boost the reasoning power of large language models (LLMs), but they typically incur heavy computational costs and lack convergence guarantees. To overcome these challenges, we recast multi-LLM coordination as an incomplete-information game and seek a Bayesian Nash equilibrium (BNE), in which each agent optimally responds to its probabilistic beliefs about the strategies of others. We introduce Efficient Coordination via Nash Equilibrium (ECON), a hierarchical reinforcement-learning paradigm that marries distributed reasoning with centralized final output. Under ECON, each LLM independently selects responses that maximize its expected reward, conditioned on its beliefs about co-agents, without requiring costly inter-agent exchanges.
We mathematically prove that ECON attains a markedly tighter regret bound than non-equilibrium multi-agent schemes. Empirically, ECON outperforms existing multi-LLM approaches by 11.2% on average across six benchmarks spanning complex reasoning and planning tasks. Further experiments demonstrate ECON’s ability to flexibly incorporate additional models, confirming its scalability and paving the way toward larger, more powerful multi-LLM ensembles. The code is publicly available at: https://github.com/tmlr-group/ECON.

From Debate to Equilibrium:
Belief-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
Yi Xie 12
Zhanke Zhou 2'
Chentao Cao
Qiyu Niu
Tongliang Liu
Bo Han
Abstract
Existing
Multi-agent frameworks can substantially boost the reasoning power of large language models (LLMs). but they typically incur heavy computa- tional costs and lack convergence guarantees. To overcome these challenges, we recast multi-L.LM coordination as an incomplete-information game and seek Bayesian Nash equilibrium (BNE), in which each agent optimally responds to its proba- bilistic beliefs about the strategies of others. We introduce Efficient Coordination via Nash Equi- librium (ECON). hierarchical reinforcement- learning paradigm that marries distributed reason- ing with centralized final output. Under ECON, each I.I.M independently selects responses that maximize its expected reward, conditioned on its beliefs about co-agents, without requiring costly inter-agent exchanges. We mathematically prove that ECON attains markedly tighter regret bound than non-equilibrium multi-agent schemes. Em- pirically, ECON outperforms existing multi-LLM approaches by 11.2% on average across six bench- marks spanning complex reasoning and planning tasks. Further experiments demonstrate ECON's ability to flexibly incorporate additional models, confirming its scalability and paving the way to- ward larger, more powerful multi-LLM ensem- bles. The code is publicly available at: https: / //github.c. www/moomrrrreeoooooooo//oooeeeem wttpm/ttllrggooo/oeeeeeeee group/ECON.
Amerr
Answer
Atavect
L.ME
Asgrogaler
Outrrt
Fizal
ECON
Beter'
Besef
LLMI
LLMI
Anaver
Coondruace
Fisal Curper
Belee3
100
Beller
Mesaga
Stagl
Stage?
Figure Comparison of multi-agent coordination ap- proaches. Existing MAD requires explicit message pass- ing between LLMs, incurring high communication over- head. ECON replaces it with belief-based coordination and achieves BNE-a stable state where each LLM optimizes its strategy based on belief's about other LLMs' behaviors.
from natural language understanding and generation to com- plex problem-solving. Recent work has shown that orga- nizing multiple LLMs into multi-agent framework can further amplify their reasoning power by enabling collabo- rative in-context discussions without parameter updates (Du et al., 2024; Chan et al., 2024; Liang al., 2023; Chen et al., 2023; Hong et al., 2023; Zhang et al., 2023b).
In particular, multi-agent debate (MAD) (Liang et al., 2023; Du et al., 2024) leverages structured argumentation among several L.M agents, where each participant critically evalu- ates and refines others' proposals to arrive at a more robust, consensus- driven solution. MAD has been shown to outper- form single-agent methods in various reasoning scenarios, providing clearer justifications and reducing error rates.
1. Introduction
However, existing multi- frameworks encounter three key obstacles that limit their practical deployment. First, extensive inter-agent communication consumes large num- bers of tokens, driving up computational overhead (Du et al., 2024). Second, the sheer volume of information exchanged over multiple rounds exceeds LLMs' context-window ca- pacity, impeding scalability (Liu et al., 2024b). Third, with- out well-defined coordination protocols, these systems can underperform simpler ensembling or self-consistency meth- ods (Liang et al., 2023). Moreover, recent work has shown that the consensus threshold between agents can signifi- cantly influence performance (Smit et al., 2024). Thus,
Large Language Models (LLMs) have demonstrated excep- tional reasoning capabilities across a wide range of tasks,
'Equal contribution 'Academy for Engineering and Tech- nology, Fudan University "TMLR Group, Department of Com- puter Science, Hong Kong Baptist University 'Sydney AI Cen- ter, The University Sydney. Correspondence to: Bo Han <bhanml @ comp. hompp.hkbu.cdee edu.kk..
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

From Debate to Equilibrium: Belief-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
developing a principled, scalable framework for multi-agent coordination is necessary but remains an open challenge.
usage by 21.4% on average. In scalability tests, increasing the number of Execution LLMs from three to nine yields an additional 18.1% performance gain, underscoring ECON's capacity to scale with only moderate resource overhead.
In this work, we introduce Efficient Coordination via Nash Equilibrium (ECON). novel framework that casts information game.
multi-LLMM interaction as an As shown in Figure I, conventional MAD approaches rely on heavy, round-by-round message exchanges among all agents, incurring substantial computational overhead. ECON, by contrast, replaces direct communication with belief-based coordination mechanism: each LLM maintains and updates probabilistic beliefs about its peers' behaviors, rather than sending and receiving explicit messages.
We summarize our key contributions as follows:
We formalize Bayesian Nash Equilibrium for multi-agent LLM systems, establishing theoretical foundations for effi- cient coordination without direct communication (Sec. 2).
We introduce the ECON framework to implement BNE via belief-based coordination and overcome scalability limits through local-global Nash mechanism (Sec. 3).
We empirically demonstrate that ECON outperforms both existing single-agent and multi-agent approaches, and validate its efficiency to scale larger ensembles (Sec. 4).
Notably, we frame coordination as the pursuit of a Bayesian Nash Equilibrium (BNE), where each LLM chooses its opti- mal strategy in light of its beliefs about other agents. Con- cretely, each Execution LLM uses belief network to con- vert its local observations and past trajectory into a belief state, and then produces an output that maximizes its ex- pected reward. All agents' belief states are merged by a Belief Encoder into a shared, group-level representation, which feeds into a centralized mixing network. This net- work refines the belief-network parameters for every agent, gradually guiding the entire ensemble toward BNE in which every LLM converges to stable strategy. By sub- stituting costly token exchanges with belief-driven coordi- nation, ECON dramatically cuts communication overhead while preserving and even enhancing multi-agen reasoning.
2. Theoretical Foundation
This section formalizes BNE for multi-agent LLM systems, establishing theoretical foundations for efficient coordina- tion without direct communication. We model the multi- agent setup as a DEC-POMDP in Sec. 2.1, prove BNE existence in Sec. 2.2, then develop convergence analysis with performance difference lemmas and regret bounds.
2.1. Problem Definition
We consider a framework improving collaboration under partial observability and without fine-tuning the internal pa- rameters of any LLM. In this setting, agents cannot observe each other's s outputs directly and rely on beliefs to coordi- nate. We formally model this as a decentralized partially observable Markov decision process (DEC-POMDP), de- fined Markev games (N, S, A, O, P, Q R. 7). Here, N ......NN denotes the set of agents; S represents the state space, including user queries and dialogue con- text; A = A1 X ... X AN is the joint action space, with each A, defining agent i's action as a prompt embedding a, = [Ti, p.] that controls generation behavior via tempera- ture and repetition penalty; O is the joint observation space; P and Ω define the state transition and observation func- tions; R is the reward and 7 is the discount factor.
Theoretically, we begin by proving the existence of a BNE and then bound the performance gap an optimal strategy via regret analysis. Specifically, ECON attains sublinear regret of O (NVT//1->, where N is the number of agents, T the number of iterations. and y the discount factor. By contrast, existing multi-agent frameworks lacking an equi- librium guarantee suffer linear regret O (6m)xI/1~>). This tighter bound demonstrates ECON's capacity to learn near- optimal strategies efficiently, with far lower token and com- putation costs than existing MAD methods. Consequently, ECON scales effectively, addressing the scalability limita- tion in prior research of multiagent (Wu et al., 2024; Yin et al., 2023; Lan et al., 2024; Yuan et al., 2024a) and graph reasoning (Zhou et al., 2023b;a; Li et al., 2024; Zhou et al., 2024b). Attributed to the local-global Nash coordination (cetral-corrtinnoor-cccccioo design, ECON can ensem- ble up nine LLMs with only moderate resource increases.
Here, A includes N - Execution LLMs and Coordinator LLM to achieve belief coordination. Each Execution LLM maintains a local history of = {a! o!...., of 2-1 of f-1) }, composed of its previous actions and observations for belief state updates. Execution LLM receives three inputs: the question, format and strategy from the coordinator, and the final aggregated output from Coordinator LLM. Our objective is to identify policy profile = (TI,.... AN) forming a BNE through belief' coordination, each agent's policy म; : H4 + A(A,) maps its history to an action distri- bution based on its belief about other agents' strategies, such that no agent can improve its response quality unilaterally.
Empirically, ECON reliably drives the multi-LL.M system toward a Bayesian Nash Equilibrium, fostering collabora- tive reasoning and stronger consensus among agents. On six diverse benchmarks- cenchmarks-spaaning complex reasoning and planning tasks ECON outperforms single-agent baselines by an average of 10.9% and existing multi-agent methods by 11.2%. demonstrating its superior effectiveness. Moreover, compared to 3-round MAD protocol, ECON reduces token
2

---

From Debate to Equilibrium: Belief-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
2.2. BNE Existence and Convergence
and (s) is the value under the current policy profile T -......,. mTyy at time t. The expectation accounts for randomness in both state transitions (governed by P) and policy choices. To analyze the total Bayesian regret R(T) = Σi= = (see Appendix B.3) and propose Lemma 2.2, proven in Appendix C.I. Using Lemma 2.2, we bound the Bayesian regret and provide a proof sketch below, with more details and bound comparison with MAD in Appendices C.2-C.3.
To bridge our DEC-POMDP formulation with BNE analysis, we introduce the concept of agent rypes. In our framework, each agent is associated with rype 8, = T/. which is precisely the agent's local history, capturing its internal beliefs and private observation history. Given the partial observability in our DEC-POMDP, each agent forms beliefs about other agents' types based on a common prior and its own observations. These beliefs are operationalized through belief networks in our implementation (detailed in Sec. 3).
(T))),, we impose standard assumptions
Lemma 2.2 (Performance Difference). For joint policies (Ta m-) and xt (T/ LL)) the difference in value:
Formally, a strategy profile {m; ) is a BNE if, for each i:
1
Eg_. Ui( (0,). (O))),
V. (8) (7 =
1 1 Bonder [aan Q7 (8,
2 Eo_. Us [T (0) (0
.). 0z 0-:)]: VTL,
- EaNT Q7
where 0-i (01,..., 0;- 1 0 1,... ON) denotes the types of all agents except i, utility function is defined as:
where da is the state distribution under n', and a (a,, a-;) denotes the joint action from A.
Ui( - 0-) = E Σ
vr , 0,000
Note that the Q-function Q (s, a) appearing in this lemma will be approximated by neural networks in our implementa- tion, as detailed in Sec. 3. Applying this lemma to our regret analysis yields (Jin al., 2020; Fujimoto et al., 20:
where r R(s', a²); is agent i's reward from the DEC- POMDP reward function at time t. To establish the existence of BNE in our setting, we verify three standard conditions: (1) each agent's mixed strategy space is non-empty, compact, and convex (satisfied by our bounded prompt embedding space); (2) the payoff function U,(O,a) a) is continuous in types and actions (ensured by continuous reward functions); and (3) each agent's expected payoff is quasi-concave in its own actions for fixed 0, (guaranteed by our reward design).
N
1
R(T)
Ege
E
(Be our ( a?)
Eagw Qi (Se, (8t,ae))
where x² represents the BNE policies. By bounding the estimation error e, and suboptimality òt. we show
Theorem 2.1 (Existence of Bayesian Nash Equilibrium). In this multi-agent LLM framework, if the above condi- tions are satisfied, there exists a BNE strategy profile मt (mi, -......... . x²x) by Glicksberg's 's Fixed Point Theorem (Ahmad et al., 2023). full proof is provided in Appendix B.I.
Eaj-x - 277h Q7'( ai) Burr, 1, a.) S 2c, + 8c.
where e, = OL/vit) bounds the Q-function estimation error and d = O(1/VI) measures policy suboptimality. Under standard regularity conditions, these errors can be bounded by constants C. and Co respectively, leading to
We then analyze the convergence properties of our ECON framework via Bayesian regret. Our analysis shows that combining belief networks with coordinated updates leads to efficient approach for approximating BNE, achieving sublinear regret bound O (NVT/>>>), in contrast to the linear regret of existing multi- debate methods.
R(T) S
(2C. + CCs)
Σ
T
1 Vi
o
NVT
3. BNE Implementation with ECON
To connect our theoretical analysis with the DEC-POMDP formulation, we define the value function and regret in terms of our system components. For each agent we measure learning efficiency using Bayesian regret over I' steps:
We present ECON that satisfies the DEC-POMDP architec- ture in this section. Our framework satisfies the assumptions in Appendix B.3, enabling the application of Lemma 2.2 to bound Bayesian regret. As illustrated in Figure 2, ECON adopts
T
R (T) = E S...T Σ (
Lendinatto-Executto architecture where multiple Execution LLMS operate locally under the guidance of a Coordinator LLM, and comprises two phases: Inference and Optimization. Below we first introduce the core modules that collectively implement our DEC-POMDP design, then describe the complete training process in Algorithm 1.
where (s()s denotes optimal value function under BNE:
Vi*(s) max E.,
Σ RR(s "TR(ss,a')) (a))))a))))))) 1s" =,=,,
Ris -
0
3

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
Inference Phase
Ground Truth
SD Loss
Optimization Phase Belief Network
Final Output
Reward
TT) Loss
TH
Question
Mixing Network
Inpur for
Coordinator LLM
F
F?
Multi Ilead Attention
Q
Exccution
Concatenate
Strategy Fomual
Answer
Answer
Answer
E
Attention
Belief
Group Info
E'
Execution 1IM!!
Bellet Network
Belief Fnonder
Q (t)
Network?
Execution LLM
b)
6
b,"
MLP
b/
GRU
b,"
Q Belief Netwook
@ Delief Network
Belief Network
Belsef
(O
MT P
Execution LLM
Figure 2: ECON Framework: The inference procedure (left) shows how the Coordinator LLM processes and manages
Execution LLMs' responses. The optimization procedure (right) illustrates the parameter process of the belief network.
3.1. Inference Phase
where o()) is the sigmoid function. Here, Tj modulates the temperature of token sampling, and pj sets a penalty threshold for repetition. The belief network B, has two outputs: (1) the prompt embedding e, that serves as the action in our DEC-POMDP framework, and (2) local Q- value ( et; <b:) that estimates the expected return from the current belief state. The belief state b, is also passed to the belief encoder for group-level processing.
During the inference phase of ECON, a Coordinator LLM generates an informative strategy and a format based on the input question. These are then disseminated to the Execu- tion LLMs, which independently produce their respective answers. Then Coordinator L.LM aggregates these answers to form final output, as illustrated in the left part of Fig- ure 2.A detailed case study demonstrating this inference process is provided in Appendix F. To implement the theo- formu-
To optimize the belief network, we apply a TD loss to update the parameters 0/5 as illustrated the right part of Figure 2. The belief network parameters 0 {Ør, Wr, br, Wp, bp} include Q-value function parameters o, and prompt embed- ding parameters. Thus, we have:
retical policy x; : Hi - ((A)) from lation, each Execution LLM maintains a belief network that maps its local history actions. The following subsections detail how ECON collectively approximates towards BNF.
(i ED (1t
(1 7 max (
3.2. Individual Belief Network
e (0)
Each Execution LLM maintains a belief network Bi(T], 05) that implements its policy म, by mapping local trajectory 7f and current observation of O to belief state b; E Idd. The belief state captures the agent's understanding of the environment and other agents' behav- iors under partial observability, enabling strategic decision- making without direct access to others' outputs. The belief state b, is further used to generate the action of E A,, rep- resented prompt embedding e, [T]. p.]. While e; is 2-dimensional vector, term it "embedding" it embeds the agent's 's strategic decisions into the prompt control space, influencing the LLM's generation. We define:
Qi(ri,
where r/
R(s', a²); is the local reward signal and of denotes target network parameters (updated via soft update). By minimizing LtD, Execution LLM i refines its belief state to improve local decision making.
3.3. Belief Encoder
A shared belief encoder, Je(:; 0n), aggregates the belief states from all agents to produce a group-level representa- tion E = fc((b)) f2). This encoder enables agents to implicitly coordinate their beliefs, facilitating convergence to BNE. We employ multi-head attention with H heads to
T: Tiin + (Tmux Tmin) (Wr b, + by),
P: Pmin (Pmax Pmiin (W, b; + bp)

---

From Debate to Equilibrium: Belief-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
capture inter-agent dependencies in belief states:
sim(u; C)
C:CC (Zhu et al., 2023). The Task- Tu |
Specific Reward r! = min( eval(u;, task)) evalu- ates domain -emaain-peccfic performance through normalized scor- ing (Hao et al., 2023). The Collaborative Contribution Reward r = minnBmasse quality (w,, {uy) assesses each agent's contribution to the collective solution (Xie et al., 2024b), The total reward is computed as ri airAl + agr! + aagre, where an + 02 + ag 1. These weights are dynamically adjusted via gradient updates or + ag %a ÖLar/Oo,, with Lar = >i= actual expecnd )
head,, Attention ? b, W b. WXb): b):
where b = [bj;...; bx] E RVd concatenates {b; 1=1 The final output is E = Concat(head,,.. ..... headæ) W with { W W as learnable parameters and W as the output projection. The belief encoder can also be regular- ized via (02) = LDD (6) + >e > (i)( where LTD the global TD objective. This encoder captures higher- level dynamics among Execution LLMSs for coherence.
3.6. Early Stopping
3.4. Centralized Mixing Network
To ensure efficient optimization and convergence to stable solutions, early stopping is implemented based on three key criteria. First, final output stability is achieved when the change in the coordinator's 's output satisfies |||||| = C++1 - C+ S tc. Second, reward convergence is moni- tored such that the average reward across agents reaches a predefined threshold, + -i=1
The Centralized Mixing Network coordinates the integrated belief information from all Execution LLMs, facilitating global optimization toward BNE. Specifically, each agent's ' prompt embedding [ is processed via self-attention to capture agent-agent dependencies, yielding intermediate embeddings [w]] We then combine [w]].w] with the group-level representation E' to produce feature transfor- mations { This combination allows the network to integrate individual strategic decisions (captured in e with collective belief dynamics (captured in E`), enabling coordinated optimization. The local Q-values {Q)) i=1 and the transformed features { { are jointly fed into multi- head attention layers to compute a global Q-value Qt This global Q-value function accounts for local-global interac- tions, ensuring that improvements in individual behavior also contribute. To train the mixing network, we minimize:
-N 1 r, 2 Rahreshodd- Lastly, loss
convergence is ensured when the total loss stabilizes, satisfy- ing [ T I <,. where Tust is the sum of individual agent losses N Li, execution loss Le, and the mixing loss Lmix- These criteria monitor the optimization process, ensuring both strategic alignment and performance while preventing premature termination.
Algorithm Belief Network Training Algorithm
Require: Question, (ExecLLM, LN 1- Coordinator LLM, thresholds {ec, er.)
N
Lmix = CTD (bb) + LSD + Am
Σ 10
- Vioe 1?,
Ensure:
Optimized parameters (0/) Ji=1, t.,
I: Initialize: (0(²) /N 1. Az. &
2: if not converged then for each LLM do
where CTb(() aligns Oia with the global reward Froe:
4: b, BI( o;; 0.)
(Ø) = ED (Nior + 7
max (o N M
(
(et 1ett11
MeettIy IN
)
5:
e; < ComputeEmbecdting(b)
6:
u, < E ex)
(Ta ( ())))=)))))))
7:
Ti = aIrAL a2r TS + aar
8:
end for
while a similarity difference loss LSD (e.g. do (1- sim(F; ?)) aligns agent features with the coordinator's final output C. ||Q Qiull ensures local Q-values remain consistent with the global estimate. Target parameters ø' are updated by soft update rule ø' t TØ + (1 o) As result, the mixing network optimizes local policies to improve the global objective, promoting stable convergence (see Appendix B.5 for monotonicity proof).
9:
E + feb, 02)
10:
C Coordinatoo (fu, )= E)
11: Update (0 Ji=1 via LiDi O. via Lei @ via Lmix
12: converged +
+Ic -C- -pre- I Ec and Ravg Rib
and |Lror SEL
13: end if
14: return (0,5) Oc. Φ
3.5. Reward Design
4. Experiment
The reward function Rassign consists of three components, all bounded by Rmax per Assumption B.2. The Ac- sum C))
In this section, we present the experiment setup in Sec. 4.1, demonstrate the method effectiveness in Sec. 4.2, validate the heterogencous results in Sec. 4.3, test scalc-up capability in Sec. 4.4, and conduct ablation studies in Sec. 4.5.
tion Likelihood Reward raN- = measures final output consistency via cosine similarity
5

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
4.1. Setups
Mistral Miteral Rx2281
Limua 700
Flama 4150
C5D Beel
leet
4.1.1. MODELS AND DATASETS.
100
Average
80
We evaluate 6 released opensourced LLMs: LLaMA3.1
8B (Dubey et al., 2024), LLaMA3.1 70B, Mistral-7B (Jiang
Accurac((5))
60
et al., 2023), LLaMA3.1 405B, Mixtral-8x22B (Jiang
40
et al., 2024) and Qwen1.5 1.5 110B (Yang et al., 2024)
across 5 reasoning tasks, including 4 mathematical datasets
20
(GSM8K (Cobbe et al., 2021), GSM-Hard (Gao et al., 2023),
MATH (Hendrycks et al., 2021), SVAMP (Patel et al., 2021)) and one commonsense reasoning dataset (StrategyQA (Geva et al., 2021)). Then, we evaluate GPT4 turbo) (Achiam et al., 2023) in very challenging planning task (Travelplan- ner (Xie et al., 2024a)) to further validate the performance. The details of benchmarks can be found in Appendix C.5.
Zon Taat Cor
Fer thst Cal
SCimnele
(Stot
Tol
RAP
TW-LLM
PRO-MCTS
EonNarh
Figure 3: Average results of five reasoning datasets: GSMBK,GSM-Hard, SVAMP, Strategy QA and MATH.
4.2. Main Result
4.1. COMPARED METHODS AND BENCHMARKS.
Figure 3 presents the average accuracy comparison of each method across four mathematical reasoning datasets and one commonsense reasoning dataset. Detailed accuracy comparisons for each dataset can be found in Appendix E. The empirical results demonstrate that ECON outperforms most baselines on all complex reasoning benchmarks. On average, ECON outperforms the single-round method Zero- shot CoT by 25.6%, Few-shot CoT by 6.3%, multi-round CoT prompting SC by 10.9%, multi-round self-improving approaches ToT by 11.2%, and multi-LLM reasoning frame- works rStar by 6.4%.
We compare ECON against several strong baseline types widely adopted: (i) single-round CoT prompting, includ- ing zero-shot and few-shot CoT (Kojima et al., 2022; Wei et al., 2022); (ii) multi-round CoT prompting, Self Con- sistency (SC) (Wang et al., 2023) method, where we sam- ple answers 64 times and employ majority voting for an- swer selection; (iii) value-guided search approaches with learned action-value functions, including TS-LLM (Feng et al., 2023) which leverages AlphaZeroostyle value net- works for MCTS, and PPO-MCTS (Liu et al., 2024a) which learns value models to evaluate generation quality in tree search; (iv) multi-round self-improving approaches, using ToT (Yao et al., 2023), RAP (Hao et al., 2023) and Re- act (Yao et al., 2022), with BFS and MCTS for tree search, respectively, following their original answer selection; and (v) multi-LLM reasoning frameworks, including rStar (Qi et al., 2024) and multi-agent debate (Du et al., 2024).
Furthermore, when evaluated on the very challenging Trav- elplanner benchmark using GPT-4-Turbo in Table 1, ECON enhanced the final pass rates 7.2% on the validation set and 9.3% the test set, while compared to 2.3% and 3.7% achieved by a three-round multi-agent debate.
These results demonstrate that ECON effectively lever- ages the capabilities of more powerful models and outper- forms alternative reasoning optimization methods in com- plex tasks. Additionally, we provide a corresponding ex- ample for MATH, which are available in Appendix F, Note that ECON uses fewer tokens compared to multi-round CoT prompting SC, multi-round self- self-improving approaches ToT, and MAD, meanwhile achieved performance improvements. 4.3. Model Configuration and Cost Efficiency Analysis
4.1.3. ECON SETUPS.
In this section, the ECON framework includes one Coor- dinator and three Execution LLMs The hyperparameters for training can be found in Appendix C.6. To ensure a fair comparison with the baseline, we use four identical models for these L.Mss. For heterogeneous results, we also evaluate ECON with different models in Table 2. All evalu- ations are conducted in zero-shot setting, with a general prompt provided in Appendix F. Notably, while we set 50-token constraint for the coordinator's 's strategy genera- tion, considering that LLMS may not strictly follow length instructions (Yuan et al., 2024b), who showed that 95% of responses stay within 1.4x and 50% within 1.0x of the specified length, we implement a 70-token hard cutoff with regeneration mechanism to controls the token usage.
To evaluate the impact of both the Coordinator LLM and Ex- ecution LLM performance on the ECON framework and find whether heterogeneous Execution LLMs can also achieve a BNE, we conducted two types of experiments: one pairing strong Coordinator LLM with weaker Execution LLMs, and another pairing a weak Coordinator LLM with stronger Execution L.M.s. These experiments were further divided into homogeneous and heterogeneous execution groups for detailed analysis. To ensure a fair comparison, the Coordi- nator LLM was consistently set to LLaMA3.1 70b across all experiments. For the heterogeneous execution group, we
6

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
Table I: Empirical results on the TraveIPlanner dataset, along with some leaderboard rankings, are presented.
Validation (#180)
Test (#1,000)
Commonsense
Hard Constraint
Commonsense
Hard Constraint
Delivery Rate
Final
Delivery Rate
Final
Pass Rate
Pass Rate
Pass Rate
Pass Rate
Pass Rate
Pass Rate
Miero
Macro
Micro
Macro
Micro
Macro
Miero
Macro
Groedy Search
100
74.4
0
60.8
37.8
0
100
72.0
52.4
31.8
Two-stage
Mixtral-8x7BB -x-ral-
49.4
30.0
1.2
0
51.2
32.2
0.4
Gemini Pro
28.9
18.9
0
0.5
0.6
0
39.1
24.9
0.1
GPT-3.5-Turbbu
86.7
54,0
0
0
0
91.8
57.9
0
0.5
0.6
0
GPT-4-Turbo
89.4
61.1
2.8
15.2
10.6
0.6
93.1
63.3
2.0
10.5
5.5
0.6
Debate (GPT-4) @ 3round
95.2
67.3
6.7
22.7
13.1
2.3
97.8
72.4
11.3
17.4
12.1
ECON (GPT-4)
100
71.4
15.6
32.1
25.7
7.2
100
82.1
26.6
32.4
17.6
9.3
Sole-plame Sole-planning
Diractart-a.5-Turs
100
60.2
4.4
2.8
100
59.5
4.4
CoTGGT335-Turo
100
66.3
3.3
11.9
100
64.4
3.8
ReActGrr-3.5-
82.2
47.6
3.9
11.4
6.7
0.6
81.6
45.9
2.5
10.7
3.1
0.7
ReflexionGGra..ssuts
93.9
53,8
2.8
11.0
2.8
0
92.1
52.1
2.2
9.9
3.8
0.6
100
68.1
5.0
3.3
1.1
0
99.3
67.0
3.7
3.9
1.6
0.7
Directriemii Pro
93.9
65.0
8.3
0.6
93.7
64.7
4.7
2.1
DirectorT-4 Turbo
100
80.4
17.2
47.1
22.2
4.4
100
80.6
15.2
44.3
23.1
4.4
Debate (GPT-4)
97.7
78.9
15.6
43.3
20.6
6.7
98.2
79.5
18.8
41.7
22.9
7.1
ECON (GPT-4)
100
83.3
22.2
51.7
27.8
12.9
100
84.2
23.5
49.8
28.7
15.2
Table 2: Performance of different configurations.
We would like to point out that ECON does not eliminate inter-agent communication entirely, but rather adopts an incompleteinnoomattinn perspective that minimizes com- munication. ECON's optimization objective focuses on achieving consensus among Exccution LLMs, consensus achieved as the result of implicit communication. We make additional experiment demonstrate incorporating direct complete interaction into ECON (which makes it become complete information formation). The performance improve 1.1 % while the token consumption increase 42.4%
Method
GSM-Hard MATH
Baselines
ECON
51.43
81.47
LLaMA (Fow-shot CoT)
42.23
62.71
ECON Configurations
Homo. LLaMA3.1 8B)
67.70
Homo. (3x LLaMA3.1 405B)
61.29
89.24
Hetero. (LLaMA3.1 LLaMA3 8B, Mixtral 7B)
45.24
74.24
Hetero, (Mixtral 8x22B, 5wen1.5 1108, LLaMA3.1 405B)
55.73
85.46
used the following configurations: LLaMA 3.1 8b, LLaMA 3 8b, and Mixtral 7b, as well as another configuration con- sisting of Mixtral 8x 22b, Qwen1.5 10b, and LLaMA3.1 405b. For the homogeneous execution group, two configu- rations were tested: one with three weak models (LLaMA 3.1 8b), and another with three strong models LLaMA 3.1 405b. Experimental results indicate that stronger Execution LLMs improve performance by providing higher-quality answers and achieving BNE more efficiently. Additionally, heterogeneous model perform worse than homogeneous models due to increased challenges in reaching BNE, but still outperform baseline Few-shot CoT.
4.4. Scale Up Result
We analyzed the impact of varying the number of agents fur- ther to validate ECON across broader range of LLMs. We conducted three sets of experiments on the MATH, GSM- Hard, SVAMP, and Strategy QA datasets, aiming to address three key questions: (1) To what extent can weaker LLMMs be enhanced? (examined on LLaMA 3.1 8B), (2) Can stronger LLMs be further improved? (using LLaMA 3.1 70B), and (3) Should the number of Coordinator LLMs be increased along with the number of Execution LLMs? Starting from three Execution LLMs (as in the main results), we gradu- ally increased the number of agents to nine. We used the few-shot CoT as the baseline (in grey line) as Figure 4. The results suggest that beyond four Execution LLMs, perfor- mance improvements were minimal, and in cases, per- formance even declined. We attribute this to the challenge faced by the Coordinator LLM in managing excessive number of Execution LLMs, making it difficult to achieve coordination by information from additional agents.
To assess the cost efficiency of the ECON framework, Ta- ble presents the average token usage of ECON, Multi- Agent Debate, RAP, and SC across the MATH, GSM8K, and GSM-Hard datasets for three models: LLaMA 3.1 70B, Mixtral 8x7B, and Mixtral 8x22B. The results demonstrate that ECON reduces token consumption by average of 21.4% compared to Multi-Agent Debate (3 rounds). No- tably, when the Coordinator LLM provides detailed strate- gies with answer(as shown in the token consumption data in Table 3), token usage increases an average of 112% higher as the full strategy processing.
Instead of simply increasing the number of Execution LLMs. we enhance scalability by forming a global Nash Equilib-

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
LLaMA 3.1 8B
Performance Comparison GSM8K
Performance Compurison in GSM-Hlard
Performunce Comparison Math
Performance Comparison in SVAMP
90
40
Perfermanc
30
60
30
75
75
3
8
9
3
4
5
7
8
3
4
7
3
4
5
7
8
Agent Number
Agent
Agent Number
Agent Number
Figure 4: Scaling up our framework with single coordinator while increasing the number of Execution LLMs in 5 datasets.
Performance Comparison in GSM8K
Performance Comparison in GSM-Hard
Performance Comparison in Math
Comparison SVAMP
05-
95
Performance
40
85
35
30
25
75
55
3
4
5
6
7
5
Agent Number
Ageot Number
Agent Number
Ageat Number
Figure 5: Scaling up by increasing the number of coordinators in proportion to the number of Execution LLMs in 5 datasets.
rium through local Nash equilibria by introducing additional coordinators. This setup ensures that each Coordinator handles reasonable amount of data. Specifically, each Coordinator manages up to 4 Execution LLMs, forming outputs and guiding them toward local Nash equilibria. Fur- thermore, a central LLM was introduced to coordinate the multiple coordinators, facilitating the transition from local Nash equilibria to global Nash Equilibrium (details in Ap- pendix 2). We observed sigmificant improvements across benchmarks, both for weaker models (LLaMMA 3.1 8B) and stronger models (LLaMA 3.1 70B). Compared to system with 3 Execution LLMs and one coordinator, the scaled-up system with 9 Execution LLMs, 3 coordinators, and cen- tral LLM achieved 18.1% improvement Figure 5, which has potential to further scale up.
the Coordinator LLM provides strategy without giving a direct answer, ensure the validity of our architecture. All experiments were conducted with LLaMA 3.1 70B, tested the MATH benchmark. Specifically, R₁ refers to the action likelihood reward, R2 to the task reward, and Ra to the self-evaluation reward. S₁ represents the setting where the coordinator does not provide any strategy, while S₂ represents the setting where the coordinator provides detailed strategy, S3 represents ECON as our baseline.
Theoretically, the design of our mixing network optimizes local policies to improve the global objective, Monotonicity proofs are demonstrated in Appendix A.5. We make addi- tional ablation experiments to validate this by removing the concatenation and removing the belief encoder in Table
5. Related Work
4.5. Ablation Study
In the additional experiments, heterogeneous Execution LLMs experienced slight performance decline. An in- tuitive explanation for this observation that implement- ing BNE is more challenging for heterogeneous Execution LLMs. To verify the actual performance differences of the ECON framework before and after achieving BNE, we con- ducted experiments on three math reasoning benchmarks: GSMSK, GSrdH and MATH. Results in Table demon- strate that our framework achieved an average performance improvement of 14% after implementing BNE.
Prompting Large Language Models to Reason. Large language models are significantly more capable of complex reasoning with the advancement of prompt techniques (Wei et al., 2022; Kojima et al., 2022; Zhou et al., 2022; et al., 2022; Zhang et al., 2023c; Wang et al., 2023; Li et al., 2023; Chia et al., 2023; Cao al., 2024; Zhou al., 2024a; Wang et al., 2025; Zhou al., 2025). Wei et al. (2022) introduced Thought (CoT) prompting, which presents step- by-step reasoning examples within the prompt. This enables the model to engage in explicit reasoning, enhancing its ability to follow the logical progression that leads to the correct answer. Various extensions of CoT have been devel- oped to improve reasoning performance further. Zero-shot
ohain-of-Thought
Additionally, performed ablation studies various sub- modules, including the reward design and the setting where
8

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
Table 3: Average token usage and performance comparison in the MATH, GSM8K, and G
LLaMA3.1 70B
Mixtral 8x7b
Mixtral 8x22b
Dataset
Inference Strategy
Token Usage
Performance
Token Usage
Performance
Token Usage
Performance
Multi-Agent Debate rounds)
2154.87
71.58
1462.12
31.28
$345.56
67.41
RAP
2653.27
68.71
1737.73
33.99
6668.55
62.53
MATH
ECON (with detailed strategy)
3270.06
72.38
2150.23
26.18
8054.03
68.25
Self Consistency (64 rounds)
11917.00
67.39
8066.21
31.58
29616.13 13
62.21
ECON
1629.79
81.47
1128.23
35.02
4270.86
72.29
Multi-Agent Dehate rounds)
1391.57
86.32
1463.40
70.19
5714.05
81.95
RAP
1907.86 86
81.33
1248.66
72.03
6517.77
76.97
GSM8K
ECON (with detailed strategy)
2772.24
85.17
1188.13
65.37
9341.60
81.46
Self Consistency (64 rounds)
9574.25
89.56
6601 .34
71.08
24671.91
86.24
ECON
1131.65
92.70
1284.98
76.97
4715.31
88.20
Multi-Agent Dehate (3 rounds)
3030.73
41.98
1478. 14
20.04
9250.78
45.21
RAP
1768.72
38.97
1036.11
22.47
6464.52
42.79
GSM-Hard
ECON (with detailed strategy)
3662.64
44.12
2239.07
18.52
11464.98
41.04
Self Consistency (64 rounds)
16724.69
39.76
11668.19
22.47
74544.25
44.19
ECON
1518.76
51.43
1271.53
25.76
7101.62
47.58
Table 4: Comparison between Token Consumption and Effi
Chan et al., 2024; Liang et al., 2023; Chen et al., 2023; Smit et al., 2024; Zhang et al., 2023a; Pham et al., 2023; Li et al., 2023) have been developed strengthen the reasoning abil- ity of LLMs. Du al. (2024) introduced an approach where multiple instances of LLMs propose their individual rea- soning processes, engaging in multiple rounds of debate to reach a consensus on the final answer. This method not only significantly enhances reasoning performance across a variety of tasks but also reduces the occurrence of hallucina- tions. Some studies (Chan et al., 2024; Liang et al., 2023) incorporate vole-playing into multi- multi-agent debate strategies using role-specific prompts, foster divergent thinking and en- hance the reasoning capabilities of LLMs. However, current multi-agent debate strategies face high computational costs and lack theoretical guarantees for convergence. In this work, we introduce an incomplete information perspective to enhance the scalability multiple LLMs to ensure inde- pendent reasoning by each Execution LLM, while ensure convergence through rigorous theoretical analysis.
ciency of ECON w/o complete information in LLaMA 3.1.
Dataset & Model
Complete Info
Consumption
GSM8K- LLaMA 3.1
81.4
80.3 (+35
GSM8K L.LaMA 3.1 70B
96.1
96.7 (+42.7%)
GSM-Hard L.L.MA 8B
30.2
29.9 (+62.3%)
GSM-Ilard LLaMA 3.1 70B
53.6
51.4 (+40.9%)
MATH LLaMA 3.1 8B
59.6
60.4 (433.8%%)
MATH LLaMA 3.1 70B
83.1
81.5 (+39.4%)
Table Ablation on different reward and strategy settings.
Reward
Strategy
Ra
Ra
Ra
ECON
S1
S2
S3
ECON
x
77.55
/
71.35
74.32
72.31
x
76.21
X
81.47
Random
62.71
Table 6: Ablation Study on Concatenate and Belief Encoder.
Dataset & Model
No Concatenate
No belief encoder
GSMSK- LLaMA 8B
85,9 (-1.8)
84.1 (-3.4)
GSMSK I.L.MA 3.1 70B
93.6 (-3.1)
91.1 (-5.6)
GSM-IHard LLLMA 3.1 8B
24.9 (-5.0)
(-8.2)
6. Conclusion
GSM-Hard LLaMA 3.1 70B
47.2 (-4.2)
(-8.8)
MATH LLaMA 3.1 8B
(-1.8)
(-7.1)
In this work, introduce ECON, novel collaborative soning framework in multi-LLM systems. ECON constructs a hicrarchical coordination mechanism, enabling multiple Execution LLMs to engage in distributed reasoning guided by Coordinator LLM. The hierarchical coordination mech- anism allows each Execution L.LM to operate independently with its own belief network, receiving only the question and strategy from the Coordinator LLM. This enables multi- ple Execution LLMs to engage in distributed reasoning to achieve BNE. Experimental results across six benchmarks demonstrate ECON outperforms single-agent approaches by 10.9% in average. confirming the robustness and effi- ciency of our framework. Moreover, ECON demonstrate great potential of scalability while maintain reasonable cost.
MATH I.I.MA 3.1 70B
(-..4)
75.3 (-6.2)
CoT (Kojima et al., 2022) eliminates the need for manually constructing exemplars, prompting models with phrases like "Let's think step by step" to encourage reasoning. Wang et al. (2023) proposed self-consistency (SC) sampling, where mul- tiple reasoning paths are sampled, and the final answer is determined by majority voting. To enable LLMs to engage in deliberate decision-making Tree of Thoughts (ToT) (Yao et al., 2023) generates multiple potential answers at each reasoning step, building a tree of possible solutions.
Multi-agent Debate for Large Language Models Reason- ing. Various multi-agent debate strategies (Du et al., 2024;

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
Acknowledgements
R., et al. Training verifiers solve math word problems. arXiv preprint arXiv:21 10. 14/68, 2021.
ZKZ, CTC, and BH were supported by RGC Young Col- laborative Research Grant No. C2005-24Y, NSFC General Program No. 62376235, Guangdong Basic and Applied Basic Research Foundation Nos. 2022A15 101 1652 and 2024A1515012999 HKBU Faculty Niche Research Ar- cas No. RC-FNRA-IG/22-22SSCUU, and HKBU CSD Departmental Incentive Scheme. TLL is partially sup- ported by the following Australian Research Council projects: FT220100318, DP220102121, I.P220100227, LP220200949, and IC190100031.
Du, Y., Li, S., Torralba, A., Tenenbaum, B., and Mordatch, I. Improving factuality and reasoning in language models through multiagent debate. In ICML, 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Feng, X., Wan, Z., Wen, M.. McAleer, S. M., Wen, Y., Zhang, W., and Wang, Alphazero-like tree-search can guide large language model decoding and training. arXiv preprint arXiv:2309.1 171799 2023.
Impact Statement
This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Foerster, J., Chen, R. Y., Al-Shedivat, M., Whiteson, S., Abbeel, P., and Mordatch, I. Learning to model other minds: A deep learning framework for social intelligence. In NeurIPS, 2018.
References
Fu, Y., Peng, H., Sabharwal, A., Clark, P., and Khot, T. Complexity-based prompting for multi-step reasoning. arXiv preprint arXiv:2210.00720 2022.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L.,, Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., al. Gpt-4 technical report. arXiv preprint arXiv:2303.08874, 2023.
Fudenberg, D. and Levine, D. K. The Theory of Learning in Games. MIT Press, 1998.
Ahmad, J., Al-Mazrooei, A. E., and Rassias, T. M. Com- mon fixed point theorems with applications to theoretical computer science. International Journal of Nonlinear Analysis and Applications, 2023.
Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In ICMI., 2018.
Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. Pal: Program-aided language models. In ICML, 2023.
Ben-David, S. and Blais, E. A new minimax theorem for randomized algorithms. Journal of the ACM, 2023.
Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., and Berant, J. Did aristotle use laptop? question answering benchmark with implicit reasoning strategies. Transac- tions of the Association for Computational Linguistics, 2021.
Borkar, V. S. Stochastic Approximation: Dynamicul Systems Viewpoint. Cambridge University Press, 2009.
Cao. C., Zhong, Z., Zhou. Z., Liu, Y., Liu, T., and Han, B. Envisioning outlier exposure by large language models for out-of-distributioo detection. In ICML, 2024.
Hao, S., Gu, Y., Ma, H., Hong, J. J., Wang, Z., Wang, D. Z.,, and Hu, Z. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305. 14992. 2023.
Chan, C.-M., Chen, W., Su, Y., Yu, J., Xue, W., Zhang, S., Fu, J., and Liu, Z. Chateval: Towards better llm-based evaluators through multi-agent debate. In ICLR, 2024.
Hazan, E. Introduction Online Convex Optimization. Now
Chen, J. C.-Y,, Saha, S., and Bansal, M. Reconcile: Round-table conference improves reasoning via consen- sus among diverse llms. arXiv preprint arXiv:2309.13007, 2023.
Publishers Inc, 2016.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Mcasuring math- ematical problem solving with the math dataset. arXiv preprint
Chia, Y. K., Chen, G., Tuan, L. A., Poria, S., and Bing, L. Contrastive chain-of-thought prompting, arXiv preprint arXiv:2311.09277 2023.
2021.
Hong, S., Zheng, X., Chen, J., Cheng, Y., Wang. J., Zhang, C., Wang, Z., Yau, S. K. S., Lin, Z., Zhou, L., et al. Metagpt: Meta programming for multi-agent collabora- tive framework, arXiv preprint arXiv:2308.0 00352, 2023.
Cobbe, K. Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, I.., Plappert, M., Tworek, J., Hilton, J., Nakano,
10

---

From Debate to Equilibrium: Belief-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
Jaques, N., Lazaridou, A., Hughes, E., Gulcehre, C., Ortega, P. A., Strouse, D., Leibo, J., and De Freitas, N. Social influence intrinsic motivation for multi-agent deep reinforcement learning. In ICML, 2019.
efficiency of multi-agent debate using group discussion. arXiv preprint arXiv:2409. 14051, 2024b.
Lowe, R., Wu, Y. I., Tamar, A., Harb, J., Pieter Abbeel, O., and Mordatch, I. Multi-agent actor-critic for mixed cooperative-compptiivv environments. In NeurIPS, 2017.
Jiang, A. Q., Sablayrolles, A., Mensch, A.. Bamford, C., Chaplot, D. S., Casas, D. d. I., Bressand, F., Lengyel. G., Lample, G., Saulnier, L.., et al. Mistral 7b. arXiv preprint arXiv:2310.0682. 2023.
Patel, A., Bhattamishra, S., and Goyal, N. Are nip models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191 2021.
Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. I., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint ar arXiv:2401.04085 2024.
Pham, C., Liu, B., Yang. Y., Chen, Z., Liu, T., Yuan, J., Plummer, A., Wang, Z.,, and Yang, H. Let models speak ciphers: Multiagent debate through embeddings. arXiv preprint arXiv:2310.0000 2023.
Jin, C., Yang, Z., Wang, Z., and Jordan, M. Provably efficient reinforcement learning with linear function ap- proximation. In COLT, 2020.
Qi, Z., Ma, M., Xu, J., Zhang, L. L.,, Yang, F, and Yang, M. Mutual reasoning makes smaller Ilms stronger problem- solvers, arXiv preprint S.KY:22408.06199 2024.
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners, In NeurlPS, 2022.
Smit, A. P., Grinsztajn, N., Duckworth, P., Barrett, T. D., and Pretorius, A. Should we be going mad? look at multi-agen debate strategies for llms. In ICML, 2024.
Lan, X., Gao, C., Jin, D., and Li, Y. Stance detection with collaborative role-infused Ilm-based agents. In ICWSM, 2024.
Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. MIT Press, 2018.
Wang. Q. Zhou, J. P., Zhou, Z., Shin, S., Han, B., and Weinberger, K. Q. Rethinking IIm unlearning objectives: A gradient perspective and go beyond. In ICLR, 2025.
Lanctot, M., Zambaldi, V., Gruslys, A., Lazaridou, A., Tuyls, K., Pérolat, J., Silver, D., and Graepel, A uni- fied same-theoretic approach multiagent reinforcement learning. In NeurlPS, 2017.
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency im- proves chain of thought reasoning in language models. In ICLR, 2023.
Li, X., Zhou, Z., Zhu, J., Yao, J., Liu, T., and Han, B. Deepinception: Hypnotize large language model to be jailbreaker. arXiv preprint 2.X0v:2311.03399, 2023.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of ofaiin-of-though prompting elicits reasoning in large language models. In NeurlPS, 2022.
Li, X., Zhou, Z. Yao, J., Rong. Y. Zhang, L., and Han, B. Neural atoms: Propagating long-range interaction in molecular graphs through efficient communication chan- nel. In ICLR, 2024.
Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., J.an, L., Zhang, X., Zhang, S., Liu, J., et al. Autogen: Enabling next-gen Ilm applications via multi-agent conversation. In ICLR Workshop, 2024.
Liang, T. He, Z., Jiao, W., Wang, X., Wang, Y., Wang. R., Yang, Y., Tu, Z., and Shi, S. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305. 19/ 19/18,11111111 2023.
Xic, J., Zhang, K., Chen, J., Zhu, T., Lou, R., Tian, Y., Xiao, Y., and Su, Y. Travelplanner: A benchmark for real-world planning with language agents. arXiv preprint arXiv:2402.01622, 2024a.
Liu, J., Ding, Z., Liu, Y. and Wang, X. Decentralized multi- agent reinforcement learning with networked agents: Re- cent advances. Foundations and Trends in Machine Learn- ing, 2022.
Xie, Y., Kawaguchi, K., Zhao, Y., Zhao, J. X., Kan, M.-Y., Hc, J., and Xic, M. Self-evaluation guided beam search for reasoning. NeurlPS, 2024b.
Liu, J. Cohen, A., Pasunuru, R., Choi, Y., Hajishirzi, H., and Celikyilmaz, A. Don't throw away your value model! generating more preferable text with value- value-guided monte- carlo tree search decoding. In CoLM, 2024a.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D.,, Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang. J., Yang, J., Tu, J., Zhang, J., Ma, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K.,
Liu, T., Wang, X., Huang, W., Xu, W., Zeng, Y., Jiang, L., Yang, H., and Li, J. Groupdebate: Enhancing the

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
Chen, K., Yang, K., Li, M., Xue, M., Ni, N., Zhang, P., Wang, P., Peng, R., Men, R., Gao, R., Lin, R., Wang. S., Bai, S., Tan, S., Zhu, T., Li, T., Liu, T., Ge, W., Deng, X., Zhou, X., Ren, X., Zhang, X., Wei, X., Ren, X., Fan, Y. Yao, Y., Zhang, Y., Wan, Y., Chu, Y., Liu, Y. Cui, Z., Zhang, Z., and Fan, Z. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024.
Zhou, Z., Zhou, C., Li, X., Yao, J., Yao, Q. and Han, B. On strengthening and defending graph reconstruction attack with markov chain approximation. In ICML, 2023b.
Zhou, Z., Tao, R., Zhu, J., Luo, Y., Wang, Z., and Han, Can language models perform robust reasoning in chain- of-thought prompting with noisy rationales? In NeurIPS, 2024a.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, Narasimhan, K., and Cao, Y. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.0360 10.0029,, 2022.
Zhou, Z., Zhang, Y., Yao, J., Yao, Q. and Han, B. Less is more: One-shot subgraph reasoning on large-scale knowledge graphs. In ICLR, 2024b.
Zhou, Z., Zhu, Z., Li, X., Galkin, M., Feng. X., Koyejo, S., Tang, J., and Han, B. Landscape of thoughts: Visualizing the reasoning process of large language models. arXiv preprint arXiv:2503.22166 2025.
Yao, S., Yu, D., Zhao, J., Shafran, L. Griffiths, T., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. In NeurIPS, 2023.
Yin, Z., Sun, Q., Chang, C., Guo, Q., Dai, J., Huang, X.- J., and Qiu, X. Exchange-of-thowee thttgghh Enhancing large language model capabiliiiess through cross-model com- munication. In EMNLP, 2023.
Zhu, B., Jordan, M., and Jiao, Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In JCML, 2023.
Yuan, S., Song, K., Chen, J., Tan, X., Li, D., and Yang, D. Evoagent: Towards automatic multi-agent gen- eration via evolutionary algorithms. arXiv preprint arXiv:2406. 14228, 2024a.
Yuan, W., Kulikov, I., Yu, P., Cho, K., Sukhbaatar, S., We- ston, J., and Xu, J. Following length constraints in in- structions. arXiv preprint arXiv:2406. 17744, 2024b.
Zhang, J., Xu, X., and Deng, Exploring collaboration mechanisms for I1m agents: A social psychology view. arXiv preprini 12200:2310.02/24, 2023a.
Zhang, K., Yang, Z., and Başar, T. Multi-agent deep re- inforcement learning: A survey. IEEE Transactions on Artificial Intelligence, 2021.
Zhang, Y., Zhang, F., Yang, Z., and Wang, Z. What and how does in-context learning learn? bayesian model averag- ing, parameterization, and generalization. arXiv preprint arXiv:2305. 19420, 2023b.
Zhang, Z., Zhang, A., Li, M., and Smola, A. Automatic chain of thought prompting in large language models. In ICLR, 2023c.
Zhou, D., Schärli, N., Hou, L.,, Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q., al. Least-to-mon most prompting enables complex reasoning in large language models. arXiv preprint 2055:2200..10625, 2022.
Zhou, Z., Yao, J., Liu, J.. Guo, X., Yao, Q., He, L.,, Wang, L., Zheng, B., and Han, B. Combating bilateral edge noise for robust link prediction. In NeurIPS, 2023a.
12

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
Appendix
Notations
14
Theoretical Proof
16
B.1 Proof of Theorem 2.1
16
B.2 Proof of Convergence to BNE
17
B.3 Assumptions...
18
B.4 Scaling Up the System
20
B.5 Proof of Mixing Network Monotonicity
21
Detailed Proofs
22
C.1 Proof of Lemma 2.2
22
C.2 Bounding the Bayesian Regret
23
C33 Comparison with Multi-Agent Debate
25
C.4 Detailed Reward Setting
27
C.5 Task Setups
28
C.6 Hyperparameter
29
D Together API Integration for ECON
32
D.1 Parallel Invocation and Rate Limits
32
D.2 Prompt Construction and Truncation
32
D.3 Online vs. Offline Modes
32
D.4 Sample Workflow
32
D.5 Error Handling
32
E Prompt and Additional Result
32
Example
37
F.1 Case Study
37
F.2 Strategy Examples
37
13

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
A. Notations
This section summarizes the key notations used throughout the appendix and the main paper where applicable.
Notation
Description
General Game Theory and Reinforcement Learning
N
Number of agents.
Index for an agent, E N.}
m
Strategy or policy for agent i. Strategies of all agents other than Bayesian Nash Equilibrium (BNE) strategy profile.
T-1
II;
Set of all admissible strategies for agent i.
a,
Action taken by agent i.
Actions taken by all agents other than i.
(os. a-i)
Joint action profile. Action space for agent i.
A
8,8
State the environment. Payoff utility function for agent i.
U.()
(T-(T-))
Best response correspondence for agent to --ş-
r
Reward received by agent i.
Rmax
Uniform upper bound for rewards.
Discount factor for future rewards, 0 S < 1. Loss function for agent (e.g.. TD loss), parameterized by 0,
(2,0.)
(,(s,a;00)) 0.)
Q-value function for agent for state and action d; parameterized by 0,. Optimal Q-value function.
(s,
0,
Parameters of agent i's network (e.g., Q-network, belief network). Also used denote agent type in BNE proof.
0
Parameters of agent i's belief network.
0.
Parameters of the coordinator's belief aggregation function.
0,
Parameters of the target Q-network for agent i.
T), 1/0, "1, v1', Ylglobal, Ba
Learning rates.
D2
Replay buffer or data distribution for agent i. Historical data up to time
D
(T((s) Vo(),)
Value function for agent under policy (or at time for observation O,).
(s((s)
Optimal value function for agent i.
B, or B.
Bellman operator at time t. True Bellman optimality operator.
B
(T)
Regret for agent i over 7 time steps.
R(T)
Total regret for all agents over time steps.
T
Total number of time steps or episodes.
Small positive constant (e.g., exploration rate, error margin, convergence threshold). Kullback-Leiblee divergence. Expectation operator.
(-a(--l-))
E[]
V
Gradient operator.
MA-LLM Framework Specifies (ECON)
Type space for agent i (in BNE proof). Observation for agent i, typically O = [[e,, es, b:]T.
O
61
Task encoding global observation.
0s
Coordinator's strategy (embedding).
b;
Belief state (embedding) for agent
e;
Prompt embedding generated by agent (considered as its action in some contexts).
Continued on next page
14

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
Table 7 continued from previous page
Notation
Description
Br,
Belief network function for agent
T
Historical trajectory local information for agent i. Coordinator's 's belief aggregation function.
E
Aggregated group information//mbeeddng from coordinator.
C
Coordinator's 's final output (embedding).
Peea(E) )
Posterior distribution over group information determined by the coordinator. Belief distribution over group information maintained by an Execution LLM. Belief entropy at time
RIM(E 1.)
He
T(X;Y) Z)
Conditional mutual information between random variables X and Y given Z. A function related to coordination, depending on e, and E.
E(e,, E)
y
Game regularity constant (distinct from learning rate).
k
Concentrability constant.
u4
Output (e.g., text, solution) generated by Execution LLM i. Number of clusters the hierarchical scaling-up framework. Global Coordinator L.M..
Coordgiobal
Coord
Local Coordinator LLM for cluster k.
S
Global strategy (embedding) from Coordglobal- Local strategy (embedding) for cluster k from Coorde-
C₁
Cluster of Execution LLMs (also used for local output from Coordg, context dependent).
EC,EL
Convergence thresholds for final output and losses respectively.
Rehreshold
Reward threshold for early stopping in scaling-up framework. Action Likelihood Reward.
AL
is
Task-Specifts Reward.
rfc
Collaborative Contribution Reward.
01,02, 0s
Weights for combining reward components.
Que
Total Q-value from the mixing network.
h
Hidden state of layer / in the mixing network.
WL. b
Weights and biases for layer of the mixing network. Non-decreasing activation function for layer of the mixing network.
(-(-)
dk
State distribution at step k induced by policy
T.,ps
Temperature and nucleus sampling (top-p) parameters for LLM i.
Car
Loss for dynamic reward weight adjustment.
Proof Specifics (Regret Bounds, Convergence)
Ei E2(t)
Error terms in Q-value decomposition (related to Q-function estimation error).
A(t)
Policy suboptimality term in Qavuuu decomposition.
a(),a)
Q-function estimation error [Qr(s, (s, a) - Q" (s, a) at time f.
C2,
Generic constants appearing in convergence rate and regret bounds.
& E.
Martingale difference noise term in stochastic approximation proofs. Bound on the variance of the noise term Çt.
o²
P
Policy space (assumed convex and compact in proofs). Policy objective function (e.g., expected discounted return of policy π))
((T)
L
Lipschitz constant (e.g., for policy gradient).
D(P)
Diameter of the policy space P.
8min
Lower bound on persistent policy suboptimality in competitive (debate) settings. Value the game for agent i (in zero-sum games).
0
(mT))
Entropy of policy Ry. Minimum entropy required for policies in competitive settings to avoid exploitation.
hmin
15

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
B. Theoretical Proof
B.1. Proof Theorem 2.1
Proof. We aim to prove the existence of a Bayesian Nash Equilibrium (BNE) in our multi-agent LLM framework under the specified conditions. The proof proceeds by verifying the conditions of Glicksberg's Fixed Point Theorem, which guarantees the existence of fixed point continuous games with infinite-dimensional strategy spaces.
Step 1: Define the Best Response Correspondence
For each agent i, define the best response correspondence BR, as:
( = { E II m maximizes U3 (0s, Ti, T-1)),
where IT, the set of admissible strategies for agent and T- denotes the strategies of all other agents.
Step Verify the Conditions of Glicksberg's Fixed Point Theorem
To apply Glicksberg's Fixed Point Theorem, need to verify the following conditions for each agent d:
1. Strategy Space Compuctness and Convexity:
The strategy space II, is non-empty, convex, and compact in the topology of pointwise convergence.
2. Continuity of Payoff Functions:
The payoff function U,(O;, Ti, m-)) .) continuous in (T,, n-1) for each fixed 0z.
3. Quasi-Concavity of Payoff Functions:
The payoff function U0,0,, Te. --4) is quasi-concave in π, for each fixed 0g and --
Verification:
1. Strategy Space Compactness and Convexity:
The strategy space πI; consists of all measurable functions mapping types to actions a, in A,, Since O and Ai are compact metric spaces, and strategies measurable functions from one compact space another, the space of such functions II, can be endowed with the topology of pointwise convergence, making it compact by Tychonoff's Theorem. Convexity follows because the set of mixed (probabilistic) strategies is convex, and any convex combination of measurable functions is measurable.
2. Continuity of Payoff Functions:
For fixed 0,, the payoff function U.(0,, Te, -i) depends continuously T; and T-z due to the continuity of U, in actions and types. Specifically, since U; is continuous in (a;, a-i) and the strategies nis Th map continuously from types to actions, the composition (1,O,, T. (0,),, ग (0 .)) is continuous in (T,,
3. Quasi-Concavity of Payoff Functions:
For each 0 and m- the function म; + U.(O,, Ri, T-i) is quasi-concave because U, is quasi-concave in a, and the strategies are linear in the space of mixed strategies. Therefore, any convex combination of strategies does not decrease the utility, satisfying quasi-concavity.
Step 3: Establish Upper Hemicontinuity and Non-Empty, Convex-Valuedness of Best Response Correspondences
We need show that (R((TT))) is upper hemicontinuous with non-empty, convex values.
1. Non-Empty, Convex Values:
For each n_;, since II, is compact and convex, and U, continuous and quasi-concave T, the Weierstrass Theorem ensures that the maximum exists; hence, BR, (T=4) is non-empty. Convexity follows from the quasi-concavity of U, in me implying that any convex combination of best responses also a best response.
16

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
2. Upper Hemicontinuity:
Upper hemicontinuity of BR; means that for any - n- and any F. (T- there exists " (mT,,) such that x - .T.- This property holds because the payoff function U; is continuous in (Tz, m-2), and the strategy spaces are compact.
Step 4: Application of Glicksherg's Fixed Point Theorem
Having verified all the conditions, we can apply Glicksberg's Fixed Point Theorem, which states that if each player's strategy set is compact and convex, and their payoff functions are continuous and quasi-concave in their own strategies, then the game has least one Nash Equilibrium in mixed strategies.
Step 5: Conclusion
Therefore, there exists strategy profile म|" = (Ti, T2 ... x.) such that for each agent
(R((
meaning that no agent can unilaterally deviate to improve their expected payoff, given their beliefs about other agents' types and strategies. This strategy profile constitutes a Bayesian Nash Equilibrium in our multi-agent L.LM framework.
B.2. Proof of Convergence BNE
Proof. We aim to show that, by minimizing the TD loss for each agent's Q-network, the agents' strategies converge to Bayesian Nash Equilibrium (BNH).
Assumptions:
1. The Q-networks Qi(s, 0j; 0s) are parameterized by prompt embeddings 0,, and the mapping from 0 to Q, is continu- ously differentiable.
2. The exploration strategy ensures sufficient coverage of the state-action space (e.g., c-greedy with decaying c).
3. The loss function (,,),) is convex or has Lipschitz continuous gradients with respect to 0,.
4. The gradient Vo, (())) is Lipschitz continuous.
5. The learning rate 7/, is chosen such that it satisfies the Robbins- conditions: Eee Te oc and E.e 11 < oc.
Step 1: Defining the TD Loss Function The TD loss function for agent is:
( * max (i d4: 0,) Qi az 0.) } (1:+
(,(+:)= = E(s.a)
of
This loss measures the discrepancy between the predicted Q-value and the target Q-value based on the reward and the estimated optimal future Q-value.
Step Gradient Descent Update Agent i updates its Q-network parameters according
0 = 0 T Ve, Ve, (e))).
The gradient of the loss function with respect to the parameters is:
V (.0)) = (( (2 (Tx rr max of
Q, (3,,, Qi Q((s,a;;))) (
(-Vo.(((s.a,:0))))
Step 3: Convergence of Gradient Descent with TD Loss Under the assumptions that L,(0;) has Lipschitz continuous gradients and the learning rate dt satisfies the Robbins-Momn conditions, stochastic gradient descent converges stationary point 0 of L,(+:):
lim 0% = 07- 100
17

---

Qi(s, azi 02) = r; + max Q,'s', de; From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
At convergence, the gradient vanishes:
Vo. T. 0,
which implies:
(( (r+ max d, m (,''',d;;')) Q((s,a;03)) df
Ve, Vo. )
= 0.
Assuming that the Q-network parameterization is such that the above condition holds only when:
(;) = + max Q, (7)
the Q-network accurately estimates the expected cumulative rewards, aligning the agent's policy with the optimal response to other agents" strategies.
Step 4: Characterizing the Stationary Point At the stationary point 8,, the Q-network satisfies the Bellman optimality condition:
of
This condition ensures that the agent's policy ग. (a: s: 0?) is a best response to the current policies other agents, maximizes the expected cumulative reward.
Step 5: Establishing Bayesian Nash Equilibrium Since each agent's policy is a best response to the policies of others, the set of policies } constitutes Bayesian Nash Equilibrium. Each agent maximizes its expected utility given its belief' about other agents' types and strategies, fulfilling the definition of BNE.
B.3. Assumptions
Our theoretical analysis relies on four key assumptions that are both common in multi-agent systems (Zhang et al., 2021; Liu et al., 2022) and specifically relevant to our MA-LLM framework.
Definition B.1 (System Components). In our MA-LLM framework:
Each agent i's observation O; (ex, es: b.jT, where encodes the task, represents the coordinator's strategy, and by is the belief state
Each agent's action is its prompt embedding e, generated by belief network (1 0g;
The coordinator aggregates beliefs through fe((b, J i 02) into group information
Assumption B.2 (Bounded Rewards). The rewards from coordinator final output uniformly bounded: |r:(O,, e,, E) < Rmaxx for all O,, e, E.
This assumption is standard in reinforcement learning (Sutton & Barto, 2018) and critical since it ensures numerical stability in the learning process of LLMs, preventing reward explosion that could lead to unstable training.
Definition B.3 (Historical Data and Posterior). Given historical data D, {(O), cf C*) C
Prest E Dr, Oi, ex) is the posterior distribution over group information determined by the coordinator
(PIM(E) E Dr, O,, c,) is the belief distribution maintained by each Execution LLM
Assumption B.4 (Approximate Posterior Alignment). Execution LLMs aim to align with the posterior distributions determined by the Coordinator LLM within an acceptable error margin >>
DKL (PLLM D., e,), Ppost E Dr, O, e;)) Se.
where DKL. denotes the Kullback-Leihle divergence.
18

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
This approximate alignment acknowledges that perfect alignment impractical but strives for close approximation:
The Coordinator LLM acts centralized distributor of strategic guidance.
Exccution LLMs maintain belief alignment through prompt (detailed in Section 2).
Monotonic guarantee in ECON mixing optimization network B.5.
Such alignment has been shown in (Foerster et al., 2018; Jaques et al., 2019) enhance coordination.
Definition B.5 (Belief Entropy). For given time the belief entropy H, is defined as the Shannon entropy of the aggregated belief embeddings:
N
H1
Eb, B, [b, log bi],
where B; represents the belief network of agent Assumption B.6 (Game Regularity). There exists vj >> 0 such that for any 4 < 12, if He, H22 S llog2 2, then
I E(e,, E) Da,) I 4n E(ee, E) Dta);
for all agents i, where 0 are the belief network parameters.
This information-theoreti assumption serves multiple purposes in our framework:
It ensures the stability of belief updates between LLMs over time by bounding the entropy difference of belief states.
The mutual information term ((e,, quantifies how much an LLM's belief network parameters affect its coordination through prompt embeddings.
The bound 41 controls the rate at which LLMs can adapt their belief states based on observed interactions and coordinator guidance.
Definition B.7 (Value Function and Bellman Operator). For each Execution LLM i:
The value function V2(O,) = E r 1 = Oi] estimates the expected cumulative rewards.
The optimal prompt embeddings e," maximize the Q-function Q,(O,, ex; 0,5) at time t.
The Bellman operator B, transforms value function another: (B,))(O:) max., B(r, + +V(O)))
Assumption B.8 (Concentrability). There exists λ < oc such that
E
M (BB BT) of
of E-t)
$ TT,
where 3 is the true Bellman operator.
This assumption is fundamental to our theoretical guarantees:
It ensures that the value function estimates by each LLM converge to their true values at an appropriate rate.
The constant x bounds the cumulative estimation error across all LLMs, critical for establishing our regret bounds.
In our MA-LLM system, this translates to the stability of response quality improvements during training.
Collective Impact: Together, these assumptions enable us to:
Establish the existence of BNE in MA-LLM system (Theorem
Derive meaningful regret bounds for the learning process (Lemma
Guarantee the convergence of our iterative training procedure (Proposition 1)
19

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
B.4. Scaling Up the System
While ECON efffetivvly handles moderate-scale scenarios, large-scale multi-LLM systems require additional structure maintain efficiency and theoretical guarantees. To that end, employ hierarchical design in which Execution LLMs organized into clusters, each governed by Local Coordinator LLM, and Global Coordinator LLM oversees the entire set of clusters. Each local cluster establishes a local Nash equilibrium, which is then integrated into global equilibrium.
Algorithm 2 Scaling-Up Framework for ECON
Require: Global Coordinator Coordglobal: Local Coordinators {Coord thresholds {ec, Ruh EL} Ensure: Hierarchical Nash Equilibrium across K clusters
1: Initialize: cluster embeddings ] belief networks {B,(0.)), prompt embeddings {c:}
2: while not converged do
3:
> Inference Phase Hierarchical execution without updates
4: S C-oodggoobaal
Global strategy generation
5: for each cluster & = 1 to K in parallel do
6: Os [et, S, Ex]
Combine global info with cluster state b Local strategy refinement
7:
Sk + Coordy
8:
for each Execution LLM i E C₁ in parallel do
9:
b, + Bi(Ti, oz; 0
D. Compute belief state
10:
C; f ComputeEmbeddinng(
D Generate prompt embedding Local observation > Generate output
11:
Oj + [e,, Sk, b,]
u, t ExecLLM((query, e;)
13:
r; airAl + azr;''
.S + agr.c
Local reward
14:
Store (Ti, ei, u;) in cluster buffer
15:
end for
16:
+ Coorda (f (EEc.)
> Aggregate cluster output
17:
end for
18: + Coordgiooaal ( 1K 1=1)
Global final output D Optimization Phase Update parameters for hierarchical BNE
20:
for each cluster & 1 to K do
21:
R& + Relobal(sim(() (Ck; C))
Global reward for cluster Improve cluster coordination
22:
Update Coord, parameters using Rş
23:
Update cluster embedding Eş via belief encoder
24:
for each LLM i E C₁ do Update 0 via CTD end for
25:
Update individual belief network
27: end for
28: Update global mixing network & via Lmix
29:
Convergence Check
30: converged + ||C++1 C+ll s ca and E R 2 Ru,. and ALwol I
31:
end while
32:
return Hierarchical Nash Equilibrium with optimized {0 {Ex). ó
B. DETAILED EXPLANATION
The hierarchical scaling framework operates through structured two-phase process designed to maintain both efficiency and theoretical guarantees at scale. In the inference phase, the system executes hierarchically from global strategy generation down to individual L.M outputs without any parameter updates, ensuring stable execution. Subsequently, the optimization phase updates parameters in a bottom-up manner to achieve hierarchical BNE, where each cluster reaches local cquilibrium while contributing to global coordination. This separation of concerns allows the system to handle large-scale multi-LLM deployments while preserving the convergence properties established the base ECON framework.
Initialization.
20

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
Clustering. Partition Execution LLMs into K clusters (C1, .,.,..,, C e.g., by task similarity.
Local Coordinators. Each cluster C uses a Local Coordinator LLM Coord, to manage within-cluste interactions.
Global Coordinator. A Global Coordinator LLM Coordglobal aligns all clusters by forming a global final output C.
Parameters. Initialize belief networks (B:(0)), prompt embeddings (ei): ) and cluster embeddings (Ex).
Inference Phase. During inference, the system executes hierarchically without updating any parameters:
The Global Coordinator generates high-level strategy S from the global observation
Each Local Coordinator Coord, refines S into local strategy Sy based on cluster state Eg--
Execution LLMs compute belief states b, and generate outputs u, using their current parameters.
Local Coordinators aggregate within-cluster outputs to produce cluster outputs {cu}
The Global Coordinator aggregates cluster outputs to form the global final output C.
Optimization Phase. After inference, the system updates parameters hierarchically to achieve BNE:
Each cluster receives a global reward Rg = Rgtobal (sim(ca, C)) measuring alignment with C.
Local Coordinators update their parameters based on R2 to improve global consistency.
Individual belief networks (0 are updated via TD loss CTD-
Cluster embeddings Eş are updated via the belief encoder.
The global mixing network updates via Liix to coordinate across clusters.
The system converges to hierarchical Nash equilibrium when: (1) the global output stabilizes (I|C++1 Cell S (c), (2) the average global reward exceeds Ra, and (3) the total loss converges (tolt S cg). This hierarchical approach maintains ECON's theoretical guarantees while enabling efficient parallelization across clusters.
Convergence.
B.5. Proof of Mixing Network Monotonicity
Proposition B.9 (Monotonicity of Mixing Network). The mixing network Qui is monotonic in each individual Q-value Qú ensuring that improvements in Q; lead to improvements in Q
Proof. The mixing network is designed using positive weights and non-decreasing activation functions. Specifically, let the mixing network be composed of layers where each layer computes:
h' = ='d'''' +.6),
where:
n10 = 1Q.,,
W has nen-negative entries.
Φ non-decreasing activation function (e-g., ReLU).
21

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
We proceed by induction show that each component of i is non-decreasing function of Q
Base Case: At layer 1 = 0, h2? Qi so
aQ; ō.j 2 0.
ah
Inductive Step: Assume a
2 0 for all k. Then, for each component n
aQ:
h2 - 0'
Σ W W + 03
Since Wie 2 0 and of is non-decreasing:
W Oh 11 iQ,
Σ W W + +b}
2 0. Σ &
aQ,
because d² 2 0 and
Bh: 8Q. 2 0 by the inductive hypothesis. Therefore,
8Qu 8Q;
> 0, ensuring monotonicity.
This monotonicity property is crucial as it ensures that improvements in individual agent performances contribute positively to the overall system performance, aligning local and global objectives within ECON.
C. Detailed Proofs
C.1. Proof of Lemma 2.2
Proof. Consider the value functions under policies T' and x:
20 rr ax) 100
-0 5.. (T) = Ex
8
ViT (8) Ex'
Σ
7
(skk ax)
880 = 8
Their difference is:
(iT(s) - V27(s) E.g' Σ Arissks au)
Σ (r()
n#th (Esendy, (ri(sk. ax)] Esandt (r)()))
k=
Assuming the difference in state distributions is negligible (justified under Assumption B.8), focus on action differences. Using the Q-function definition:
= as, a..) + YBsap (V (5)),
we can write:
Σ
x E Bo d(.) V (sx)).
- =
k=0
Since V,7 Ba(wn(ss)) [QT (sk, ]z)], have:
(s) Σ
B
Fog (ew)
[QT (sk, a/) Bay~s((ww) 1Q7 (Sk, ak)]
k=0
Switching the order of expectations and summing over hi we get:
1
Vil (s) - (a) = 1-> Eand, (Q (s,,4,a (7, ai a.;))
22

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
C.2. Bounding the Bayesian Regret
We provide a rigorous proof the O(NVT/(1- 7)) regret bound through stochastic approximation theory and convex optimization analysis.
C.2.1. PROBLEM SETUP AND REGRET DEFINITION
Starting from the regret definition for agent over I steps:
(T) = Es,,.
VZ
where the expectation is over the randomness in state transitions and policies.
Applying Lemma 2.2:
- (s)) = 1- Bas Bagtat att 10, (se, (s,
C.2.2. Q-VALUE DECOMPOSITION
We decompose the Q-value difference into three components:
Q (st, of a²i))= - a'i)
= (0,7 (st, et a²) - Q7 (st, ai ,a"1))
Error (1)
a (st,
(5t oi a²
Policy Suhoptimality: ((t) + (Q als a²,) Q (8t, dit
Error Term 2: Bz(()
C.2.3. Q-FUNCTION CONVERGENCE ANALYSIS
Consider the Q-learning update rule with learning rate 1/; vo/Vt:
Q++1(s, = Qc(s, + (r(s,a) a) + y Qe(s', (z(s,a)) a)
a²
Define the estimation error (x a) = (Q((s, Q*,ssa))) a)l.
Lemma C.1 (Q-function Convergence Rate). Under the following conditions:
(i) Learning rate schedule т) o/o/ /tt with Tjo >
(ii) Bounded rewards: [r(s, a)l S Rmax. for all (s, a)
(iii) Standard stochastic approximation conditions (Robbins-Monro)
The Q-function estimation error satisfies:
C (((((s,)) S
Vi
where C1 00 (
Riwae )-
Proof. Following the stochastic approximation framework of Borkar (2009), we analyze the error recursion:
(+1(s,,) a) < (1 - -n)))(ss a) + mey m a') + me
o²
23

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
where & the noise term with F 0.
Taking expectations and using the contraction property:
Elec+1] $ (1 7/ - 7)) + t
where 2² bounds the variance of the noise.
With 7/+ 77//v solving this recursion yields:
Bled] C1
Vii
where the constant C₁ depends on the initial error, discount factor, and noise variance.
C.2.4. POLICY CONVERGENCE ANALYSIS
For the policy update, we consider gradient-based methods in the convex policy space:
Te-1 = 11p [Tt + V (Te),,
where 1p is the projection onto the policy space P.
Lemma C.2 (Policy Suboptimality Bound). Under the following conditions:
(i) Convex and compact policy space P
(ii) L-Lipschitz continuous policy gradient: [[ - VJ(T)) L|TT x'1l
(iti) Learning rate schedule = -7//
The policy suboptimality satisfies:
C2 Vi
max
where C2 = 0 (D(P)) (LD(P) and D(P) is the diameter P. v/
Proof. Following the online convex optimization framework of Hazan (2016), the regret gradient descent with learning
rate 1/- = To/ Vl satisfies:
-(((*) J( D²(P)
Vt
2110
2
Dividing by t and taking the limit:
C2 ΣΣ s
S
Since J(x) = Exnar [v) and by the performance difference lemma, this translates to the Q-value suboptimality bound.
C.2.5. COMBINING ERROR TERMS
From Lemmas C.I and C.2, we have:
|Ei(0) < 5 CI/VI
|Ez(t) S Sa/VE
A(t) S Co/Vt
Therefore:
201 + +C2 C ((.
(st, - (,,
Vt V
24

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
C.2.6. FINAL REGRET BOUND
Substituting back into the regret expression:
N
R(T) = Σ (T)
==1
T
Σ
1
Σ
E
(st, ,a")))
Q," (st, a".))
i=1
1=1
1 c Σ
$ Σ 1
Vi
NC T 1
Σ
Vi
Lemma C.3 (Harmonic Sum Bound). For the harmonic with exponent 1/2:
T
Σ
Vx de 2 2V2TT.
t=1
Therefore, the total regret satisfies:
2NCVT R(T) S 1-7
NVT
=0
1-7
This completes the rigorous proof of the sublinear convergence rate, explicitly showing how the learning rate schedule n// Vi ensures the ((1-1/2) convergence of both Q-function estimation errors and policy suboptimality, leading to the final -))) regret bound.
C.3. Comparison with Multi-Agent Debate
We now analyze the regret bound in multi agent debate settings to contrast with our cooperative framework. The key difference lies in the persistence of policy suboptimality due to the competitive nature of debate.
C.
SETUP FOR DEBATE
In multi-agent debate settings, agents operate in competitive environment characterized by:
Zero-sum constant-sum reward structure: Ei" rr(ss a) for all as)
Strategic uncertainty: each agent must model opponents' strategies
No coordination mechanism: agents independently optimize their policies
C.3.2. FUNDAMENTAL LIMITS COMPETITIVE SETTINGS
Lemma C.4 (Persistent Suboptimality in Competitive Games). Consider a multi-agent debate setting with N agents in zero-sum game. Under the following conditions:
(i) The game has no pure strategy Nash equilibrium
(ii) Agents use no-regret learning algorithms
(iii) Each agent faces strategic uncertainty about opponents
25

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
Then there exists constant min
O such that for all 2 To:
E max (s,, a; d' Qf di d'xe)
8win
Proof. In zero-sum games without pure strategy equilibria, the Nash equilibrium requires mixed strategies. By the minimax theorem (Ben-Davic & Blais, 2023):
max min J (T = min max Ji( x Vi: TA R-
x-c mh
where v; is the value of the game for agent i. For any deterministic policy पं. at time t, there exists an adversarial response Th, such that:
( m- S 5
where is the exploitability gap. Since agents use no-regret learning, they must maintain sufficient randomization to avoid exploitation. Following Fudenberg & Levine (1998), the minimum entropy required is:
(m,) Amin > 0.
This entropy constraint directly implies a lower bound on policy suboptimality:
el1min
Omii 2(1 7)
C.3.3. REGRET ANALYSIS FOR DEBATE
Following the same decomposition framework from Section C.2, have:
1
(st) (si)
I Eayo. (2T (St, at a -, (sc. a',ه)
1 (Ei + Adebute (t) + Ez(t))
where:
(t), Ci/V2 (Q-function estimation errors, same before)
Adebate (t) 8min (persistent policy suboptimality from Lemma C.4)
Computing the regret for agent ;:
Relebate (7) = (s)) - v.))
E
1 Σ 8min 2C1
Vt
For sufficiently large there exists To such that for all & 2 To: 6min > 2220/ Vt. Therefore:
201
8min
Relebate (T)
8min
Vt
(=1++1
5 8min
T
To)
2(1 7)
T
1
Summing over all agents:
N Σ
NT
R (T) =
Rdebute (T)
S
1->)
1
iml
26

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
C.3.4. COMPARATIVE ANALYSIS
The fundamental difference between ECON and debate settings is:
ECON (Cooperative Learning):
Policy suboptimality: AECON (t) O(1/vZ) 0
Total regret: (TC
Convergence to Bayesian Nash Equilibrium through coordination
Multi-Agent Debate (Competitive Learning):
Policy suboptimality; Adchate (t) 2 6min >
Total regret: Raebase (T) (2(TT((1 - -)))
Persistent randomization required to avoid exploitation
This O(VT) vs (2()) gap demonstrates that competitive debate settings suffer from:
1. Strategic Cycling: Agents continuously adapt opponents, preventing convergence optimal deterministic policies
2. Exploration-EEploottttoo Conflict: Need for defensive randomization conflicts with exploitation of learned knowledge
3. Information Inefficiency: Lack of coordination prevents efficient use of collective information
In contrast, ECON's cooperative framework with Bayesian policy optimization enables agents to:
Share information through the communication phase
Coordinate strategies toward Bayesian Nash Equilibrium
Achieve diminishing policy suboptimality through joint optimization
This theoretical analysis is supported by empirical evidence in competitive multi-agent settings (Lanctot et al., 2017; Lowe et al., 2017), where agents exhibit persistent strategic cycling and fail to achieve sublinear regret bounds.
C.4. Detailed Reward Setting
The reward function f provides feedback on each agent's performance while respecting Assumption B.2, ensuring all reward components are uniformly bounded by Rmax- Drawing inspiration from maximum entropy inverse reinforcement learning (Zhu et al., 2023), we define the Action Likelihood Reward ra = minxamax. (u, C)), where sim( u. C*) = C,-C' [u,|[|[[] measures the consistency between an agent's output u, and the coordinator's final output C. Following Hao et al. (2023), the Task-Specific Reward r!TS min eval (u; task)) evaluates domain-specific objectives through the coordinator's assessment, where eval computes normalized scores considering solution correctness in mathematical problems or response relevance in planning tasks. Building upon Xie et al. (2024b), the Collaborative Contribution Reward , = min( Rmaxx quality (us, {uy 1)+))) enables the coordinator to assess each agent's 's output quality within the multi- agent context, where quality evaluates the response's coherence and creativity while considering contribution to the collective solution. The total reward combines these components as r; airA + a2r? TS + asr CC where the weights 01 + a2 + 03 ensure the total reward is bounded by Rmax. To enhance adaptability and learning efficiency, we introduce dynamic mechanism adjusts these weights using gradient- based updates a + ag 7a dLst Dog, where Lar N. actual r expoctedy2 measures the discrepancy between actual and expected rewards.
27

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
C.5. Task Setups
GSM8K is a benchmark for mathematical reasoning that requires multi-step problem solving. Given context description and question, it requires step-by step mathematical reasoning and computation arrive at a final answer. The dataset contains approximately 7.5K problems in the training set and 1.3K problems in the test set. Problems range from basic arithmetic to complex word problems, testing both mathematical and logical reasoning capabilities.
SVAMP is a challenging mathematical word problem dataset specifically designed to test the robustness of language models in solving arithmetic problems. It contains 1,000 elementary math word problems, carefully curated to probe for specific vulnerabilities in mathematical reasoning systems. The problems require understanding both mathematical concepts and natural language semantics, with focus structural variations that test genuine problem-solving capabilities rather than pattern matching.
Strategy QA is question answering dataset that focuses on multi-hop reasoning and strategic thinking. It consists of 2,290 yes/no questions, each requiring implicit multi-step reasoning and background knowledge to arrive at the correct answer. Unlike traditional QA datasets, Strategy QA questions cannot be answered by simply retrieving and combining explicit facts, making an effective benchmark for testing complex reasoning capabilities.
MATH a comprehensive mathematics dataset spanning various topics from algebra to calculus. It contains approximately 12K problems across different difficulty levels, with detailed step-by-step solutions. The dataset structured into multiple categories including algebra, counting and probability, geometry, intermediate algebra, number theory, prealgebra, and precalculus, making it particularly effective for evaluating mathematical problem-solving capabilities across different domains.
GSM-Hard a specialized subset of mathematical word problems specifically designed to test advanced reasoning capabilities. It contains problems that are significantly more challenging than standard GSM8K problems, requiring more complex multi-step reasoning and mathematical operations. The dataset focuses on problems that typically have lower success rates with standard approaches, making particularly useful for evaluating the upper bounds of model performance.
TravelPlanner is benchmark crafted for evaluating language agents in tool-use and complex planning within multiple constraints. The dataset comprises 1,225 queries in total, divided into training (45 queries), validation (180 queries), and test (1,000 queries) sets. The benchmark incorporates three types of constraints: environment constraints for testing adaptability to real-world conditions, commonsense constraints for evaluating practical reasoning, and hard constraints for assessing the ability satisfy specific user requirements such as budget limitations. This structure makes TravelPlanner particularly effective for evaluating both reasoning capabilities and practical planning skills in real-world scenarios.
28

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
C.6. Hyperparameter
C.6.1. LLAMA 3.1 8B ON MATH
C.6.2. LLAMA 3.1 ON GSM8K
Table 8: Hyperparameters (8B, MATH)
Table 9: Hyperparameters (8B, GSM8K)
Parameter
Value
Parameter
Value
Training Configuration
Training Configuration
Episodes per Task
120
Episodes per Task
100
Buffer Size
32
Buffer Size
32
Batch Size
16
Batch Size
16
Update Interval
8
Update Interval
8
Optimizer
Adam
Optimizer
Adam
Learning Rate (7)
0.001
Learning Rate (1))
0,001
Learning Rate (7ond))
0.0005
Learning Rate (Nowndd
0.0005
Discount Factor (7)
0.99
Discount Factor (7)
0.99
Network Architecture
Network Architecture
Entity Dimension (d)
256
Entity Dimension (d)
256
Belief State Dimension (ds)
128
Belief State Dimension (d,)
128
Attention Heads (H)
4
Attention Heads (H)
MLP Hidden Size
256
MLP Hidden Size
256
Transformer Blocks
2
Transformer Blocks
2
Key/Query Dimension
64
Key/Query Dimension
64
Feed-forward Size
1024
Feed-forward Size
1024
Dropout Rate
0.1
Dropout Rate
0.1
Layer Norm Epsilon
le-5
Layer Norm Epsilon
le-5
Temperature & Sampling
Temperature & Sampling
Tiin
0.1
Tiin
0.1
max
2.0
T Tmax
2.0
Pmin
0.1
Pmin
0.1
Pinx
0.9
Pmux
0.9
Reward Configuration
Reward Configuration
Ryax
1.0
Rmax
1.0
(1, 02 0:
0.4, 0.4, 0.2
01, 02 03
0.4, 0.4, 0.2
Loss Weights
Loss Weights
As. X, Am
0.1, 0.1, 0.1
As. X, Am
0.1, 0.1
Early Stopping
Early Stopping
EC
0.01
EC
0.01
er.
le-4
6r. Rehreshold
le-4
Ratresbold
0.7
0.7
T putiience
5
T putience
5
Model Size
Model Size
Learnable Params
1.7M
Learnable Params
~.77
29

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
C.6.3. LLAMA 3.1 70B ON MATH
C.6.4. LLAMA 3.1 70B ON GSM8K
Table 10: Hyperparameters (70B, MATH)
Table 11: Hyperparameters (70B, GSM8K)
Parameter
Value
Parameter
Value
Training Configuration
Training Configuration
Episodes per Task
150
Episodes per Task
100
Buffer Size
32
Buffer Size
32
Batch Size
16
Batch Size
16
Update Interval
8
Update Interval
8
Optimizer
Adam
Optimizer
Adam
Learning Rate (7)
0.001
Learning Rate (7)
0,001
Learning Rate (ooord
0.0005
Learning Rate (Reandd
0.0005
Discount Factor (?)
0.99
Discount Factor (?)
0.99
Network Architecture
Network Architecture
Entity Dimension (d)
256
Entity Dimension (d)
256
Belief State Dimension (ds)
128
Belief State Dimension (d,)
128
Attention Heads (!))
4
Attention Heads (H)
MLP Hidden Size
256
MLP Hidden Size
256
Transformer Blocks
2
Transformer Blocks
2
Key/Query Dimension
64
Key/Query Dimension
64
Feed-forward Size
1024
Feed-forward Size
1024
Dropout Rate
0.1
Dropout Rate
0.1
Layer Norm Epsilon
1e-5
Layer Norm Epsilon
1e-5
Temperature & Sampling
Temperature & Sampling
mii
0.1
Tii
0.1
maa
2.0
T Tmax
2.0
Pmin
0.1
Pmin
0.1
Pmax
0.9
Pmax
0.9
Reward Configuration
Reward Configuration
Rmax
1.0
Rmax
1.0
01, 0: 0g
0.4, 0.4, 0.2
Q1, 0z2, 0g
0.4, 0.4, 0.2
Loss Weights
Loss Weights
As X, Am
0.1, 0.1, 0.1
Nt. X, Am
0.1, 0.1, 0.1
Early Stopping
Early Stopping
6C
0.01
6C
0.01
6L
le-4
6L
1e-4
Ratreshold
0.7
Rehreshold
0.7
T ppatience
5
T patieenc
5
Model Size
Model Size
Learnable Params
1.7M
Learnable Params
~1.7
30

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
C.6.5. LLAMA 3.1 405B ON MATH
C.6.6. LLAMA 3.1 405B ON GSM8K
Table 12: Hyperparameters (405B, MATH)
Table 13: Hyperparameters (405B, GSM8K)
Parameter
Value
Parameter
Value
Training Configuration
Training Configuration
Episodes per Task
200
Episodes per Task
150
Buffer Size
64
Buffer Size
64
Batch Size
32
Batch Size
32
Update Interval
8
Update Interval
8
Optimizer
A
Optimizer
A
Learning Rate (7)
0.0005
Learning Rate (7)
0,0005
Learning Rate (7onnd)
0.0003
Learning Rate (Nannd)
0,0003
Discount Factor (7)
0.99
Discount Factor (γ)
0.99
Network Architecture
Network Architecture
Entity Dimension (d)
512
Entity Dimension (d)
512
Belief State Dimension (ds)
256
Belief State Dimension (d4)
256
Attention Heads (I))
8
Attention Heads (H)
8
MLP Hidden Size
512
MLP Hidden Size
512
Transformer Blocks
4
Transformer Blocks
4
Key/Query Dimension
64
Key/Query Dimension
64
Feed-forward Size
2048
Feed-forward Size
2048
Dropout Rate
0.1
Dropout Rate
0.1
Layer Norm Epsilon
le-5
Layer Norm Epsilon
le-5
Temperature & Sampling
Temperature & Sampling
mii
0.1
Tin
0.1
max
2.0
T Tmax
2.0
Pmin
0.1
Pmin
0.1
Pmax
0.9
Pmax
0.9
Reward Configuration
Reward Configuration
Rmax
1.0
Rmax
1.0
01, 0: 0g
0.3, 0.5, 0.2
01, 02 03
0.4, 0.4, 0.2
Loss Weights
Loss Weights
bs A, Am
0.1, 0.1, 0.1
Ag. X, Am
0.1, 0.1, 0.1
Early Stopping
Early Stopping
6C
0.01
EC
0.01
6L
le-4
6L
1e-4
Ratreshold
0.75
Rehreshold
0.7
T patience
8
T patience
5
Model Size
Model Size
Learnable Params
2..5
Learnable Params
~2.5
31

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
D. Together API Integration for ECON
This subsection elaborates on how we invoke Together API to query different LLMs (LLaMA3.1 8B/70B/40SB, Mistral-7B, GPT4 turbo) our ECON framework across GSM8K, GSM- Hard, MATH, SVAMP, and StrategyQA.
D.I. Parallel Invocation and Rate Limits
We employ four LLMs in ECON (one Coordinator and three Executions). To respect Together's Requests per Minute (RPM) and Tokens per Minute (TPM) limits, we maintain:
job queue that sends LLM requests mini-batches if concurrency could exceed the allowed RPM.
A global token counter to ensure we do not surpass the daily or per-minute token quota. If query risks going over, we delay or split the request.
A backoff mechanism that retries an LLM call up to 3 times if we encounter rate-limit or network errors, incrementally increasing the wait period.
D.2. Prompt Construction and Truncation
Each Execution LLM receives a prompt string containing:
1. Local Belief State bj. This is textual or embedded summary of the agent's partial view of the environment, updated via the belief network.
2. Coordinator Strategy The high-level guidance from the Coordinator LLM.
If the total token count (prompt expected output) could exceed the Together API's per-request cap (e.g., 2048 tokens), truncate repeated instructions compress partial states. Similarly, we impose 50-token (soft) and 70-token (hard) limit for the Coordinator outputs, aligning with Sec.4.1.
D.3. Online vs. Offline Modes
By default, we adopt an offline training procedure each dataset's training split (see Sec.4.1). During training, repeated queries are sent to gather transitions for the TD and mixing losses. For test evaluation, we freeze all parameters and still use the same prompt construction pipeline, but no further updates are performed.
D.4. Sample Workflow
An ECON iteration typically goes as follows:
Coordinator LLM call: We pass aggregated local outputs from the previous iteration (or initial context) to the Coordinator, which returns strategy e, or final output C.
Execution LLM calls (parallel): Each Execution LLM is invoked with the prompt described above. They produce outputs ug, which we collect, compute rewards r,, and record (0), at, r() transitions.
Update Phase: Belief networks and mixing networks optimized offline, then the next iteration starts. If are in test phase, we skip updates and only retrieve final solutions.
D.5. Error Handling
Rate Exceeded or Network Timeout. We wait 10-30 seconds, then retry.
Invalid Output. If the Execution LLM responds with incomplete reasoning message, store special placeholder of =< INVALID > with reward of = 0, and proceed.
E. Prompt and Additional Result
32

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
Coordinator Prompt(for Strategy')
"You are coordinator in a multi-agent system responsible for devising effective strategies to solve a given problem. Based on the following problem, generate concise high-level strategy in English, no more than 50 tokens:
Problem: {question}
Please provide strategy considering the following points:
1. elements and objectives of the problem
2. Possibll solutions or steps
3. Potential challenges or limitations
eey aspects to focus on Strategy:"
Figure 6: Coordinator Prompt(for Strategy)
Coordinator Prompt(for Commitment)
"You are a coordinator in multi- celti--gentta system responsible for reviewing the answers of multiple execution LLMs based on a given strategy. Your tasks are:
1.Form Commitment: Integrate the best aspects of all answers to ensure consistency in the solution process and accuracy in the final result. 2. Evaluate each answer: A ssess the similarity of the solution process to the Commitment and the accuracy of the final result. Based on these criteria, assign a reward score between 0 and to each answer.
Strategy: {strategy}
Execution LLMs' nswers:
LLM1: (answerl}
LLM2: {answer2}... ...
Figure 7: Coordinator Prompt(for final output)
33

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
Execution LLM
"You are an execution LLM multi multi-agent system, responsible for deriving solutions based on given strategy and your own belief network. Each LLM has different beliefs but cannot access the outputs of other LLMs. Your tasks 1.Fomm your belief based on the strategy: A ssume other LLMs will follow certain potential solutions. Your goal is to generate the optimal solution without global information.
2.Output the best answer: Considering your belief about other LLMs' outputs, derive the optimal solution for the current environment.
3.Bayesian Nash Equilibrium: Your output should maximize expected utility under incomplete information, aligning with the strategy.
4. Feedback adjustment:
Ensure your solution is coherent under uncertainty and optimized for the best result. Strategy: {strategy }
Please follow these steps: a. Review the strategy and form your belief on how other LLMs might output. Based on your belief, derive and oulput your optimal solution. c. Ensure your solution aligns with Bayesian Nash Equilibrium, maximizing expected utility.
Final answer."
Figure 8: Execution LLM
MATH
87.5
80.3
83.5
$2.9
80.2
81
Mistral-sx22B
76
LLAMA3. 1-700
73.8
74
LLAMA3 1-405B
68.2
68,
66.5
65.2
62.2
62.
60.4
60
Accurazy (9))
37.
33,4
34.0
34
25.2
Zao-butCot
Fow-shut CoT
SC@mmafial
/Strr
ToT
BAP
TS.LI
PRO-MCTS
ExaNesi
Figure 9: Accuracy Comparison of MATH
34

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
GSM-Hard
60.1
60
Mistml-8=7B
55.2
Mistal-8+22B
$2.9
52.8
LLaMA3,1-70B
19.9
1LaMA3.1-4055
47.4
46.8
46.4
41.1
41.7
42.8
42.9
40
36,
Accuracy
32
30-
26,
24.9
20.2
Zan-dut CoT
Few-sbut CUT
SCGimajis
/Stoy
Toy
RAP
TSALM
PRO-MTS
Ecoosll
Figure 10: Accuracy Comparison of GSM-Hard
GSM8K
100
Mistral-x7B
92.4
92.7
92.1
92.
Mistral-=22B
LLAMA3.1-70B
81.9
1LaMA3.1-405B
78.8
72.
74.9
72.0
74.2
73.5
Accuracy
0
Too-tbat GoT
Fowahat CoT
SCi5majis
cibr
ToT
RAP
TS-LLM
PRO-MCTS
EosNadi
Figure 11: Accuracy Comparison of GSMSK
35

---

From Debate to Equilibrium: Belief-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
SVAMP
100
91.4
95.
92.
95.9
95.5
9.164.2
97
966
Mistra-8+22B
Mistal-8=7B
866
85.
85.2
LLMMA3,1-70B
LLaMA3,1 14055
80
Aucuracy (%)
60
40
20
0
Zeo-but CoT
Fow-shet CoT
SCBmmajs4
rabr
ToT
RAP
TS-1LM
PRO-MTS
EvtNadl
Figure 12: Accuracy Comparison of SVAMP
StrategyQA
94.3
Mistral-S=7B
86, 787.9
87.9
89.2
Mistral-&-22B
85.3
81 12.1
84.58-12
$289
8253
LLaMAS.1-70B
75.2
78.6
LLLMA3.1 4055
68.
65.5
62.8
Accurncy (9%
60-
40
Zan-dbtt det CoT
Fow-thot CoT
Scaamaite
18tor
Tor
RAP
TS-LLM
PRO-MCTS
Eashash
Figure 13: Accuracy Comparison of Strategy
36

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
F. Example
F.I Case Study
Question: Let Vf(x) = /eftty Vbegin arm Hcl) ax+3, &eteett if xx>2, Wx-5 tttxtt| if 2 Ve lle 2, 2x-b text if } <-2, lend (array } (righ Sa+bs if the piecewise function is continuous (which means that its graph be drawn without lifting your pencil from the paper).
Input
continuity by matching function values x = and = 2, then solve for a and b.
Coordinator
Strotogy Question
B
For continuity x= a = -2
Execul
Atx=2, 2-5-- -
Thus, b-
Answerl
Balie
To make f(x) continuous, =3-
at x-5 2x xt
A nswerr
Execu²
(oolvnng gives = -3 and 33. Therefore, a =
Belie a
Continuity requires f(2-) = f(2++ and f(-2-) f(-2++). Solving these equations yields 3 and 3. Hence, a =b
Execu
Answer3
Integrate correct calculations ensuring continuity at = and = 22, accurately solving for and b to determine + =0.
Commitment
Coordinator
Figure 14: case study of math
F.2. Strategy Examples
F.2.1. GSM8K
GSM8K Problem 1:
Q1: John buys 3 pizzas for S12 each. he gives the delivery person 20% tip the total, how much did he spend in total?
S1: Calculate pizza subtotal first. Add 20% subtotal tip. Sum for final amount.
1. Pizza cost = S? x ?
2. Tip =? x subtotal
3. Total subtotal
Strategy Formal 35tokens
GSM8K Problem 2:
Q2: Janet twice much money Tom. Tom saves $45 per week, how much does Janet save in 5 weeks?
S2: Find Janet's weekly savings relative Tom's. Multiply by number of weeks.
1. Janet weekly= = x Tom
2. Total = weekly x weeks Strategy Format 28tokens
37

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
GSM8K Problem
Q3: A factory produces 150 cars per day. If they increase production by 15% next month, how many cars will they produce a 30-day month?
S3: Calculate production increase. Add to original. Multiply by days in month.
F3:
1. Inerease original 15%
2. New daily original increase
3. Monthly = daily days
Strategy + Format 36tokens
GSM8K Problem 4:
Q4: Alex has 240 marbles and gives $ of them Sarah. Sarah then gives 1 of her marbles Tom. How many marbles does Sarah have left?
S4: Calculate Sarah's initial share. Find amount she gives to Tom. Subtract.
F4:
1. Sarah gets total x
2. Sarah gives = her marbles x
3. Remaining = initial given
Strategy Formal 39tokens
GSM8K Problem 5:
Q5: A train travels 60 mph for 2.5 hours, then increases speed mph for 1.5 hours. What's the total distance traveled?
S5: Calculate distance for each speed separately using d Sum distances.
1. First distance = speedi x time)
2. Second distance speed, time?
3. Total d1 + du
Strategy + Format 36tokens
F.2.2. MATH
MATH Problem
Q1: In bag of marbles, are blue and? are red. The remaining 11 marbles green. How many marbles are in the bag?
S1: Convert fractions
denominator. Find the fraction for remaining color. Use given find total.
1. Convert to common denominator
2. Add converted fractions
3. Subtract from whole
4. Use remaining
find total
Strategy + Format 32tokens
MATH Problem
Q2: Find the area of triangle with vertices at (0,0), (4,0), and (2,5).
$2: Use coordinate geometry method for area. Set up calculation matrix. Take final result.
F2:
1. Set up coordinate matrix
2. Calculate determinant
3. Apply area formula
Strategy + Format 28tokens
38

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
MATH Problem
Q3: If (z) = and (t) = 4, find log)(zy).
S3: Apply logarithm properties. Combine given values. Express final result.
F3:
1. Write multiplication property
2.
Substitute given values
3. Simplify result
Strategy + Format 26tokens
MATH Problem 4:
Q4: A circle has radius 6, Find the area of the sector formed by 40 angle at the center.
S4: Convert angle measurement. Apply sector formula. Simplity result.
F4:
Convert to radians
2. Write sector formula
3. Calculate final area
Strategy Format 27tokens
MATH Problem 5:
Q5: Solve the equation: 2x + 5x 12 = O.
85: Identify quadratic components. Apply standard formula. Solve for variables.
F5:
1. Identify coefficients
2. Sctup quadratic formula
3. Calculate solutions
Strategy + Format 28tokens
F.2.3. SVAMP
SVAMP Problem
Q1: There are 56 books on the shelf. Tom puts 14 more books and Jane removes 22 books. How many books are on the shelf now?
S1: Track sequential changes. Apply additions and subtractions in order.
F1:
1. Add new books
2. Subtract removed books Strategy + Format 25tokens
SVAMP Problem 2:
Q2: A hox has 3 rows of chocolates. Each row has chocolates. If chocolates were eaten, how many are left?
$2: Calculate initial total. Subtract consumed amount.
F2:
1. Find total chocolates
2. Subtract eaten ones
Strategy + Format 23tokens
39

---

From Debate to Equilibrium: Belicf-Driven Multi-Agent LLM Reasoning via Bayesian Nash Equilibrium
SVAMP Problem
Q3: Mary has times as many stickers John. John has 12 stickers. How many stickers do they have together?
S3: Calculate second person's amount. Sum both quantities.
F3:
1. Find Mary's stickers
2. Add both totals
Strategy Format 24tokens
SVAMP Problem 4:
Q4: A garden has 35 flowers. ( are roses and ( are tulips. How many flowers are neither roses nor tulips?
S4: Sum known fractions. Find remaining fraction. Calculate final count.
F4:
1. Add type fractions
2.
Find remaining fraction
3. Calculate flower count
Strategy + Format 27tokens
SVAMP Problem
Q5: Each child needs 3 pencils. If there are 23 children, how many boxes of 10 pencils should the teacher buy?
85: Calculate total need. Convert to required units. Round appropriately.
F5:
1. Calculate total pencils
2. Divide by box size
3. Round to whole boxes
Strategy + Format 28tokens
Note Token Counts:
All problems follow consistent format: strategy step-by-step format
Strategy statements aim to be concise yet clear
Format points provide framework without giving solutions
Token ranges:
Shortest: 23 tokens (SVAMP Q2)
Longest: 39 tokens (GSM8K Q4)
Average: (~)30 tokens
40


Reviews:
[{"review_id": "2cdZoeTorK", "reviewer": "Reviewer_7mra", "summary": "This paper introduces ECON, a multi-agent framework designed to enhance the reasoning capabilities of LLMs. ECON models the multi-LLM setup as a DEC-POMDP with incomplete information, employing a Bayesian Nash Equilibrium perspective. Specifically, multiple “Execution LLMs” reason in parallel, each maintaining its own belief network and generating local solutions under partial information. A separate “Coordinator LLM” orchestrates consensus by aggregating and evaluating these local solutions, issuing a guidance to all agents. This structure aims to achieve a BNE, in which no single LLM agent can unilaterally improve its outcome, given its beliefs about the other agents.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 7mra:\nThanks for your constructive review, we provide point to point response regarding your question:\n\n### Q1: The Clear Definition of DEC-POMDP and the ECON Framework\n\n**Reply:** We provide a more intuitive and detailed explanation of our framework below:\n\n----\n\n### Detailed DEC-POMDP Component\n\n**Partial Observation:**\nIn our framework, each agent receives **only three types of information: the question it needs to answer, the format and strategy provided by the coordinator, and the final commitment** (aggregated response) after the coordinator combines all LLM answers. This setting prevents Execution LLMs from directly accessing the outputs of others.\n\n**Local History:**\nLocal history includes its actions and received observations. This local history enables the agent to learn from past behaviors and environmental feedback, refining its situational understanding and updating its belief state accordingly.\n\n**Action: Prompt embedding**\nThe **action space** is represented by the **prompt embedding**, which encodes the agent's action output. The prompt embeddings directly influence the LLM's generation process, adjusting the balance between exploration and exploitation.\n\n**State Transition:**\nThe state transition in our framework is essentially the update of the agent's **belief state**. The belief network updates the belief state based on the agent's local history and current observation, influencing future actions.\n\n----\n### Intuitive Framework Explanation\n\nDuring the **inference phase** of ECON, a Coordinator LLM generates an informative strategy and a format based on the input question. These are then disseminated to the Execution LLMs, which independently produce their respective answers. Finally, the Coordinator LLM aggregates these answers to form a final commitment. A detailed case study demonstrating this inference process is provided in Appendix D.1.\n\nIn the **optimization phase** of the ECON framework, we update each Execution LLM's belief state through its belief network, aggregate these belief states via the belief encoder to form group-level information, then the mixing network outputs the final Q value. After the current answer is completed, we calculate loss and reward based on the commitment, which are then used to update the parameters of each Execution LLM's belief network and the belief encoder. \n\n---\n\n### Q2: Explanation of the Learning Target of ECON Framework\n\n**Reply:** We clarify how setting BNE as the learning target leads to better aggregated answers as follows:\n\nAs mentioned in MAD, the key improvement in Multi-Agent Debate lies in **agreement intensity,** indicates the degree to which agents agree with each other can provide significant performance gains. This principle underlies our approach, where **we set BNE as the optimization target to establish consensus among the agents.** We then analyze the total Bayesian regret of the joint policy (i.e., the aggregated answer output by the MA-LLM system) based on learning towards BNE in **Lemma 2.2** and **Appendices B.2–B.3**.\n\nTo further validate that learning towards BNE can lead to better aggregated answers, we provide additional experiments showing the actual performance differences of the ECON framework before and after achieving BNE as follow:\n\n\n| Dataset | LLaMA3.1| Without BNE (%) | With BNE (%) |\n|-|-|-|-|\n| GSM8K |8B | 74.38 | 80.33 |\n| |70B | 82.12 | 96.61 |\n| | 405B | 92.36 | 99.17 |\n| GSM-Hard | 8B | 21.73 | 30.71 |\n| | 70B | 43.58 | 60.26 |\n| | 405B | 51.54 | 65.91 |\n| MATH | 8B | 55.92 | 71.45 |\n| | 70B | 74.47 | 87.31 |\n| | 405B | 82.31 | 94.78 |\n\n----------\n\n### Q3: Discuss the Issue of ECON Scalability \n\n**Reply:**  We would like to clarify how we make our framework scalable:\n\n- Our solution enhances scalability by forming a **global Nash equilibrium through local Nash equilibria** by introducing additional coordinators. Simply increasing the number of Execution LLMs would cause performance degradation since coordinator LLMs cannot handle excessive information (especially for weaker models).\n- As shown in **Figure 4**, merely increasing Execution LLMs causes performance decline as coordinator LLMs struggle to manage numerous agents, particularly with weaker models.\n- Results in **Figure 5** demonstrate that our scaled-up system (9 Execution LLMs, 3 coordinators, 1 central LLM) achieved an **18.1% improvement** over the baseline system (3 Execution LLMs, 1 coordinator), indicating potential for further scaling.\n- More details about ECON's scalability can be found in **Section 3.4**, with detailed explanation and pseudocode in **Appendix A.4**.", "claims_and_evidence": "Yes. In general, the claims are supported by (1) regret bounds derived from RL theory and DEC-POMDP frameworks, and (2) empirical results across diverse tasks.", "methods_and_evaluation_criteria": "Yes. The hierarchical reinforcement learning approach and the evaluation (both theoretical and empirical) are", "theoretical_claims": "I have skimmed through the proof of the theoretical results and believe they are intuitively correct.", "experimental_designs_or_analyses": "The authors test on well-established datasets, including math reasoning, commonsense QA set, and planning, using models with different sizes.", "relation_to_broader_scientific_literature": "In general, this extends prior “multi-agent debate” by proposing incomplete-information modeling and rigorous regret analysis, bridging a gap between purely heuristic debate approaches and formal MARL frameworks.", "essential_references_not_discussed": "This work did a good job in reviewing and discussing related references.", "other_strengths_and_weaknesses": "I have a mixed feeling about this work, stated in the following:\n\nStrength:\n- I strongly believe that incorporating a more principled approach into multi-agent LLM collaboration (e.g., a game-theoretical one) is highly valuable. This work makes advances in this direction and can inspire future works.\n\n- This work is a nice combination of theory and practice. In particular, both theoretical regret analyses and empirical test results are provided.\n\nWeakness:\n- The presentation can be largely improved, in particular, providing more clarity and intuition on the framework. For example, the mapping into DEC-POMDP is a key step, which is not presented in a very abstract way without explanation of related concepts (e.g., how is the partial observation received, what is the state transition, why is the action \"generating prompt embeddings\"). With the lack of these, the understanding of the connection is very hard.\n\n- Also, the learning target should also be explained better. In general, we wish to have the system perform well in an aggregated fashion (i.e., we only care about the final output from the aggregator), while this work focus on BNE. A better connection should be drawn.\n\n- The scalability of the proposed training method should also be discussed. The previous works, while less principled, can be flexibly extended without training. This work, however, requires certain training steps to enable collaboration. It is unclear whether such training is valuable in terms of flexibility and efficiency.", "other_comments_or_suggestions": "NA", "questions_for_authors": "My name concerns are listed in \"Other Strengths And Weaknesses\". It would be much appreciated if the authors can discuss these aspects.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5HTOI7SggM", "reviewer": "Reviewer_sVjj", "summary": "The paper introduces ECON, a hierarchical reinforcement learning framework that optimizes multi-agent reasoning in Large Language Models (LLMs) by leveraging Bayesian Nash Equilibrium (BNE) under incomplete information. By modeling collaboration as a Decentralized Partially Observable Markov Decision Process (DEC-POMDP), ECON ensures each LLM agent independently generates solutions based on local beliefs and a shared coordinator strategy, minimizing inter-agent communication. The framework employs a Coordinator-Executor architecture: Execution LLMs produce answers using belief networks that model probabilistic expectations of others’ behaviors, while a Coordinator LLM aggregates responses into a global commitment. Theoretical contributions include proving BNE existence and achieving a sublinear regret bound, outperforming linear regret in non-BNE methods. Key innovations include belief networks for reducing token overhead, dynamic reward mechanisms balancing task performance and collaboration, and hierarchical Nash coordination for scalability.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer sVjj, we'd like to thank you for your careful readings and valuable comments, we provide point to point response as follow:\n\n### Q1: The proof of sublinear convergence rate (Appendix B.2)\n\nWe acknowledge that the proof in Appendix B.2 would benefit from a more rigorous presentation. In the revised manuscript, we enhance this proof with the following improvements, while we cannot include the rewritten proof in this rebuttal due to space limitations and URL limitation, we provide the procedure as follow:\n\n- Formally justify the $O(t^{-1/2})$ convergence rates for both Q-function estimation errors and policy suboptimality through stochastic approximation theory and convex optimization results.\n- Explicitly demonstrate how our learning rate schedule $\\eta_t = \\eta_0/\\sqrt{t}$ ensures these convergence properties under stated assumptions.\n- Include the complete mathematical derivation from regret definition to final bound, showing:\n    - Precise Q-value decomposition into error terms\n    - Rigorous bounding of each error component\n    - Explicit summation across agents and time steps\n    - Formal harmonic sum bound: $\\sum_{t=1}^T 1/t^{1/2} \\leq 2\\sqrt{T}$\n- We present the complete derivation of the final $O(N\\sqrt{T}/(1-\\gamma))$ bound with all intermediate mathematical steps included.\n\n### Q2: The token consumption of ECON\n\nYes, the token usage reported in Table 3 including the coordinator LLM's output strategy and formatting instructions, the execution LLMs' answers, and the coordinator LLM's aggregated answers. We provide a case study about ECON Inference process in Appendix D.1 and a token usage for strategy formulation and formatting in Appendix D.2.\n\n### Q3: Regarding the confusion caused by Table 4\n\nWe apologize for the presentation of Table 4. You are correct that R3 and S3 are missing from the table, which makes it difficult to interpret. We have provided a revised Table 4 as follow:\n|$R_1$|$R_2$|$R_3$|ECON|\n|-|-|-|-|\n|✓|✗|✓|77.55|\n|✓|✗|✗|74.32|\n|✓|✓|✗|76.21|\n|Random|||62.71|\n\n|$S_1$|$S_2$|$S_3$|ECON|\n|-|-|-|-|\n|✓|✗|✗|71.35|\n|✗|✓|✗|72.31|\n|✗|✗|✓|81.47|\n### Q4: Explanation of notations\n\n> I suggest the author to explain what the local history and partial observation consist of.\n> \n\nWe make a more detailed explanation about local history and partial observation, as the limitation of space, **please refer to the response of reviewer 7mra Q2**.\n\n> What is U in the definition of Sec 2.2, can you write the math form? What R consists of?\n> \n\n**Utility Function**: In our framework, the utility function $U_i$ represents the expected cumulative reward for agent i, calculated as:\n\n$$U_i(\\pi_i(\\theta_i), \\pi_{-i}(\\theta_{-i}), \\theta_i, \\theta_{-i}) = \\mathbb{E} \\left[\\sum_{t=0}^{\\infty} \\gamma^t r_i^t \\mid \\pi_i, \\pi_{-i}, \\theta_i, \\theta_{-i} \\right],$$\n\nwhere $r_i^t$ is the reward at time step $t$, the components of this reward function are detailed in Sec 2.3 under \"Reward Design.\" And $R$ consists of $[r_i]$, i.e., $r_1, r_2, ..., r_n$.\n\n> Can you write the math definition of θ? \n> \n\nYes you are right, the θ in Sec 2.1 defined as  local history $\\mathbf{\\tau}_i^t$ and observation $O_i^t$.\n\n### Q5: Clarification of Sec 2.3\n\n> I wonder why this can be called \"embedding\", and can you explain why you concatenate it with Ei\nIt is confusing where ϕ comes from.\n>\n\n#### **Regarding Prompt Embeddings**\n\n- The prompt Embedding actually the action generated by belief state, we use this expression because it embeds control parameters that directly influence the LLM's generation process. We will rename this in the revised manuscript to avoid confusion about embedding of answer output.\n- Our approach encodes the action $e$ and the global state $E_t$ to jointly optimize the global $Q$-function $Q_{tot}$. The former focuses on local action decisions, enhancing diversity and aiding cooperative exploration, while the latter captures comprehensive group context to foster more efficient inter-agent coordination. We make additional ablation experiments to validate this point by removing the concatenation of $e$ and remove belief encoder; **please refer to the response of reviewer t4DV Q4**.\n\n#### **Regarding φ**: You are correct that there is an unclear relationship between $\\phi$ and $\\theta_i^B$. In our revised manuscript, we will explicitly denote $\\phi_i \\subset \\theta_i^B$ to indicate that Q-value function parameters are a subset of the belief network parameters.\n\n### Q6: Deeper analysis of weaker models\n\n> Though deeper analysis of weaker models' contributions would strengthen validity.\n\nWe have conducted additional experiments with smaller language models and found that our framework still provides significant improvements over baseline approaches, the comparison is made with few-shot CoT.\n|Dataset|Model|Few-shot CoT (%)|ECON (%)|\n|---|---|---|---|\n|GSM8K|QWEN2.5 3B|79.1|84.9|\n||LLaMA3.1 8B|84.5|87.7|\n|GSM-Hard|QWEN2.5 3B|19.7|21.3|\n||LLaMA3.1 8B|27.6|29.9|\n|MATH|QWEN2.5 3B|42.6|49.7|\n||LLaMA3.1 8B|51.9|60.4|", "claims_and_evidence": "**Sublinear Regret**\n\nThe existence of Bayesian Nash Equilibrium (BNE) is rigorously proven using Glicksberg’s Fixed Point Theorem (Appendix A.1), with assumptions like strategy space compactness and quasi-concave payoffs explicitly stated. The sublinear regret bound is derived under standard RL assumptions (bounded rewards, concentrability), supported by a detailed decomposition of Q-value differences (Appendix B.2).\n\nCritical assumptions like concentrability (Assumption A.8) and posterior alignment (Assumption A.4) are central to the regret analysis but not empirically validated. \n\n**Empirical Results**\n\nPerformance gains over single-LLM (10.9%) and multi-agent baselines (11.2%) are validated across six benchmarks (Tables 1–3, Figures 3–5), with detailed task setups (Appendix B.5) and hyperparameters (Appendix B.6). Token efficiency (21.4% reduction vs. Multi-Agent Debate) is demonstrated via token usage tables (Table 3), though prompts and strategies are standardized (Appendix D).\n\nWhile performance improvements are reported, statistical significance tests (e.g., confidence intervals, p-values) are absent.", "methods_and_evaluation_criteria": "**Benchmarks**:\n\nTasks like GSM8K, MATH, and TravelPlanner are standard for evaluating reasoning and planning in LLMs, ensuring comparability with prior work.Including heterogeneous model experiments (Table 2) tests robustness, though deeper analysis of weaker models’ contributions would strengthen validity.\n\n**Metrics**:\n\nAccuracy and token consumption directly measure performance and efficiency, addressing the paper’s core claims. Scalability tests (up to 9 LLMs) validate the framework’s practical utility for large ensembles.", "theoretical_claims": "I have checked the proof, but I think the proof is hard to understand. For example, in Appendix B.2, the proof is an outline rather than an rigorous proof. I think this part should be rewritten because it is one of the key theoretical contributions: sublinear convergence rate.", "experimental_designs_or_analyses": "I have checked the its main experiments (Sec 3.2), weaker or stronger execution llms (Sec 3.3), ans scaling up to multiple execution llms by hierarchical coordination (Sec 3.4). and ablation study on different compoent of the model (Sec 3.5). I think the results are promising.", "relation_to_broader_scientific_literature": "- Introduces Bayesian Nash Equilibrium (BNE) to formalize consensus in multi-agent LLMs,\n\n- Proposes a Coordinator-Executor architecture where a Coordinator LLM guides Execution LLMs via strategies, combining CTDE’s centralized coordination with the flexibility of LLM-based reasoning. This extends hierarchical RL to static, non-trainable LLM agents.\n\n- Bridges game-theoretic equilibria with modern LLM ensembles, offering a blueprint for principled multi-agent reasoning.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "No.", "other_comments_or_suggestions": "In the Appendix, the section numbering is incorrect. For example, \"Together API Integration for ECON\" should be section C I think.", "questions_for_authors": "- Does the results of ECON in Table.3 contains the token usage of coordinator models? \n- Table 4 seems to be confusing for me, why R3 does not appear in the column name, and where is S3 in the table?\n- I suggest the author to explain what the local history $\\tau$ and partial observation $O$ consist of.\n- What is U in the definition of Sec 2.2, can you write the math form? What $\\mathcal{R}$ consists of? I think $\\mathcal{R}=\\{ r_{coordinator}, r_1,\\cdots, r_n \\} $? I also notice that the notation $R$ is used in both the reward design and regret $R(T)$, please avoid reusing notation names.  \n- is $\\theta$ in Sec 2.1 related to $\\theta^B$ in Sec 2.2? It seems that $\\theta^B$ is model's parameters but $\\theta$ is something like the concatenation of $b_i$ and $\\tau_i$. Can you write the math definition of $\\theta$? I think it should be $(\\tau_i, O_i)$, i.e., past history and current obervation. \n- The equation in Sec 2.3 is a little bit confusing. $B_i$ output $Q_i$ and $e_i$.  \n    - $e_i$ be called a prompt embedding? Based on your writing, I think $T_i$ and $p_i$ are scalars and $e_i$ is a two-dimensional vector, I wonder why this can be called \"embedding\", and can you explain why you concatenate it with ${\\bf E}_i$. As stated in the paper, $T_i$ and $P_i$ control the sampling, so why do you connect it with belief encoding?\n    - $Q_i$ is the output of $B_i$, therefore, $\\phi$ should be part of $\\theta^B$, I think? But the current writing does not relfect this relation and it is confusing where $\\phi$ comes from.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7mULXBfwaw", "reviewer": "Reviewer_hmqe", "summary": "This paper proposes a Multi-agent LLM framework (ECON) to improve the communication efficiency and consensus. It formulates multi-agent LLM as Decentralized Partially Observable Markov Decision Process. It reduces the token consumption from incomplete-information perspective, and optimizes towards Bayesian Nash Equilibrium to improve the consensus. The proposed method has a lower regret bound, making it possible to scale up effectively. Experimental results on six reasoning tasks show that ECON surpasses single-agent solutions and outperforms existing multi-agent approaches with a lower token consumption.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer hmqe:\nThanks for your constructive review, we provide some response regarding your question:\n\n### Q1: About the inference phase of ECON\n\n> Section 2 mainly focuses on the optimization phase and it is not clear **how the inference phase works in ECON**. What is the observation of each agent, what is the strategy provided by the coordinator, how Coordinator aggregates the answers, etc are not mentioned.\n\nWe provide a more intuitive and detailed explanation of our framework below:\n\n1. Intuitive Framework Explanation\n\nDuring the **inference phase** of ECON, a Coordinator LLM generates an informative strategy and a format based on the input question. These are then disseminated to the Execution LLMs, which independently produce their respective answers. Finally, the Coordinator LLM aggregates these answers to form a final commitment. A detailed case study demonstrating this inference process is provided in Appendix D.2.\n\nIn the **optimization phase** of the ECON framework, we update each Execution LLM's belief state through its belief network, aggregate these belief states via the belief encoder to form group-level information, then the mixing network outputs the final Q_tot. After the current answer is completed, we calculate loss and reward based on the commitment, which are then used to update the parameters of each Execution LLM's belief network and the belief encoder. We describe our specific DEC-POMDP components setting as follows:\n\n1. Detailed DEC-POMDP Components\n\n**Partial Observation:**\nIn our framework, each agent receives **only three types of information: the question it needs to answer, the format and strategy provided by the coordinator, and the final commitment** (aggregated response) after the coordinator combines all LLM answers. This setting prevents Execution LLMs from directly accessing the outputs of others.\n\n**Local History:**\nLocal history includes its actions and received observations. This local history enables the agent to learn from past behaviors and environmental feedback, refining its situational understanding and updating its belief state accordingly.\n\n**Action: Prompt embedding**\nThe **action space** is represented by the **prompt embedding**, which encodes the agent's action output. The prompt embeddings directly influence the LLM's generation process, adjusting the balance between exploration and exploitation.\n\n**State Transition:**\nThe state transition in our framework is essentially the update of the agent's **belief state**. The belief network updates the belief state based on the agent's local history and current observation, influencing future actions.\n\n### Q2: Clarification about prompt embedding:\n\n>the prompt embedding e_i = [T_i, p_i], where T_i is the temperature and p_i is the threshold for repetition. How do T_i and p_i construct the token embedding for the prompt?\n\nThank you for your insightful comments. We agree that our description of the prompt embedding requires clarification.\n\n- The prompt Embedding actually the action generated by belief state, we use this expression because it embeds control parameters that directly influence the Execution LLM's generation process. We will rename this in the revised manuscript to avoid confusion about embedding of answer output.\n\n### Q3: About the scalability of ECON:\n\n> How does ECON scale up to more agents? Does it require additional training?\nWe would like to clarify how we make our framework scalable and yes, scalable ECON require additional training:\n    \n- Our solution enhances scalability by forming a **global Nash equilibrium through local Nash equilibria** by introducing additional coordinators. Simply increasing the number of Execution LLMs would cause performance degradation since coordinator LLMs cannot handle excessive information (especially for weaker models).\n - As shown in **Figure 4**, merely increasing Execution LLMs causes performance decline as coordinator LLMs struggle to manage numerous agents, particularly with weaker models.\n- Results in **Figure 5** demonstrate that our scaled-up system (9 Execution LLMs, 3 coordinators, 1 central LLM) achieved an **18.1% improvement** over the baseline system (3 Execution LLMs, 1 coordinator), indicating potential for further scaling.\n- More details about ECON's scalability can be found in **Section 3.4**, with detailed explanation and pseudocode in **Appendix A.4**.\n\n## Q4: Figure Presentation Issues\n\n> In Figure 2, the red gradient flow is missing. In Figure 3, it would be more effective to group the bins by LLMs rather than baseline methods, as this would better highlight differences between baselines rather than between different LLMs.\n\n**Response:**\nWe appreciate your constructive feedback. We will address these visualization issues in the revised manuscript by adding the missing red gradient flow to Figure 2 and reorganizing Figure 3 to group bins by LLMs, which will indeed better emphasize the comparative performance.", "claims_and_evidence": "This paper claims the proposed ECON can ensure the Multi-agent LLM system can converge towards Bayesian Nash Equilibrium, thus enhancing the degree of consensus and performance.", "methods_and_evaluation_criteria": "ECON uses a Coordinator-Executor architecture. Each Execution LLM keeps a brief network that updates its belief state with the local trajectory and the current observation. A shared belief encoder aggregates the belief states from all agents to model coherent joint behavior.  \n\nHowever, Section 2 mainly focuses on the optimization phase and it is not clear how the inference phase works in ECON. What is the observation of each agent, what is the strategy provided by the coordinator, how Coordinator aggregates the answers, etc are not mentioned. This make the method part confusing.   \n\nThe evaluation metric is accuracy on four mathematical reasoning datasets and the common sense reasoning dataset. On TravelPlanner, the metric is the final pass rates.", "theoretical_claims": "I didn't check the correctness of the correctness of the proof", "experimental_designs_or_analyses": "The experimental section can be divided into 4 parts:  \n(1) performance against baseline models across 5 reasoning tasks with 3 llama models and 1 mistral model, and complex planning on Travelplanner with GPT-4 turbo   \n(2) strong coordinator with weak execution & weak coordinator with strong execution to evaluate the heterogeneous Execution   \n(3) the result of scale-up agent numbers. \n(4) ablation study. \n\nThese experiments and analyses are comprehensive and reasonable, demonstrating the effectiveness of ECON in different settings.", "relation_to_broader_scientific_literature": "It is related to the multi-agent reinforcement learning and methods in game theory", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths: \nThe idea is interesting and novel.    \nTheoretical proofs are provided to guarantee the effectiveness of ECON.   \nThe experiments are comprehensive and the findings are insightful.\n\nWeaknesses: While Section 2 provides the formulation and optimization of ECON, it is not clear how ECON works during the inference phase. The meaning of notations such as O_i and e_i and their instantiation in a concrete task should be introduced (e.g. text or hidden representation)", "other_comments_or_suggestions": "Figure 2, there is no red gradient flow.   \nFigure 3, it is better to group the bins by LLMs instead of the baseline methods because it should highlight the difference between different baselines rather than different LLMs.", "questions_for_authors": "the prompt embedding e_i = [T_i, p_i], where T_i is the temperature and p_i is the threshold for repetition. How do T_i and p_i construct the token embedding for the prompt?  \nHow does ECON scale up to more agents? Does it require additional training?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A7ch5E8lSq", "reviewer": "Reviewer_t4Dv", "summary": "The paper introduces ECON, a hierarchical reinforcement learning framework designed to optimize multi-agent reasoning in Large Language Models (LLMs) by leveraging Bayesian Nash Equilibrium (BNE). ECON replaces inter-agent communication with a belief mechanism to save communication costs and make it easier to scale up the population of agents. LLMs integrate distributed reasoning and achieve centralized commitment by a Coordinator. Experiments show the proposed method surpasses the existing methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer t4Dv, thanks for the feedback and suggestions, we will add clarification where needed as space permits.\n\n### Q1:Explain the Implementation of BNE in Sec 2.3\n\n**Regarding the definition of belief**: belief represents each Execution LLM's policy derived from partial observations and local history, we use belief network to update belief. To enable coherent joint behavior, belief encoder aggregates individual belief states into group-level representation. Then the mix network coordinates this integrated belief information for global Q-value and compute loss and reward after commitment made. This architecture aligns with our formal definition of BNE in Sec 2.2, where optimizes agent policies through belief network updates until reaching a strategy profile where no agent can unilaterally improve its payoff given its beliefs.\n\n**Regarding the effectiveness** of belief-based coordination, we conduct additional experiments to response this concern. **The result of performance comparison with /without achieving BNE demonstrated an average performance improvement of 14%,** which can be found at the **response to reviewer 7mra Q1**. \n### Q2: Explain the Result in Figure 4\n\nWe would like to clarify that Figure 4 in Sec 3.4 intentionally demonstrates that **simply increasing the number of execution LLMs without appropriate coordination mechanisms can actually decrease performance.** We attribute this to the challenge faced by the Coordinator LLM in managing an excessive number of Execution LLMs in Sec 3.4.\n\n**To address this scalability challenge, we propose a global Nash equilibrium through local Nash equilibria by introducing additional coordinators in our manuscript (Sec 3.4).** This setup ensures that each Coordinator handles a reasonable amount of data, as demonstrated by the improvements reported in Figure 5. We provide the corresponding pseudocode and detailed explanation for scaling up ECON in Appendix A.4.\n\n### Q3: Regarding The Inter-agent Communication in ECON\n\n> I believe the direction of this area should improve the efficiency of inter-agent communication instead of giving it up.\n\nWe would like to clarify that ECON does not eliminate inter-agent communication entirely, but rather **adopts an incomplete-information perspective that minimizes communication** (as recognized by Reviewers sVjj and hmqe). ECON's optimization objective focuses on achieving consensus and constructing implicit communication among execution LLMs.\n\n**Although ECON is implicit inter-agent communication, it is not conflicted with explicit inter-agent communication**, we make additional experiment to demonstrate that by incorporating explicit interaction into ECON (which makes it become a complete information formation). The performance improve 1.1% while the token consumption increase 42.4% on average, demonstrate potential to achieve stronger performance with sufficient token budget.\n|Dataset|LLaMA3.1|Complete Info (%)|ECON (%)|Token Consumption|\n|---|---|---|---|---|\n|GSM8K|8B|81.4|80.3|+35.6%|\n||70B|96.1|96.7|+42.7%|\n|GSM-Hard|8B|30.2|29.9|+62.3%|\n||70B|53.6|51.4|+40.9%|\n|MATH|8B|59.6|60.4|+33.8%|\n||70B|83.1|81.5|+39.4%|\n### Q4: The Effectiveness of Different Modules of the Mix Network\n\nTheoretically, the design of our mixing network optimizes local policies to improve the global objective, monotonicity guarantee are demonstrated in Appendix A.5. **We make additional ablation experiments to validate this point by removing the concatenation of $e$ and remove belief encoder** to claim that each parts of ECON is essential, as shown in the experiments.The baseline result is ECON.\n|Dataset|LLaMA3.1|No concat(%)|No encoder(%)|\n|---|---|---|---|\n|GSM8K|8B|85.9(-1.8)|84.1(-3.4)|\n||70B|93.6(-3.1)|91.1(-5.6)|\n|GSM-Hard|8B|24.9(-5.0)|21.7(-8.2)|\n||70B|47.2(-4.2)|42.6(-8.8)|\n|MATH|8B|55.6(-4.8)|52.3(-7.1)|\n||70B|77.0(-4.4)|75.3(-6.2)|\n### Q5: Why are Multi-LLM Methods are Even Worse Than Single Ones?\n\nOur abstract needs clarification: **the 10.9% for MA-LLM is averaged across six datasets, while the 11.2% for Single is across five datasets (exclude TravelPlanner)**. Since single agent approach with open-source model have 0% pass rate on TravelPlanner, ECON's improvement ratio cannot be calculated. With TravelPlanner excluded, ECON's comparative results on five datasets are reported in Sec 3.2 which indicates the single-LLM methods are worse than multi-LLM methods.\n\n### Q6: The Missing References Need to Discuss\n\nWe agree that it's necessary to incorporate discussions of these works. We will update our related work section in the revised version.\n\n### Q7: Clarification about Figure 2\n\nFigure 2 illustrates both the inference phase and optimization phase of ECON. The inference phase (left part) is primarily based on LLMs, where the Coordinator LLM and Execution LLMs work together to generate solutions. For a more detailed understanding of this process, we provide a comprehensive case study of the inference process in Appendix D.1.", "claims_and_evidence": "1. The paper claims that \"they conceptually formalize BNE in Multi-LLM systems and instantiate it through a hierarchical optimization framework to improve collaborative reasoning\", but the paper seems not to clearly describe how BNE is implemented in practice in the section 2.3 Framework of ECON.\n2. One of the main claims of the paper is the scalability of the proposed method, while in Figure 4, the performance does not improve while increasing the number of Execution LLMs.", "methods_and_evaluation_criteria": "1. I agree that inter-agent communication is one of the key concerns in LLM agents, but it does not mean it is better to remove such communication entirely. The belief mechanism proposed in the paper seems to follow the design in MARL, which lacks interpretability and generability. I believe the direction of this area should improve the efficiency of inter-agent communication instead of giving it up.\n2. Benchmarks (GSM8K, MATH, TravelPlanner) are standard for reasoning tasks. Metrics (accuracy, token usage) align with the goals of efficiency and performance.", "theoretical_claims": "I did not find errors here.", "experimental_designs_or_analyses": "1. The Abaltion study does not analyze the effectiveness of different modules of the mix network. It may not be clear why the mix network should be designed in this way.\n2. The paper mentions the method \"outperforms single-LLM approaches by 10.9% and surpasses existing multi-LLM methods by 11.2%\". Why the multi-LLM methods are even worse than single ones?", "relation_to_broader_scientific_literature": "1. The paper incorporates the belief mechanism from MARL into LLM agents.\n2. The paper further improves multi-agent debate method to executor-coordinator architecture.\n3. The paper introduces Nash Equilibrium from game theory to LLM agents.", "essential_references_not_discussed": "1. Belief mechanisms in MARL are not discussed, e.g. [1].\n2. Multi-LLM-agent collaboration methods beyond debates are not included, e.g. [2][3].\n\n[1] Wang, Yuanfei, Jing Xu, and Yizhou Wang. \"ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind.\" International Conference on Learning Representations.\n[2] Li, Guohao, et al. \"Camel: Communicative agents for\" mind\" exploration of large language model society.\" Advances in Neural Information Processing Systems 36 (2023): 51991-52008.\n[3] Guo, Xudong, et al. \"Embodied LLM Agents Learn to Cooperate in Organized Teams.\" Language Gamification-NeurIPS 2024 Workshop.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper explores a novel method to optimize the multi-agent reasoning with LLM.\n\nWeaknesses:\n1. The definition and effectiveness of belief in such tasks are not clear. \n2. In Figure 2, the components of the framework are not clear enough. Which parts are based on LLM?", "other_comments_or_suggestions": "1. A space is missing on Page 1 \" challenge(Liu et al., 2024b).\"\n2. On page 6 \"Figure4\"\n3. On page 37 \"Figure 14: case study of math\"", "questions_for_authors": "Refer to the previous parts.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: After discussion with the authors and updating their scores, reviewers unanimously agree on accepting this paper, with final scores of [3,3,4,3]. 

Strengths identified by the reviewers include:
- Theoretical results
  - “extends prior “multi-agent debate” by proposing incomplete-information modeling and rigorous regret analysis, bridging a gap between purely heuristic debate approaches and formal MARL frameworks.” (7mra)
  - “The existence of Bayesian Nash Equilibrium (BNE) is rigorously proven using Glicksberg’s Fixed Point Theorem (Appendix A.1), with assumptions like strategy space compactness and quasi-concave payoffs explicitly stated. The sublinear regret bound is derived under standard RL assumptions (bounded rewards, concentrability), supported by a detailed decomposition of Q-value differences (Appendix B.2).” (sVjj)
  - “The proposed method has a lower regret bound, making it possible to scale up effectively” (hmqe)
- Empirical results
  - experiments are “comprehensive and reasonable” (hmqe)
  - “The authors test on well-established datasets” like GSM8K, MATH, and TravelPlanner, enabling comparison with prior work (sVjj, 7mra)
  - “Performance gains over single-LLM (10.9%) and multi-agent baselines (11.2%) are validated across six benchmarks (Tables 1–3, Figures 3–5), with detailed task setups (Appendix B.5) and hyperparameters (Appendix B.6). Token efficiency (21.4% reduction vs. Multi-Agent Debate) is demonstrated via token usage tables (Table 3), though prompts and strategies are standardized (Appendix D).” (sVjj)
- “This work did a good job in reviewing and discussing related references” (7mra)
- “The idea is interesting and novel.” (hmqe)
- “the findings are insightful.” (hmqe)

Many of the weaknesses identified by the reviewers related to the clarity of the paper:
- “the paper seems not to clearly describe how BNE is implemented in practice in the section 2.3 Framework of ECON.” (t4Dv)
- “I have checked the proof, but I think the proof is hard to understand. For example, in Appendix B.2, the proof is an outline rather than an rigorous proof. I think this part should be rewritten because it is one of the key theoretical contributions: sublinear convergence rate”. (sVjj)

In the rebuttal, the authors provided clarifications to the reviewer’s questions, and spelled out a high-level strategy for a revised proof to answer reviewer sVjj’s concern. Several errors (e.g. with Table 4) were corrected. They also provided several additional experiments with different ablations, using smaller models with CoT, scaling up to achieve even better performance improvements, etc. 

Given this evidence, I am recommending accepting the paper, but would strongly encourage the authors to revise the paper to clarify the mistakes and issues that were identified in the review process.
Decision: Accept (poster)