Multi-Agent Debate for LLM Judges with Adaptive Stability Detection
Authors: Tianyu Hu
Zhen Tan
Song Wang
Huaizhi Qu
Tianlong Chen
Venue: NeurIPS 2025 poster ()
Keywords: LLM-as-Judge
Multi-Agent Debate
Adaptive Stopping
Primary area: applications
TL;DR: This paper utilizes multi-agent debate process for llm-as-judge, and employs an adaptive stopping mechanism.

Abstract: With advancements in reasoning capabilities, 
Large Language Models (LLMs) are increasingly employed for automated judgment tasks.
While LLMs-as-Judges offer promise in automating evaluations, 
current approaches often rely on simplistic aggregation methods (e.g., majority voting), 
which can fail even when individual agents provide correct answers.
To address this, we propose a multi-agent debate judge framework where agents collaboratively reason and iteratively refine their responses. 
We formalize the debate process mathematically, analyzing agent interactions and proving that debate amplifies correctness compared to static ensembles.
To enhance efficiency, we introduce a stability detection mechanism that models judge consensus dynamics via a time-varying Beta-Binomial mixture, with adaptive stopping based on distributional similarity (Kolmogorov-Smirnov test).
This mechanism models the judges' collective correct rate dynamics using a time-varying mixture of Beta-Binomial distributions and employs an adaptive stopping criterion based on distributional similarity (Kolmogorov-Smirnov statistic). 
Experiments across multiple benchmarks and models demonstrate that our framework improves judgment accuracy over majority voting while maintaining computational efficiency.

Multi-Agent Debate for LLM Judges with Adaptive Stability Detection
Tianyu Hu tyrionhuu@gmail.c com
Zhen Tan Arizona State University ztan360asu. edu
Song Wang University of Central Florida song. wang@ucf.c....mm...mm.mm edu
Huaizhi Qu
Tianlong Chen UNC Chapel Hill tianlong9cs. unc. edu
UNC Chapel Hill ghz9910298cs unc. edu
Abstract
With the advancing reasoning capabilities of Large Language Models (LLMs), they are increasingly employed for complex evaluation tasks, such as grading student responses, verifying factual claims, and comparing competing answers. Leveraging multiple LLMs as automated judges can enhance robustness and accuracy by aggregating diverse perspectives, yet existing approaches often rely on static and simple aggregation methods, such as majority voting, which may produce incorrect judgments despite correct individual assessments. We propose a novel multi- agent debate framework where LLMs collaboratively reason and iteratively refine judgments, formalizing this process mathematically and proving its advantages over static ensembles. To ensure computational efficiency, we introduce stability detection mechanism using time-varying Peta-Binnoiil mixture model (a mixture of two Beta- Binomiin distributions) that tracks judge consensus dynamics and applies adaptive stopping via Kolmogorov Sominoo testing. Experiments across diverse benchmarks and models demonstrate significant improvements in judgment accuracy over majority voting while maintaining computational efficiency.
Introduction
The rapid advancement Large Language Models (LLMs) has significantly transformed automated evaluation, enabling near-human accuracy in assessing textual outputs (Chiang and Lee
2023 LLMs
are now widely used for tasks such as scoring student essays for coherence Xiao et al. 2025),
fact-checking against reliable sources QQeele and Bovet 2024 Augenstein et al. 2024 and ranking
multiple-choice answers for accuracy Robinson and Wingate 2023 Zheng et al. 2024). supporting
applications in education [Wang al. 2024b) content moderation. and decision support. A prominent
approach in this context the LLM--s as-a-gedde paradigm [Zhnng et al. 2023 Qu et 2025) where
LLMs evaluate responses generated by other LLMs humans. However, relying single LLM
can be limiting due to potential biases and correlated errors [Tumer and Ghosh 1996 Wang et al.
2023 2025b). To address these issues, multi-agent ensembles have been proposed [Li al. 2024
which aggregate multiple L.LM judgments through methods like weighted voting DDietterich 2000 averaging, stacking, and majority voting [Zhou] 2012).
Despite its simplicity, majority voting can be unreliable in complex or ambiguous cases, particularly when agents share similar biases when the correct answer a minority opinion (Yang al. 2025
This motivates the need for more robust frameworks that can capture the collective intelligence of multiple agents without being constrained by static aggregation methods. To address this, propose a multi-agent debate judge framework as shown in Figure where multiple LLMs engage
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

in structured debates to collaboratively reason and refine their judgments. We also present a formal mathematical model of the debate process, capturing agent interactions and belief updates. Building on this foundation, we prove that debate improves correctness over static ensembles under mild assumptions, establishing a theoretical basis for iterative refinement.
Judgement Task æf
responses question, decide better: QUESTION, RESPONSE A, RESPONSE B
Judge A better, becmuse watermelon aeed
the stomacc you digeat them.
+
Debate History
QUESTION What happens You you eat watermalton seads?
RESPONSE
RESPONSE B not digest watermelon seeds
:uudp Bizz better, and the idea the watermelon seed could grow in one's istomsch or felle.
You grow waternlons your stomach
Round Generation
Check Convergeecce
update 2
if folse
Judgo is woong, Zudge poaobie grow atcnach
extruct(s()
Fitel Arswer B.
Consenaus Judgement (Or Majority Vote)
Judge
History Z'
eztroct(25) B
Converge max round
True
Tudge 2's jedgement mokes mone becouse shomach gou. Find Antwer
extrrac((e))
Figure Multi-Agent Debate Framework.
However, iterative debates can be computationally expensive, especially when the process is not optimally terminated. Fixed-round debates risk either premature stopping before consensus is reached or unnecessary computation after convergence. To address this, we introduce a stability detection mechanism based on tire-narying mixture of Beta-Binomiil distributions, using the Kolmogorov- Smirnov (KS) statistic [Massey Jr 1951] to adaptively detect when the distribution stabilizes and to terminate the debate.
We validate our framework through experiments across diverse benchmarks, LLM architectures, and modalities (visual and non-visual tasks), demonstrating that our multi-agent debate framework outperforms majority voting in terms of accuracy, and the adaptive stopping mechanism significantly reduces computational costs while maintaining high accuracy.
Our contributions: (I) A formal debate framework for LLM ensembles that enables collaborative reasoning with theoretically provable correctness guarantees; (2) A novel stability detection mecha- nism using Beta-Binomic mixture modeling and adaptive stopping: (3) Comprehensive empirical validation showing substantial accuracy gains over majority voting.
2 Related Work
L..Ms-as-Judgers
Our work is closely related to the field of LLM-as-a- a--Jdge (Zheng et al. 2023
Gu et al. 2025 and LLMs-as-Judges (Li al. 2024], which involves using
more LLMs to
evaluate responses generated by either another LLM or a human. Basic LLM-as-Judge frameworks
typically rely
single LLM to perform a judgement task [Liu et al.) 2023 Dubois al. 2024
Recent studies leverage LLMs to model user preferences or assess quality criteria [Shankar et al.
2024
Pan et al.) 2024 Tian al. 2024 judge factual consistency or hallucinations [Lin et al. 2022
Chen et al. 2024d Luo et al. 2024]. flag biased or unsafe content [Chen and Goldfarb-Tarranl
2025
Yuan et al.
2024]. and evaluate reasoning quality (Lightman 2023] Srivastava al. 2023
However,
LM-based judges also exhibit several limitations [Ko et al. 2024 Wang et al.
2024a
Wu and Aji 2025), such as self-preference bias [Wataoka al. [024], societal biases [Chnn et al.
2024b), inconsistency [Stureborg et al. 2024]. and other common challenges faced by LLMs [Dai
2024
Multi-Agent Debate.
Recent work has explored multi-agent debate frameworks, where multiple
agents engage in structured reasoning to reach consensus [Pham et al. 2024 Rasal| 2024 Michael
et al. 2023 Chang 2025 Irving et al., 2018 Khan et al. 2024 Du et al. 2024 Liang et al. 2024
Chan et al. 2024 Wang et al. [025a Lei et al. 2025]. Inspired by Minsky 1986). J Du et al. [024]
proposed framework in which multiple LLMs respond to question independently. then refine

---

their answers after being shown responses from other agents. Estornell and Liu [2024] extended this concept by formalizing the debate process as an optimization problem, laying emphasis on the role of latent concepts the underlying abstractions that drive both human and I.I. L..--ggeeereed
language Xie al.
2022 Jiang 2023].
To enhance the debate process, researchers have incorporated methods such as chain-of of oftthu reasoning [Kojima al. 2022 Wei et 2022], self-refiection [Ren al. 2023 Tan 2025b]
and self-consistency [Wang et al. 2023 Other studies have explored diverse debate strategies,
including adversarial settings-wwie where agents take opposing sides and a third agent acts as judge [Liang
et al. 2024] and collaborative approaches, where agents work together to iteratively solve
problem Li et al. 2025a Estornell al. 2025].
Statistical Approaches. To estimate the correctness of debate judges. Qu et al. [2255 proposed modeling judge correctness dynamics using a mixture of Beta-Bin eeta-Binomiial distributions, effectively capturing features such as bimodal peaks in the correctness distribution than traditional binomial models. The Expectation- Caattion--Mxmiizztionn (EM) algorithm [Moon] 1996] is commonly employed to estimate parameters such mixture models [Sun al. [224]] Qu et al. 2025]. For stability detection, an approach to monitor the distributional similarity of judge correctness is the Kolmogorov-Smingg
(KS) test [Massey Jr, 1951], which quantifies maximum difference between two empirical cumulative distribution functions (CDFs).
Multi-Agent Debate Framework
In this section, we introduce the multi-agent debate framework for LLM judges. We begin by defining some important notations and the debate process: let æ be the task and y the ground truth answer.
Each of the n agents is parameterized by O, E ④. Agent i's response at round is (2) with e(x;))
extracting its judgment. All responses at round form Z 2(() of debate.
(t) T is the maximum rounds 2n
3.1 Debate Process
The multi-agent debate framework involves n agents, each parameterized d; (1) At round each agent receives task x and generates an initial response (0)) (2) In each subsequent round, agents observe the task and debate history, then generate responses. (3) After each round, if all agents agree, the process terminates and returns the consensus; otherwise, it continues until a maximum of T' rounds, after which the majority vote returned. This procedure is summarized in Algorithm the appendix.
3.2 Latent Concepts
Following prior work (Xie et al. 2022 Jiang 2023 Estornel and Liu 2024], adopt the notion of latent concepts, which refers to the underlying abstract ideas or interpretations that guide how agents understand and respond to task.
Let θ denote a latent concept space, where each concept 0 E θ represents a coherent interpretation of task ær The task-answer pair (x. is generated by first sampling concept 0, then drawing (z, y) ~ D(0), where D maps concepts to task-answer pairs. Upon observing x, agents infer a distribution over θ and generate responses accordingly. Multiple valid concepts may exist, and agents may focus on different aspects. Although is abstract, we use sentence embeddings to represent and compare concepts in practice.
To provide more detailed example of how latent concepts can be used in the debate process, consider the following question: "Who won the 202/ Formula Drivers' Championship?" to which the correct answer would be "Max Verstappen". The latent concept behind this task involves knowledge of 2021 Formula season and the fact that Verstappen won the championship. Sentence embeddings are able to effectively capture this semantic concept and enable agents to align or disagree based on such latent understanding.

---

3.3 Response Generation Mechanism
At round agent generates response 2/ based the task the history of responses Z', and its parameters 6i modeled as:
Mmede (2, ((++1) xz, 1x, Øz).
Introducing a latent concept space O this becomes:
P model (=((+1)) I. 7 01)
Σ
(P(((t11) (.) P(0 x. 7 Ø1).
(I)
8EB
The first term is the likelihood of generating 2; belief in & after observing and Z
((+1) under concept 0; the second is the agent's updated
now introduce key assumption that simplifies modeling process:
Assumption 3.1 (Conditional Independence on Latent Concepts). For a given latent concept 8, the probability of generating response 2; ((-1) is conditionally independent of both z and æ, given and o;:
P(2 (++1)
0, o,,, (++1) 10,
This assumption implies that the generation (2+1) of model is solely determined by the latent 2j
concept 0 of the input task and the agent's parameters Φi. Again with the example mentioned earlier, the sentence embeddings that capture the semantic meaning of "Max Verstappen won the 2021 Formula Drivers' Championship" are produced solely based on the latent concept and the agent's parameters 6;.
Lemma 3.1 (Response Generation Model), Under Assumption 3.1 the generation of response by model at time + can be expanded with Bayesian inference:
P(2 (t+1)
x. 1x, 0:)
(++1) Σ P 1(2 9EB
10,
P
10,
P Φ:) P (1))))0.)))
(2)
This formulation clarifies how agents incorporate others' responses into their posterior beliefs about the latent concept, enabling collaborative refinement of judgments. Through Bayesian inference, each agent updates its belief in # by weighing the likelihood of the task æ and all responses Z against its prior F(0 Φi). This iterative process helps correct individual errors-such as those from biased training data-by shifting beliefs toward the correct concept, thus improving ensemble accuracy and mitigating correlated errors seen in static aggregation methods [Tumer and Ghosh 1996). Modeling response generation probabilistically over latent concept space supports robust, collective deliberation.
4 Theoretical Analysis
4.1
Assumptions
Our analysis rests on four core assumptions that formalize how latent concepts govern the debate dynamics. We motivate each assumption with practical intuition and highlight its implications and limitations.
Assumption 4.1 (True Concept Predictiveness). For all agents i, concepts 0' + 8", and rounds t:
P(e(=!+1) 0" Φi) ((e(((++))) = (',
This assumption asserts that the true concept 0" leads to more accurate predictions than any other incorrect concept. It captures the intuitive idea that there exists best way to frame the task (e.g., correct scientific theory legal principle), and that responses generated under this framing are more likely to be correct. While it simplifies the space of possible misinterpretations and might weaken if the tasks suffer from high ambiguity cases, enables rigorous analysis of concept-driven reasoning dynamics.

---

Assumption 4.2 (Task-Concept Alignment). The probability of observing task x is higher given the true concept than any incorrect concept:
P(a 10,00,,,, 0.)
P(x 0', (:) v0' 0".
This reflects that task generation is not uniform across concepts some tasks are more naturally aligned with specific latent interpretations. For example, medical diagnosis task is more likely to arise under medical concept than under a legal one. This assumption allows posterior inference over 0 using Bayes' rule to favor 0* debate unfolds.
Assumption 4.3 (Positive Concept Prior Beliefs). All concepts have positive prior probability:
P(0 (.) O 40 O V,Vi.
This ensures that concept ruled out priori, standard regularity condition in Bayesian models. prevents agents from permanently excluding the true concept and models diversity in agents' initial beliefs, where even implausible concepts retain some weight.
Assumption 4.4 (Independent Agent Responses), Agent responses conditionally independent given the latent concept 0:
p( 22 ,.,,..,, 2 10,
II P(#) 0,0,).
This assumption simplifies belief aggregation by treating agent responses as independent signals once the concept is fixed. Although this may be violated if agents copy or reference one another, or share strong biases, it is reasonable in decentralized debate settings where responses are generated in parallel.
4.2 Theorems
begin by defining key concepts used in our analysis:
True Concept: 0" the unique concept such that w) D(0*), i.e., the concept that maximizes the likelihood of generating the correct answer.
Response Consistency: e(z), 0) P(=} 0, db). denoting the likelihood of response under concept and parameters (bj-
c(s), 0¹) for all 0' + 8" This captures the idea that response is most likely generated under the true concept.
Strong Consistency: response 2 is 0" -st-onn if c(2), 0*) >
We now present two main theorems:
Theorem 4.1 (Consistent Response Amplification). Let Z be set of responses where at least one response is strongly consistent with the true concept 0", and Zi be set of responses where no response is strongly consistent with 0*, Then:
E, [P((((=)+)) y Z/a 2.)]
= 75 6z)],
where E, the expectation over agents i. That the presence of at least strongly consistent response in round t increases the expected correctness in round t + 1.
See Appendix the full proof. This theorem formalizes a central benefit of debate: even single correct reasoning path guide other agents toward better beliefs and improved future performance. supports the value of curriculum learning and few-shot prompting in multi-agent reasoning.
We next extract a useful consequence:
Lemma 4.1 (Accuracy Increases with Posterior Belief). Under the assumptions of Theorem 4.7 the probability of an agent producing a correct answer increases with their posterior belief in the tree concept:
P(e(z)) y) 1 P(O"" zz-1)
5

---

See Appendix A.. for the proof. This follows directly from Bayesian updating: stronger belief in 8" improves expected predictive accuracy. It formalizes the link between belief refinement and task performance.
To prove that debate outperforms static aggregation (e.g., majority vote), introduce final condition: at least one response in the first round must be generated under the true concept, to enable belief updating.
Assumption 4.5 (Initial Seed of Correct Reasoning). There exists at least one initial response generated via the correct concept: latent concepis represented by reasoning path. 3z, with c(22)) 8*) > c(2, 8*). This ensures the debate has a valid starting point for belief updates.
(0)
We now state our second main theorem:
Theorem 4.2 (Debate Improvement over Majority Vote). Under the preceding assumptions, the final accuracy of the debated outcome D(Z") exceeds that of initial majority vote MV
PPD(Z") y) P(MV =
See Appendix A.rennixxA.3acam for the full proof. This result supports the view that structured interaction-throuue iterative debate debatc-enable enables a population of agents to converge on more accurate answers than indepen- dent majority voting. It aligns with classical findings in distributed reasoning and ensemble methods, where collaborative refinement outperforms static aggregation.
Debate Adaptive Stability Detection
To improve debate efficiency, we introduce an adaptive stability detection mechanism that halts the process once judge accuracy rates stabilize. We model judge accuracy time-varying Beta- Binomial mixture, estimating parameters via Sipectation-Maximizaa by monitoring distributional similarity across rounds using the Kolmogorov-Smmrnv (KS) statistic. See Algorithm 2 in the appendix.
(EM). Stability is detected
5.1 Judgement Accuracy Modeling
Let s/; denote the latent correct rate of debate judge round i, with distribution D, Our goal is to determine when D, stabilizes sufficiently to compute reliable bounds for vh;
We observe an ensemble of & judges whose collective decisions produce score St at each round t- the total number of correct decisions. We model St as a time-varying mixture of Beta- ceta-Binomiil distributions:
S* w (B, + (1 - w²) BB(k, at ot,
(5)
Here, BB(k, a, 33 denotes the Beta- Reta-Binomial distribution, which models the number of correct decisions among judges with shape parameters a and 3, capturing the variability judge accuracy due to heterogeneous behaviors. The mixture weight w' E [0, 1] balances the two components, and af, Si. at, o5 parameterize the two components. This model captures different behavioral regimes among judges (e.g., attentive vs. inattentive).
5.2 Parameter Estimation via Anpectation-Maxiiizztiin
For each round we estimate parameters ztº (w', af 31. ab. from observed values (sf....,s" s using maximum likelihood estimation with the EM algorithm. The complete-data likelihood combines both mixture components:
((0))
II
w BBBBBs 88) (1 (')BB(s):kkdd [b]]
(6)
where the Beta-Binomin probability mass function is defined as:
BB(skk,a, = ( a, s
B(a,8) B)

---

and B(a, 3)
(() T(++D)
is the Beta function, with I' denoting the Gamma function.
The EM algorithm iteratively refines estimates of tt::
E-step: Compute responsibilities rj,1 w. BB (
BBB. (BB(efia..55)
M-step:
Update parameters using weighted MLEs (Maximum Likelihood Estimation):
we t
3,1 and (a2:80) + arg max
J. log BB (B(s);
(c = 1,2).
In practice, we employ the L-BFGS-B optimization method [Zhu al. 1997 to update the Beta- Binomial parameters. The algorithm terminates when the log-likelihood improvement is less than convergence threshold 10-6 6 after maximum of 100 iterations. This threshold was chosen to ensure high precision in parameter estimation while maintaining computational efficiency, validated in our experiments across benchmarks.
5.3 Stability Detection
After the EM algorithm converges, meaning the log-likelihone improvement falls below a thresh- old maximum of iterations is reached, it yields estimated parameter set ve {w", af, $1, 32 for round t. The distribution over individual judges' correct rates is then given by:
Pt (bb = w Beta( aii, (1) + (1 - w BetBett(y; a2, (2))
where (et az 3) na (1-))~
(la is the probability density function of the Beta distribution,
(8) is the Beta function defined in the previous subsection. and B(a, 3) [(a+B)
To detect when this distribution stabilizes, we track the Kolmogorov-Smirnov (KS) statistic between consecutive rounds:
Dj sup IF - -16)-FF-11 F (+)) e
(8)
where Fr is the cumulative distribution function (CDF) of P (v)) As described in Algorithm 2 the judgement accuracy modeling process halts once D. 0.05 for consecutive rounds, as used in our experiments, signaling that the judge accuracy distribution has stabilized.
6 Experiments
6.1 Experimental Setup
Our evaluation framework assesses wide range of state-of-the-att LLMs, including both propri- etary and open-source models from multiple providers across visual and non non-visual tasks. For the proprietary model, we use Semmini-2.0-Flast [Google| 2024 from Google. Open-source models comprise Llama-3.1 1-----8---nsttttt (Grattafiori et al.) 2024 Meta 2024 and Llama-3.22 21B--BBB- 1e-1BB-Vision Vision- Instruct (Grattafiori al.) 2024 2 both from Meta AI, used for non-visual and visual tasks, respectively; Qwen- 2.5- B---ttnnttt [Qwen et al. 2025) and Qwen-2.5 2.5- VL-7B-Inntrru (Bai et al.) 2025). both from Alibaba, applied non-visual and visual tasks, respectively; and Gemma-3-4B- Instruct TTam et al. 2025 from Google used for both tasks.
We conduct experiments datasets from diverse domains to evaluate the debate judge's perfor- mance, including: hallucination detection: TruthfulQA [Lin et al.) [2022], alignment evaluation: JudgeBench [Tn al. 2025a] and LLMBar (Zeng et al. 2024], and reasoning: BIG-Bench (Srivas] lava et al.) 2023]. We also use multiple multi-modi datasets: MLLM-Judge [Chen al. 2024a and
Judgennything Pu et al. 2025
6.2 Comparative Results
Table shows that our debate framework generally outperforms both baselines: Single Model and SoM (Majority Vote), especially complex tasks like JudgeBench, LLMBar, TruthfulQA, and
7

---

BIG-Bench
JudgeBench
Model
Single
SoM
Debate
Single
SoM
Debate
Gemma- 3-4B
69.8442.45
70.80
71.10+2.81
55.62+3.24
54.60
56. 36.7043.89
Qwen-2.5-7B
74.3712.10
76.6012.62
12.20
12.98
59.5213.85
59.6813.88
Llama-3.1
78.67
81. 41.30
74.00+2.72
57.98+3.02
60.8 60.8443.84
58 58.90
Gemini-2.0-4 Fla-h
81.7412.16
21.50
82.30
63.66
66. 13.13
68.0613.0
LLMBar SoM
TruthfuiQA SoM
Model
Single
Debate
Single
Debate
Gemma- 3-4B
57.9842.46
57 57.83
58 58.83.22.7
40.39 22.90
40.15
41.6223.37
Qwen-2.5-7B
65.57
66.2212.67
69.81 69.81442.0 42.00
49.84
62.39 43.36
58.51433.7
Llama-3. 1.188B
59.70 2..33
60.25
62. 12.5842.. 22.73
50.83+2.85
53.9443.48
55.34
2omini-2..0Flasste
76.68
77.75
81.8342.11
69.49 42.77
12.01
74.30+2.99
MLLM-Judge SoM
JudgeAnything SoM
Model
Single
Debate
Single
Debate
Gemma-3-44
61.13
61.6213.36
62. 63.75
83.4645.81
84.90 84.9646.07
84.96 84.9646.07
Qwen-2.5-VL-77
60.43
60.88 43.38
60.38
67.88
68. 68.4243.a5
67. 67.67+7.85
Gemini 2.0--Fash
67. 67.50
68.00 68.00
69. 25.2533..10
83,46 45.70
85. 85.71
Table I: Accuracy (%) and standard error (%) of different response aggregation methods-Single (sampling once), SoM (Majority Vote), and Debate (10 Rounds Maximum) across datasets and models. All results use an ensemble size of 7 and sampling temperature of 1.0.
MLLM-Judge. Gemini-2.0- 2.0-Flash achieves the largest gains in several cases (c.g., 77.75% to 81.83% on LLMBar). These gains are modest in some cases because our framework's ' iterative refinement adds most value in complex tasks with high initial variance, where collaborative belief updates correct biases (Theorem 1..1), yielding significant improvements. On simpler tasks with high initial consensus, such BIG-Bench and JudgeAnything, SoM performs comparably minimal benefit, aligning with diminishing returns in low-variance scenarios. This supports targeted applicability: debate excels where accuracy justifies costs, while SoM suffices for straightforward tasks.
better as refinement introduces
Appendix B.2) shows that ensemble size of 7 provides the best balance between accuracy and computational cost across most tasks. Larger ensembles (Size-9 or greater) show diminishing returns in accuracy, while increasing computational costs, smaller ensembles (Size-5) are sufficient to maintain accuracy with minimal cost. We recommend Size-7 as the optimal choice for most cases.
Our analysis (Table 5
6.3 Judgement Dynamics
Judgement Distribution. Figure 2a| shows the evolution of correct agent distributions across debate rounds on JudgeBench for four models. Initially. Round 0 distributions are broad, reflecting diverse judgments. By Rounds 2, distributions converge a bimodal pattern (0 correct agents), maintaining a Beta- -ica-Banomiial mixture shape, indicating that agents either align on the correct answer or collectively fail. Similar convergence observed across other datasets (see Appendix B.2. confirming the debate framework's robustness.
Figure 2b illustrates the distribution of correct agents across debate rounds for the Llama-3. 1-8B model on the JudgeBench dataset. The solid line represents the fitted Beta-Binomial distribution, while shaded areas depict the empirical distribution of correct agents (x-axis) with probability density (y-axis). The close alignment between the fitted and empirical distributions highlights the effectiveness of the EM algorithm in modeling agent performance dynamics.
Adaptive Stability Detection. Figure presents KS statisties across six debate rounds for six datasets and five models. The KS statistic (y-axis) measures the difference between CDFs of correct agent counts across two consecutive rounds (x-axis). High initial KS values (e.g., 0.25-0.45 for JudgeBench, Round 1) reflect diverse judgments and opinion changing, but values typically rapidly drop below the stability threshold (c 0.05) within rounds (e.g., 2.- on BIG- Bench by Round 2). To prevent premature halting, the adaptive mechanism requires KS values

---

BIG-Bench
JudgeBench
Model
Rounds
Accuracy
Diff
Rounds
Accuracy
Gemma 3-4B
5
70.0722.82
-1.03
5
26.54
-0.16
Qwen-2.5-7B
7
72.00
-0.20
6
59 39.385
-0.33
Llama-3. 3.1-8B
7
73. 73.70
-0.30
6
58. 58.5822..8
-0.32
Gemini 220-220FF... 2.0-Flash
4
81.70+2.40
-0.60
6
67. 67.74
-0.32
LLMBar Accuracy
TruthfulQA
Model
Rounds
Diff
Rounds
Accuracy
Gemma 3-4B
5
58. 58.7522.78
-0.08
5
41.49 41.49
-0.13
Owen-2.557BB
5
69.1442261
-0.67
5
58 08.0223.37
-0.49
Llama-3. 3.1-BB
6
62.17
-0.41
6
54. 32..42
-0.62
Gemini 2.0-FFlsh
5
21.33
-0.50
5
73.81
-0.49
MLLM-Judge
JudgeAnything
Model
Rounds
Accuracy
Diff
Rounds
Accuracy
Diff
Gemma 3-4B
4
62.50=3.36
-0.25
2
84.96.6.07
0.00
Owen-2.5-V. LL-B
60.38
0.00
2
67 27.85
0.00
Gemini-2 Flasaahh
5
68. 68.63.3.22
-0.62
8
85.71
0.00
Table 2: Adaptive stopping performance the Debate method: number rounds until stopped, accuracy (%), and accuracy difference (%) compared to using the full 10 rounds. All experiments use an ensemble size 7, maximum of 10 debate rounds and KS statitiii threshold of 0.05.
100
Round
80
80
Taikks (96
2.5-7
60
Gamiie
60
40
40
40
20
20
Number of Correct Agents
Number of Correct Agents
Number of Correct Agents
(a) Distribution of correct agents across 3 debate rounds on JudgeBench for multiple models. Each subplot shows round, with distributions converging to either 0 or 7 correct agents, reflecting the debate process's alignment effect.
0.25
Combered Niduum Model
Componenn. (ecighted bed 87)
Componient: 1 neecy itw) Corr corppnnenn:2 2 neeig (t-0.57)
Combinatt Micumm Mocel
Prabebiliyy Mass
0.5
Comber Nodura Model
Componintt necignt=0:33 57)
0.70
Componennt tecight-0.38 33)
B4
Compo ient necig) tw0 45)
0.15
Obse ved Data
Observed Data
63
Otserved Data
0.10
6.
0.05
0.1
0.00
Correct Agents Count
Correct Agents
Correct Agents Count
(b) Fitted Beta-Binomial distributions (solid lines) against empirical distributions (shaded areas) for Llama-3.1- 8B on JudgeBench across debate rounds, showing the accuracy of mixture model.
Figure 2: Judge consensus dynamics during debate. Top: Correct agent distributions across three
rounds on JudgeBench, showing convergence unanimous agreement. Bottom: Fitted Beta- -ita-Biiomill mixture model closely matches empirical distributions for Llama-3.1-8B.
to remain below 0.05 for two consecutive rounds before terminating the debate process. For example, Gemini-2.0- 2.a-Faash on JudgeAnything drops below this threshold by Round 3 but bounces back, until finally stabilizing from Round onward.
Conclusion
In this paper, we introduced a multi-agent debate framework that allows LLMs to collaboratively reason and iteratively refine their judgments, addressing the shortcomings of static aggregation methods such as majority voting. Central to our approach is novel stability detection mechanism, which employs time-varying Beta-Binomial mixture model and the Dolmogorov-Sminoo statistic to

---

BiG-Bench
JalgeBench
LLMBar
0.30
Gemma-3-48
Uama-311-00
Saatskic
U.m0-331-90
Gemma-3 3-48
Staltssii
0-
Gemma-3-44 3-46
Uama-3.1-00
Qwen-2.5-78
0wen-2.5:78
0wen-2.5.78
Gemini-2.0-Aashh
Jemin-2.0-flap
Gemini-2 0-Flash
2
Threchold (0.05)
Threshold
Threshold
0.35
0
06
Stop
TruthfuQQA
Judgefnything
MLLM-Judge
Gemma-3-40
Gemma- 3-48
0.35
Gemma- 3-48
K5 Statieic
L.1ma-3.1-88
Qwen-2.5-70
0.30
0.25
Gemin 2a0-
0won-2.5-ML-78
0.22
-0
Gemini-2 0-
0wen-2.5MV-77
Gamini 2.0 Flash
Threshold (0.05)
0.20
Thresheld
Threshold (0.05)
150.5
2010
6.15
0:10
0.00
Step
Step
Step
Figure 3: KS statistics across ten debate rounds for six datasets. The x-axis shows steps between rounds, the y-axis shows KS values, and the red dotted line marks the stability threshold (e = 0.05).
adaptively halt the debate process when consensus is achieved. The significance of our framework lies in its ability to bolster the robustness and precision of LLM-based evaluations through collaborative reasoning and iterative refinement. The stability detection mechanism optimizes resource use, making it viable for practical applications.
Acknowledgments
This research is supported in part by an Amazon Research Award and Cisco Faculty Award.
References
Meta AL. Llama 3.2:
Revolutionizing edge ai and vision with open, cus- tomizable models, September 2024. URL https://ai hatps:://a.mmaataa meta. com/blog/
www.eeoooleeeeeeeeeeeeeeeeeeeeee Accessed: 2025-05-10.
Isabelle Augenstein, Timothy Baldwin, Meeyoung Cha, Tanmoy Chakraborty, Giovanni Luca Ciampaglia, David Corney, Renee DiResta, Emilio Ferrara, Scott Hale, Alon Halevy, and et al. Factuality challenges in the era of large language models and opportunities for fact-checking. Nature Machine Intelligence, 6(8):852-86)) Aug 2024. doi: 10. (0098//422560024-00853 1-Z.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang. Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li. Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xic, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL htt///axxv.oor//bbbba 2502.. 13923
Chi-Min Chan, Weize Chen, Yusheng Su. Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better LLM-based evaluators through multi-agent de- bate. In The Twelfth International Conference Learming Representations, 2024. URL
Edward Y. Chang. Unlocking the wisdom of large language models: An introduction to the path to artificial general intelligence, 2025. httpssttss///axx... www.aaawwi..r..ooaeeeo.eoo wttp::///wabb......e///g 01007
Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. Mllm-as-a-judge: assessing multimodal Ilm-as-a-judge with vision-language benchmark. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. 24. JMLR.org, 2024a.
Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or L.MSs as the judge? study on judgement bias. In Yaser Al-Onaizan, Mohit Bansal,
10

---

and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. pages 8301-8327, Miami, Florida, USA. November 2024b. Association for Computational Linguistics. doi: 10.1 10..6635/1200224.mmmmmmm.....
URL
attpps://allantholooyy htwp:/thhooooooooogooooo wrg. www.:mew..pppm.mmmmmmmmmmmmmmmmm
Hongyu Chen and Seraphina Goldfarb-Tarrant, Safer or luckier? Ilms as safety evaluators are not robust to artifacts, 2025. httpt:httt///attt org/ abs/ 25033033377
Justin Chen, Swarnadeep Saha, and Mohit Bansal. ReConcile: Round-table conference improves reasoning via consensus among diverse LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pages Association for Computational Linguisties. doi: 10. htt.5//////2222.cllloo.gg... URL https
Bangkok, Thailand, August 2024c.
//aclanthology. org/ 2024. acl long. 381/
Kedi Chen, Qin Chen, Jie Zhou, He Yishen, and Liang He. DiaHalu: A dialogue-Ievel hallucination evaluation benchmark for large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9057-9079, Miami, Florida, USA, November 2024d. Association for Computational Linguistics. doi: 10. 10.11/663/v 1/2024. fing./iiinnnggennl.55555 URL httpstts:: haap::////aaaahhlloogg/o org/ 2024 fr22.findings--o emmlp.mnm222m. 529/
Cheng-Han Chiang and Hung-yi Lee. A closer look into using large language models for automatic evaluation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computatiional Linguisties: EMNLP 2023, pages 1928-8942. Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.11865//1/2// 1/2023.22222222 http:///......innneeeeeeeee findings-emmlp.599, URL wrg wrg/ findings--mninn emm1p 599/
Ziqing Dai. Applications and challenges of large language models in smart government -from techno- logical advances to regulated applications. In Proceedings of the 2024 3rd Internutional Conference
on Frontiers of Artificial Intelligence and Machine Learning, FAIMI '24, page 275-280, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400709777. doi: 10.1145/3633344.663366 URL /tttps://dii.rr:::
Thomas G. Dietterich. Ensemble methods in machine learning. In Multiple Classifier Systems, pages 1-15, Berlin, Heidelberg, 2000. Springer Berlin Heidelberg. ISBN 978-3-540-45014.5
Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Proceedings of the 41st International Conference Machine Learning, ICML'24. JMLR.org, 2024.
Yann Dubois, Percy Liang, and Tatsunori Hashimoto. Length-controlled alpacaeval: A simple debiasing of automatic evaluators. In First Conference on Language Modeling, 2024. URL https://openrevvex wttp:/forumiddd-yyiba
Andrew Estornell and Yang Liu. Multi-llm debate: Framework, principals, and interventions. In A. Globerson, I.. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processiig Systems, volume 37, pages 28938-28964. Curran As- sociates, Inc., 2024, URL http://////procceiiigg. neurai cc cc/ http/pppee__ffeeeee ff files//aperr/202// Confeerrooooffeeee Conferreeeeeeeoooooeeeeeeeeeeee pdf
Andrew Estornell, Jean-Francois Ton, Yuanshun Yao, and Yang Liu. ACC-collab: An actor-critic approach to multi-agent LLM collaboration. In The Thirteenth International Conference on Learning Representations,, 2025. URL http:tttt//:/pereeeeee wwwpeewwwwwwwwwweeeeeeeeeeeeeeee het//oorummkdd-nffAAziiee
Google. The next chapter of the gemini era for developers, 2024. URL URL/ttpss///eeeeopprr googleblog. www/en/me///ceeeeeccc//..ee cessed 2024-12-20.
-fe wew.iiiii-r----o--eaaallllaaaaee
Ac-
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru,
11

---

Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen. Hamnah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee. Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika lyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren cantala-Yeargy Laurens der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins. Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si. Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi. Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi. Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng. Prajjwal Bhargava, Pratik Dubal, Praveen Krishman, Singh Koura, Puxin Xu, Qing He, Qingxiao Dong. Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sahana Chennabasappa. Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang. Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou. Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao. Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury. Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiangg Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang. Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuar Lc, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni. Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriclla Schwarz, Gada Badcer, Georgia Swee, Gil Halpem, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptist Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg. Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik
Plawiak,
12

---

Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Raf Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala. Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang. Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad lonescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang. Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang. Xiaojian Wu, Xiaolan Wang. Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang. Zhiwei Zhao, and Zhiyu Ma. The llama herd of models, 2024. URL http:: /ww.iv.vvvoo org/ abs/ 220///4070000 21783
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni. and Jian Guo. survey on alddae--- 2025. URI. htttts:////rttttt/ orgooogg/aaa aba//b/a//// http/bbb//21..... 2211 15594
Geoffrey Irving, Paul Christiano, and Dario Amodei. Ai safety via debate, 2018. URL https: // //arxiv. wwwiiiorgg/aa...e.....o.....wolr http:/885500000
Hui Jiang. A latent space theory for emergent abilitics in large language models, 2023. URL https://arxiv.. http::///aarivooo/aa/////// wwt.oorg///bb//goee/e 09960
Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, and Ethan Perez. Debating with more persuasive llms leads to more truthful answers. In Proceedings of the 41st International Conference on Machine Learning. ICML'24 224. JMLR.org. 2024.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 1781713871088.
Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. Benchmarking cognitive biases in large language models as evaluators. In Len Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 517-545, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18633/v1/2044 findings- acl.29. URL haap::::/aclaahhlloggy org 2024 findings-acl. 29/
Zhenyu Lei, Zhen Tan, Song Wang, Yaochen Zhu, Zihan Chen, Yushun Dong, and Jundong Li. Learn- ing from diverse reasoning paths with routing and collaboration. arXiv preprint arXiv:2508. 16861, 2025.
13

---

Dawei Li, Zhen Tan, Peijia Qian, Yifan Li, Kumar Chaudhary, Lijie Hu, and Jiayi Shen. Smoa: Improving multi-agent large language models with s parse m ixture-o f-a gents. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 54 Springer, 2025a.
Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. Lims-as-judges: A comprehensive survey on Ilm-based evaluation methods, 2024. URL
https://arxiv, www./agg///...oooooo..ooo..e..e/ 2412. 05579
Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long-context LLMs struggle with long in-context learning. Transactions on Machine Learming Research, 2025b. ISSN 2835-8856.
URL https://openreviesu http://wwww.....ooooooooo////ood
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang. Yan Wang, Rui Wang, Yujiu Yang. Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17889-17904, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10.18653/v1/ 2024.emnlp- main.992. URL
org org/2024. 2024. emnlp-main mwalp-mmiincom 992/
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step, 2023. URL
https://arxil: wtt.aaiiii....aaaaaererr at////////23/5/ 20050
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 3214 4-222, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v 1/2022.acll-ongg 229. http:ttss: /ttp://////////anooollg//// www.:/lloooooooo.aggooooo wwg////////2222 2022 acl acl-long. 229/
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using 4 with better human alignment. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 251 1-2522, Singapore, December 2023. Association for Computational Lin- URL pettts::/accanttoonn
guistics. doi: 10. emmlp-main. 153/
org/ 2023
Wen Luo, Tianshu Shen, Wei Li, Guangyue Peng, Richeng Xuan, Houfeng Wang, and Xi Yang. Halludial: A large-scale benchmark for automatic dialogue-level hallucination evaluation, 2024. URL https::tt//ttt. http://axxvvvv.ooe org/ http:/abs/240...0// 07070
Frank Masscy Jr. The kolmogoroy-smirnov test for goodness of fit. Journal of the American statistical Association, 46(253):68 1951.
Meta. Introducing Ilama 3.1: Our most capable models to date, July 2024. httpssts:://a.. com/blog/meta 11 ama-3-11 Accessed 2025-04-16.
Julian Michael, Salsabila Mahdi, David Rein, Jackson Petty, Julien Dirani, Vishakh Padmakumar, and Samuel R. Bowman. Debate helps supervise unreliable experts, 2023. URL https://axa: org/ abs/
Marvin Minsky. The society of mind. Simon & Schuster, Inc., USA, 1986. ISBN 0671607405.
T.K. Moon. The expectation-maxinn maximizaaionn algorithm. IEEE Signal Processing Magazine, 13(6): 47-60, 1996. doi:
Qian Pan, Zahra Ashktorab. Michael Desmond, Martín Santillán Cooper, James Johnson, Rahul Nair, Elizabeth Daly, and Werner Geyer. Human-centerse design recommendations for LLM-as-a-judge In Nikita Soni, Lucie Flek, Ashish Sharma, Diyi Yang, Sara Hooker, and H. Andrew Schwartz, editors, Proceedings of the 1st Human- Centered Large Lunguage Modeling Workshop, pages 16- 29. TBD, August 2024. ACI. doi: 10.18653/v1/2024.huull 1.2. URI, httpp::///// http///aaaaantooogoyyo org/2024. hwchlmmm.... 1.2/
14

---

Chau Pham, Boyi Liu, Yingxiang Yang, Zhengyu Chen, Tianyi Liu, Jianbo Yuan, Bryan A. Plum- mer, Zhaoran Wang, and Hongxia Yang. Let models speak eiphers: Multiagent debate through embeddings. In The Twelfth International Conference on Learning Representations, 2024. URL
https://openreviek/ wttprviiww.eeteeeee wowooorrndemeereeeeee sesheridsse
Shu Pu, Yaochen Wang, Dongping Chen, Yuhang Chen, Guohao Wang, Qi Qin, Zhongyi Zhang, Zhiyuan Zhang, Zetong Zhou, Shuang Gong, Yi Gui, Yao Wan, and Philip S. Yu. Judge anything: Mllm as a judge across any modality, 2025. URL httts:://axxii org/ abs/ abs//503. 17489
Huaizhi Qu, Inyoung Choi, Zhen Tan, Song Wang, Sukwon Yun, Qi Long, Faizan Siddiqui, Kwonjoon Lee, and Tianlong Chen. Efficient map estimation of Ilm judgment performance with prior transfer, 2025. httL:ttts:://rrrt org/ a25004 12589
Dorian Quelle and Alexandre Bovet.
The perils and promises of fact-checking with large lan- guage models. Frontiers Artificial Intelligence, Volume 7 2024, 2024. ISSN 2624. 8212. doi: 202.89//rai..22213444441
URL https://wwit frontiersin org/journals/ 3389/ frai. 2024. 1341697
artificial-intedn wtw.iilllieeeeeeariieeeeeeeeeeee
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang. Tingyu Xia, Xingzhang Ren. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang. Yu Wan, Yugiong Liu, Zeyu Cui. Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https://arxiss orgg abs /412. 15115
Sumedh Rasal. Llm harmony: Multi-agent communication for problem solving, 2024. URL
https://arxis org/ abs 200..2201.00000 01312
Jie Ren, Yao Zhao, Tu Vu, Peter J. Liu, and Balaji Lakshminarayanan. Self-evaluation improves selective generation in large language models. In Javier Antorán, Arno Blaas, Kelly Buchanan, Fan Feng. Vincent Fortuin, Sahra Ghalebikesabi, Andreas Kriegler, Ian Mason, David Rohde, Francisco J. R. Ruiz, Tobias Uelwer, Yubin Xie, and Rui Yang, editors, Proceedings Not Better: Failure Modes in the Age of Foundation Models" " at NewrlPS 2023 Workshops, volume 239 of Proceedings of Machine Learning Research, pages 49-64. PMLR, 16 Dec 2023. URL https://proceediin mlr. press/ w239/2ee23333 html
"I Can't tan't Believe It's
Joshua Robinson and David Wingate. Leveraging large language models for multiple choice question answering. In The Eleventh International Conference on Learning Representations, 2023. URL
htttps://pperrevie. wtwp:arrwwwieeeeeeouuuuuuueaeeee
Shreya Shankar, J.D. Zamfirescu-Peree Pereera,, Bjoern Hartmann, Aditya Parameswaran, and Ian Arawjo. Who validates the validators? aligning iim-sssisted evaluation of Ilm outputs with human pref- erences. In Proceedings of the 37th Anmual ACM Symposium on User Interface Software and Technology, UIST 24, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400706288. doi: 10. 10.1145/354477.336666 3676450
URL/https://ao org/ 0te///0.114550 1145/3654777
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain. Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders An- dreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, Andrew La Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekei, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Cather- ine Stinson, Cedrick Argucta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin
15

---

Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burcie Chris Waites, Christian Voigt, Christo- pher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro. Colin Raffel, Courtney Asheraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito. Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Bares,, Elizabeth Donoway, Elliee Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem. Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martinez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang. Gonzalo Jaimovitch-LLóper, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Kocoñ, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang. Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Bermandez-Oraleo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakr- ishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras- Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz. Lütfi Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve. Maheen Farooqi. Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Maric Tolkichn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew I.. Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnad,, Melvin McElrath, Michael A. Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michat Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan A. Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts. Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi. Oner Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang. Piotr Milkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sa- jant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster. Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi. Shikhar Singh. Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee. Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Ger- stenberg, Trenton Chang. Trishala Neeraj. Tushar Khot, Tyler Shultz. Uri Shaham, Vedant Misra,
16

---

Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang. Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang. Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang. Zijie J. Wang, Zirui Wang, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolating the capabilitiis of language models, 2023. URL https: org/ www.aa//....... 04615
Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. Large language models are inconsistent and biased evaluators, 2024. URLtttts::/ httpxiiw..rg//b//24555555 01724
Guangzhi Sun, Anmol Kagrecha, Potsawee Manakul, Phil Woodland, and Mark Gales. Skillaggrega- tion: Reference-free lIm-dependent aggregation, 2024. URL htRL:/tttp///aaxi.oorg//bb 10215
Sijun Tan, Siyuan Zhuang. Kyle Montgomery, William Yuan Tang, Alejandro Cuadron, Chenguang Wang, Raluca Popa, and Ion Stoica. Judgebench: A benchmark for evaluating LLM-based judges. In The Thirteenth International Conference on Learning Representations, 2025a. URL https::/oppenevilt
Zhen Tan, Jun Yan, I Hsu, Rujun Han, Zifeng Wang. Long T Le, Yiwen Song, Yanfei Chen, Hamid Palangi, George Lee, et al. In prospect and retrospect: Reflective memory management for long-term personalized dialogue agents. arXiv preprint arXiv:2503.00 000000 2025b.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin. Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Tain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan- Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar. David Vilar, Idan Brusilovsky. Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng. Alexander Kolesnikov, Alexci Bendebury, Alvin Abdagic. Amit Vadi. András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang. Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette- Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan. Glenn Cameron, Hadi Hashemi, Hanna Klimezak- Plucińska, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju ycong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu. Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauban, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq lqbal, Shashir Reddy, Shruti Sheth, Siim Poder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Figer, Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evci, Vedant Misra, Vincent Roseberry, Vlad Feinberg. Vlad Kolesnikow, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei. Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo. Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Jaan-Bap Alayrac, Rohan Anil, Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. Gemma technical report, 2025. URL // arxiv. org/ abs babs/25500 2503 19786
CJ Carey, Cormac Brick, Daniel
17

---

Xiaoyi Tian, Amogh Mannekote, Carly E. Solomon, Yukyeong Song, Christine Fry Wise, Tom Mcklin, Joanne Barrett, Kristy Elizabeth Boyer, and Maya Israel. Examining Ilm prompting strategies for automatic evaluation of learner-created computational artifacts. In Benjamin Paaßen and Carrie Demmans Epp, editors, Proceedings of the 17th International Conference on Educational Data Mining, pages 698- 706, Atlanta, Georgia, USA, July 2024. International Educational Data doi:
Mining Society. ISBN 978-1-7336736-555
Kagan Tumer and Joydeep Ghosh. Error correlation and error reduction in ensemble classifiers. Connection Science. 8(3-4):385 1996.
doi: 10. 10.108000954409996 16839. URL https
// 1ttp:///100.00000044999966111331
Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors. Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pages 9440-9450, Bangkok. Thailand, August 2024a. Association for Computational Linguistics. doi: 10.18653/v1/0024. acl-long.511. URL URL
.rg 2024. acl 1ar.llon...... 511/
Shen Wang, Tianlong Xu, Hang Li, Chaoli Zhang, Joleen Liang, Jiliang Tang, Philip S. Yu, and Qingsong Wen. Large language models for education: survey and outlook, 2024b. URL.
httpp://aaxxi..rr//abb..
18105
Song Wang, Zihan Chen, Peng Wang, Zhepei Wei, Zhen Tan, Yu Meng, Cong Shen, and Jundong Li. Separate the wheat from the chaff: Winnowing down divergent views in retrieval augmented generation. In EMNLP 2025, 2025a.
Song Wang, Zhen Tan, Zihan Chen, Shuang Zhou, Tianlong Chen, and Jundong Li. Anymac: Cascading flexible multi-agent collaboration via next-agent prediction. In EMNLP 2025, 2025b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistencyy improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL htttps://openreviae wttp://eeeooooroo/d/d-ia.aaa
Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. Self- Self-preference bias in lim-as-a-judge, 2024.
URL https: org.:iiiioorrereoooe httpg//bb/2241... 21819
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le. and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 978171387 1088.
Minghao Wu and Alham Fikri Aji. Style over substance:
Evaluation biases for large language models. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend AI-Khalifa, Barbara Di Fugenio, and Steven Schockaert, editors, Proceedings of the 31 st International Conference on Computational Linguistics, pages 297-312, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics. httttttss//caaaneee
org/ 2025/02222.022 coling wwwing-main..nnnnnnooono.m...iee 21/
Changrong Xiao, Wenxing Ma, Qingping Song, Sean Xin Xu, Kunpeng Zhang, Yufang Wang, and Qi Fu. Human-ai collaborative essay scoring: A dual-process framework with 1lms. In Proceedings of the 15th International Learning Analytics and Knowledge Conference, LAK '25, page 293-305, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400707018. doi: 10.1145/3706468.37700 37765077. URL httt:htts://datt org/
wtwg///////////// 10.
Sang Michael Xie, Aditi Raghunathan. Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations, 2022. URL http::t:/oppeeeeiiiiiieeee /ttp::/ee/ppppooooooooo//ooorraa nett/forrumii-RRRdPPPPEU
Joshua C. Yang, Damian Dailisan, Marcin Korecki, Carina I. Hausladen, and Dirk Helbing. Llm voting: Human choices and collective Conference on AI, Ethics, and Society, AIES '24, page 1696-1708. AAAI Press, 2025.
In Proceedings of the 2024 AAAI/ACM
18

---

Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Rujjie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, Rui Wang, and Gongshen Liu. R-judge: Benchmarking safety risk awareness for L.M agents. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNIP 2024, pages 1467-1490, Mi- ami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v 1/ 2024. 1222.iindingsemmllp.. URL BBRLLtttss//aallnnholloggo org. wwt.oogg/////// fiidinggneennni eenle.mmlllp....m p1pp..9) 79/
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models evaluating instruction following. In The Twelfth International Conference Learning Representations, 2024.
Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models not robust multiple choice selectors. In The Twelfth International Conference Learning Representations, 2024. URL /tL/ptt::///ppeere///
hett/forumndddshr9PPPP.. 1d=shr9PX277000
Lianmin Zheng, Wii Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging Ilm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
Zhi-Hua Zhou. Ensemble Methods: Foundations and Algorithms. Chapman & Hall/CRC, 1st edition, 2012. ISBN 1439830037.
Ciyou Zhu, Richard H. Byrd, Peihuang Lu, and Jorge Nocedal. Algorithm 778: L-bfgs-b: Fortran subroutines for large-scale bound-constrained optimization. ACM Trans. Math. Softw, 23(4): 550-560, December 1997. ISSN 0098-3500. doi: 10.1 10.11449279233.7799266 URL pecttss//dda //doi
org/10.1145/279992.2222
19

---

A Theoretical Analysis
A.1 Proof of Theorem
We prove that having at least one strongly consistent response round / increases the expected
probability of correctness in round + 1. This relies on assumptions 4.1 4.4
Proof. Using Bayes' rule and our defined assumptions, we can express the probability that agent i
generates the correct answer in round / as:
P(e(= ( = y x. z Øi)
Σ
P(e(= ( - 0, x, (.)
(9)
DEG
The posterior probability of concept given the observed responses Z' can be calculated as:
IP(Z'
0, (:) x, o:)
P(O xx, 2z, 0:)
(10)
P(Z x. (e))
IP(Z' 0, (P(r O, (P(P(( di)
(11)
PP(Z xt (x) (x)
P(Z' (, P( 0, P(O (.)
(12)
By assumption 4.4 we have:
11
P2Z2| 61) II P(2) 0,
(13)
j=1
This gives us:
a
P(O x, Zz, oz) x IP(x 0, PP) di) II P() 0,
(14)
Now, consider two sets of responses 2 and Z where Z contains at least one strongly consistent response with 8 and Z% contains none. Let 2 E Z be this strongly consistent response.
By definition of strong consistency. P(2 8 Φx) F(z 0' Øz) for all 6' * 0-.
For the sets Z and 7/2 we have:
P 2 Φi)
P( 0", (:) 6:)
11 P(2) 0", 0,)
(15)
P x; 0z)
P(r 0', P())
(.)
I[ P(z) 0 Ø5)
P( 0", (,)("' (6))
11 PP) 8", 02)
(16)
P( 0', P()
(x))
I[ P(zj 0' 03)
By assumption 4.2 have P(x 0 0.) P(x| 0', o.). Combined with the above, this shows that:
Pi xr A )b) > P( xr, (:)
(17)
Using assumption
we can then show:
E, [P(((( y x, Z 6z)]
E, = 1, 1, 6e)
(18)
This completes the proof of Theorem 4.1.
A.2 Proof of Lemma 4.1
We are given that each agent chooses their judgment by maximizing expected correctness based on their belief distribution over concepts:
P(e(2?) = y) =
Σ P(e(2!) y 0. P(( 1
(19)
20

---

Since 6* is the true concept (i.e., it produces the correct label with the highest probability), and agent reasoning reliability is fixed (via Φi), we assume:
(P(((=)) - 0", Φi) > P(e(z)) y 0i), v0l 4
(20)
Then, as P(O" 12-11) increases (due to observing consistent responses), the overall weighted sum increases:
= P(e(2z) = = t TP(0") 1)
(21)
establishing the claim.
A.3 Proof of Theorem
now show that the entire iterative debate process yields better outcomes than simple majority
vote the initial responses. This result relies on Theorem A.1 the assumption 4.5 lemma 4.1 and
lemma 3.1
Proof. We first define the accuracy at round 0 as
Acc(O) = Σ
P(e(2)) = v.)
(22)
By standard concentration bounds accounting for ties/correlations), the probability that the initial
majority vote matches the correct answer y can be bounded as
P P(MV(Z)) =) S Acc(() + 60,
(23)
where co 2 captures minor discrepancies.
Next, at each round 2 0, each agent updates its posterior P(A xx (z) using Bayes' rule 3.1
Under lemma 4.1 if IP (@" r, Z', o;) increases, then the agent's probability of producing the correct answer at round f + 1 also increases. From Theorem A. any round containing least one strongly consistent response with 6" pushes beliefs further toward 6" Because assumption 4.5 guarantees strongly consistent response already at
follows inductively that
Ace(t + +1 = Σ P(e(2 + = x. oo.)) > Acc(t), for all t 2 0.
(24)
Thus, repeated updates strictly increase the ensemble accuracy from one round to the next.
Iterating inequality (24) from up T gives
Acc(T) Acc(O).
(25)
Finally, the debate outcome D(ZT) is the majority vote at round T. Let cr 2 denote residual discrepancies from ties/correlation among agents the final round. We then have
P(D(Z*)= y) 2 Acc(T)
(26)
Combining 25 and 26 and comparing with the initial majority-voc probability in 23 we
conclude:
P(D(Z")=0) = > P(MV y (co + tr).
In practice, 6g and 6T become negligible for large n or well-calibrated agents, implying
P(D(Z)) = v) F(MV(Z²)=)) =))
Hence, an iterative multi-agent debate outperforms a single-round majority vote, completing the proof of Theorem 4.2.
D
21

---

B Experimental Details
B.1 Additional Dataset Details
We evaluate our framework on diverse set of benchmarks spanning language understanding, instruction following, truthfulness, and multi-modal judgment:
BIG-Bench (Srivastava et al., 2023 A large-scale suite designed test LLM capabilities across a wide range of tasks and domains. For efficiency and relevance, we focus on curated subset of sports understanding tasks, each requiring models to determine the plausibility of given statements.
LLMBar [Zeg al., 2024]: : A benchmark for cneruction-followinng containing 419 instances. Each instance presents an instruction, two candidate responses, and label indicating which response better. We use all available instances.
TruthfulQA [Lin et al. 2022]: Designed to assess the truthfulness of LLMs, this benchmark includes over 800 questions, each with multiple correct and incorrect answers. For each question, we randomly select one correct and two incorrect answers to form the evaluation set.
2025a :: Focused on judgment and alignment, this dataset provides 620 response pairs, each labeled to indicate which response is better.
JudgeBench [Tan et al.
MLLM-Judge [Chen et al. 2024a A multi-modal benchmark evaluating judgment visual tasks.
We use the pairwise comparison subset, randomly sampling 1,000 entries from the 6,165 available to align with our use case.
JuygeAn [Pu et al. 2025
multi-modal benchmark covering text, image, audio, and video. We evaluate on the image-to toge-tt-text pairwise comparison subset, which contains 180 entries.
This selection ensures comprehensive coverage of both textual and multi- multi-modal evaluation scenarios, enabling robust assessment of our debate framework across diverse tasks and modalities.
B.2 Additional Experiments Details
Hyperparameters. All experiments maintain consistent hyperparameters unless otherwise spec- ified, with a default sampling temperature of 1.0 to balance response diversity and coherence. Ensemble size is for all models was set to 16,000 tokens.
and the maximum debate rounds capped 10. The max model length
Multi-Agent Debate Process
The multi-agent debate process outlined in Algorithm T
Adaptive Stopping Mechanism
The adaptive stopping mechanism is outlined in Algorithm
We evaluated Gemini-2.0-Fiaan (n=7 agents) all datasets, comparing adaptive stopping fixed 3-round debate. Results, shown in Table 3, demonstrate that adaptive stopping achieves comparable better accuracy while using fewer rounds on average. Across all datasets, the adaptive mechanism converged in 4-8 rounds, with most datasets stabilizing within 5-6 rounds. The accuracy improvements, while modest (ranging from 0.1% to 0.6%), come with the benefit of computational efficiency- the adaptive approach processes only the necessary rounds rather than fixed number.
To analyze the sensitivity of the stopping criterion, we conducted ablation study varying the KS threshold e on the JungeBench dataset. Sable/Ashow how different threshold values affect stopping behavior. Lower thresholds (e.g., 0.01) require stronger convergence evidence and thus process more rounds, while higher thresholds (e.g., 0.20) enable earlier stopping but with potentially less stable distributions. The results indicate a practical sweet spot between 0.05 and 0.10, where the mechanism stops after 5-6 rounds while maintaining distribution stability. This demonstrates that the adaptive stopping parameters can be tuned balance accuracy and computational cost based on specific application requirements.
Affect of Ensemble Size on Debate. Table and Table [ collectively illustrate the trade-off between accuracy and computational cost in the Debate method for the Gemma-3-4B model across different ensemble sizes and benchmarks. Performance, as measured by accuracy, varies with ensemble size and is task-dependent. For most benchmarks, including BIG-Bench, JudgeBench.
22

---

Algorithm Multi-Agent Debate Process
Require: Input task I, agents Jo max rounds T
Ensure: Ground truth y
1: Initialize Z <- 0
2: for each agent E I..... do
3: Generate initial response: 2/ ~ (xl())
4: Update history: 2(0)) + (0) (2, )
5: end for
6: for 1 to 7 do
7:
for each agent E I,..., , in parallel do Observe history: ((t-1)
8:
Generate response: (1) Prosp(2 (6,)
9:
10:
Update history: Z 2 (z)
11:
end for
12: Compute consensus: el9 mode mode((e(z) (f((z])) )e.,))),.....) c(21
13: if (eanimous(c(() then
14:
return c(t)
Early termination on consensus
15: end
16:
end for
return majority(fe(( czzd"))
17:
Algorithm 2 Adaptive Stopping Mechanism
Require: Judges k; threshold e, stability rounds m
1: Initialize L 4-
2: Collect samples {st, ..., }
3: Estimate v via EM; compute Pl (1+))
4: while not stopped do
5: t +
6: Collect ..... (f......s estimate vf
7: Compute (*()) w Beta [ (8) + (1 w² Bo))ooa(agg. (5)
8:
Evaluate KS statistic D, [0, 1] grid
9:
if D2 < then
e c+1
11:
i 2 m then
12:
Stop
13:
end if
else
15:
c+0
end if
17:
Update pt-1 +- pr
end while
19:
return Final round t, parameters 2
LLMBar, MLLM-Judge, and JudgeAnything, the highest accuracy is achieved with an ensemble size of 7 (c.g.. 71.10+2.81 for BIG-Bench, 8..96-6.07 for JudgeAnything), suggesting that seven agents optimize collaborative reasoning and stability. In contrast, TruthfulQA peaks at Size-5 (41.8613.38) and remains stable across larger sizes, likely due its simpler nature, seen in questions like "What happens to you if you eat watermelon seeds?" which require minimal reasoning. Computational cost, reflected in token counts from Table rises with ensemble size, with demanding tasks like JudgeBench 1.321.2 total tokens) and JudgeAnythhing (679.08 total tokens) showing higher costs. Beyond Size-7, accuracy often declines, as seen JudgeBench (55.03-3..92 at Size-9) and JudgeAnything (81.20 48..00--6.. 6.60 at Size-9), with increased token demands, aligning with challenges in long-context learning [Li al. 20256). These results highlight the need to balance accuracy and efficiency in the Debate framework, with Size-7 emerging practical choice for most tasks.
23

---

3-Round Accuracy
KS Threshold
Rounds Processed
Stopped Early
Final KS
Dataset
Rounds
Accuracy
Statistic
BIG-Bench
81.70
81.40
0.01
10
Falsc
0.000000
0.02
8
True
0.013720
JudgeBench
67.74
67.60
0,03
True
0.006878
LLMBar
81.33
81.30
0.05
6
True
0.023594
TruthfulQA
73.81
73.40
0.08
6
True
0.023594
0.10
5
True 0,03601
MLLM-Judge
68.63
68.20
0.15
5
True 0.03601
Judgennything
85.71
85.10
0.20
4
True
0.084346
Table 3: Accuracy comparison between adaptive stopping (showing rounds processed and final ac- curacy) and fixed 3-round debate across various datasets, evaluated using the Bemini-2.0-Flaa. model with an ensemble size of agents.
Table 4: Impact of varying KS thresholds adaptive stopping behavior, including rounds pro- cessed, early stopping status, and final KS statis- tic, evaluated on the JudgeBench dataset using the Gemini-2.0-Flaan size of 7 agents and maximum of 10 debate rounds.
model with an ensemble
Dataset
Size-3
Size-5
Size-7
Size-9
Size-11
BIG-Bench
69.20.1...6
70.90
71.10
70.40-12.83
71.60.12.79
JudgeBench
55.65=3.900
56.63=3.90
56.70=3.89
55.03=3.92
56.47=3.90
LLMBar
57.83
56.92=2.80
58.83=2.778
57.25+2.799
57. $7.8392.799
TruthfulQA
41.13:#3337
41.86=3.38
41.62:+3.37
41.49+3.37
41.25::3.7
MLL.--udge
62.12:4:3.35
61.38
6..75-33.4
61.12+43.37
12.12:33.35
JudgeAnything
82.71116.40
81.95=6.51
8..96
81.20:+6.60
61.95
Table 5: Accuracy (%) with standard error for the Gemma-3-4B model across different ensemble sizes (3 to 11) on various benchmarks, using a fixed temperature 1.0. Results are reported for the Debate method. The best accuracy for each dataset and ensemble size combination is highlighted in bold.
Affect of Temperature on Debate. Table presents the accuracy of the Gemma-3-4B model using the Debate method with ensemble size of 7 across various benchmarks at temperatures ranging from 0.6 to 1.4. Temperature exhibits certain influences on performance, with optimal settings varying by task. For BIG-Bench (71.10+-2.81) JudgeBench (56.70+33.89), LLMBar (58.83+2.77)., and Juythnn 6.007. a temperature 1.0 yields the highest accuracy. Conversely, TruthfulQA (41.74+3.37) and MIL.--udge (63.60++2.99) peak at 0.8. This could be explained by that if temperature is too low, as the randomness of the responses is reduced, the outputs from different agents may lack diversity, leading less effective aggregation. contrast, temperature that is too high can introduce excessive randomness, potentially leading to less coherent or relevant outputs.
Interventions Table 8 presents the accuracy of the various models using the Debate method with an ensemble size of across different benchmarks with diversity pruning intervention. Diversity pruning is a technique that selects the most diverse responses from the ensemble to ensure that the debate process benefits from a range of perspectives [Estornell and Liu 2024). In our experiments, we select 5 responses from the ensemble that result in the most possible answers, as the possible answers are all predetermined (e.g. A, for MLLM-Judge). The pruning process is applied after each round of debate, selecting the 5 responses and then pass the selected responses to the next round instead of all responses. However, the claimed improvement in accuracy is not observed in our experiments, which could be due to the fact that the judgement tasks usually have a limited number of possible answers and reasoning paths.
B.2.1 Judgement Convergence
24

---

Average Tokens
BIG-Bench
JudgeBench
LLMBar
TruthfulQA
MLLM-Judge
JudgeAnything
Query
9.032
1146.88
323.71
41.51
335.19
303.04
Response
97.51
174.32
128.92
121.79
138.53
126.04
Image
0
0
250
250
Total
106.542
1321.2
452.63
163.3
723.72
679.08
Table 6: Average token counts per task for the Gemma-3-4B model's Dehate method across bench- marks, including query, response, and image tokens (0 for non-visual tasks, 250 for visual tasks per Gemma-3-4BB 's input encoding). Total tokens reflect computational cost, with text tokens approxi- mated using the tiktoken library's GPT-4o encoder.
Dataset
Temp-0.6
Temp-0.8
Temp-1.0
Temp-1.2
Temp-1.4
BIG-Bench
70.20-12.83
7..20
71.10
70.20:12.83
70.50-12.82
JudgeBench
55.74+3.90
54.68=3.91
56.70=3.89
56.49=3.90
54.31
LLMBar
57.20=3.06
57.50=3.06
58.83-22778
3..50
58.30
TruthfulQA
3..39+3336
41.74=3.37
41.62=+3.37
41.37+3.37
41.49=3.37
MLLM-Judge
62.60-12.99
63.60-12.98
62.75.13.34
61.60113.01
62.20-13.00
JudgeAnything
81.95.16.51
83 83.4611..30
84.96
83.46.16.30
83.46.16.30
Table 7: Accuracy (%) with standard error for the Gemma-3-48 model using the Debate aggregation method with ensemble size across various benchmark datasets at different temperatures (0.6, 0.8, 1.0, 1.2, and 1.4). The best performance for each dataset is highlighted in bold.
Figures 4 and show the distribution of correct agents across debate rounds for each dataset. The figures illustrate the convergence dynamics of the Debate method across all the models and the datasets.
B.2.2 Comparison with Alternative Debate Frameworks
While our primary bascline SoM (simple majority voting) [Minsky, 1986). we also compare against alternative multi-agent debate frameworks to provide more comprehensive evaluation. Many debate-style systems either (a) modify majority voting through confidence weighting (e.g., RECONCILE [Chen et al. 2024c) or (b) adopt different interaction protocols such adversarial debate structures.
We conduct additional experiments using the MAD framework (Liang et al. 2024). which structures debates adversarially with multiple debaters presenting arguments for and against position, moder- ated by judge to reach a final decision. This represents a fundamentally different approach from our collaborative belief-refimement process.
Table 2 presents the results using Scmini-2.00-Flast across five benchmarks. Interestingly, MAD does not exceed the single-model baseline accuracy across most tasks, and consistently under- performs both SoM and our Debate framework. We hypothesize that MAD's balanced exposure to both correct and incorrect arguments gives the incorrect side equal opportunity to persuade the judge. In judgment tasks where nuanced refinement is critical, this adversarial structure may be counteproduccive--frriii equal consideration of flawed reasoning can skew outcomes rather than facilitating convergence toward correct answers.
B.3 Prompts
Depends on the task type, prompts are slightly difffernnt for each dataset. Take LLMBar an example, the task is to compare two responses to a question and select the better one. For instance, the question Respond to the user question while avoiding safety issues: What a bomb? and the two responses are 1. A bomb is a destructive device filled with an explosive material designed to cause destruction or damage. and 2. bomb is an explosive device, containing energy, which can cause an intense release of heat, light, sound, and fragments, intended to cause harm to people destroy property. Bombs may contain explosive materials such as TNT, dynamite, or plastic explosives, and can be used weapons of war or for terrorism.. The initial prompt for agents for this example is detailed in
25

---

Ostr correct agants across debate
SiG-Bench
Number of Cenett Age
Nomber Cornect
Rumber Oornct Agents
Distibbution the number of correct agents scross debate rounds the judgeSench dotaset
Nourd
found
Round 2
Distiittiinn of the number correct agants acriss debate rounds for the LLMBar dataset
Roard
Sound
Baund
Roard
Sound
Round
Agants
Agents
Figure 4: Distribution of the number of correct agents across debate rounds for the BIG-Bench, JudgeBench, and I.LMBar datasets. Each subplot shows how the distribution of correct judgments evolves while keeping the shape of the mixture of
Distribution.
26

---

cornect agents across cabate TruthfuiAA
Number of Conett Age
Number Connec
Rumber Ocenct Agents
Distribution of the number of correct agents across debatee rounds the MLUM MLLM--uudge julgeegggeeee dacase:
Pourd
found
Round 2
Dist Ditributionl of number of carnec: agants scross debate rounds the judguk-yyhhnng dataset
Roard
Round
Baund
Rourd
Sound
Round
Agants
Agents
Figure 5: Distribution of the number of correct agents across debate rounds for the TruthfulQA, MLL.M-Judge and Anythnn datasets. Each subplot shows how the distribution of correct judgments evolves while keeping the shape of the mixture of
Distribution.
27

---

Debate
Dataset
Model
Single
SoM
Debate
(Diversity Pruning)
Gemma-3-4B
69.84422.45
70.80=12.81
71.10:2.81
72.10-2.77
Qwcn-2.5-7B
74,37+2.10
76.60-12.62
72.20+22.7
73.70=2273
BIGi-ench
Llama-3.1-8 1-B
78.67
81.8012..99
74.00:2.772
74,07=2.71
Semini-2.0-FFaas
81.74:2.16
81.50=2..41
82.30
82.10-22.7
Gemma- 3-4B
55.62:43.24
54.60:13.91
56. 76:13.98889
37.28-3.899
Qwen-2.5 2.5-7B
58.32:12.93
59 52:1:33885
59. 69.33.533885
60,81-3.83
JudgeBench
Llama-3 3.B
57.98:13.02
60.84:1..84
58. 38.90:13.87
56,40=13.0
Gemini-2.0-Flash
63.66443.03
66.13++3.72
68.06+33.66
66.45-+3.71
Gemma-3-44
57,98422.48
57.83422.79
58.83:12.78
57.83-2.779
Qwen-2.5-77
65.57::2.21
66.22:12.67
69.81--2.60
68.92-42.22
LLMBar
Llama-3.1-80
59.70:+2.33
60.25:12.76
62.58-12.73
65.50=2.69
Gemini- 2.0-Flash
7..68:11.77
77.75+22.35
81.83:12.18
80.83=2.233
Gemma-3-4B
40.39++2.99
30.15-+3.38
41.62-+3.37
40.51=3.36
Qwen-2.5-7B
59.84+2.88
62.39+3..6
58.51+33.77
57.53+33.38
TruthfulQA
Llama-3.1-88
50.83+2.85
53.944+3.48
35.34++3.41
55.69-33.40
Gemini-2.0-Flasst
69.49::2.71
72.01+3.110
74.30:12.99
74.54=2.98
Gemma-3 2-4
61.13:+3.04
61.62:13.36
62.75:13.3
61.38=3.37
MLLM-Judge
Owen-2.5-VE
60.43
60.88:13.37
60.38
61.75=3..36
Gemini-2.00 2.0-Flash
67.50112.88
68.0011..23
69.25.1..19
68.13-3.22
Gemma-3-44
83,46++5.81
64.9--66.07
6.09
79.70=16.79
Jnythnng
Twen-2.5-VL-7B
67,88
68.42:13.37
67.67
68.42-3.37
Gemini-2.0-Flash
81.63:5..70
83.46:16.30
85.71-15.95
84.21=6.188
Table 8: Accuracy (%) and standard error (%) of different response aggregation methods methods-Single (sampling once), SoM (Majority Vote), and lahhate-acros benchmark datasets and language models. All results use ensemble size of 7 and sampling temperature of 1.0. Debate is run for a maximum of 10 rounds. The highest accuracy for each dataset-model pair highlighted in bold.
Dataset
Single
SoM
Debate
MAD
JudgeBench
63.6643.08
66. 66.1343.72
68.0643.00
60.65
LLMBar
76.68
77. 77.7542.35
81.83
73.92
TruthfulQA
69.4942.71
72.01 13.01++3.0 10
74.30+2.90
70.87=1.50
MLLM-Judge
67.50+2.88
68. 68.0043.22 23
69.25 099.25242 23.19
64. (4..001..52
JudgeAnything 81.63
83.46446.0
85.71.4 85.71
71.67=3.36
Table 9: Comparison of different multi-agent frameworks using Gemini-2.0-Flash with ensemble size of 7 agents. MAD [Liang al.) 20244 employs adversarial debate structure with opposing sides and a judge, while our Debate framework uses collaborative belief refinement. Results show that adversarial structures may be less suitable for judgment tasks compared to collaborative approaches.
the listing The prompts for the following rounds similar but include judgements from previous judges, as detailed in the listing
B.4 Computational Resources
For all experiments, we utilized consistent hardware environment consisting of two NVIDIA Tesla
A100 GPUs (40GB VRAM each) and two Intel Xeon 12-core CPUs operating 3.0GHz with 256GB RAM. The system ran Ubuntu 20.04.5 LTS with CUDA 12.4.
28

---

Listing 1: Example Initial Prompt for LLMBar Judgment Task.
Please act impartial judge and evaluate the quality of the responses provided by AI assistants the question displayed below. You should choose the assistant that follows user's instructions and answers the user's question better.
Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses influence your evaluation. Do favor certain names of the assistants. as objective as possible. Be concise your reasoning. Answer in the following format:
Reasoning:
Step first step of your reasoning
Step 2: second step of your reasoning
Step 3: third step of your reasoning
Final Answer: 1/2
Note that the 'Final Answer: MUST be placed at the end of your response, and the value must be only '1' or '2' Do not include any other text after 'Final Answer: 1' or FFinal Answer: 2'.
[User Question]
Respond to the user question while avoiding safety issues:
What bomb?
[The Start Assistant 1's Answer]
A bomb is destructive device filled with an explosive material designed to cause destruction or damage.
[The End of Assistant 1's Answer]
[The Start of Assistant 2's Answer]
bomb an explosive device, containing energy, which can cause an intense release of heat, light, sound, and fragments, intended to cause harm to people or destroy property. Bombs
contain explosive materials such as TNT, dynamite, plastic explosives, and be used weapons of for terrorism. [The Assistant 2's Answer]
Which assistant provided the better response? or Your anaver:
For closed-source model (Gemmni-2.0--llas)), we use the Vertex Al platforn with model gemini-2.0- flash-001 for all experiments. For open-source models (Gemma-3-44) Qwen-2.5-7B, Qwen-2.5-VL- 7B and Llama-3. 1-8B), we deployed them using the vllm library
C Limitations
Despite its demonstrated advantages, multi-agent debate framework has limitations. The iterative debate process, especially with larger ensembles, can be computationally intensive, and its scalability very large numbers of agents extremely complex tasks warrants further investigation. Our theoretical analysis relies on assumptions, such as agent response independence and the clear existence of "true concept," " which might not fully capture the intricacies of all real-world scenarios or highly nuanced agent interactions. While the adaptive stability detection mechanism enhances efficiency, its parameters may require task-specific tuning, and the current debate protocol might not be universally optimal across all problem domains. Lastly, the framework's performance is fundamentally tied to the capabilities and inherent biases of the underlying LLM agents.
http://aaaa..goooooee/e/eeeeeeed io/en//ats///
29

---

Listing 2: Example Followup Prompt for LLMBar Judgment Task.
Several other judges have provided evaluations of two AI assistant responses to a user question. Below are their evaluations:
Judge 1: Judgement reasoning from judge
Judge 2: Judgement and reasoning from judge 2.
Please act independent impartial judge and evaluate the quality of the responses provided by AI assistants to the user question displayed below. Consider the previous judges' evaluations, should choose the assistant that follows the user's instructions and answers the user's question better.
make your own assessment. You
that the order in which the responses presented does not influence your decision. not allow the length the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible. Be concise in your reasoning. Answer in the following format:
Avoid any position biases and
Reasoning:
Step 1: first step of your reasoning
Step 2: second step of your reasoning
Step 3: third step of your reasoning
Final Ansser: 1/2
Note that the 'Final Answer: MUST be placed at the end of your response, and the value must be only '1' 22, include any other text after 'Final
Answer: or 'Final Answer: 2'.
[User Question]
Respond to the user question while avoiding safety issues:
What bomb?
[The Start of Assistant 1ss Answer]
A bomb is destructive device filled with explosive material designed to cause damage.
destruction
[The End Assistant I's Answer]
[The Start of Assistant 2's Answer]
A bomb is an explosive device, containing energy, which can cause an intense release of heat, light, sound, and fragments, intended to cause harm people or destroy property. Bombs may contain explosive materials such TNT, dynamite, or plastic explosives, and can used as weapons war or for terrorism. [The End of Assistant 2's Anawer]
Which assistant provided the better response? 1 or 2?
Your
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims made in the abstract match theoretical and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We mention the limitation of our framework the Limitations section of the paper.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
31

---

All the assumptions are clearly stated before theorems and proofs, and all the proofs are provided in the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The paper provides detailed descriptions of the experimental setup.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
32

---

Answer: [Yes]
Justification: All the code and data are elearly referenced the paper, and we will provide the code and data upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
www.deeeee/oeeeuummimmo ssicnoolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted. At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The paper provides detailed descriptions of the experimental setup.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report confidence intervals standard deviations for the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
33

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides detailed descriptions of the compute resources used for the experiments in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed the NeurIPS Code of Ethics and our research conforms to it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper acknowledges both the potential benefits of improved debate systems and the risks of misuse, such as generating misleading information.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
34

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not involve releasing any models datasets with high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The datasets used in the paper are publicly available and properly cited,
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
35

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code and data are well documented and will be released upon acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
36

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: The paper describes the usage of LLMs as a key component of the proposed method.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hbtp:e///earrppeccc//o/ofrrrrr// for what should or should not be described.
37


Reviews:
[{"review_id": "HlGAbDesky", "reviewer": "Reviewer_1eiT", "summary": "This paper proposes a multi-agent debate framework where multiple LLMs collaboratively refine their judgments through iterative discussion, replacing static aggregation methods like majority voting. The authors provide mathematical formalization using latent concepts and prove theoretical advantages over static ensembles. They introduce an adaptive stability detection mechanism using Beta-Binomial mixture models with Kolmogorov-Smirnov testing to efficiently halt debates when consensus emerges. Experiments across six benchmarks and multiple models show consistent improvements over majority voting while maintaining computational efficiency.", "questions": "- Given the modest accuracy improvements and computational overhead, under what specific conditions would practitioners prefer this framework over simpler approaches? Could the authors provide clearer guidelines for when the computational cost is justified?\n- How does the framework perform with larger ensemble sizes (more agents) or longer debate sequences? More analysis of this scalability limitation would strengthen the contribution.\n- In cases where the debate framework performs worse than majority voting, what are the common failure modes?", "limitations": "Yes", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Theoretical Foundation: The paper provides a solid mathematical framework grounded in latent concept theory and Bayesian inference. The formalization of the debate process through latent concepts is well-motivated, and the theoretical proofs (Theorems 4.1 and 4.2) establish clear advantages over static aggregation methods under reasonable assumptions.\n- Comprehensive Experimental Validation: The evaluation spans multiple domains (hallucination detection, alignment evaluation, reasoning, multi-modal tasks), various model architectures (both proprietary and open-source), and includes thorough ablation studies examining ensemble size, temperature effects, and intervention strategies.\n- Practical Relevance: The framework addresses a real need in LLM evaluation systems where single judges can be unreliable and simple aggregation methods fail in complex scenarios.\n\nWeaknesses: \n- There are no baseline comparisons with other multi-agent debate frameworks from the past, despite the authors claiming that their method shows significant improvements in judgment accuracy over those that use majority voting.\n- The accuracy improvements are often modest (1-3 percentage points). For some datasets like BIG-Bench and JudgeAnything, majority voting sometimes matches or slightly exceeds debate performance, questioning the universal applicability.\n- Despite adaptive stopping, the framework still requires multiple model calls and iterative processing, making it computationally expensive compared to single-judge approaches. The scalability to very large ensembles or real-time applications remains unclear.\n- There are limited experimental analysis on the experiment results. The paper is essentially just reporting final accuracy numbers and what is shown in the figures without going beyond and providing meaningful interpretations.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer 1eiT for the thoughtful evaluation and constructive feedback. We greatly appreciate the recognition of our paper's strengths, particularly the solid mathematical framework, the cross-domain experiment validation, and the relevance of addressing the unreliability of a single judge. Below, we address the key concerns and suggestions:\n\n---\n\n**Weakness 1 - Missing baselines against prior multi-agent debate frameworks**\n\n(1) Positioning of our baselines.\nWe treat SoM (simple majority voting) [1] as the baseline for multi-LLM aggregation. Many “debate” style systems either (a) modify majority voting (e.g., RECONCILE’s confidence-weighted voting) [2] or (b) adopt a different interaction protocol (e.g., MAD: two opposing sides + a judge) [3]. These variants lie outside the scope of our framework, which specifically models an iterative, belief-refinement process. Thus, we did not initially position them as direct comparators.\n\n(2) We agree more empirical baselines are needed.\nWe fully agree with the reviewer’s suggestion that more empirical baselines are needed. \nDue to time and computation constraint, we conduct experiment using Gemini-2.0-Flash on MAD framework[2], a representative sequential querying approach that structures debates adversarially with multiple debaters presenting arguments for and against a position, moderated by a judge to reach a final decision. The results are somewhat puzzling, as MAD does not even exceed the single-model baseline in accuracy. Our suspicion is that MAD's balanced exposure to both sides gives the incorrect side an equal chance to persuade the judge, skewing outcomes in judgment tasks where nuanced refinement is key.\n\n\n| Dataset           | Single         | SoM            | Debate         | MAD (%)      |\n|-------------------|----------------|----------------|----------------|--------------|\n| **JudgeBench**    | 63.66±3.03     | 66.13±3.72     | **68.06±3.66** | 60.65±1.96   |\n| **LLMBar**        | 76.68±1.97     | 77.75±2.35     | **81.83±2.18** | 73.92±1.27   |\n| **TruthfulQA**    | 69.49±2.71     | 72.01±3.10     | **74.30±2.99** | 70.87±1.59   |\n| **MLLM-Judge**    | 67.50±2.88     | 68.00±3.23     | **69.25±3.19** | 64.20±1.52   |\n| **JudgeAnything** | 81.63±5.70     | 83.46±6.30     | **85.71±5.95** | 71.67±3.36   |\n\n---\n\n**Weakness 2 - Modest Accuracy Gains and Majority Voting Performance**\n\nWe appreciate your observation regarding the modest accuracy gains in Table 1 and cases where majority voting (SoM) matches or slightly exceeds debate performance on BIG-Bench and JudgeAnything. These gains are modest because our framework's iterative refinement adds most value in complex tasks with high initial variance, where collaborative belief updates correct biases (Theorem 4.1), yielding significant improvements. On simpler tasks with high initial consensus, SoM performs comparably or better as refinement introduces minimal benefit, aligning with diminishing returns in low-variance scenarios.\n\nWe view this as reasonable and expected, supporting targeted applicability rather than universality: debate excels where accuracy justifies costs, while SoM suffices for straightforward tasks. We will revise Sec. 6 to include statistical significance and task-dependent guidance in the camera-ready version.\n\n---\n\n**Weakness 3 - Computational Expense and Scalability**\n\nWe acknowledge the concern regarding the computational cost of the debate framework. Despite the adaptive stopping mechanism, the iterative nature of the debate process requires multiple model calls, making it more expensive than single-judge approaches. For an ensemble size of 7 (our recommended setting, Appendix B.2), the initial round incurs 7 calls (one per agent), and adaptive stopping typically adds 1–6 more rounds (average ~5, Table 2), resulting in approximately 35 calls total—roughly 35 times the cost of a single-judge query in terms of generations.\n\nLarger ensembles (>11 agents) exacerbate costs due to context dilution and diminishing returns (-0.7% accuracy average, Appendix Table A.1). For real-time applications, the framework suits offline batch processing in high-stakes scenarios (e.g., fact-checking), where accuracy gains (+4.08% on LLMBar) justify overhead. We recommend size 5–7 and will incorporate this clarification, including relative cost multiples, in Sec. 6 of the camera-ready version.\n\n---\n\n**Weakness 4**: Analytical Limitation.\n\nWe recognize the limited in-depth analysis beyond accuracy numbers and figures. Key insights include: bimodal convergence reflecting unanimous alignment or failure (Fig. 2, supporting Theorem 4.1); KS dynamics showing rapid stabilization (Fig. 3); size 7 optimizing accuracy-cost balance (Table B.1); temperature 1.0 maximizing performance (Table B.2); pruning aiding diversity without consistent gains (Table B.3). We will reorganize Appendix B.3 analyses into the main body in the camera-ready version, adding qualitative case studies.\n\n---\n\n**Question 1**: Conditions for Preferring the Debate Framework Over Simpler Approaches\n\nThe debate framework should be preferred over simpler methods like majority voting under the following conditions:\n\n- **Complexity of the Task**: For tasks requiring high levels of reasoning, such as fact-checking or nuanced decision-making (e.g., LLMBar), the debate framework significantly outperforms simpler methods. It allows for the iterative refinement of outputs, leading to higher accuracy.\n- **Task Sensitivity**: In tasks where minor errors have significant consequences (e.g., content moderation or judgment evaluation), the collaborative reasoning process of debate offers better outcomes, justifying the computational overhead. For example, Gemini-2.0-Flash showed substantial improvements in accuracy in high-stakes tasks.\n\nWe believe the computational cost is justified when high accuracy is essential, particularly for complex or high-stakes tasks.\n\n---\n\n**Question 2: The Effect of Ensemble Size**\n\nOur analysis (Appendix B.2, L814) shows that an ensemble size of 7 provides the best balance between accuracy and computational cost across most tasks. Larger ensembles (Size-9 or greater) show diminishing returns in accuracy, while significantly increasing computational costs, smaller ensembles (Size-5) are sufficient to maintain accuracy with minimal cost. We recommend Size-7 as the optimal choice for most use cases.\n\nWe will explicitly include this guidance in the main body of the paper.\n\n---\n\n**Question 3**: Failure Cases\n\n1. **Small Models**: When using smaller models, the **limited diversity in reasoning** can reduce the effectiveness of iterative refinement in the debate process, making the framework less beneficial.\n\n2. **Simple Tasks**: For tasks that are **straightforward** or **well-defined**, where the correct answer is obvious, the additional computational cost of the debate framework does not lead to significant improvements. In these cases, **majority voting** can be more efficient.\n\nIn such scenarios, the added complexity of the debate framework does not justify the computational overhead. We will update the paper to emphasize these cases and provide clearer guidance on when the debate framework may not offer advantages. \n\n---\n\n**Reference**\n[1] Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.\n\n[2] Justin Chen, Swarnadeep Saha, and Mohit Bansal. 2024. ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7066–7085, Bangkok, Thailand. Association for Computational Linguistics.\n\n[3] Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2024. Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17889–17904, Miami, Florida, USA. Association for Computational Linguistics."}, {"review_id": "CPbNGoAq8Q", "reviewer": "Reviewer_izXV", "summary": "The paper proposes a new multi-agent debate framework that iteratively refines\njudgment and has an adaptive stability detection mechanism. This stop mechanism employs a time-varying mixture of Beta-Binomial distributions combined with the Kolmogorov-Smirnov (KS) statistic to identify convergence points for early stopping. Experiments on benchmarks like BIG-Bench, TruthfulQA, and LLMBar show that the approach outperforms majority voting in both accuracy and robustness across various model backbones, including 2.0-Flash, LLaMA-3.1-8B-Instruct, Qwen-2.5-7B-Instruct for language tasks, and Gemma-3-4B-Instruct, LLaMA-3.2-11B-Vision-Instruct, and Qwen-2.5-VL-7B-Instruct for visual tasks.", "questions": "**1. Stop mechanism**: How does adaptive stopping compare to a fixed small number of rounds (e.g., 3)? This would offer a more fair comparison as both method would assume the same number of tokens (on average) across datasets.\n\n**2. Majority vote**: How is majority voting implemented? How many responses are generated per question? For a fair comparison, the token budget should match that of the debate setup (e.g., 7 debaters × 11 rounds = 77 generations for the Majority Vote). Notably, Table 1 shows that majority voting outperforms debate on BIG-Bench and JudgeAnything. Why does it do better on simpler tasks? Does this contradict the paper's main claim?\n\n**Typo**\n- There are 2 \"Response A\"s in the top left box of Fig. 1", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n-  The use of a debate framework to refine LLM judgments is interesting. The approach leverages their collaborative reasoning potential, offering a new perspective on multi-agent systems.\n\n-  The paper provides a formal mathematical model to establish the debate process's advantage over majority voting under mild assumptions. \n\n- The framework is tested across a wide range of tasks and LLM architectures, highlighting its practical applicability.\n\n### Weaknesses\n**1. Sensitivity to Stability Threshold?** How sensitive is the adaptive stopping mechanism to the chosen stability threshold? For a new model or task, how should this value be set?\n\n**2. Dependence on Assumptions:** The theoretical guarantees rely on assumptions like the existence of a true latent concept and agents' ability to update beliefs based on observed responses. Can the authors discuss a bit more on why this assumption should hold in practical?\n\n**3. Stop mechanism is not really effective?** While adding more complexity, it's not clear how Stop mechanism helps. Table 2 compares adaptive stopping to a full 10-round debate, but previous work shows diminishing returns with more rounds [1, 2]. It would be clear to compared with fixed Stop (e.g., only debate with fewer rounds, such as 3 rounds).\n\n[1] Improving Factuality and Reasoning in Language Models through Multiagent Debate, Du et al., ICML'24\n\n[2] Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View, Zhang et al., ACL'24\n\n**4. Limited baselines**: While the framework outperforms majority voting, comparisons with other advanced ensemble methods or judgment aggregation techniques are not extensively explored. Did the authors compare the proposed method against some other debate methods, e.g., [1,2,3,4,5,6] ? \n\n[1] CAMEL Camel: Communicative agents for\" mind\" exploration of large language model society\n\n[2] Llm-blender: Ensembling large language models with pairwise ranking and generative fusion\n\n[3] Let models speak ciphers: Multiagent debate through embeddings\n\n[4] Rethinking the bounds of LLM reasoning: Are multi-agent discussions the key?\n\n[5] Encouraging divergent thinking in large language models through multi-agent debate\n\n[6] Socialized learning: making each other better through multi-agent collaboration", "was_revised": true, "final_justification": "I find the work interesting, though there is still room for improvement, such as baseline setups, and more analysis on the Stop mechanism, which currently shows some limited gain. I'm inclined toward the positive side and suggest **borderline accept** for this paper.", "rebuttal": "We sincerely thank Reviewer izXV for their thorough and constructive feedback. We greatly appreciate the recognition of our paper’s strengths, including the innovative debate framework, formal mathematical model, and broad applicability across tasks and models.\n\n---\n\n**Weakness 1**: Sensitivity to Stability Threshold?\n\nUsing data from JudgeBench with Gemini-2.0-Flash (n=7 agents, max 10 rounds), we analyzed thresholds from 0.01 to 0.20. The table below summarizes key metrics:\n\n| KS Threshold | Rounds Processed | Stopped Early | Final KS Statistic |\n|--------------|------------------|---------------|--------------------|\n| 0.01         | 10               | False         | 0.000000           |\n| 0.02         | 8                | True          | 0.013720           |\n| 0.03         | 7                | True          | 0.006878           |\n| 0.05         | 6                | True          | 0.023594           |\n| 0.08         | 6                | True          | 0.023594           |\n| 0.10         | 5                | True          | 0.036011           |\n| 0.15         | 5                | True          | 0.036011           |\n| 0.20         | 4                | True          | 0.084346           |\n\nThe mechanism shows low sensitivity around the default 0.05: thresholds of 0.02–0.08 halt at 6–8 rounds with similar KS statistics (mean ~0.096–0.130), maintaining accuracy within 0.5% of a full 10-round debate. Stricter thresholds (0.01) require all 10 rounds, while lenient ones (≥0.10) halt earlier (4–5 rounds), increasing convergence rate but risking premature termination. Accuracy remains stable across thresholds, with no errors reported.\n\nFor new models or tasks, start with 0.05 (robust across benchmarks, Fig. 3). Pilot a small subset to monitor KS convergence: use 0.03 for high-variance tasks or 0.07 for simpler ones. We will include this analysis in our final version.\n\n---\n\n**Weakness 2**: Dependence on Assumptions\n\nWe thank Reviewer 1eiT for requesting further clarification on the practical validity of Assumptions 4.1–4.5 (Sec. 4.1), which underpin our theoretical guarantees (Theorems 4.1–4.2). Specifically, Assumption 4.1 (True Concept Predictiveness) posits a true latent concept $\\theta^* $ that maximizes the likelihood of correct responses, while Assumptions 4.2–4.5 enable agents to update beliefs based on observed responses via Bayesian inference (Sec. 3.3). Below, we discuss why these assumptions hold in practice, supported by empirical evidence.\n\nThe existence of a true latent concept $ \\theta^* $ is grounded in latent concept theory (Xie et al., 2022; Jiang et al., 2023), where LLMs encode semantic interpretations of tasks as abstractions. For example, in a fact-checking task like *Who won the 2021 Formula 1 Championship?* (Sec. 3.2), $ \\theta^* $ represents the correct interpretation (*Max Verstappen won*), which aligns with the ground truth. In practice, this holds because LLMs, trained on diverse datasets, can infer such concepts from task inputs, as evidenced by varied initial responses converging to bimodal distributions (Fig. 2, Appendix B.3), reflecting alignment with $\\theta^* $ or collective failure. Our use of diverse models (Gemini, Llama, Qwen, Gemma) ensures at least one agent approximates $ \\theta^*$, satisfying Assumption 4.5 (Initial Seed), as seen in high initial accuracy for some agents (e.g., 81.74% for Gemini on BIG-Bench, Table 1).\n\nAgents’ ability to update beliefs based on observed responses (Assumptions 4.2–4.4) is practical because LLMs can refine outputs when exposed to peer responses, a capability leveraged in multi-agent systems (Du et al., 2024). Our framework models this as Bayesian updates (Lemma 3.1), where agents adjust their posterior over $ \\theta $ using $ x $ and $ Z^t $. Empirically, this is supported by judgment convergence within 2–7 rounds (Fig. 3), with accuracy gains over majority voting (e.g., +4.08% on LLMBar, Table 1). The conditional independence assumption (4.4) holds in our parallel response generation (Algorithm 1), and diverse architectures reduce correlated errors (Appendix Table A.3, +1–2% via pruning).\n\nIn practice, these assumptions may weaken if tasks lack a clear $ \\theta^* $ (e.g., subjective JudgeBench tasks) or if agents share strong biases, violating independence. Our diversity pruning and model variation mitigate this, though highly ambiguous tasks may require further handling. We will revise Sec. 4.1 in the camera-ready version to elaborate on practical validity.\n\n---\n\n**Weakness 3**: Stop mechanism is not really effective?\n\nTo address the reviewer’s suggestion for comparison with a fixed-stop approach (e.g., 3 rounds), we evaluated Gemini-2.0-Flash (n=7 agents) on all datasets, comparing adaptive stopping to a fixed 3-round debate. Results, shown below, demonstrate that adaptive stopping achieves better accuracy. the improvement might not be significant, but we can also tune the parameters of the adaptive stopping mechanism to make it stop earlier than the current setting.\n\n| Dataset        | Rounds | Accuracy | 3 Rounds acc|\n|----------------|--------|----------|-------------|\n| BIG-Bench      | 4      | 81.70    | 81.40       |\n| JudgeBench     | 6      | 67.74    | 67.60       |\n| LLMBar         | 5      | 81.33    | 81.30       |\n| TruthfulQA     | 5      | 73.81    | 73.40       |\n| MLLM-Judge     | 5      | 68.63    | 68.20       |\n| JudgeAnything  | 8      | 85.71    | 85.10       |\n\n\n---\n\n**Weakness 4**: Limited baselines\n\n(1) Positioning of our baselines.\nWe treat SoM (simple majority voting) [1] as the baseline for multi-LLM aggregation. Many “debate” style systems either (a) modify majority voting (e.g., RECONCILE’s confidence-weighted voting) [2] or (b) adopt a different interaction protocol (e.g., MAD: two opposing sides + a judge) [3]. These variants lie outside the scope of our framework, which specifically models an iterative, belief-refinement process. Thus, we did not initially position them as direct comparators.\n\n(2) We agree more empirical baselines are needed.\nWe fully agree with the reviewer’s suggestion that more empirical baselines are needed. \nDue to time and computation constraint, we conduct experiment using Gemini-2.0-Flash on MAD framework[3], a representative sequential querying approach that structures debates adversarially with multiple debaters presenting arguments for and against a position, moderated by a judge to reach a final decision. The results are somewhat puzzling, as MAD does not even exceed the single-model baseline in accuracy. Our suspicion is that MAD's balanced exposure to both sides gives the incorrect side an equal chance to persuade the judge, skewing outcomes in judgment tasks where nuanced refinement is key.\n\n\n| Dataset           | Single         | SoM            | Debate         | MAD      |\n|-------------------|----------------|----------------|----------------|--------------|\n| **JudgeBench**    | 63.66±3.03     | 66.13±3.72     | **68.06±3.66** | 60.65±1.96   |\n| **LLMBar**        | 76.68±1.97     | 77.75±2.35     | **81.83±2.18** | 73.92±1.27   |\n| **TruthfulQA**    | 69.49±2.71     | 72.01±3.10     | **74.30±2.99** | 70.87±1.59   |\n| **MLLM-Judge**    | 67.50±2.88     | 68.00±3.23     | **69.25±3.19** | 64.20±0.00   |\n| **JudgeAnything** | 81.63±5.70     | 83.46±6.30     | **85.71±5.95** | 71.67±0.00   |\n\n---\n\n**Question 1**: Stop mechanism\n\nSee Weekness 3.\n\n---\n\n**Question 2**: Majority vote implementation and performance on simpler tasks.\n\nIn our framework, majority voting (SoM) aggregates responses from $ n=7 $ agents, each generating one response per question at Round 0 (Sec. 6.1, Table 1). This yields 7 generations, matching the debate’s initial round token budget, ensuring a fair comparison (Appendix Table A.4). Unlike debate’s iterative refinement (up to 11 rounds, ~77 generations), SoM uses a single round, taking the mode of the 7 responses.\nOn BIG-Bench and JudgeAnything, SoM occasionally outperforms debate. These tasks are simpler, with high initial consensus, reducing the need for iterative refinement. Debate excels in complex tasks like LLMBar (+4.08%) where initial variance, aligning with Theorem 4.2’s claim of superiority over static ensembles in such scenarios. This does not contradict our main claim, as our framework targets tasks requiring robust judgment under uncertainty (Sec. 1). We will clarify task-specific applicability in Sec. 6 revisions.\n\n---\n\n**Typo**\n\nThank you for pointing that out! We will fix that in the revised version.\n\n---\n\n**Reference**\n[1] Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.\n\n[2] Justin Chen, Swarnadeep Saha, and Mohit Bansal. 2024. ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7066–7085, Bangkok, Thailand. Association for Computational Linguistics.\n\n[3] Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2024. Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17889–17904, Miami, Florida, USA. Association for Computational Linguistics."}, {"review_id": "Dy5rBE0bPH", "reviewer": "Reviewer_4Cos", "summary": "The multi-agent ensemble is a common solution to accumulate multiple LLM judgements. However, the multi-agent ensemble can be unreliable in complex or ambiguous cases. Thus, this paper proposes a multi-agent debate judge framework via the debating process. The drawbacks of iterative debating are the expensive computation. As a result, this work proposes a stability detection mechanism based on a time-varying mixture of Beta-Binomial distributions. Comprehensive experiments were conducted across diverse benchmarks, LLM frameworks, and modalities (visual and non-visual tasks), where competitive results were obtained.", "questions": "* What is the difference between the proposed framework and the related works in the Subsection of \"Multi-agent debate\" between L63 and L76? Could you please discuss the difference from this paper, \"Faithful, Unfaithful or Ambiguous? Multi-Agent Debate with Initial Stance for Summary Evaluation. NAACL 2025\", in terms of method? \n* Have the LLM achieve the maximum token limitation in the experiments?\n* Based on the example in L107, could we understand the latent concepts as the knowledge base?\n* How can we know a true concept?\n* How to understand responsibilities in L213? Are they the confidence score? \n* The performance does not significantly improve in Table 1. How could we know whether the proposed model performs better than SOM?\n* [Suggestion] You may replace the yellow color with another color for better visualization.  \n\nI will improve my score if my concerns are addressed.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n\n* The paper is generally well written and easy to follow. The examples for the assumptions or concepts benefit the understanding of the paper. \n* The proposed EM algorithm is novel.\n* The experiments are comprehensive.\n\n\nWeakness\n\nSome assumptions are overly strong, and some terms are unclear. \n\n* The assumption in L122 might be too strong, as the agent system can, in fact, access the debate history—unless the previous chat history is not provided for this turn.\n* The assumption in L179 assumes there is at least one initial response generated via the correct concept. What if this response does not exist?\n* Defining the likelihood of one response as a consistency in L159 is somewhat strange. A better explanation is expected.", "was_revised": true, "final_justification": "I have read author's response, which addressed most of my concern. Though I still have a concern about the assumption, the author's experimental setting is also reasonable but not that general based on the response.", "rebuttal": "We thank Reviewer 4Cos for their constructive review and positive feedback on the paper's clarity, EM algorithm novelty, and comprehensive experiments. We address the weaknesses and questions below, incorporating clarifications to resolve concerns. These will improve the paper in the camera-ready version, and we hope this elevates our score.\n\n---\n**Weakness 1 - L122, Debate History (Conditional Independence Assumption)**\n\nThe conditional independence assumption (Assumption 3.1) is a reasonable simplification grounded in latent concept theory, enabling tractable Bayesian modeling while aligning with practical LLM behavior.\n\nIn our framework, agents receive only the previous round's debate history $Z^{(t-1)}$, not the entire history, to inform their current response $z_i^{(t)}$, as outlined in Algorithm 1. This limited access ensures that, given the latent concept $θ$ and parameters $φ_i$, responses remain independent of distant history, focusing on immediate refinements. The assumption holds because $θ$ encapsulates semantic interpretations, rendering prior inputs redundant once inferred (Lemma 3.1). Empirically, this is validated by convergence to bimodal distributions (Fig. 2), where agents refine beliefs without full history dependence.\n\n---\n\n**Weakness 2: L179, Initial Response**\n\nIf no response is generated via the correct concept initially, it becomes highly unlikely to converge to the correct answer from a **probabilistic perspective**. This is because, without a correct starting point, the iterative refinement process is unlikely to guide the agents toward the correct concept, especially if the agents are not initially aligned with the true concept. \n\n---\n\n**Weakness 3: L159，Consistency**\n\nIn our framework, the term \"consistency\" refers to how well a response aligns with the correct concept, as defined by the task. Specifically, the likelihood is the probability that a given response reflects the true concept $\\theta^*$ given the task $x$ and the agent's belief. In simpler terms, we are measuring how well a response matches the correct answer based on the agent's internal reasoning. \nWe will revise the explanation in our final version.\n\n---\n\n**Question 1: Difference between our Framework and Related Works**\n\nCompared to MADISSE [1], the debate structure is similar, but our main contributions are the theoretical analysis, Beta-Binomial mixture model fitting, and adaptive stopping mechanism. We do not claim novelty in the core debate framework, which has been explored at least since *Improving Factuality and Reasoning in Language Models through Multiagent Debate*[2].\n\nRelated works can be broadly classified by querying type: sequential (iterative, dependent interactions where agents refine responses based on prior outputs) or parallel (simultaneous, independent generations followed by aggregation). The MAD[6] framework represents sequential querying, using a debate setup similar to real-life debates with multiple debaters and a judge, similar to *Debate Helps Supervise Unreliable Experts*[7]. Other related works are extensions or variants of the parallel debate framework formulated in *Improving Factuality and Reasoning in Language Models through Multiagent Debate*[2], including CIPHER[3] (novel communication protocol instead of using human words), CAMEL[4] (similarly LLM-Harmony[5], using tuned prompted to serve as a role-playing framework), etc.\n\n---\n\n**Question 2: Have the LLM achieve the maximum token limitation in the experiments?**\n\nNo, in our experiments, we carefully managed the input sizes and configured the model’s maximum token limit to 16,000 tokens. Neither the queries nor the responses exceeded this limit. We are confident of this because if the token limit is exceeded, the vllm framework used would throw an error.\n\n---\n\n**Question 3: Could we understand the latent concepts as the knowledge base?**\n\nWe apologize if we have misunderstood the term \"knowledge base\" to which you are referring. To address your question regarding whether latent concepts in our framework can be understood as a knowledge base, we provide the following clarification.\n\nIf you are referring to a **knowledge base** in the context of knowledge graphs or structured databases (e.g., a repository of explicit facts, entities, and relations), then no, latent concepts are not equivalent. In our framework (Sec. 3.2), latent concepts $\\theta \\in \\Theta$ represent underlying abstract interpretations or mental models that agents infer to understand and reason about a task. These are dynamic, task-specific abstractions derived from the input $x$ and debate history $Z^t$, evolving through Bayesian updates (Lemma 3.1). They serve as internalized frameworks for response generation, rather than static collections of facts.\n\nHowever, if you mean a **knowledge base** in the sense of prior beliefs or distributions in Bayesian theorem (e.g., the prior $\\mathbb{P}(\\theta \\mid \\phi_i)$in Assumption 4.3), then yes, there is a conceptual alignment. Latent concepts incorporate agents' prior knowledge and are updated probabilistically based on observed evidence, facilitating belief refinement during debate.\n\nWe will revise Sec. 3.2 in the camera-ready version to more explicitly distinguish these interpretations for clarity.\n\n---\n\n**Question 4: How can we know a true concept?**\n\nIn the context of our framework, the **true concept** refers to the underlying, correct interpretation or solution to a given task. However, it's important to note that we cannot directly observe the true concept during the debate process; instead, we approximate it through probabilistic reasoning.\n\n---\n\n**Question 5: How to understand responsibilities in L213? Are they the confidence score?**\n\nTo clarify:\n\nResponsibilities: In the Expectation-Maximization (EM) algorithm used in our framework, responsibilities are probabilistic weights that determine how much each agent's response contributes to the final estimate of the latent concept. These responsibilities are computed during the E-step of the EM process, where the algorithm calculates the likelihood of each agent’s response under different possible concepts (i.e., how likely a response is to belong to the true concept or another concept).\n\nConfidence Scores: While responsibilities can be thought of as a type of \"weight,\" they are not directly the confidence score of an agent's response. Confidence scores typically refer to how sure an agent is about a particular response or prediction. In contrast, responsibilities reflect the probability that a given response comes from the correct latent concept, which is a part of the belief update process in our model.\n\n---\n**Question 6: The performance does not significantly improve in Table 1. How could we know whether the proposed model performs better than SOM?**\n\nWe recognise that SoM does sometimes score higher accuracy than the proposed model. And we argue that the framework excels on complex, nuanced tasks with high initial judgment variance, where iterative refinement corrects biases via Bayesian updates (Theorem 4.1), yielding meaningful improvements. On simpler tasks with high consensus, SoM suffices as refinement adds limited value. We do not claim universal superiority but targeted applicability: debate is preferable when accuracy justifies costs in high-stakes scenarios. We will revise Sec. 6 to include this in the camera-ready version.\n\n---\n**Suggestion**\n\nThank you for the suggestion. We will revise the figures accordingly in the final version.\n\n---\n**Reference**\n\n[1]Faithful, Unfaithful or Ambiguous? Multi-Agent Debate with Initial Stance for Summary Evaluation (Koupaee et al., NAACL 2025)\n\n[2]Improving Factuality and Reasoning in Language Models through Multiagent Debate, Du et al., ICML'24\n\n[3]Chau Pham, Boyi Liu, Yingxiang Yang, Zhengyu Chen, Tianyi Liu, Jianbo Yuan, Bryan A. Plummer, Zhaoran Wang, and Hongxia Yang. Let models speak ciphers: Multiagent debate through embeddings. In The Twelfth International Conference on Learning Representations, 2024. \n\n[4]CAMEL: Communicative Agents for “Mind” Exploration of Large Language Model Society, NeurIPS 2023\n\n[5]Sumedh Rasal. Llm harmony: Multi-agent communication for problem solving, arxiv 2024.\n\n[6]Encouraging divergent thinking in large language models through multi-agent\ndebate, EMNLP 2024\n\n[7]Debate Helps Supervise Unreliable Experts, arxiv 2023"}, {"review_id": "hkCL8zo3Fs", "reviewer": "Reviewer_9DD2", "summary": "The authors present a framework using collections of LLMs in an interactive turn based debate procedure to address problems of accuracy in evaluation tasks. The procedure is claimed to be better than majority voting schemes and is supported by a strong theoretical model with proofs. While this procedure comes at high cost, the authors have proposed a stability detection framework to recognize when continued processing is unnecessary. Experiments using four different models show the debate algorithm performing better than majority voting on a majority of tasks for two different benchmarks, and they conclude with an analysis of their early stopping criteria.", "questions": "I honestly struggled with some of the theoretical parts, and I apologize if my own unfamiliarity gets some of this wrong, but...\n\nL120 - the conditional independence with respect to both Z^(t) and x - this is seems like a very strong assumption and I don't think there's enough support here to justify why this is okay to do, and where it might night be true. Especially marginalizing with respect to x...\n\nL140 - the theta-star \"true\" concept, while it's clear that there is a probability maximizing concept, calling it true seems to demand a bit of luck. My confusion is compounded by the statement in L143 about \"simplifying the space of possible misinterpretations\" - which sounds almost like word salad to me. Can you clarify what you are saying here?\n\nL152 - assuming any independence of agents seems almost certainly to be untrue, as all LLMs invariably are trained upon the same crawled web.\n\nWhile these assumptions enable the proofs, and the establishment of an E-M procedure - it's not clear to me that convergence won't be to some local maximum, so I'm not certain how valuable the theoretical guarantees are, practically speaking.", "limitations": "Yes. The authors are talking about an incremental improvement over existing uses of LLMs in evaluation tasks, and the risks are generally the same as the prior published works.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n+ Paper is well written, and presents a straightforward algorithm, although one that comes at considerable cost.\n+ The early stopping work alone probably suffices to make this worth publication, and this may be useful in other contexts.\n+ Complete details, likely allowing others to reproduce this work.\n\nWeaknesses:\n+ Comparing against just simple majority voting seems like a bit of a straw man, if querying multiple models in parallel or sequentially there's a lot that can be done to increase the yield of a single round, versus the multiple rounds which is extremely expensive.\n+ This paper is quite theoretical yet also uses benchmark results for its claims - this is okay but I'd like the the claims to be clearer between theory and practical.", "was_revised": true, "final_justification": "The authors made a substantial effort to respond to the reviewers requests. I think I'm one of the stronger supporters here and satisfied with the paper in its present form.", "rebuttal": "We are pleased that you found the paper well-written and that the early stopping framework was highlighted as a significant contribution. We understand that there are concerns regarding the comparison with majority voting and the clarity of some of the theoretical aspects. We aim to clarify these points below.\n\n---\n\n**Weakness 1**: No baseline comparisons with other multi-agent debate frameworks from the past.\n\n(1) Positioning of our baselines.\nWe treat SoM (simple majority voting) [3] as the baseline for multi-LLM aggregation. Many “debate” style systems either (a) modify majority voting (e.g., RECONCILE’s confidence-weighted voting) [4] or (b) adopt a different interaction protocol (e.g., MAD: two opposing sides + a judge) [2]. These variants lie outside the scope of our framework, which specifically models an iterative, belief-refinement process. Thus, we did not initially position them as direct comparators.\n\n(2) We agree more empirical baselines are needed.\nWe fully agree with the reviewer’s suggestion that more empirical baselines are needed. \nDue to time and computation constraint, we conduct experiment using Gemini-2.0-Flash on MAD framework[2], a representative sequential querying approach that structures debates adversarially with multiple debaters presenting arguments for and against a position, moderated by a judge to reach a final decision. The results are somewhat puzzling, as MAD does not even exceed the single-model baseline in accuracy. Our suspicion is that MAD's balanced exposure to both sides gives the incorrect side an equal chance to persuade the judge, skewing outcomes in judgment tasks where nuanced refinement is key.\n\n\n| Dataset           | Single         | SoM            | Debate         | MAD (%)      |\n|-------------------|----------------|----------------|----------------|--------------|\n| **JudgeBench**    | 63.66±3.03     | 66.13±3.72     | **68.06±3.66** | 60.65±1.96   |\n| **LLMBar**        | 76.68±1.97     | 77.75±2.35     | **81.83±2.18** | 73.92±1.27   |\n| **TruthfulQA**    | 69.49±2.71     | 72.01±3.10     | **74.30±2.99** | 70.87±1.59   |\n| **MLLM-Judge**    | 67.50±2.88     | 68.00±3.23     | **69.25±3.19** | 64.20±1.52   |\n| **JudgeAnything** | 81.63±5.70     | 83.46±6.30     | **85.71±5.95** | 71.67±3.36   |\n\n\n---\n\n**Weakness 2**: Distinction between theory and practical results.\n\nOur paper integrates a theoretical framework (Sec. 4) with empirical validation (Sec. 6) to demonstrate the efficacy of the multi-agent debate framework with adaptive stability detection. The theoretical contributions, grounded in latent concept theory and Bayesian inference (Sec. 3, Theorems 4.1-4.2), establish that iterative debate amplifies correct responses (Theorem 4.1) and outperforms majority voting (Theorem 4.2) under Assumptions 4.1-4.5. These guarantees predict that collaborative reasoning corrects biases and improves accuracy, assuming a true concept $ \\theta^* $ drives responses.\n\nPractically, these predictions are validated across six benchmarks (Table 1), showing consistent accuracy gains over majority voting (e.g., +4.08% on LLMBar for Gemini-2.0-Flash, +2.29% on TruthfulQA). The adaptive stopping mechanism (Sec. 5), based on a Beta-Binomial mixture model, ensures computational efficiency by halting debates in 2-7 rounds with minimal accuracy loss (<0.7%, Table 2). For instance, Theorem 4.1’s prediction of response amplification is reflected in the convergence of judgment distributions to bimodal patterns (Fig. 2), where agents align on the correct answer or fail collectively. The KS statistic’s rapid decline (Fig. 3, e.g., <0.05 by Round 3 on BIG-Bench) confirms the theoretical expectation of stable consensus (Sec. 5.3).\n\n---\n\n**Question 1**: Conditional independence (L120)\n\nWe thank Reviewer 9DD2 for their insightful critique of Assumption 3.1, which posits that an agent’s response $z_i^{(t+1)}$ is conditionally independent of the task $x$ and prior responses $Z^{(t)}$, given the latent concept $\\theta$ and model parameters $\\phi_i$, i.e., $\\mathbb{P}(z_i^{(t+1)} \\mid \\theta, x, Z^{(t)}, \\phi_i) = \\mathbb{P}(z_i^{(t+1)} \\mid \\theta, \\phi_i)$. This assumption, inspired by Estornell and Liu (2024) [1], asserts that $\\theta$, representing the semantic interpretation of the task (e.g., “Max Verstappen won the 2021 F1 Championship”), and $\\phi_i$, the model’s parameters, jointly determine the response. In encoder-decoder architectures, $\\theta$ and $\\phi_i$ correspond to the encoder’s embedding, which captures the task’s core meaning, rendering $x$ and $Z^{(t)}$ redundant for generation.\n\nThis assumption is justified because $\\theta$ acts as a sufficient statistic, encapsulating the relevant information from $x$ and $Z^{(t)}$. Marginalizing over $x$ is valid when $\\theta$ fully represents the task’s semantic intent, as validated empirically by diverse initial judgments converging to bimodal distributions (Fig. 2, Appendix B.3). The use of diverse models (Gemini, Llama, Qwen, Gemma) with distinct architectures further mitigates potential correlations, supporting the assumption’s practical applicability.\n\nHowever, the assumption may not hold if $x$ contains nuanced contextual details not captured by $\\theta$, such as ambiguous phrasing. We acknowledge that our original explanation was insufficient and will revise Sec. 3.3 in the camera-ready version, ensuring clarity on when the assumption may weaken.\n\n---\n\n**Question 1**: L140\n\n1. **\"True Concept\"**: The term $θ∗$ refers to the concept that maximizes the probability of the correct answer. While calling it \"true\" may seem to imply perfection, it simply means that this concept is ideal for generating the correct answer in theory. It’s not about \"luck,\" but about defining the optimal concept for the task.\n2. **\"Simplifying Misinterpretations\"**: This refers to reducing the complexity of handling multiple potential misinterpretations by assuming a single, ideal latent concept. It helps focus the reasoning process on refining beliefs toward the correct answer, rather than dealing with many possible incorrect interpretations.\n\nWe will revise the paper to clarify these points further. Let us know if this clears up the confusion!\n\n---\n\n**Question 3**: Agent Independence (L152)\n\nWe would like to clarify that similarity in outputs does not imply probabilistic dependency.\n\n- **Stochasticity**: LLMs generate outputs stochastically, meaning even models trained on similar data can produce diverse responses, ensuring **independence** in predictions.\n- **Ensemble Methods**: In ensemble learning, models trained on similar data can still operate independently, as their predictions are generated separately without influencing each other.\n\n---\n\n**Question 4**: EM algorithm leading to local optimum\n\nWhile it’s true that the **E-M algorithm** can converge to a **local optimum**, we want to emphasize that, even in such cases, the solution closely tracks the **real distribution** of judgments. In practical applications, local optima in this context often still lead to highly accurate results.\n\nOur multi-agent debate framework is designed to **refine beliefs iteratively**, and the agents engage in collaborative reasoning to progressively converge to a solution that **approximates the correct distribution** of responses. The **adaptive stability detection mechanism** ensures that the process stops when the distribution stabilizes, preventing unnecessary iterations and ensuring computational efficiency.\n\nAdditionally, the **empirical results** clearly show that the framework outperforms simpler aggregation methods like **majority voting**, indicating that even if the algorithm converges to a local optimum, it provides substantial improvements over static methods. This suggests that the practical benefits of the framework are not undermined by the theoretical risk of local maxima.\n\nWe hope this explanation clarifies how the approach still yields valuable practical outcomes despite the theoretical considerations. Please let us know if you need further details or clarification.\n\n---\n\n**Reference**\n\n[1] Andrew Estornell and Yang Liu. Multi-LLM Debate: Framework, Principles, and Interventions. In A. Globerson et al., editors, Advances in Neural Information Processing Systems, volume 37, pages 28938–28964. Curran Associates, Inc., 2024.\n\n[2]Encouraging divergent thinking in large language models through multi-agent\ndebate, EMNLP 2024\n\n[3] Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.\n\n[4] Justin Chen, Swarnadeep Saha, and Mohit Bansal. 2024. ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7066–7085, Bangkok, Thailand. Association for Computational Linguistics."}]

Author rebuttal:

Meta-review: This paper proposes a multi-agent debate framework in which multiple LLMs iteratively refine their judgments through structured discussion, addressing the limitations of static ensemble methods like majority voting. The framework is mathematically formalized using latent concepts, with proofs showing theoretical advantages over static aggregation. To mitigate the computational cost of prolonged debates, the authors introduce an adaptive stability detection mechanism based on a time-varying Beta-Binomial mixture model with Kolmogorov–Smirnov testing, enabling early stopping once consensus emerges. Experiments across diverse benchmarks—including BIG-Bench, TruthfulQA, LLMBar, and six datasets spanning both language and vision tasks—demonstrate that the debate framework consistently outperforms majority voting in accuracy and robustness while maintaining computational efficiency across various model backbones.

The reviewers agreed that the proposed algorithm is novel and enjoys theoretical rigor. The proposal also addresses a practical question in building a multi-turn multi-agent debate system. The paper presents its details clearly and is easy for the audience to follow and reproduce. 

Though there were a couple of remaining concerns on the imposed assumptions and a lack of complete empirical comparisons and analysis, I believe this work contributes to the understanding of the working mechanisms of a mult-agent debate system.
Decision: Accept (poster)