Title: Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?
Venue: NeurIPS 2025
Accepted: yes
Keywords: multi-agent debate

majority voting

LLM

Agentic AI
Abstract: Multi-Agent Debate (MAD) has emerged as a promising paradigm for improving the performance of large language models through collaborative reasoning. Despite recent advances, the key factors driving MAD’s effectiveness remain unclear. In this work, we disentangle MAD into two key components–Majority Voting and inter-agent Debate–and assess their respective contributions. Through extensive experiments across seven NLP benchmarks, we find that Majority Voting alone accounts for most of the performance gains typically attributed to MAD. To explain this, we propose a theoretical framework that models debate as a stochastic process. We prove that it induces a martingale over agents’ belief trajectories, implying that debate alone does not improve expected correctness. Guided by these insights, we demonstrate that targeted interventions, by biasing the belief update toward correction, can meaningfully enhance debate effectiveness. Overall, our findings suggest that while MAD has potential, simple ensembling methods remain strong and more reliable alternatives in many practical settings. Code is released in https://github.com/deeplearning-wisc/debate-or-vote.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the core components of Multi-Agent Debate (MAD), arguing that its performance gains stem primarily from the aggregation of multiple agent outputs (i.e., Majority Voting) rather than the iterative debate process. While prior work has already raised empirical questions about the universal effectiveness of MAD compared to other ensembling methods [2] and has highlighted the power of voting protocols over extended debate rounds [3], this paper's main contribution is the introduction of a formal theoretical framework to explain why this is the case. The authors model the debate as a martingale process over agent beliefs, demonstrating that debate alone does not systematically improve expected correctness. \n\nThe modeling assumes that each agent follows a Dirichlet-Compound-Multinomial (DCM) process. The belief update rule is based on counting responses from neighboring agents. However, using simple count vectors for belief updates can make the complex interactions in LLM-based debates, and this modeling choice receives no further empirical validation. Only two observations inform the introduction of the assumption: the stochasticity of LLM outputs and the interaction between agents. The only supporting evidence is the stability of average accuracy over rounds, which is a minimal sign of the martingale behavior claimed.\n\nThe authors propose two MAD variants that bias belief toward correct responses. They observe locking correct answers, which breaks the martingale property and improves performance. Since this process needs access to ground truth, they propose heuristic methods using majority-voted responses. However, these methods often perform similarly to or worse than majority voting, limiting their practical value.\n\nOverall, the paper presents a clean theoretical framework, but the modeling assumptions are strong and under-justified, and the proposed improvements offer limited advantage in practice.\n\n[1] Tongxuan Liu, et al. \"GroupDebate: Enhancing the Efficiency of Multi-Agent Debate Using Group Discussion.\" arXiv preprint arXiv:2409.14051 (2024)\\\n[2] Andries Smit, et al. \"Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs.\" Proceedings of the 41st International Conference on Machine Learning (2024)\\\n[3] Lars Benedikt Kaesberg, et al. \"Voting or Consensus? Decision-Making in Multi-Agent Debate.\" arXiv preprint arXiv:2502.19130 (2025)\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n**Novel Theoretical Framework**: The primary strength of this paper is its novel theoretical framework. By modeling the multi-agent debate as a DCM process and martingale argument it provides a principled and rigorous explanation for why majority voting explains the success of MAD. \n\n**Clarity and Organization**: The paper is exceptionally well-written and organized. The theoretical concepts, including the Dirichlet-Compound-Multinomial (DCM) model and the martingale proof, are presented with clarity, making the core arguments accessible.\n\n### **Weaknesses**\n\n**Limited Empirical Novelty**: The paper's central empirical conclusion, that majority voting is often as good as or better than complex debate while important, is not entirely novel. Recent studies have already shown that MAD is highly sensitive to hyperparameters and does not reliably outperform other prompting strategies [2]. Furthermore, the effectiveness of voting protocols and the potential negative effects of increasing debate rounds have been systematically demonstrated [3]. Thus, the paper's experiments seem as a confirmation of these existing findings rather than a new discovery.\n\n**Strong and Underjustified Modeling Assumptions**: The theoretical framework, while elegant, rests on the strong assumption that the complex, semantic-rich interactions of LLM agents can be simplified to a Dirichlet-Compound-Multinomial (DCM) process updated by simple response counts. However, even standard prompt tuning stragies like reformulating question or refining response can adjust DCM $\\alpha$ and marginal probability. The paper provides limited empirical validation that this model accurately captures the dynamics of real-world LLM debates, beyond showing that the average accuracy remains stable over rounds.\n\n**Limited Practical Advantage of Proposed Methods**: The proposed interventions (e.g., MAD-Conformist), designed to break the martingale property, do not consistently or significantly outperform the simple majority voting baseline. This limits the practical impact of the theoretical insights, as the simpler, more efficient method remains the more compelling choice in most scenarios, a point also relevant in the context of improving MAD's efficiency [1].\n\nQUESTIONS:\n**1**. The modeling assumes that belief updates can be represented by simple response counts, using the Dirichlet-Compound-Multinomial process. Can you provide empirical evidence or ablation results showing that this update rule reflects how real LLM responses evolve during debate?\n\n**2**. The only support for the claimed martingale behavior is the stable mean accuracy over rounds. Did you check other signals such as $p_t$ directly or further support the theoretical claim?\n\n**3**. The proposed methods that use majority-voted results often perform similarly to or worse than plain majority voting. How do you interpret this outcome, and under what conditions would your method provide clear advantages?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper disentangles multi-agent debate (MAD) into two key components – Majority Voting and inter-agent Debate to assess the contribution arising from each. It demonstrates that the vast majority of the performance comes from Majority Votes, rather than from any agent debate. The paper goes on to introduce a mathematical framework and two theorems that help explain their empirical results, and highlight why in these homogeneous-agent Q&A tasks, inter-agent debate will not improve the expected correctness, i.e. Debate induces a martingale over agents’ belief trajectories, which in expectation remains unchanged across rounds.\n\nSTRENGTHS AND WEAKNESSES:\nOn Quality: \n\nThe paper presents strong motivation, and is well written. Clear and concise background is provided on the Majority Voting and Multi-Agent Debate, citing relevant works. It was a pleasure to read. \n\nThe paper does well to couple an empirical study (with 7 datasets and multiple strong LLM agents), with a theoretical formulation that explains the empirical findings. This is a particularly strong part of the paper, as to the best of my knowledge, many papers in this field do not formally define MAD with as much care as taken here. For example, careful attention is given to define MAD strategies as undirected graphs, where edges indicate which agents can observe one another. \n\nThe empirical evaluation is also conducted with care, and although the code is not released, it feels highly reproducible given the level of detail provided.\n\nThe Extended Experiments section preempted multiple of my questions – although a more thorough experiment into Open-Ended tasks would be welcomed (see below questions). \nHow would one consider extending the Bayesian update rule to account for the additional context an agent could provide to another, going beyond a simple count vector $c_{i,t}$ which would be required for open-ended tasks? \n\nOn Impact: \n\nI believe that the formulation introduced in the paper could have a moderate-to-high impact on the MAD community – a field which has suffered from imprecise and ad-hoc approaches in the past. \n\nHowever, I believe the paper can be more forthright with the assumptions made in the Theoretical Analysis section – which lead to the main conclusions of the paper. \nFor example, Line 172: “We assume that the correct answer has the largest belief $\\alpha^{(1)}$”. \nThis feels like an overly strong assumption, and one that removes much of the \nsurprise of the findings.\nDoes this imply that the probability of the correct response is lower-bounded by Theorem 1 only in cases when the homogenous agent correctly assigns the largest $\\alpha^{(1)}$ belief to that answer in round 0? If this is the case, are we surprised that majority voting provides the largest contribution to the aggregated response - given that the homogenous agent already “knows” the answer?  \n \nSurely the more interesting setting is the one where the underlying agent (or subset of agents) does not assign the largest belief to the correct response in the initial round – and how their belief can be updated over multiple rounds of debate? The theorem does not cover this more challenging case, and therefore one should be careful with the claims made, Line: 60: “we prove formally that majority vote does essentially all the work, which explains our empirical findings.” \n\nSimilarly, when we consider the agent’s Dirichlet parameter update, (Line 190) we are only considering the counts associated with the $K$ possible responses – this seems reductive and overly restrictive. For example, one would want an agent to be persuaded by the additional context and explanation another agent provides, biasing its belief away from its largest $\\alpha$ value towards the correct response. Do you consider this setting?  \n\nFor these reasons, it’s not clear to me that the theoretical results presented will hold when more complex and open-ended tasks are performed. How would a consensus even be defined in long form NPL tasks? \n\nOn Clarity: \n\nHeterogeneous Agents: The manuscript would benefit from more context provided in this section. For example, what are “optimal persona sets” and how is this implemented in practice? \nIt would also be interesting to expand the theoretical analysis to this setting, but appreciate that might be future work.\n\nQUESTIONS:\nHow do the empirical and theoretical findings presented in the paper align with paragraph “Pros of Multi-Agent Debate”? \n\nFor example, the conclusions of the theoretical analysis is that MAD induces “a martingale process, which preserves the expected success probability of each agent over time.” However, also stated on page 9: MAD “has the potential to enhance both factual accuracy and reasoning quality”. Please can you clarify how both these passages can hold?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the effectiveness of Multi-Agent Debate in improving the performance of large language models. It contrasts MAD with simple Majority Voting through extensive experiments across seven NLP benchmarks. The key findings are that Majority Voting accounts for most of the performance gains typically attributed to MAD, and that debate alone does not systematically improve expected correctness. The paper also presents a theoretical framework modeling debate as a stochastic process and proves that it induces a martingale over agents’ belief trajectories. Based on these insights, the authors propose interventions to enhance debate effectiveness by biasing belief updates toward correct signals.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. This paper puts forward an interesting and significant argument that Majority Voting is responsible for most of the performance gains usually associated with MAD, and it substantiates this argument through both experimental and theoretical analysis.\n\n2. The paper compares Majority Voting and various MAD techniques across over seven NLP datasets, and the experimental results largely align with the theoretical proof.\n\n3. The conclusions drawn in this paper hold certain implications for enhancing the design of MAD architectures.\n\nWeaknesses:\n\n1. The paper mainly conducts comparative experiments on the Qwen2.5-7B-Instruct and Llama3.1-8B-Instruct base models. The experimental results on Llama3.1-8B-Instruct show a certain degree of instability, and experiments on larger-scale bases are only performed on some benchmarks, making it impossible to fully understand the model's patterns. Additionally, the authors do not test on other closed-source LLMs.\n\n2. The conclusions of the paper mainly target closed-ended question answering tasks, and the applicability of Majority Voting techniques to open-ended text generation tasks is limited. Whether there is a Majority Voting architecture suitable for text generation tasks remains a topic of discussion.\n\nQUESTIONS:\n1. Why is the Dirichlet-Compound-Multinomial distribution chosen for modeling LLM response generation? Are there related papers or statistical data supporting this choice?\n\n2. What is the base model used for Figure 3?\n\n3. Can an average result column be added to Table 1 and Table 2?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors breakdown MAD into two components, majority voting and inter-agent debate. Through extensive experiments across 7 NLP benchmarks, they show that majority voting is responsible for most, if not all ,of the improvement seen in MAD techniques vs single agent numbers. \n\nTo explain this, they develop a theoretical framework to model the debates as a stochastic, peer-influenced process. They prove that an unguided debate process induces a martingale over the agents' belief trajectories, i.e. expected correctness does not improve withe debate alone. Empirical results in the paper, support this as well. \n\nBased on this, the paper proposes MAD-Conformist and MAD-Follower which use majority votes to guide the belief update process to bias the belief update process towards correctness. Both methods show a strong improvement over standard MAD techniques across a variety of tasks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n - This paper does a great job of breaking down why MAD does and doesn't work as expected. It provides strong empirical proof for majority voting being the largest contributor to any MAD improvements and develops a sound mathematical explanation for why that is the case. Modeling the agents as DCMs and the debate process as a martingale provides helps understand how MAD works. \n - The paper's claims are substantiated by a comprehensive set of experiments. It multiple models of different sizes and model families, diverse benchmarks (from reasoning to QA), and various MAD configurations (decentralized, sparse, etc.), which strongly supports the generalizability of the findings.\n - The paper includes simple but impactful improvements to MAD, incorporating majority voting to bias the agents' belief update process towards correctness. \n - This paper shows that sometimes simpler is better, and serves a reminder of the crucial role a good baseline plays in contextualizing improvements. \n\n\n### Weaknesses\n - The theoretical framework and most empirical results are based on homogenous agents. While this is useful for simplicity, it somewhat defeats the purpose of a debate. Since all agents share the same beliefs, there is less variety in the responses generated by them. [Estornell et. al.](https://proceedings.neurips.cc/paper_files/paper/2024/file/32e07a110c6c6acf1afbf2bf82b614ad-Paper-Conference.pdf) show that a lack of diversity in responses or model capabilities is detrimental to the debate process.\n- The lack of diversity is somewhat addressed through the heterogenous agents experiment in section 6 but the models are all still similarly capable. It would also be helpful to include a t-SNE visualization of the response embeddings to confirm that the diverse personas result in truly diverse responses. \n - The experimental setup is based on a relatively simple MAD design (simultaneous-talk, simple debate). The paper doesn't evaluate against more sophisticated methods involving structured debates, where the results might not generalize. \n - Limited novelty for MAD-Conformist and MAD-Follower: [Smit et. al.](https://arxiv.org/pdf/2311.17371) propose an agreement modulation mechanism to precisely control what proportion of responses agree with each other and show that that high inter-agent agreement leads to improved correctness.  The paper would be more impactful if it contextualized its proposed methods against similar related work.\n\nQUESTIONS:\nIn a majority of the results, it seems that correctness scores decrease with more debate rounds. Do the authors have an insight into why that is the case? Is it due to the degradation of initial strong majorities or a few agents changing their response causing weaker majorities to fade? Is it an overall breakdown of agreement, i.e. as the debate progresses, the models disagree with each other more rather than agree?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}