Title: Weighted Iterative Society-of-Experts for Robust Multi-Agent Debate
Primary area: foundation or frontier models, including LLMs
Keywords: multi-agent debate; mathematical reasoning;

Abstract: Recent large language models (LLMs) are trained on diverse corpora and tasks, leading them to develop complementary strengths. Multi-agent debate (MAD) has emerged as a popular way to leverage these strengths for robust reasoning, though it has mostly been applied to language-only tasks, leaving its efficacy on multimodal problems underexplored. In this paper, we study MAD for vision-and-language mathematical reasoning. Our setup enables generalizing the debate protocol with heterogeneous experts that possess single- and multi-modal capabilities. To this end, we present Weighted Iterative Society-of-Experts (WISE), a generalized MAD framework that partitions the agents into Solvers, that generate solutions, and Reflectors, that verify correctness, assign weights, and provide natural language feedback. To aggregate the agents’ solutions across debate rounds, while accounting for variance in their responses and the feedback weights, we present a modified Dawid–Skene algorithm for post-processing that integrates our two-stage debate model. We evaluate WISE on SMART-840, VisualPuzzles, EvoChart-QA, and a new SMART-840++ dataset with programmatically generated problem instances of controlled difficulty. Results show that WISE consistently improves accuracy by 2–7% over state-of-the-art MAD setups and aggregation methods across diverse tasks and LLM configurations.

TL;DR: A novel way to conduct multi-agent debate using ranking