RECONCILE: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs (2024)

Abstract: Large Language Models (LLMs) still struggle with natural language reasoning tasks. Motivated by the society of minds (Minsky, 1988), we propose RECONCILE, a multi-model multiagent framework designed as a round table conference among diverse LLM agents. RECONCILE enhances collaborative reasoning between LLM agents via multiple rounds of discussion, learning to convince other agents to improve their answers, and employing a confidenceweighted voting mechanism that leads to a better consensus. In each round, RECONCILE initiates discussion between agents via a ‘discussion prompt’ that consists of (a) grouped answers and explanations generated by each agent in the previous round, (b) their confidence scores, and (c) demonstrations of answerrectifying human explanations, used for convincing other agents. Experiments on seven benchmarks demonstrate that RECONCILE significantly improves LLMs’ reasoning – both individually and as a team – surpassing prior single-agent and multi-agent baselines by up to 11.4% and even outperforming GPT-4 on three datasets. RECONCILE also flexibly incorporates different combinations of agents, including API-based, open-source, and domainspecific models, leading to an 8% improvement on MATH. Finally, we analyze the individual components of RECONCILE, demonstrating that the diversity originating from different models is critical to its superior performance.1

heading: 1 Introduction
text: A large body of recent work has focused on improving the reasoning capabilities of Large Language Models (LLMs) by imitating various human cognitive processes (Wang and Zhao, 2023; Park et al., 2023; Sumers et al., 2023; Ye et al., 2023). These include phenomena like reflecting on and critiquing one’s own predictions, being receptive to feedback, and learning from feedback. Of note,
1Code: https://github.com/dinobby/ReConcile
self-reflection is an introspective process that allows the model to improve its outputs by generating feedback from the model itself (Madaan et al., 2023; Shinn et al., 2023). However, self-reflection suffers from Degeneration-of-Thought – when the model is overly confident in its answer, it is unable to generate novel thoughts even after multiple rounds of feedback (Liang et al., 2023).
To promote more diverse thoughts, past work has drawn inspiration from the concept of society of minds in multi-agent systems (Minsky, 1988; Zhuge et al., 2023). It highlights the importance of communication and collaboration between multiple agents for complex decision-making tasks. While such collaborative frameworks like multi-agent debate (Liang et al., 2023; Du et al., 2023) increase the reasoning diversity through the process of a debate, multiple agents have typically been limited to different instances of the same underlying model like ChatGPT (OpenAI, 2022).2 This results in an inherent model bias, a restricted knowledge scope, and a lack of external feedback from other models due to identical pre-training data and model architectures across all agents. In general, when multiple agents propose solutions to a problem, the success of such a multi-agent system is fundamentally reliant on (a) the diversity of the solutions, (b) the ability to estimate each agent’s confidence, and (c) accordingly, convince other agents (with explanations) to reach a better consensus. This puts forward the question: if multiple diverse LLMs collaboratively solve a task, are they capable of discussing their solutions with each other to reach a better consensus?
We aim to solve reasoning problems by learning from diverse insights and external feedback, originating from agents that belong to different model
2In this work, we refer to multi-agent as multiple instances of the same underlying model (e.g., ChatGPT), whereas multimodel model-agent refers to different models (e.g., ChatGPT, Bard and Claude2) as agents.
7066
Q X Q + Exp V
yes
( , )
( , )
yes no
0.8 0.2
𝑓(𝑝)
families. Collaborative processes such as brainstorming, group meetings, and discussions play a pivotal role in reaching a consensus and arriving at more refined solutions to complex problems (Li et al., 2022b). Effective discussion also entails the selection of stances, voting, convincing, exchange of information, and a diversity of opinions. Thus, we propose RECONCILE, a framework of round-table conference for obtaining better consensus among diverse LLM agents. RECONCILE consists of multiple discussion rounds between diverse LLM agents who try to convince3 each other to either rectify their answers or become more confident of their initial correct answers (see Fig. 1 for a broad overview).
Given a reasoning problem, RECONCILE begins with each agent first generating an answer, its uncertainty, and a corresponding explanation (as a Chainof-Thought (Wei et al., 2022)) for the answer. Then all agents enter a multi-round discussion phase. Each discussion round consists of all agents generating a revised explanation and answer based on all other agents’ explanations and answers from the previous round. In particular, RECONCILE initiates a discussion by designing a discussion prompt for each agent, that lets it condition on (1) grouped answers from all agents, (2) corresponding explanations generated in the previous round, and (3) demonstrations of answer-rectifying human expla-
3When we say that an agent tries to convince another agent, we mean that it learns (based on corrective explanations) to defend or argue for its stance while still being receptive to the other agent’s argument.
nations for convincing other agents. We leverage them in an in-context learning framework to teach models to generate their own convincing explanations (see Fig. 3). Even in cases where an agent initially offers an incorrect answer and explanation, it can consider another agent’s convincing explanation and amend its response accordingly. In each discussion round, we estimate an agent’s uncertainty via a confidence-estimation prompt (Tian et al., 2023; Xiong et al., 2023a). Once all agents converge to the same answer (i.e., a consensus has been reached), we employ these confidences to compute a weighted vote as the team answer.
We primarily develop RECONCILE with three state-of-the-art LLMs: ChatGPT (OpenAI, 2022), Bard (Anil et al., 2023), and Claude2 (Anthropic, 2023). We also demonstrate the flexibility of RECONCILE with variants that employ a much stronger GPT-4 (OpenAI, 2023), an open-source LLaMA-2-70B (Touvron et al., 2023), or a domainspecific DeepSeekMATH (Shao et al., 2024) model as an agent. Across seven benchmarks spanning commonsense reasoning, mathematical reasoning, logical reasoning, and Natural Language Inference (NLI), RECONCILE outperforms prior singleagent (e.g., Self-Refine (Madaan et al., 2023) and Self-consistency (Wang et al., 2023b)) and multiagent baselines (Debate (Du et al., 2023) and Judge (Liang et al., 2023)) that are built on top of the same underlying models. For example, RECONCILE, (1) on a date understanding task, outperforms the leading multi-agent debate baseline by
Self-Consistency (SC, Wang et al. (2023b)); Debate (Du et al., 2023) and Judge (Liang et al., 2023). means supported and means not supported. RECONCILE supports multi-model multi-agent discussion with confidence estimation and convincingness. * = Du et al. (2023) primarily experiment with multiple instances of ChatGPT as different agents and conduct an initial investigation with 20 samples using ChatGPT and Bard as the two agents.
11.4%, (2) on StrategyQA, also outperforms GPT4 by 3.4%, and (3) on MATH, outperforms both GPT-4 and a specialized DeepSeekMath model by 8%. Moreover, detailed analyses of the individual components of RECONCILE demonstrate that leveraging diverse LLM agents leads to maximum gains, and we further validate their higher response diversity via a BERTScore-based diversity metric (Zhang et al., 2019). Finally, we show that RECONCILE not only leads to better team performance but also enables each agent to improve individually via the discussion process.
In summary, our primary contributions are: • We propose RECONCILE, a reasoning framework
involving diverse Large Language Models in a Round Table Conference. • We conduct extensive experiments on seven benchmarks to show that RECONCILE outperforms strong baselines (including GPT-4 on some benchmarks) and also generalizes to different combinations of agents. • We study the role of diversity, confidence estimation, and an agent’s ability to convince others (by learning from corrective explanations) in multiagent discussion systems.

heading: 2 Related Work
text: Reasoning with LLMs. Progress in LLMs has led to the development of advanced prompting and fine-tuning techniques for solving reasoning problems. Representative methods include Chain-of-Thought (CoT) (Kojima et al., 2022; Wei et al., 2022; Wang et al., 2023a) and Treeof-Thought prompting (Yao et al., 2023a), selfconsistency (Wang et al., 2023b), meta-reasoning over multiple paths (Yoran et al., 2023), use of scratchpads (Nye et al., 2021), training verifiers (Cobbe et al., 2021), self-collaboration (Wang et al., 2023c; Schick et al., 2022; Li et al., 2023a; Feng et al., 2024), self-reflection (Shinn et al., 2023; Madaan et al., 2023; Wang and Zhao, 2023; Yao et al., 2023b), improved math reasoning (Yue et al., 2023; Luo et al., 2023) and fine-tuning via bootstrapping models (Zelikman et al., 2022; Lewkowycz et al., 2022; Li et al., 2023b). Eliciting reasoning from a single agent, while promising, is fundamentally limited by a lack of diverse insights. Reasoning in Multi-Agent Systems. A recent line of work has explored student-teacher frameworks with the goal of distilling reasoning capabilities from a stronger teacher to a weaker student (Magister et al., 2023; Fu et al., 2023; Ho et al., 2023; Saha et al., 2023; Mukherjee et al., 2023). As opposed to a teacher teaching weaker agents, we seek to develop a multi-agent system where different LLM agents have their unique strengths and try to collaboratively improve performance by reaching a better consensus. Notable prior works include multi-agent debating frameworks (Du et al., 2023; Liang et al., 2023; Chan et al., 2023; Xiong et al., 2023a; Khan et al., 2024) but such efforts are still largely limited to multiple instances of the same underlying language model. We argue that relying on a single model limits the potential of complementary benefits from different model families and the advantage of ensemble learning. Moreover, estimating the confidence of each agent and being able to defend or improve one’s opinions become more prominent components in such multi-model multi-agent systems because of the individual differences. Overall, Table 1 summarizes RECONCILE’s key differences compared to prior single-agent and multi-agent reasoning methods. Ensembling Large Pretrained Models. Large pre-trained models, by virtue of being trained on
different data and with architectural variations, exhibit distinct capabilities. This has led to the development of ensembles (Sagi and Rokach, 2018) in multimodal learning (Zeng et al., 2023; Li et al., 2022a). Mixture of Experts, a popular ensemble learning technique, trains multiple smaller specialized models to improve robustness and overall accuracy (Jacobs et al., 1991; Shazeer et al., 2017; Du et al., 2022). Specific to language models, Self-Consistency (Wang et al., 2023b) generates diverse reasoning paths using CoT and chooses the most consistent answer as the final output. Jiang et al. (2023) propose LLM-Blender, a method to rank and fuse generations from different models. Different from these, we study communication via explanations between distinct LLM agents and their ability to discuss and convince each other in order to improve collective reasoning.

heading: 3 Problem Setup
text: We assume that we are given a test problem Q and there are n agents A = {Ai}ni=1 participating in a round table discussion. Each agent is a distinct LLM, potentially trained with different pre-training data and model architectures. All agents are capable of generating an answer and a corresponding Chain-of-Thought explanation (Wei et al., 2022) for the test problem. For each agent Ai, we utilize a small number of k demonstrations
of convincing samples Ci = {c(i)j }kj=1. Each convincing sample c(i)j = (q (i) j , a (i) j , e (i) j ) for an agent Ai is an instance of a question q (i) j , gold answer a (i) j , and a human explanation e (i) j that helps rectify an agent’s initial incorrect answer (see more details in Sec 4). The objective of RECONCILE is to improve the team performance on a given task by holding multiple rounds of discussion between the agents, quantifying the uncertainty associated with each agent, and convincing other agents to reach a better consensus. Note that convincing samples serve as an additional performance enhancer; even when the dataset lacks human explanations, our method can still yield performance gains independent of this (more details below).

heading: 4 RECONCILE: A Collaborative Discussion Framework
text: RECONCILE operates in three phases: initial response generation, multi-round discussion, and team answer generation. The overview of our method is demonstrated in Fig. 2 and Algorithm 1.
Phase 1: Initial Response Generation. RECONCILE operates with each agent Ai initially generating an answer a(0)i , an explanation e (0) i , and an associated confidence p(0)i ∈ [0, 1] for the generated answer. Each agent conditions on a zero-shot
prompt that instructs it to reason about the problem ‘step-by-step’. See ‘Phase 1’ in Fig. 2 and the prompt is shown in Fig. 5 in Appendix A.2.
Phase 2: Multi-round Discussion. RECONCILE then enters a discussion phase, consisting of R rounds (see ‘Phase 2’ in Fig. 2). In discussion round r, for each agent Ai, RECONCILE develops a discussion prompt D(r)i (as shown in Fig. 5), consisting of the following three components. (a) Grouped responses of all agents from the previous round. D(r)i consists of the answers {a(r−1)j }nj=1 and explanations {e (r−1) j }nj=1 of all agents from round (r − 1). To foster better discussions, RECONCILE summarizes this information by grouping the answers into distinct categories and appends all plausible explanations for each answer, as shown in our discussion prompt (Appendix Fig. 5) and on the left side of Fig. 2. (b) Confidence associated with the answers. All agents are not equally confident in their answers. Hence, an effective discussion should also consider each agent’s uncertainty. For all black-box models, we estimate its confidence p(r)i in round r by directly prompting the agent to verbally quantify its uncertainty, which in past work has been shown to be effective (Xiong et al., 2023b). See Appendix Fig. 5 for the usage of confidence in discussion. (c) Convincing samples from all other agents. Finally, the prompt contains convincing samples Cj
for all other agents Aj ̸=i.4 When an agent tries to reassess its reasoning in light of the reasoning provided by other agents, we hypothesize that it should benefit from conditioning on demonstrations that can convince other agents. In order to obtain such convincing samples for an agent Aj , we select a small number of samples (4 in our experiments) for which the agent’s initial answer is wrong but conditioning on the corresponding human explanation, rectifies the answer (see Fig. 3). For datasets that do not come with human explanations (e.g., the date understanding task in our experiments), we develop RECONCILE without using any convincing sample in the discussion prompt and still obtain large improvements (see §6.2 for details).
We now define the discussion prompt D(r)i = {a(r−1)j , e (r−1) j , p (r−1) j , Cj ̸=i}nj=1 for each agent Ai in round r, based on the above three components. The agent conditions on it to generate an updated answer a(r)i , explanation e (r) i , and confidence p (r) i , to be used in the next round. Demonstrations of convincing explanations enable the agent to generate explanations that are more likely to convince other agents to reach a better consensus.
Phase 3: Team Answer Generation. RECONCILE continues the discussion for a maximum of R rounds or terminates it as soon as a consensus is reached (i.e., all agents agree on the same answer). At the end of any round r, RECONCILE generates the team answer â(r) for that round using a weighted voting scheme (see the right side of Fig. 2). In particular, we recalibrate each agent’s confidence using a function f(·) and then use these as weights to compute the team answer, as follows:
â(r) = argmax a
∑
i
f(p (r) i )1(â (r) i = a)
where a is a distinct answer generated by any of the agents, p(r)i is the original confidence of agent Ai in round r and f(p(r)i ) is the corresponding recalibrated confidence. While an unweighted majority vote and uncalibrated confidence-weighted vote also work well in practice, we use the calibrated weighted vote because it not only obtains slightly better results but the same recalibration strategy also works out-of-the-box for all seven tasks that
4We did not include an agent’s own convincing samples in the prompt because an agent is expected to specifically convince other agents. We also verify this empirically – additionally including self-convincing samples in the prompt leads to comparable performance.
we experiment with (see Appendix B.5 for more details of our recalibration function f(·)).

heading: 5 Experimental Setup
text: Agents in RECONCILE. We primarily implement RECONCILE with ChatGPT, Bard, and Claude2 as the three agents, engaging them in up to three rounds of discussion. Later in §6.1, we also show the generalizability of our RECONCILE framework with different choices of agents, including APIbased (GPT-4), open-source (LLaMA-2-70B), and domain-specific (DeepSeekMath) agents. Datasets. We evaluate RECONCILE on seven benchmarks, including two commonsense, three math, one logical reasoning, and one NLI task. These are: (1) StrategyQA (Geva et al., 2021), (2) CommonsenseQA (CSQA; (Aggarwal et al., 2021; Talmor et al., 2019)), (3) GSM8K (Cobbe et al., 2021), (4) AQuA (Ling et al., 2017), (5) MATH (Hendrycks et al., 2021), (6) Date Understanding (BIG-bench collaboration, 2023), and (7) ANLI (Nie et al., 2020). Baselines. We compare RECONCILE to prior works in three categories: • Vanilla single-agent methods. In this category,
we experiment with (1) zero-shot CoT prompting (Kojima et al., 2022) with one of the interacting LLMs, and (2) eight-shot CoT with Claude2 where the number eight matches the number of convincing samples used in RECONCILE. • Advanced single-agent methods. Next, we compare with (1) Self-Refine (SR) that iteratively generates feedback and refines the output leveraging the model itself (Madaan et al., 2023), (2) Self-Consistency (SC) that samples multiple reasoning paths and generates the most consistent answer (Wang et al., 2023b), and (3) their combination, SR+SC, that first conducts multiple iterations of refinement, followed by a majority vote. Note that in RECONCILE, the number of LLM calls per instance can vary between 3, 6, and 9 based on the number of discussion rounds. Hence, for a fair comparison, we implement SC with the same average number of LLM calls as in RECONCILE. Later in Appendix B.3, we show that RECONCILE even outperforms 9-way SC (that equates to the worst-case LLM calls in RECONCILE). • Multi-agent methods with a single backbone model. Our final baselines are two multi-agent debating methods: a multi-agent debate between
multiple ChatGPT instances (Du et al., 2023) and a debate with judge method (Liang et al., 2023). These methods use multiple instances of the same underlying model (ChatGPT) as different agents.
Implementation Details. Owing to the cost associated with API-based models and the limit imposed on the number of API calls, we follow many prior works (Du et al., 2023; Bian et al., 2023; Besta et al., 2023; Yao et al., 2023a) to experiment with a subset of 100 samples (from the validation set for StrategyQA and the test set for all other datasets). Later in Appendix B.1, we also experiment on the full test sets of StrategyQA and Date understanding and find similar trends. We report accuracy and its standard deviation. For each experiment, we conduct at least three runs on the same test samples with the same prompts, primarily accounting for the variance caused by the decoding strategy. Other implementation details can be found in Appendix A.1.

heading: 6 Results

heading: 6.1 Main Results
text: RECONCILE outperforms single-agent and multi-agent baselines. We first evaluate the overall reasoning capabilities of RECONCILE in Table 2 with ChatGPT, Bard, and Claude2 as the three agents. For fair comparisons, all iterative methods go through 3 rounds of iteration and all singlemodel multi-agent baselines are implemented with three agents with a sufficiently high temperature of 1.0 for maximizing diversity. Across all five datasets, RECONCILE outperforms all single-agent and multi-agent baselines that are built on top of the same models (see last row). Notably, without using GPT-4 as an agent, our method outperforms GPT-4 on commonsense tasks like StrategyQA and CSQA and obtains comparable performance to GPT-4 on most other tasks. GPT-4’s especially strong results on GSM8K could be attributed in part to the inclusion of some of GSM8K’s training samples in GPT-4’s pre-training data (OpenAI, 2023). While multi-agent debate with ChatGPT (Du et al., 2023) improves results on math benchmarks, debate with multiple Bard or Claude2 instances is not effective, possibly because the responses (generated from the same model) are not sufficiently diverse. When they team up with ChatGPT in a multi-round discussion, RECONCILE outperforms debate frameworks. It obtains maximum gains of
outperforms GPT-4. All results are on a random subset of 100 samples. The agents are GPT-4, ChatGPT,
model like GPT-4, or an open-source model like
LLaMA2-70B.
11.4% (75.3% → 86.7%) on date understanding and 7.7% (71.3% → 79.0%) on StrategyQA when compared to the strongest baseline (multi-agent debate with Claude2). Improvements in the math reasoning tasks are relatively moderate, because of ChatGPT’s initial strong performance. However, as demonstrated later in Table 4, integrating a specialized math reasoning model into RECONCILE significantly boosts team performance.
RECONCILE generalizes to agents of varying strengths. Next, we vary the agents in RECONCILE to study its generalization as a multi-agent framework. In particular, we either include (a) a stronger GPT-4 model, or (b) an open-source LLaMA-2-70B-chat model in the discussion. As shown in Table 3, in both these scenarios, RECONCILE outperforms the best single-agent and multiagent baselines, notably even outperforming the zero-shot GPT-4 performance by 12.1% (75.6% → 87.7%) on StrategyQA. This highlights the potential of a stronger agent to also obtain useful external feedback from comparatively weaker agents.
like DeepSeekMath and improves on a challenging mathematical reasoning benchmark, MATH.
RECONCILE generalizes to domain-specific agents. So far, we have experimented with RECONCILE variants that employed general-purpose models like ChatGPT as agents. Our next result in Table 4 shows that even for tasks that require substantial domain knowledge (e.g., the MATH benchmark (Hendrycks et al., 2021)), RECONCILE is flexible enough to utilize and improve upon specialized, domain-specific models. Recently, Shao et al. (2024) proposed DeepSeekMath, a 7B model pre-trained on a large number of mathrelated web corpus and improving over GPT-4. Notably, RECONCILE with GPT-4, Claude2, and DeepSeekMath as agents significantly outperforms zero-shot DeepSeekMath and GPT4-based Debate by 7.8% and 9.6% respectively. In summary, RECONCILE shows consistent improvements across a wide range of agent combinations (involving APIbased, open-source, and domain-specific models).
RECONCILE also improves Natural Language Inference. While all our previous results were with reasoning tasks, we also demonstrate RECONCILE’s effectiveness on ANLI (Nie et al., 2020),
a challenging Natural Language Inference benchmark. Table 6 shows that RECONCILE on ANLI outperforms Debate by a significant 9.4%, pointing to its widespread applicability.

heading: 6.2 Ablations and Analysis of RECONCILE
text: Each component of RECONCILE improves reasoning. In Table 7, we evaluate individual components of RECONCILE on StrategyQA. In particular, we compare four variants: (1) w/o Multiple Models: We use ChatGPT as the backbone for all three agents, (2) w/o Grouping: We simply concatenate the responses from different agents without grouping their answers, (3) w/o Convincingness: We remove convincing samples from all prompts, and (4) w/o Confidence Estimation: We do not use any confidence estimates during the discussion and compute majority vote as the team answer. We show that each component has a positive impact on RECONCILE with varying capacities. The effect of different models as agents is particularly significant and we observe a 6.8% improvement compared to only using ChatGPT as all three agents. This reinforces our hypothesis (and further verified below in ‘Diversity Analysis’) that diverse LLMs have complementary strengths and when put together in a round table discussion, they can learn from diverse external feedback from other agents and refine their responses to reach a better consensus. Notably, convincing samples lead to a 4.5% improvement in accuracy. In Appendix B.2, we study the role of convincing samples to show that (1) they also improve other interaction frameworks, and (2) even in the absence of such examples, RECONCILE outperforms debate baselines.
Different models enhance response diversity. As was shown in Table 7, RECONCILE obtains the most improvements via its multi-model component. This surpasses RECONCILE with multiple ChatGPT instances, even when the generations sampled from these instances are encouraged to exhibit high diversity with a sufficiently high temperature. To further validate the importance of having multiple models and the diversity brought about by them, we develop a diversity metric. We hypothesize that if explanations from different models are indeed more diverse than those generated from multiple instances of the same model (e.g., in Multiagent Debate), then our diversity metric should capture that. With that goal, we define diversity between multiple agents as the summation of the pairwise diversity between agents: D(A1, A2, A3) = D(A1, A2)+D(A1, A3)+D(A2, A3), where A1, A2, and A3 are the three agents’ initial responses (either belonging to the same underlying model or different models). We then measure pairwise diversity by computing the cosine similarity between the response embeddings with BERTScore (Zhang et al., 2019). Note that lower similarity scores will mean greater diversity. With the diversity metric defined, we compute this metric for three variants: (a) paraphrased responses of a single ChatGPT to serve as a baseline, (b) responses from RECONCILE using three instances of a single ChatGPT model, and (c) responses from RECONCILE with ChatGPT, Bard, and Claude2 as agents. In Table 5, we show that responses from different models exhibit the highest diversity (yielding the lowest similarity score of 0.8739) and also the highest accuracy (79.0%), followed by the single-model variant
(with a similarity score of 0.9102) and the paraphrased variant (with a similarity score of 0.9398). Thus, the higher diversity of (multi-model) RECONCILE means that agents have access to alternate solutions and external feedback, leading to better discussion and reasoning accuracy. We also present a case study in Appendix C.5 to illustrate that the debate baseline sometimes struggles with echo chambers, stemming from a lack of external feedback, supporting the need for external feedback for improving LLMs (Huang et al., 2023).
RECONCILE improves all agents individually. We showed that the team performance of the agents improves through discussion. Next, in Table 8, we also present the accuracy of each agent after every round, as well as the overall team accuracy for StrategyQA. Evidently, the individual performance of each agent also improves alongside the team’s performance.
RECONCILE Reaches Faster and Better Consensus. RECONCILE terminates the discussion when a consensus is reached. More discussion rounds are costlier due to the increased API calls. Hence, achieving faster consensus while maintaining comparable accuracy gains is more efficient. To study this, in Fig. 4(a), we plot the accuracy trends after each round; in Fig. 4(b), we plot the fraction
of samples for which consensus has been reached; and in Fig. 4(c), we analyze accuracy as a function of consensus. From the first plot, we make two important observations: (1) RECONCILE improves accuracy for two rounds, following which the accuracy saturates, (2) Compared to the debate baselines, RECONCILE is not only superior after every round but also peaks at a highest accuracy of 79.0% (vs 71.3% for the baselines). Next, from Fig. 4(b), our observations are also two-fold: (1) In the initial rounds (0 and 1), RECONCILE’s consensus percentage is lower because the discussion takes place between diverse LLMs. Diverse agents lead to more differences in opinions initially. (2) However, as the discussion proceeds, RECONCILE establishes consensus for all samples by round 3, while in the baseline, 13% of the samples do not converge even after round 4. Finally, Fig. 4(c) shows that for the samples that enter the discussion phase (i.e., their initial answers did not have a consensus), accuracy is positively correlated with consensus. In other words, as a greater number of samples reach a consensus, accuracy proportionally improves. In summary, RECONCILE reaches faster and better consensus compared to baselines.

heading: 7 Conclusion
text: We presented RECONCILE, a multi-agent framework for reasoning with diverse LLM agents, engaged in multiple rounds of discussion via confidence estimation and generating explanations that can correctively convince other agents. RECONCILE demonstrated strong results on multiple reasoning benchmarks, consistently outperforming prior single-agent and multi-agent baselines and even improving upon GPT-4 on some benchmarks.

heading: Limitations
text: For the API-based models used in RECONCILE, we note that we lack complete knowledge of the data that these models have been exposed to, and their scales in terms of parameters. Moreover, due to the API access, we do not possess complete control over their behavior. Depending on API-based models also necessitates the need to prompt these models to estimate their confidence. While this approach proves effective as evidenced by our results, we note that these estimates remain post-hoc in nature. Nevertheless, it is worth highlighting that these limitations could potentially be mitigated in the future should more open-sourced models emerge and demonstrate robust capabilities in adhering to long instructions.

heading: Acknowledgments
text: We thank Peter Hase and Elias Stengel-Eskin for useful feedback and suggestions regarding experiments. This work was supported by NSF-CAREER Award 1846185, NSF-AI Engage Institute DRL2112635, DARPA MCS Grant N66001-19-2-4031, Accelerate Foundation Models Research program, and a Google PhD Fellowship. The views contained in this article are those of the authors and not of the funding agency.

heading: A Additional Details of RECONCILE
text: A.1 Implementation Details We provide more implementation details of RECONCILE in this section. During decoding, we set the temperature to 0.7 for ChatGPT and Bard and use the default setting for Claude2. All implementations involving ChatGPT are using gpt-3.5turbo-0613 from Azure OpenAI.5 We retrieve results from Claude2 by posting requests to their webpage6, and for Bard, we use chat-bison-001 from PaLM2 API7. For each agent, we use four demonstrations of convincing samples. In addition, we provide the workflow of RECONCILE in Algorithm 1. Required input contains a test problem Q, maximum number of discussion rounds R, n agents A = {Ai}ni=1, and convincing samples C = {Ci}ni=1 for each agent. The output would be the team answer â(r). For the open-source models LLaMA2-70B and DeepSeekMath, we use four RTX A6000 GPUs, each with 48GB memory to generate output from them.

heading: Initial Prompt

heading: Discussion Prompt
text: A.2 Initial Prompt and Discussion Prompt We show the prompts used in RECONCILE in Fig. 5. The initial prompt encompasses (1) the convincing samples that demonstrate how to convince other agents, (2) the test question, and (3) a requirement for ‘step-by-step’ reasoning. The prompt also instructs the agent to express their confidence level,
5https://oai.azure.com/ 6https://claude.ai/chats 7https://developers.generativeai.google/products/palm

heading: Model StrategyQA Date
text: ranging from 0.0 to 1.0, indicating the likelihood of their answer being correct. The discussion prompt is an extension of the initial prompt, instructing the agent to review and express agreement or disagreement with other agents’ solutions. To facilitate discussions, we design a grouping scheme that aggregates information based on the current opinions at the table. For instance, if two agents affirm that the answer to a given question is ‘yes’ while the third agent disagrees with a ‘no’, the designed grouping mechanism in the discussion prompt consolidates this information rather than simply concatenating all responses.

heading: B Additional Results

heading: B.1 Results on Full Test Sets
text: In Table 2, we reported results with 100 test samples following several previous works and due to budget constraints. Upon experimenting on the full test sets of StrategyQA and Date Understanding, we confirm similar trends. Specifically, in Table 9, we compare RECONCILE to all of our major baselines and show that RECONCILE continues to outperform all baselines.

heading: B.2 Convincing Samples Improve Both RECONCILE and Multi-agent Debate
text: Recall that RECONCILE selects a sample as convincing if the corresponding human explanation
Algorithm 1 RECONCILE: A Group-Discuss-And-Convince Framework Require: Test Problem Q, Discussion Rounds R, Agents A = {Ai}ni=1, Convincing Samples C = {Ci}ni=1
function RECONCILE(Q,R,A, C) r ← 0 while r ≤ R and not CONSENSUS(Q, {a(r−1)i }ni=1) do
S ← [], P ← [] for each Ai ∈ A do
if r = 0 then PI ← (Q, C) ▷ Initial prompt consists of question and convincing samples a (0) i , e (0) i , p (0) i ← Ai(PI) ▷ Generate initial answer, explanation, and confidence else PD ← (Q, a(r−1)i , e (r−1) i , p (r−1) i , C) ▷ Discussion prompt
a (r) i , e (r) i , p (r) i ← Ai(PD) end if S ← S + [a(r)i ], P ← P + [p (r) i ] ▷ Append each agent’s answer and confidence end for â(r) ← WEIGHTEDVOTE(S, P ) ▷ Get team answer through a confidence weighted vote
end while return â(r)
end function
rectifies an agent’s incorrect answer. Based on this, Table 7 showed that by collecting only four human explanations, we can obtain significant improvements (‘w/o Convincingness’ row). Next, we consider a scenario where no human explanations are present. Table 10 shows that even then, RECONCILE outperforms the debate baseline by absolute 7.8 points (second row). If random (i.e., general human explanations that may not necessarily ensure answer rectification) are available (third row), we obtain some small improvements; but our convincing samples that are selected based on our novel answer-rectification criterion (fourth row) improve the results substantially. See Sections C.3 and C.4 for illustrative examples. Being able to convince another agent is also a generic concept that can be applied to other multi-agent systems, as demonstrated by improvements in the debate baseline (last row).

heading: B.3 Comparison with Other Methods
text: In Table 11, we compare RECONCILE to two other single-agent variants. While in our main Table 2, we experimented with a random 8-shot Claude2 baseline, here we replace the in-context samples with our convincing samples. Even then, RECONCILE exhibits superior performance on all datasets except for GSM8K, again highlighting the importance of collaboration between diverse models. Next, we also report results for 9-way Self-Consistency which in terms of LLM calls represents the worst-case scenario of RECONCILE – even for a more open-ended dataset like GSM8K, 9 LLM calls (i.e., 3 discussion rounds) happen in
only 12% of the samples and an even lesser 9% on multiple-choice QA dataset like Date understanding. That said, RECONCILE continues to outperform 9-way SC by a large margin on most datasets.

heading: B.4 Recalibration Strategy of RECONCILE
text: Directly using confidence scores as the voting weights is less effective due to the overconfidence problem of LLMs (Xiong et al., 2023b; Tian et al., 2023; Mielke et al., 2022). Specifically, LLMs tend to produce consistently high confidence scores, which can make it challenging to discern subtle distinctions in confidence levels across different outputs. To address this, we employ a simple yet effective rescaling technique, facilitating better differentiation of confidence levels. This is expressed as:
f(p (r) i ) =    1.0, if p(r)i = 1.0 0.8, if 0.9 ≤ p(r)i < 1.0 0.5, if 0.8 ≤ p(r)i < 0.9 0.3, if 0.6 < p(r)i < 0.8 0.1, otherwise
where p(r)i is the original confidence of agent Ai in round r and f(p(r)i ) is the corresponding adjusted score. To decide the optimal weights, we compare with a variety of settings including the majority vote and the uncalibrated confidence-weighted vote. The results are summarized in Table 13. We denote the weight we used in our main experiment as w∗ = [1.0, 0.8, 0.5, 0.3, 0.1] where each value corresponds to the recalibrated confidence score. We further compare with other settings:
• w1 = [1.0, 0.9, 0.7, 0.5, 0.3] • w2 = [1.0, 0.9, 0.5, 0.3, 0.1] • w3 = [1.0, 0.8, 0.6, 0.4, 0.2] • w4 = [1.0, 0.75, 0.5, 0.25, 0.0]
and the results show that our w∗ works the best across datasets. In our main experiment, we fix the weight using w∗ and it is constantly outperforming majority vote across all seven datasets. In addition, Fig. 9 shows that it helps reduce the Expected Calibration Error (ECE), a popular calibration metric (Naeini et al., 2015). While we note that recalibration can also be achieved through a learned model (e.g., Platt Scaling (Platt et al., 1999)), we refrain from using such models because RECONCILE is primarily designed as a few-shot method, and developing a recalibration model would necessitate access to a substantial number of annotated samples. Therefore, we use f(p(r)i ) to perform a weighted vote to generate the team answer.

heading: B.5 Comparison of Different Voting Strategies
text: At the end of any round r, every agent in RECONCILE generates its answer. Here we explore three voting strategies: (1) maximum confidence vote, where the agent’s answer with the maximum confidence score would be the final team answer,
(2) unweighted majority vote, where each vote carries equal weight, irrespective of the confidence score, and (3) weighted vote, where we use the recalibrated confidence scores as the voting weights. As shown in Table 12, weighted vote is the most effective way to aggregate the team answer.

heading: C Qualitative Examples

heading: C.1 Convincing Samples for Each Agent
text: Table 15 shows examples of convincing samples on StrategyQA for each agent.

heading: C.2 Effect of Convincing Samples
text: Here, we provide qualitative examples of how convincing samples change the way each agent responds to the question. We compare the initial responses (of each agent) with and without convincing samples in Table 16.

heading: C.3 RECONCILE w/o Convincing Samples
text: We notice that when RECONCILE operates in the absence of convincing samples, the agents tend to maintain their initial opinions more often. As depicted in Fig. 6, all three agents adhere to their original stances throughout the entire discussion and hence never converge to the correct answer.

heading: C.4 RECONCILE with Convincing Samples
text: On the contrary, when convincing samples are present, we show how the explanations of all agents change during the course of a discussion (see Fig. 7). Initially, Bard and Claude2 provide incorrect answers, but as the discussion unfolds, both
agents revise their initial predictions, ultimately arriving at the correct answer.

heading: C.5 Single-Model Multi-Agent Debate Struggles with Echo Chamber
text: In Fig. 8, we provide an illustration of multi-agent debate, implemented with multiple instances of the same underlying ChatGPT model. In this case, an incorrect answer is initially provided, but because external feedback from diverse models is lacking,
all agents persist with the same incorrect response throughout the interaction.

heading: D Benchmark Licenses
text: The licenses of datasets we used are in Table 14.

heading: Future Work
text: Nevertheless, it is worth highlighting that these limitations could potentially be mitigated in the future should more open-sourced models emerge and demonstrate robust capabilities in adhering to long instructions