More Agents Is All You Need
Authors: junyou li
Qin Zhang
Yangbin Yu
QIANG FU
Deheng Ye
Venue: Accepted by TMLR ()

Abstract: We find that, simply via a sampling-and-voting method, the performance of large language models (LLMs) scales with the number of agents instantiated. Also, this method, termed as Agent Forest, is orthogonal to existing complicated methods to further enhance LLMs, while the degree of enhancement is correlated to the task difficulty. We conduct comprehensive experiments on a wide range of LLM benchmarks to verify the presence of our finding, and to study the properties that can facilitate its occurrence. Our code is publicly available at: https://github.com/MoreAgentsIsAllYouNeed/AgentForest

Published in Transactions on Machine Learning Research (10/2024)
More Agents Is All You Need
Junyou Li Tencent
junspouH@kennennl.ccm
Qin Zhang* Tencent
adrienchaang@Steencen..ccmm
Yangbin Yu Tencent
yengbiinegu@tennentt.comm
Qiang Fu Tencent
leomfu@@tencent.ccom
Deheng Ye Tencent
dericye@@lencent.com
Reviewed on OpenReview: https: openreview. net/ nett forum? bgzUS2Baeg
Abstract
We find that, simply via sampling-and-voiinng method, the performance of large language models (LLMs) scales with the number of agents instantiated. Also, this method, termed as Agent Forest, is orthogonal to existing complicated methods to further enhance L.Ms,, while the degree enhancement is correlated to the task difficulty. We conduct comprehensive experiments on wide range of LLM benchmarks verify the presence of our finding, and to study the properties that can facilitate its occurrence. Our code is publicly available at: https:///ithuub.cm/ wwapelhnw..co/MMoooeeeeee
1 Introduction
Although large language models (LLMs) demonstrate remarkable capabilities in variety of applications (Zhao et al., 2023), such language generation, understanding, and reasoning, they struggle to provide accurate answers when faced with complicated tasks. To improve the performance of LLMs, some of recent studies focus on ensemble methods (Wang et al., 2023b; Wan et al., 2024) and multiple LLM-Agents collaboration frameworks (Du et al., 2023; Wu et al., 2023).
In these works, multiple LLM agents are used to improve the performance LLMs. For instance, LLM-Debate Du al. (2023) employs multiple LLM agents in debate form. The reasoning performance improved by creating framework that allows more than one agent to "debate" the final answer of arithmetic tasks. They show performance improvements compared to using one single agent. Similarly, CoT-SC (Wang et al., 2023b) generates multiple thought chains and picks the most self-consistent reasoning performance is improved by involving more thought chains compared to chain-of-thought (CoT) (Wei et al., 2022) which employs a single thought chain. Incidentally, from the data analysis of these works, we can notice the effects of putting multiple agents together, to some extent, can lead to a performance improvement certain problems. For example, Table the authors have reported preliminary curve: the accuracy of a math problem increases with the number of debating agents (although the number was simply increased from to 7). Also, in Wang al. (2023b), involving more chain-of-thought pipelines (termed as
the final answer. The
Section 3.3 LLM-Debate Du et al. (2023),
decoding procedure),
"Co-first authors.
1 "Corresponding author.

---

Published in Transactions on Machine Learning Research (10/2024)
Accuracy Curves In GSM8K
gpt-4 (single)
90
80
gpt-3. (single)
Accuracy (%)
70
60
lama-70B (single)
50
llama-138
40
llama-70B
30
3pt-3.5-turbo
5
10 15
20
25
30
35
Ensemble Size
Figure 1: The accuracy increases with ensemble size across Llama2-13B, Llama2-70B and GPT-3.5- GSM8K. When the ensemble size scales up to 15, Llama2-13B achieves comparable accuracy with Llama2-70B. Similarly, When the ensemble size scales up to 15 and 20, Llama2-705 and GPT-3.5- -urbo achieve comparable accuracy with their more powerful counterparts. The error bars represent the standard error.
lead to performance gain. We realize that the LLM performance may likely be improved by a brute-force scaling up of the number of agents instantiated. However, since the scaling property of "raw" agents is not the focus of these works, the scenarios connaiis/taskss tasks and experiments considered are limited. So far, there lacks dedicated in-depth study on such phenomenon. Hence, natural question arises: Does this phenomenon generally exist?
To answer the research question above, we conduct the first comprehensive study on the scaling property of LLM agents. To dig out the potential multiple agents, we propose to use a simple(st) sampling anatiiinnnn method, which involves two phascs. First, the query of the task, i.c., the input to an LLM, is iteratively fed into a single LLM, or a multiple LLM-Agents collaboration framework, to generate multiple outputs. Subsequently, majority voting is used to determine the final result. The procedure is inspired by that of the CoT-SC, but it does not rely on designing complex CoT paths. In fact, it can be used as a plug-in to further enhance CoT-based methods, as will be shown in our evaluations. Our method is termed as Agent Forest, tribute to the classic Random Forest (Breiman, 2001).
The experiments are conducted by using various LLMs of different sizes on diverse datasets covering reasoning and generation. The result indicates that LLM performance can generally be improved by increasing the ensemble size, i.e,, the number of agents, across wide range of tasks. Surprisingly, a brute-force ensemble of smaller LLMs can achieve comparable superior performance to larger LLMs, with a nutshell shown in Figure 1, which will be hurther expanded in later sections. Moreover, by combining our method with other existing methods, we find the performance can be further improved. By comparing with the performance of complicated methods, the result shows that employing our method solely can achieve comparable performance in most cases. This implies that comparable performance can be achieved without the need for additional handcraft prompt design or complex collaboration frameworks.
Additionally, the experimental results indicate that there are greater performance improvements when addressing difficult tasks and when using weaker models. To understand the reasons behind these performance improvements, we analyze the influence of problem difficulty on the effectiveness of our method. We classify difficulty into three dimensions: the inherent difficulty, the length of reasoning steps, and the prior probability of the correct answer. Through a series of experiments, we adjust these dimensions and observe their effects independently. We observe and summarize a few properties, based on which, we further develop optimization strategics that can intrigue the power of "More Agents".
Our contributions summarized follows:
2

---

Published in Transactions on Machine Learning Research (10/2024)
We present the first systematic study the scaling property of raw agents instantiated by LLMs. We find that the performance scales with the increase of agents, using the simple(st) way of sampling and voting.
We explore the compatibility of our method with existing complicated methods that stimulate the potential of LLMs, revealing that our method can enhance these methods to achieve further performance improvements.
We analyze the effectiveness of our method in tackling problems at varying difficulties and then distill the properties behind, based upon which, we propose further optimization methods that can facilitate the occurrence of our finding.
2
Related Work
Related works can be calegorized into three parts: 1) LLM self-ensemble Wang et al. (2023b), which attempts to harness multiple outputs from homogeneous LLMs to assemble the final answer; 2) heterogeneous LLM ensemble, which focuses combining heterogeneous LLMs through supervised learning to improve performance across various downstream applications; and 3) multiple LLM agents collaboration, which improves performance through interactions among LLM agents. We discuss these works below.
LLM Self-Ensemble. CoT-SC Wang et al. (2023b) harnesses diverse chain-of-thought Wei et al. (2022) prompts elicit a variety of reasoning processes from single LLM and select the final answer through majority voting. Fu et al. (2023); Li et al. (2023b); Cobbe et al. (2021b); Thoppilan et al. (2022); Lin et al. (2023) can be considered as the extensions of CoT-SC. These methods mainly focus reasoning tasks and exclusively investigate the compatibility with COT. In contrast, reasoning tasks but also in generation tasks. Moreover, our method is compatible with broader range of methods, such as prompt engineering (including CoT) and multiple LLM agents collaboration. Very recently, Lu et al. (2024) proposes a method named Blended that utilizes multiple LLMs for chat scenarios. In contrast, Blended focuses on utilizing the power of multiple LLMs, whereas our focus is on the scaling trend of adding more LLMs. Also, Blended only for limited chat scenarios evaluated via human annotations. Furthermore, explore orthogonality with other methods.
method not only validates effectiveness in
Heterogeneous LLM Ensemble. Wan et al. (2024) conducts a supervised LLM fusion framework to distill multiple heterogeneous LLMs into a single model and surpasses each of these LLMs. Jiang et al. (2023) introduces a supervised ensembling framework based on multiple heterogeneous LLMs. Chen et al. (2023b) proposes a sequential inference method for LLMs that halts when the output quality is deemed adequate. Wang et al. (2023a) addresses the ousiensof-oxperts problem by integrating outputs from models with distinct knowledge domains through supervised learning. Shnitzer et al. (2023) and Lu et al. (2023) select the most suitable LLM for new tasks by training reward-guided router. These approaches primarily employ supervised learning, necessitating task-specific annotated data, and exhibit limited generalizability. In contrast, our method is unsupervised, without the need for additional training data.
Multiple LLM Agents Collaboration. Du et al. (2023); Liang et al. (2023); Xiong et al. (2023) explore various multiple LLM agents interaction architectures, with employing static debate-style engagements among LLMs for enhanced reasoning... Liu al. (2023) enables agents to interact for multiple rounds in dynamic architecture. Li et al. (2023a); Hong el al. (2023); Wu et. al. (2023); Chen el. al. (2023caa) offer several multi-agent frameworks that enable the development of LLM applications enhance task-solving capabilities. However, these methods primarily focus on the interaction structures between LLM agents, rather than the relationship between the number of agents and performance. We also select representative methods Du et al. (2023); Shinn et al. (2023) to combine with our method, achieving further enhancements.
3 Method
In this section, we introduce Agent Forest, which implemented through a two-phase process: sampling and voting. The overview of our method is shown in Figure 2.
3

---

Published in Transactions on Machine Learning Research (10/2024)
Sampling
Voting
LLM Agent
Answer
Query
Prompts
Majority Voting
or
Query
Figure 2: Illustration of Agent Forest. The two-phase process begins by feeding the task query, either alone combined with prompt engincering methods, into LLM agents generate answers. Subsequently, majority voling is applied to these answers to determine the final answer. Specifically, an LLM agent refers to single LLM or multiple LLM-Agents collaboration framework.
Algorithm Agent Forest.
Require: Query z, number of samples N, LLM M or LLM integrated with other methods fM(()) Initialize empty set for samples S 0
for i= to N do
3: Generate sample +- M(x)) 8i + fm(()
4: Add sample to the set S + {si}
5: end for
for each sample 8; in
Initialize similarity scores (ss;) +- 0
for each sample sj in S do
if i then
10: V(s;) + Vss: + sim(ss,8)
11: end if
12: end for
13: end for
14: +- arg max,,es V(si)
15: return A
Sampling. Let æ represent the task query and M denote an LLM. In this phase, we generate N samples by solely querying the LLM M N times with each sample represented as = M(x) or by integrating with other methods fM with times executions where each sample is denoted as s = fM(x). We obtain a set of samples 5 {81, 82, at the end of this phase.
Voting. Let A represent the final answer. In this phase, we employ majority voting to consolidate the response sample set S into the final answer A. This involves calculating the cumulative similarity for each sample relative to the others, denoted as V(s:) E En-1gge sim(s;, sz). For open-ended generation tasks such as code generation, the BLEU score proposed by Papineni et al. (2002) is utilized to quantify similarity. Conversely, for close-ended tasks like multiple-choice questions, similarity is measured by occurrence frequency. The sample that exhibits the highest cumulative similarity is then chosen as the final answer denoted as A arg max,,es V(s.).
The complete process of Agent Forest is described in Algorithm 1.
4

---

Published in Transactions on Machine Learning Research (10/2024)
4 Experimental Setup
We separate the experimental setup (this section) with evaluations (next section), to introduce the coverage of sccnarios /takks compared with the most related works (for examining the comprehensiveness of our work), the backbone language models we adopted (for examining the applicability of our work), and the methods combined with ours (for examining the compatibility and orthogonality of our work).
Tasks Our method is evaluated on the following task:
Arithmetic Reasoning. Similar to Wang et al. (2023b); Fu et al. (2023); Du et al. (2023), we select the GSM8K Cobbe et al. (2021a) as one of the test sets. Additionally, we select the more challenging MATH dataset Hendrycks al. (2021b), which used by Wu et al. (2023).
General Reasoning. Similar to Du et al. (2023); Jiang et al. (2023), we select the MMLU Hendrycks al. (2021a). Additionally, we select the dataset from the chess state tracking task (Chess) which is used by Du et al. (2023); Zhang et al. (2023).
Code Generation. Similar to Liu al. (2023), we select the Human Eval Chen et al. (2021). To implement our method, we compute the BLEU score Papineni et al. (2002) among all pairs of generated candidate answers. The answer with the highest cumulative BLEU score is then selected as the final output.
Table 1: Comparing the conducted experiments with the most related works. Our comprehensive study encompasses various LLMs, multiple tasks, and the integration with multiple methods.
Tasks
Integrated with Methods
Methods
LLMs
Arithmetic
Generl
Code
Prompt
Multiple L.M--gents Collaboration
Chat
Rewsning
Rewsoning
Generation
Engineering
Co1-SC Wang et al. (20235)
v
Ouly Coll Wei et al. (2022) Only CoT Wei (2022)
Complexity.Coo CoT Bu et al. (2021)
Debate Du et al. (2023)
Blended La et al (2024)
Language models adopted We evaluate our method using language models of different scales from the Llama2 Touvron et al. (2023) and GPT series OpenAl (2022). Specifically, we evaluate two versions of alama2-Chat', optimized for conversational use cases through alignment techniques, with model sizes of 13B and 70B parameters. Additionally, we include GPT-3.5-Ture and GPT-4 in our evaluation.
Methods enhanced by our method To examine the comparability of our method, study the integration of various typical methods from two distinct categories with our method:
Prompt Engineering. Various prompt engineering methods are considered to conduct comprehensive experiments. We evaluate Chain-of-Thoune Thouught prompting (CoT) Wei al. (2022), Zero-Shot Chain-of- Thought prompting (Zero-Shot Cot. Kojima et al. (2022), and more sophisticated methods such Solo Performance Prompting (SPP) Wang al. (2023c). Initially, these methods are applied with single LLM query. We then increase the number of queries and employ majority voting to determine the most consistent answer as the final response.
Multiple LLM Agents Collaboration. We select LLM-Debate Du et al. (2023) denoted as Debate, and self-reflection Shinn et al. (2023) denoted as Rellection. Within these methods, we generate multiple samples by iteratively operating these methods and using majority voting to produce the final answer.
1 Chess State Tracking 21 21.ama2-Cha Chat
5

---

Published in Transactions on Machine Learning Research (10/2024)
Specifically, the effectiveness of our method is evaluated by averaging the results across 10 independent runs. During each run, scale up the ensemble size to 40 to ensure maximum gains. However, when integrating our method with the Debate Du et al. (2023), the ensemble size is limited to 10 due to the significant computational overhead introduced by the communication architecture. Detailed experimental scttings are provided the Appendix A.
5
Experimental Results
GSMAK
MATH
Chess
MMLU
Huma Evall
0.80
0.70
0.70
0.60
0.40
0.55
0.50-
0.30
Liamaz-139
0.20
0.45
0.30-
Liamaz 708
GPT 3u5
0.10
0.20
0.30
0.00
0.40
0.70
0.80
0.50
0.70
0.00
0.60
0.23
0.30
0.55
0.40
0.50
Llama2-135 33
0.19
0.45
0.40
0.20
0.10
10
20
30
40
10
20
30
40
10
20
30
10
26
30
40
0
10
20
30
40
Ensemble Size
Ersemble Size
Ensemble Size
Ensemble Size
Ensemble Size
Figure 3: The accuracy scales with the ensemble size of our method across different tasks with various LLMs. The error bars represent the standard error.
Table 2: Our method generally enhances performance across all tasks and LLMs. The bolded instances indicate that smaller LLMs outperform the larger LLMs. "Single" denotes that the LLM is queried only once. GPT-4 is used only for comparison with other methods, hence it only presents "Single" results. "Ours" denotes our method where the ensemble size is 40. The error bars represent the standard error.
Model
GSMBK
MATH
MMLU
HumanExa
Single
Ours
Single
Ours
Single
Qurs
Single
Ours
Single
Ours
Llama2-130 Touyron al. (2025)
0.35
0.59
1100
0.09
0.14
0.18 2--5
0.42
0.51
11.14
0.18
12am22-708 Tonvron (2023)
0.54
0.74
0.05
0.11
1e33
0.33 1e3
GPT-3.5-Taria OpenAl (2022)
0.73 :: 26-2
0.85 :: 3e-1
0.20 : 2-12
0.30 :: 2e-3
0.51 1: ae-2
0.55 2e-3
0.59 3e-2
0.70 26/3
0.07
d-2
0.73
te/2
GPT-4 OpenAl (2022)
0.88 262
0.40
0.05 262
0.77
0.88
5.1 Generalizability
Table and Figure 3 show that our method generally enhances performance across all tasks and LLMs by increasing the ensemble size. Specifically, in arithmetic reasoning tasks, the accuracy gains range from 12% to 24% on the GSM8K and from 6% to 10% on the MATH. In general reasoning tasks, the accuracy gains range from 1% to 4% on the Chess and from 5% to 11% on the MMLU. In code generation task, the accuracy gains range from 4% to 9% on HumanEval. Surprisingly, our method enables a smaller LLM to outperform a larger counterpart by simply scaling up the ensemble size. For instance, the enhanced Llama2- 13BB model achieves 59% accuracy on the GSM8K dataset, outperforming the Llama2- 20B model, which scores 54%. Additional statistical results are presented in Appendix B.2.

---

Published in Transactions on Machine Learning Research (10/2024)
Table 3: Our method outperforms other methods used standalone in most cases and always enhances other methods across various tasks and LLMs. The bolded instances indicate the highest accuracy for each task and the underlined instances indicate the highest accuracy in standalone cases.
Model
Method
GSMsK
MATI
Chess
MMLU
Human
Stenslalome
+Ohrrs
Stamiiaeeee
+Oura
Sbandske
-Ouns
Soundsko
+Ouns
Sourlaloon
+Oanx
OUT Wei al. (2022)
0.30
0.56 (40,17)
0.0%
0100 (-932)
0.18
0.28 (-0.00)
0.42
0.42 (-002)
0.13
0:20 (4102)
25-COT Kojions st al. (2002)
0.40
0.61 (+1-9))
0.05
0.08 (-010)
0.15
0.50 (-0.66)
0.42
0.48 (4014)
0.15
0:22 (+0
LlunnG118
SPP Waug (2003k))
0.311
00:22
0.21
0.26 (-0.00
0,03
0.08 (-085)
Tomcon al. (2023)
(2003)
0.38
0:65 (e0:is
(19
0.19
(10c0)
Rollection Shinn al. (2023)
0:30
0.50 (yeins)
0.01
0/03 (10 oe:
0.13
0.18 (-0
0:45
0.50 (10
0.05
(000)
0.23
0.18
0.52
0.25
(2022)
0:57
0.72 (40.0))
(4RAm)
0.01 (H400)
0:32 (1unz)
28-001 Kojiun et (20222)
0.57
0.77 (+8.10
(+0.00
0.20
0.27 (-0.00
0.54
(-4211
0:23
0.29 (4000)
Lhmm2 T0B Tomvou el al (2023)
SPP Wang ol. (2023k)
0.42
0.88 (4827)
0.03
0.00 (40.0)
0016
0.27 (-0.31
0.29
0.63 (40)4)
0.15
0:20 (4u0s)
Debaie Du al. (2023)
0.50
0.06 (+0.00
0.10
0.11 (+0,01
0.14
0.17 (-0.00)
0.56
0.58 (-0.3)
0
Hollection Shion et al. (20235)
0.52
0.77
0.02
0.00
0.15
0.26 (-0.33)
0:42
0.55 (+0.3)
0.16
0.26 (4030)
Ouns
0.74
0.11
0.13
0.80
0:33
OUT Wiri et al. (2022)
0.24
0.54 (+0.1n)
0.25
0.41 (+0:13)
0.50
0.55 (-0.00)
0.61
0.64 (-0
0.70
0.75 (+085)
2S COT Kajima et al. (2002)
0.74
0.88 (48.14)
0.25
0.40 (40:10)
0.35
0.86 (40.30)
0.38
0.09 (4030
0.67
0.74 (4487)
T-urho OpenAl (2022)
SPP Wing (2000e)
0.70
0.53
0.26
0.37
0.54 199
0.53
0.68
0.57
0.04 (o00r)
Debate Du (2123)
0.35
0.85
0.49
0.57 (-006)
0.56
0.24
Deflection Shinn al. (2023)
0.70
0.54 (4830))
(40.14)
0.44
0.57 (-0:31)
0.39
(44155)
0.38
0.73 (443))
0.55
0.05
0.73
Chess
MMLU
GSMBK
0.72
0.60
0.84
0.70
0.82
Greedy
0.50
0.68
T-0.3
Axcuras
0.66
0.800
1=0,5
2.50
0.64
0.78
T-0.7
p-0.9
3.59
0.62
0.76
p-0.95
0.74
0.52
0.58
5
10
15
20
25
30
10
15
20
25
30
0
10
35
25
30
Errvemble Stzee
Eroveblle Size
Ersvebble Size
Figure 4: Our method improves accuracy over various hyperparameters and tasks. The default T' is 1.0 and the default p is 1.0.
5.2 Compatibility
Table 3 shows that by integrating our method with other methods, the performance can be further improved across different LLMs and tasks, despite these methods have different implementations. To be specific, in arithmetic reasoning tasks, our method enhances these methods to further improvement, yielding increases between 10% and 21% on the GSMSK dataset, and between 1% and 15% on the MATH dataset. In general reasoning tasks, integration with other methods generally achieves performance gains ranging from 1% to 13% in the Choss task and from 1% to 11% in the MMLU task. In code generation task, when combined with other methods, gains range from 2% to 7%. However, two notable exceptions are observed when integrated with the debate method with the Llama2-13B and Llama2-70B models, which result in failed cases. This failure in performance is attributed primarily to the noise generated by referencing the answers of other agents during the debate process. The synthesized responses, which incorporate input from multiple agents, disrupt the coherence of the code logic, leading to the observed performance degradation. All accuracy curves are provided in the Appendix B.1.
5.3 Effectiveness
From Table 3, we find that our method outperforms other methods in standalone cases, except on the Choss task using 1lama2-13B and Llama2-70B. Additionally, based on the data from Table 3, we have calculated the average performance ranking of each enhanced method across various tasks, with the results presented in
7

---

Published in Transactions on Machine Learning Research (10/2024)
Table 4. Notably, without the need for additional prompts complex LLM collaboration frameworks, method achieves the highest average ranking across different LLMs and tasks.
Table 4:
Our method achieved the highest average ranking across different. LLMs and tasks. Rankings are derived from Table and are based on the average rank each method achieves across all five tasks for a given LLM. The bolded instances indicate the top ranking.
Method +Ours
GPT-3.5
70B
13B
Overall
COT Wei al. (2022)
2.8
3.6
3.6
3.3
ZS-COT Kojima et al. (2022)
2.8
2.4
3
2.7
SPP Wang et al. (2023e)
3.8
Debate Du et al. (2023)
3.8
4.4
5
4.4
Reflection Shinn et al. (2023)
3.3
Ours
2.6
2.2
2.5
5.4 Robustness
We conduct. ablation studies to evaluate the impact of changes various hyperparameters on the final performance. The experiment is conducted by altering the temperature Ficler Goldberg (2017) and the nucleus probability p Radford et al. (2019), using the GPT-3.5-Turbe model over an average of 20 runs. As shown Figure scaling up ensemble size improves the LLM performance consistently across different tasks, despite the variation of these hyperparameters.
5.5 Token usage
We record the token usage for different methods, with the Table presenting the token usage for a single agent. Given that method scales up by increasing the ensemble size, the token usage increases proportionally with the number of agents when combined with other methods. When addressing specilic lasks, one trade a higher token budget for improved performance. More details are presented in Appendix B.3
Table 5: Token usage of GPT-3.5
Method
GSMSK
MATH
Chess
MMLU
HumanEval
Ours (single agent)
235 + 54
326 131
138
14
247 + 91
495 + 120
COT Wei et al. (2022)
+ 63
360
134
284
76
330 + 110
330 + 116
ZS-COT Kojima et al. (2022)
228 + 56
305
: 122
132
= 12
230 + 92
305 : 132
SPP Wang et al. (2023e)
341 + 85
471
161
363
35
428 + 123
501 + 125
Reflection Shinn et al. (2023)
214 + 45
281 + 96
146 13
218 + 80
338 1 110
6 Understanding the Performance Gains
Table 2 shows that the efficacy of our method varies with the difficulty of the task. In this section, we aim to understand the underlying properties through controlled experiments.
To start the analysis, select two datasets with increasing difficulty, i.e., GSMSK and MATH, to calculate the relative performance gain. The relative performance gain is given by: Pu-Pa where Pm and P are the performances (accuracy) with our method and single LLM query, respectively. The results are shown in Table 6.
It is noteworthy that the relative performance gain is more substantial with increasing task difficulty. Specifically, we observe that within the same task, the smaller model, 2lama2-13B, gains ranging from 28%-20%, but only 8%-16% over GPT-3.5-Turbo. Moreover, the more challenging task MATH yields gains of 34%-200%, in contrast to only 16%-69% the easier task GSMSK.
8

---

Published in Transactions on Machine Learning Research (10/2024)
Table The relative performance gain (%) becomes more significant when the relative difficulty between the LLM and the task increases. is calculated basod on Table
Task
Llama2-13B
Llama2-70B
GPT-335-Turbo
GSM8K (easy)
69
37
16
MATH (hard)
200
120
34
further analyze this correlation in detail, we calegorize the difficulty a given task inlo three orthogonal dimensions: 1) the inherent difficulty of the task; 2) the number of steps required solve the task; 3) the prior probability of the correct answer. To investigate these dimensions, conduct experiments that isolate each dimension. And then, delve into each dimension in detail.
Inherent difficulty
Step 1
hard
Step
Step 3
Step
Step 5
FALSE
FALSE
TRUE
FALSE
TRUE
FALSE
Prior Probability of the correct answer
Figure 5: Illustration of three dimensions for a given task. Nodes represent steps, while dashed lines indicate alternative potential steps. The depth of nodes represents the number of steps, and the color intensity represents the level of inherent difficulty.
6.1 Isolation
To explicitly explore the impact of these dimensions, we conduct a mathematical task designed to isolate each one. Consider the task detailed below:
Find the interval A such that Σ by E As.
(1)
where:
4, are randomly chosen integers from the closed interval [-1, I defines the range of integers. 7 represents the inherent difficulty of the question. A larger value of T indicates more challenging task.
S Z is the number of terms in the summation. S represents the number of steps required to solve the problem. A larger value of S indicates more challenging task.
The result space is partitioned into K intervals A A....., AR of equal probability. K Z denotes the number of these intervals. 1/K represents the prior probability the correct answer. lower prior probability indicates challenging task.

---

Published in Transactions on Machine Learning Research (10/2024)
In the following experiments, we analyze each dimension respectively based on GPT-3.5-Turbo. Note that we use GPT-3.5-Turbo for a case study, it can also be changed to other backbone models. The relative performance gains are measured by the difference between the maximun accuracy our method can achieve (sampling 40 times) and the accuracy of single LLM query (sample once). Results are averaged over 10 runs.
6.2 Inherent Difficulty
Property 1: Gains increase then decrease by rising the inherent difficulty. We investigate the inherent difficulty by varying from 10 to 400, while keeping the values of S and K constant across four groups of different values, from small to large, respectively. Figure 6 (left) shows an initial uptick in performance gains with increases in I, indicating that our method can significantly enhance performance in line with rising inherent difficulty. The most notable gains are seen at / = 100 and I = 200, consistent across all $ and K settings. Yet, at I = 400, gains taper off, implying that excessive complexity may exceed the model's reasoning capabilities, leading to diminishing returns for our method under extreme task difficulty.
Relative Performance Gains
Relative Performance Gains
Absolute Performance
50
50
|-10 K-2
90
|-10 K-4
Percentage %
40
S=1 K=2
Percentage
40
1-100
1=100
70
80
30-
S=1
30
Accuracy
S=1 K=8
50
I=10 S=1
20
2=2 K=2
20
40
I=10 Sm4
1=200 S=1
10
10
30-
200 S-4
20:
10
100
200
400
1/32
1/16
3/10
(Inherent Difficulty)
5 (The number of reasoning steps)
1/K (Prior Probability)
Figure 6: (Left) The relative performance gains increase and then decrease with rising inherent difficulty. (Middle) The relative performance gains increase with the number of steps. (Right) The absolute performance increases with the prior probability. We analyze each dimension by fixing the other two dimensions. The error bars represent the standard error.
6.3 Number of Steps
Property 2.1: Gains increase with the number of steps. We analyze the mumber of steps by isolating S. We lune S from to 8, while keeping the values of and K constant across four groups of different values, ranging from small large, respectively. Figure 6 (middle) shows that as the number of steps increases, there is corrosponding increase in performance gain. Additionally, we find that when I and K are increased (which indicates higher difficulty), the performance gains more significant, e.g., 4%-18% gains over [T= 10, K = 2} compared to 16%-48% over (T 100, K 4).
Property 2.2: Agent Forest increases the performance for each step. We conduct a fine-grained analysis for each step of a given task. We explicitly prompt the language model to output the result of each step. Subsequently, we utilize Agent Forest at each step to derive the answer for that step. Figure 7 (left) shows that although each step has equal inherent difficulty, the accumulation of errors from previous steps lead to decrease in accuracy the number of steps increases. However, our method mitigates the performance decrease encountered with increasing steps.
Derivation. Based on Property 2, we propose Step-wise Agent Forest, which can further enhance the performance.
Step-wise Agent Forest initially prompts the LLM to decompose the task into multiple stops. It then proceeds with multi-round iterations to produce the final result. In each round, the process begins by selecting a current unprocessed step and using Agent Forest to determine the result. of that step. Subsequently, it uses
10

---

Published in Transactions on Machine Learning Research (10/2024)
Accur acy different steps
Performence gains
Absolute Performance
100
sample once
Agent Forest
Agent Forest
gains of sample 40 times (ours)
Step- sisp-wise Agent Farest
Hierarchical Agent Forest
90
Percontage (%)
30
20
50
20
10
40
homogeneeus heterogeneous
Steps
(Inherent Difficulty)
Figure 7: (Left) Our method increases the performance for each step. Blue bars show the accuracy of various steps for single sample, and orange bars show the gains for 40 samples. (Middle) Step-wise Agent Forest can further enhance the performance across different levels of inherent difficulty. (Right) Hierarchical Agent Forest can further enhance the performance with homogencous and heterogencous model combinations.
the result to update the task. This iterative process is repeated multiple times until the last step is processed. To evaluate the performance of Step-wise Agent Forest, we fix S = 8 and K = 4, and tune / from 100 to 400. Figure (middle) shows that compared to Agent Forest, Step-wise Agent Forest yields greater improvements. e.g.; we see 15%-42% gains, which increase with inherent difficulty.
6.4 Prior Probability
The performance increases with the prior probability. We investigate the influence prior probability on performance by tuning the parameter K, while maintaining constant values for I and K. As K represents the number of intervals, the prior probability defined as 1/K. We vary K from equivalently alters the prior probability from 1/4 to 1 1/32. Through four experimental groups illustrated in Figure (right), each characterized by different configurations of I and S, we find that as the prior probability increases, so does the performance.
Property 3:
32, which
Derivation. Based on Property 3, we propose Hicrarchical Agent Forcst can further enhance the performance.
As the performance is related to the prior probability, decomposing low-probability lasks into multiple high-probabilityy subtasks and addressing them hierarchically can boost performance. Moreover, subtasks with varying prior probabilitics can be addressed using different models. Additionally, cost savings can be achieved by using simpler, less expensive models for the easier, nigher-probabiliy subtasks.
In our experiments, the task is to solve the problem with K = 32. GPT-3.5- is used in homogeneous combination experiment and GPT-3.--Turbo and GPT-4 are used in heterogeneous combination experiment. The results are presented in Figure 7 (right).
In homogencous combination experiment, by employing the hicrarchical method, we start with K= 8 obtain intermediate answer and then find the solution with K 32, focusing on intervals identified by the intermediate answer. This method enhances the performance from 21% to 31%, demonstrating that the hierarchical method can further enhance the performance.
In helerogeneous combination experiment, GPT-335-Turbo is used for generating the intermediate answer with 8, and GPT-4 is then employed to solve for the final answer with K = 32. In Figure 7 (right), compared with the result of GPT-4 with K 32, the hierarchical method improves performance from 35% 47%, suggesting the deployment of different LLMs at the corresponding level of problem-solvin can improve the performance in cost-effective manner.
11

---

Published in Transactions on Machine Learning Research (10/2024)
7 Conclusions and Future Work
In this paper, we report that more agents is all you need, i.c., simply adding more instantiated LLM agents is what you need to obtain a better LLM performance in processing complex lasks, without bothering complicated methods, such as CoT pipelines, multi-agent collaboration frameworks, etc. We have conducted the first comprehensive study in the literature to understand such a "scaling law", including when holds and how to facilitate its occurrence.
The results indicate that Agent Forest generally improve the performance of LLMs by increasing the ensemble size. Importantly, this method is orthogonal to different existing methods, which can lead to further improvements when combined with them.
Furthermore, we observe that the performance gains are influenced by the difficulty of the task. To explore this correlation, we isolate and analyze three dimensions of task difficulty: the inherent difficulty, the length of reasoning steps, and the prior probability of a correct answer. We find that: 1) the performance gains increase then decrease by rising the inherent difficulty; 2) performance gains increase with the number of steps; and 3) performance increases with the prior probability. Based these properties, also develop ways to boost the elfectiveness of *More Agents".
Considering that each input remains the same when we increase the number of agents, the sampling phase can be optimized to reduce the cost. Nonetheless, such a challenge of escalating costs commonly exists in works requiring multiple LLM calls Wang al. (2023b); Du et al. (2023). This aligus with recent findings as discussed in Kapoor al. (2024) that current benchmarks focus narrowly on accuracy, leading costly agents. Additionally, the lack of adequate holdout sets and standardization in evaluation practices further complicates the development of cost-effective agents. Addressing these issues can enhance the practical efficiency of AT agents in real-world scenarios. We leave it as future work to optimize.
12

---

Published in Transactions on Machine Learning Research (10/2024)
References
Leo Breiman. Random forests. Machine learning, 45:5-33 32, 2001.
Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje F. Karlsson, Jie Fu, and Yemin Shi. 2023a.
Autoagents: A framework for automatic agent generation. CoRR,
Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. ColR, abs/2305.05176, 2023b.
Mark Chen, Jerry Tworck, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murali, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. CoRR, abss/210703377, 2021.
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents. CoRR, abs/2308. 10848, 2023c.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs abs/2110. 0ts//100..1116, 2021a.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs 0D1/2200..11688 2021b.
Yihun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. CoRR, 1//22/3551225555 2023.
Jessica Ficler and Yoav Goldberg. Controlling linguistic style aspects neural language generation. CoRR, abs/1700 07703, 2017.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity- based prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview net, 2023.
Dan Hendryeks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net net, 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Fric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sii-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NewrIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021b.
Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, and Chenglin Wu. Metagpt: Meta programming for multi-agent collaborative framework. CoRR, abs/ abss/2308.0035. 2023.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Lim-blender: Ensembling large language models with pairwise ranking and generative fusion. In Anna Rogers, Jordan L. Boyd- Graber,, and Naoaki Okazaki (ods.), Proceedings of the 61st Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long
13

---

Published in Transactions on Machine Learning Research (10/2024)
Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 14165-14178. Association for Computational Linguisties, 2023.
Sayash Kapoor, Benedikt Stroebl, Zachary S Siegel, Nitya Nadgir, and Arvind Narayanan. Ai agents that matter. arXiv preprint wxXiv2240.015000 01502, 2024.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Twasawa. Large language models are zero-shot reasoners. In NewrIPS, 2022.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. CAMEL: communicttiv agents for 'mind' exploration of large scale language model socicty. CoRR, abs/2303.11766) 2023a.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian- Guang Lou, and Weizhu Chen. Making language models better reasoners with step-aware verifier. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 5315-5333, Toronto, Canada, July 2023b. Association for Computational Linguistics.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking large language models through multi-agent debate. CoRR, 0///223051991 2023.
Lei Lin, Jia-Yi Yi Fu, Pengli Liu, Junchen Wan, Fuzheng Zhang, Zhongyuan Wang, Di Zhang, and Kun Gai. Ask one more time: Self-agreemenn improves reasoning of language models in (almost) all scenarios. CoRR, abs/233 abs/2231.08114, 2023.
Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. Dynamic Ilm-agent network: An llm-agent collaboration framework with agent team optimization. CoRR, abs/2310.00 22100.2270, 2023.
Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing to the expert: Efficient reward-guided ensemble of large language models. CoRR, abs/2311. 1. 1.06622,, 2023.
Xiaoding Lu, Adian Liusie, Vyas Raina, Yuwen Zhang, and William Beauchamp. Blending is all you need: Cheaper, better alternative to prillion-parameters 1lm. arXiv preprint arXiv:2401.02999 2024.
OpenAI. Chatgpt: Optimizing language models for dialogue, 2022. URL https:/ www com/blog/ chatgpt.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40st Annual Meeting of the Association for Computational Lin- guistics ACL 2002, Philadelphia, USA, July, 2002, pp. 311 318. Association for Computational Linguistics, 2002.
Matt Post. A call for clarity in reporting bleu scores. arxiv preprint arXiv:1804.08771, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Thirty-seventh Conference Neurol Information Processing Systems, 2023.
Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin. Large language model routing with benchmark datasets. CoRR, abs/2309.15788. 2023.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslic Baker, Yu Du, YaGuang Li, Hongrae Lec, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, Igor
14

---

Published in Transactions on Machine Learning Research (10/2024)
Meredith Ringel Morris, Tulsce Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthow Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Agūera y Arcas, Claire Cui, Marian Croak, Ed H. Chi, and Quoe Le. Lamda: Language models for dialog applications. CoRR, abs/2201.0822 0.0019,, 2022.
Krivokon, Will Rusch, Marc Pickett, Kathleen S.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blocher, Cristian Canton- Perrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswamii Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.000 00088, 2023.
Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge fusion of large language models. arXiv preprint vxXi:2201111440) 2024.
Hongyi Wang, Felipe Maia Polo, Yuekai Sun, Souvik Kundu, Eric P. Xing, and Mikhail Yurochkin. Fusing models with complementary expertise. CoRR, abs/ abs/2310.01542, 2023a.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023b.
Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Hong Ji. Unlcashing cognitive synergy in large language models: A lask-solving agent through multi-persona self-collaboration. CoRR, abs/ 20000000005,00, 2023c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. Chain-of-thought prompting clicits reasoning in large language models. In NewrIPS, 2022.
Qingyun Wu, Gagan Bansal, Jicyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Aulogen: Enabling next-gen LLM applications via multi-agent conversation framework. CoRR, 018/2308.0810 2023.
Kai Xiong, Xiao Ding, Yixin Cao, Ting Liu, and Bing Qin. Examining the inter-consistency of large language models: An in-depth analysis via debate. CoRR, abs/2305. 11595, 2023.
Jintian Zhang, Xin Xu, and Shumin Deng. Exploring collaboration mechanisms for 1l agents: A social psychology view. Xiv preprint arXiv:2310.02111 2023.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Yian-Yun Nie and Ji-Rong Wen. A survey of large language models. Xiv preprint orXiv:230s 18223, 2023.
15

---

Published in Transactions on Machine Learning Research (10/2024)
A Detailed Experiment Settings
A.1 Common Settings
In all experiments involving GPT-3.5-Turbo presented in Section 4, we utilize the model version In Table 2, the notation GPT-4 corresponds the model version gpt-4-0613. For the experiments conducted with GPT-3.5-Turbo in Section 6, we employ the model version 1pt-3.6-turbo-111 with the JSON mode enabled. Similarly, GPT-4 in this context. refers gpt4-1106-Prevae operating in JSON mode.
A.2 Experiments Arithmetic Reasoning Tasks
For the implementation of Agent Forest. to arithmetic reasoning (asks within the GSMSK and MATH datasets, exccute the initial sampling phase by Algorithm 1. Samples are extracted from the responses by matching "boxed([X] (X where "X" denotes numerical value mathematical expression.
In the subsequent voting phase, to evaluate the similarity between samples, as outlined in Algorithm 1, we employ mathematical equivalence comparisons for each sample. The most probable sample is chosen as the final answer. This answer subjected to a comparison of mathematical equivalence with the ground truth to ascertain the correctness of the result.
A.3 Experiments on General Reasoning Tasks
For general reasoning tasks, as encountered in the MMLU and Chess datasets, Agent Forest is applied following Algorithm 1 during the sampling phase. Samples are extracted by matching the pattern "(X" or "(X)", where "X" corresponds to the options A, B, C, or D in MMLU task and the chessboard position in Chess task.
During the voting phase, we calculate similarity by counting the frequency of each option's occurrence within the samples. The most frequently occurring option is then chosen as the final answer. This selected answer is compared with the ground truth to determine the accuracy of the result.
A.4 Experiments on Code Generation Task
In the code generation task, we apply the Agent Forest method to produce Python code using the HumanEval dataset. During the sampling phase, we extract Python code snippets from the model's responses.
In the voting phase, we compute the BLEU score using sacreBLEU Post (2018) to evaluate the similarity between each the generated samples. The sample with the highest cumulative BLEU score is selected as the final answer. This method ensures that the final output is the most representative and accurate piece of code determined by consensus through similarity scoring among the samples.
B Detailed Experiment Results
B.1 Additional Accuracy Curves
In this section, we provide the accuracy curves of our experiments across various datasets when utilizing different LLMs. From these curves, we demonstrrte that our method has the following properties:
Generalizability. By using method standalone, the performance be generally enhanced by increasing the ensemble size.
Compatibility. Our method can generally enhance other methods by increasing the ensemble size.
16

---

Published in Transactions on Machine Learning Research (10/2024)
GSMBK
MATH
Chess
MMUU
HumanEval
0.880
0.25-
0.08
0.50
0.50
0.23
0.20-
0.06
0.40
Duts
0.20
0.30
SPP Ours
25-C0T Durs
0.04
0.40
0.10-
Rellection Durs
0.15
0.05
20
40
10 Ersemble
30
40
0
10
20
30
40
10
30
40
23
30
Size
Erisembre Size
Enisem
Figure Accuracy curves across various datasets using the Llama2-13B model.
GSMBK
MATH
Chess
MML1
HumanEval
6.12
0.25
0.70
0.19
0.60
#
0.20
0.55
0.25-
SPP Ours
25 COT Ours
0.06
0.15
0.50
0.20-
0.50
Nuflection Dur
0.45
0.02
0.10
20
40
10
30
20
40
10
30
40
30
20
30
Ensemble
Ensombic
Ensombic
Figure 9: Accuracy curves across various datasets using the Llama2-70B model.
GSMBK
MATH
Chess
MML
HumanEval
0.85
0.40
0.55
0.65
0.75
0.50
0.60
0.70-
+
25 COT Ours
0.45
0.65-
SPP Durs
Reflection Ours
0.30
0.45
0.70
0.25
0.40
20
45
10
30
0
20
40
10
30
20
40
Figure 10: Accuracy curves across various datasets using the OPT-3.5-Turbe model.
GSMBK
MATH
Chess
MMLU
Hume HumanEval nEval
0.19
0.07
0.18
2.16
0.15
0.45
0.44
0.05
0.16
Debate Ours
0.15
0.05
Ourr
0.03
0:38
0.35
10
10
Figure 11: Debate accuracy curves across various datasets using the Llama2- 13B model.
B.2 Statistical Test Results
In this section, we performed a one-way ANOVA to determine if there are significant differences accuracy across different ensemble sizes (1, 10, 20, 30, 40), across 10 independent runs. The detailed p-values are provided in Table where all p-values are less than 0.05, demonstrating that the differences in accuracy between ensemble sizes significant.
17

---

Published in Transactions on Machine Learning Research (10/2024)
GSMBK
MATH
Chess
MMLU
HumanEval
0.68
0.90
0.25-
0.63
0.16
9.59
0.20
0.62
0.58
0.15
0.60
0.15
0.57
6.10-
0.58
Debele
0.56
0.55
0.14
0.55
5
10
10
8
10
Ensemble
Ersenblee
Erowmb
Enuemble Size
Figure 12: Debate accuracy curves across various datasets using the Llama2-70B model.
GSMBK
MATH
Chess
MMLU
HumanEval
0.54
0.56
0.82
0.80
0.34
0.64
0.50-
0.60-
0.78
0.32
Ours
Debata Ours
0.50
0.58
0.20-
0.50
10
2
10
10
10
Size
Figure 13: Debate accuracy curves across various datasets using the GPT-3.5-Turb model.
Table 7: Statistical test results (p-values) of different LLMs on different datasets. One-way ANOVA is performed where ensemble sizes are (1, 10, 20, 30, 40).
LLMs
GSMSK MATH
Chess
MMLU
HumanEval
Llama2- 13BB
1.56e-44
9.11e-32
1.97e-18
7.86e-38
5.46e-8
Llama2-70B
1.93e-43
2.16e-29
1.09e-9
1.06e-26
5.24e-7
GPT-3355- Turbo
6.03e-30
9.66e-39
4.62e-24
2.46e-39
7.47e-13
0.40
0.59
0.30
0.60
0.55
0.50
0.50
Lumma2130 2-15
0.30
0.30
0.40
Liuma2-105
0.20
2000
0.50
1003
2003
3000
4000
530
1000
1302
1050
2000
2020
4000
5.60
0.70
0.50
0.50
0.70
0.40
0.50
0.50
0.40
Libma2-130
0.10
0.20
0.40
Liama2-105
2300
5000
7500 10000
0.00
3000
10000 MMken Usags
25000
2003
4003
6000
3000
10000 Token Usage
5000 10020 15000 20303 Tokan Usage
Usage
Usage
Figure 14: Token usage accuracy.
B.3 Token Usage vs. Accuracy
Figure 14 shows the tradeoff between token budget and accuracy, where the x-axis represents the token budget and the y-axis represents the accuracy.
18


Reviews:
[{"review_id": "tMI8nVY0QG", "reviewer": "Reviewer_gVUG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The proposed method is simple and clearly explained.\n\t\n- The evaluations seem reasonably comprehensive, covering three categories of reasoning tasks and considering both Llama and GPT models of varying sizes.\n\t\n- The majority-vote trick is complementary and easy to incorporate into a variety of existing methods\n\t\n- Even the simplest variant of ensembling to which the method is applied, self-ensembling, produces notable improvements via the majority-vote strategy. \n\t\n- The sweep over ensemble sizes in e.g. Figure 4 is sufficient to conclusively show benefits to scaling the number of ensemble members as well as limitations where the performance gains saturate.\n\n### Weaknesses\n\n- The percentage improvements in Figure 6 are quite small and there are no error bars to indicate whether the trends observed are real or due to random noise. I would like to see a significance test done on the claim that the improvement at I=400 is significantly smaller than I=100 or 200.\n\n- More generally, it would be helpful to see error/standard deviation bars just to get a sense of the significance of the results.\n\n- While the additional investigations into the effect of different aspects of problem difficulty on the performance gains from the method were helpful, I would have liked to see the qualitative investigations focus more on explaining why there are such disparate improvements across the different domains. For example, the chess dataset sees relatively minimal gains, especially on Llama models — is this because the dataset is too hard? Or is it because of the output and reasoning structure that makes it a bad fit for the majority-vote similarity measure?\n\n- In section 6, it is argued that harder tasks see greater benefit from the method, but it isn’t clear to me whether this statement is a generic truth or in part an artifact of how relative performance gain is computed (for example, going from 25 to 50% accuracy is a greater improvement under this metric than improving from 50 to 99%). If instead of measuring gain as relative increase in accuracy, we instead modeled it as relative decrease in error, would the harder tasks still demonstrate greater improvement? \n\n- The term ‘prior probability’ is a bit confusing — does this refer to a uniform uninformed Bayesian prior over feasible answers to a subproblem, or does it implicitly refer to the LLM’s probability of generating the correct answer without following chain of thought reasoning first?\n\n### Questions\n\n- Did you consider alternative choices for the majority-vote clustering metric? Do different choices give different performances?\n\n- The main difference between this approach and that of Wei et al. is its broader application of majority-vote beyond CoT reasoning tasks to code generation and to intermediate outputs at CoT steps. While the latter does help reduce the risk of compounding errors in long-range reasoning tasks, it also seems like it could run the risk of prematurely committing to an incorrect reasoning step if one type of reasoning error produces outputs that are all extremely similar based on their BLEU scores. Could the authors comment on when / if ever such a situation might occur?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the effect of increasing the ensemble size of a variety of LLM sampling methods on performance. It proposes a simple majority vote strategy which leverages the similarity of generations to aggregate proposed outputs in natural language tasks with combinatorially large output spaces.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WGkZWMfcLj", "reviewer": "Reviewer_EUQF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\nThis paper studied the capability of integrating multiple results from LLMs.\n\n**Weaknesses**:\n\nI think this paper just simply generate multiple outputs by using LLMs and then ensemble results. The core idea of this paper is still the thinking of ensemble learning, and I don't see any other contributions in term of methods. Get multiple results and then using voting strategy to improve performance is not a novel idea. And this paper just uses LLMs to get results. Moreover, I think this paper just applied prompting to LLMs but claim it as LLM-based agents. I think this is an over-claim. Actually, this paper just use LLMs or LLMs plus different prompting strategies. I didn't see any unique insights in agents.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper mainly studied how to use LLMs to generate multiple results and then ensemble it to get better performance. Experimental results demonstrate the effectiveness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see my above comments.", "broader_impact_concerns": "This paper does not include broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4mzsfFbPVN", "reviewer": "Reviewer_EgWB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# strengths\n- Presents a simple method with strong results across many domains.\n- Analyzes the relative benefits of ensembling by task difficulty and base model scale / capability, providing new knowledge about how LLM inference structures scale relative to tasks.\n- Provides clear evidence of the value of majority voting ensembles to complement other methods, showing useful generality. General, simple methods are welcome additions to the LLM design toolkit.\n\n# weaknesses\n- Statistical rigor: there are no error bars in plots/tables or statistical tests of differences.\n\t- Reporting statistical variability is important to quantify the benefits of the method. My concern is not that the magnitude be large, but that the tests are missing altogether.\n- Lacks a cost-matched comparison of methods.\n\t- Ensembling incurs greater inference costs (measurable as number of tokens inferred) from a base LLM. This means adding more agents to an ensemble is an \"unfair\" comparison to models that use a fixed, smaller budget (like bigger LLMs).\n\t- Ideally the analysis would include reporting on model performance vs tokens produced. How do ensembling performance gains scale relative to token consumption? Where do the other methods compared fall along those same dimensions?\n\t- This will provide a more \"fair\" comparison and better clarify what the cost-benefit trade-offs are to ensembles. They clearly are beneficial in inference _time_ requirements: majority voting can be fully parallelized, unlike techniques like chain-of-thought or debate. This may also be useful as a comparison, though of secondary concern compared to the token consumption requirements.\n- The analysis of difficulty facets feels unrelated to the main paper.\n\t- It would help to show the step-wise and hierarchical methods can apply to at least one of the base problem domains (that is not the toy example).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "# contributions\n- A systematic exploration of how ensembling LLMs (using parallel answering and majority vote selection) can improve LLM performance across a variety of task domains including multiple choice question answering and code generation.\n\t- Includes using BLEU scores to implement majority voting for open-ended text-based outputs (specifically code).\n- An analysis of task difficulty components and how they influence ensemble scaling in a toy case study.\n\n# new knowledge\n- Simply using multiple agents together with majority voting can substantially improve LLM performance on multiple choice reasoning and code generation tasks. LLM performance scales in (simple) ensemble size.\n- Ensembling benefits (mostly) complement reasoning methods that augment base LLM predictions (like Chain-of-Thought prompting), including multi-agent approaches (like Debate).\n- Weaker LLMs gain more from this ensembling than stronger LLMs.\n- Harder tasks benefit more than easier tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# critical\n- Add error bars to all plots and tables, along with statistical tests of differences among methods (where multiple runs are available).\n\t- The current evidence is favorable, but without variance measures (LLMs are often high variance) it is hard to know how strong the results are.\n- Include reporting on token costs vs performance for the experiments.\n\t- This would directly address the question of how \"smart\" ensembling is: does it get more or less improvement in performance when consuming the same fixed inference (token) budget?\n\t- This will help characterize the scaling trade-offs being made with ensembles. Even if ensembling is more expensive for a given performance level it is still valuable knowledge to contribute and offers a meaningful baseline that other methods will need to show they outperform (in the future).\n\t- For example: a scatter plot showing the token usage vs performance for the different ensemble sizes and combined techniques. \n\t\t- This would be a plot using the data from something like Table 2 or Figure 3. \n- Apply the stepwise and hierarchical methods to one or more of the main domains in the text (not just the toy domain)\n\t- This would make the paper an obviously very strong result and reinforce the value of the difficulty facet analysis.\n\n\n# strengthen\n- \"Additionally, the experimental results indicate a correlation between the efficacy of the performance improvements and the difficulty of the problems addressed.\"\n\t- Be explicit: there are greater performance improvements on more difficult tasks and for weaker baseline models. \n\t- No reason to be vague with terms like \"correlation\" (positive or negative?) or to leave readers in suspense.\n- Figure 1\n\t- How do the larger ensembles differ to the baseline in token costs?\n\t- This additional information would help when considering what a \"fair\" comparison might be for the amount of inference used by different methods.\n- Table 2\n\t- How many agents are in the \"Ours\" ensemble? 40? This was suggested but not explicitly stated in the text (since it mentions scaling up the ensemble size while this is a single number)\n- Figure 3\n\t- This would be a stronger figure if it included all the ensembles of size 1 to 10. That is the point of steepest increase of the curve, so the extra detail would help understand how rapidly the benefits of ensembling diminish. This is clearly sharp in GSM8K and MMLU, but not obviously so in MATH, Chess, or HumanEval.\n\t- I ask as this would better characterize how ensembling helps on different tasks, informing others who may want to repeat these experiments with smaller scale on newer domains.\n- Section 5.1\n\t- Any idea why Chess shows such small gains (1% to 4%) compared to other domains?\n\t\t- Can this be related to the difficulty facets of steps or prior probability of success (or inherent difficulty, though I'm not sure how to measure that)?\n\t- Table 3 shows this domain to be challenging for \"Ours\" across base models.\n- Figure 4\n\t- Please include the default T and p values used in the caption for easy reference.\n- Section 6\n\t- It may be worth explicitly mentioning in the text that the weaker base models have greater performance gains. This is a valuable result.", "broader_impact_concerns": "The broader impact statement is generically about LLMs, but this is because the work is a very general technique.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the methods proposed in this paper are not particularly novel themselves (ensembling being a well-studied method in ML for decades), their application to agents and demonstration of effectiveness make this paper interesting and useful to the TMLR audience and will inform future work in this space. The authors clarified questions and improved the paper based on reviewer suggestions. 

The one recommendation I have for the authors is to include some concurrent work along similar lines (like https://arxiv.org/abs/2407.01502) in the discussion so that the readers may gain a more comprehensive understanding of the core ideas in the paper.
Decision: Accept (poster)