Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains
Authors: Vighnesh Subramaniam
Yilun Du
Joshua B. Tenenbaum
Antonio Torralba
Shuang Li
Igor Mordatch
Venue: ICLR 2025 Poster (2025)
Keywords: Language Models
Multi-agent Interaction
Self Improvement
Primary area: foundation or frontier models, including LLMs
TL;DR: We illustrate how multiagent finetuning can improve language models

Abstract: Large language models (LLMs) have achieved remarkable performance in recent years but are fundamentally limited by the underlying training data. To improve models beyond the training data, recent works have explored how LLMs can be used to generate synthetic data for autonomous self-improvement. However, successive steps of self-improvement can reach a point of diminishing returns. In this work, we propose a complementary approach towards self-improvement where finetuning is applied to a multiagent society of language models. A group of language models, all starting from the same base model, are independently specialized by updating each one using data generated through multiagent interactions among the models. By training each model on independent sets of data, we illustrate how this approach enables specialization across models and diversification over the set of models. As a result, our overall system is able to preserve diverse reasoning chains and autonomously improve over many more rounds of fine-tuning than single-agent self-improvement methods. We quantitatively illustrate the efficacy of the approach across a wide suite of reasoning tasks.

Published as a conference paper at ICLR 2025
MULTIAGENT FINETUNING: SELF IMPROVEMENT WITH
DIVERSE REASONING CHAINS
Vighnesh Subramaniam
Yilun Du Harvard University
Joshua B. Tenenbaum
MIT CSAIL
MIT CSAIL,, BCS, CBMM jbt @mi edu
vsub851 @mit edu
yduêseas harvard. edu
Antonio Torralba
Shuang Li!
Igor Mordatch UC Berkeley
MIT CSAIL
Stanford University
torralbaemit.edu
Lishuang@stanfordd.ed
mozzlatch@beereleyycco
ABSTRACT
Large language models (LLMs) have achieved remarkable performance in recent years but are fundamentally limited by the underlying training data. To improve models beyond the training data, recent works have explored how LLMs be used to generate synthetic data for autonomous self improvement. However, successive steps of seff-improvement can reach a point of diminishing returns. In this work, we propose complementary approach towards self-improvement where finetuning is applied to a multiagent society of language models. group of language models, all starting from the same base model, are independently specialized by updating each one using data generated through multiagent interactions among the models. By training each model independent sets of data, we illustrate how this approach enables specialization across models and diversification over the set of models. As result, overall system is able preserve diverse reasoning chains and autonomously improve over many more rounds of fine-tuning than single-agent seff-improvement methods. We quantitatively illustrate the efficacy of the approach across wide suite of reasoning tasks.
Project website at https: /
htt.:li...gett............. github.io
INTRODUCTION
Recent breakthroughs in large language models (LLMs) like GPT-3.5 and GPT-4 have demonstrated remarkable proficiency in language generation, comprehension, question answering, and transla- tion (OpenAI, 2023; Touvron et al., 2023). Despite these advancements, LLMs are fundamentally constrained by the data they are trained on, with existing models already using much of the available data the Internet (Brown et al., 2020). To further enhance the performance of LLMs, recent research on ref--improvement, where LLMs generate additional synthetic data on which they are trained on (Huang et al., 2022; Yu al., 2023).
One approach to increase the data available to LLMs is to use powerful existing frontier models like GPT-4 to generate additional supervisory data. However, this approach is limited by the inherent quality of frontier models, preventing models from becoming better than the frontier of what the best existing models can accomplish. addition, such an approach incurs high financial costs due to inference expenses of such large models and is also often legally prohibited with existing commercial-grade models.
An alternative approach is to directly leverage existing language models to generate additional synthetic data for their self-improvement (Zelikman al., 2022; Bai et al., 2022; Chen et al., 2024b; Yuan et al., 2024). In such works, language models are used to iteratively collect data that they are then finetuned on. However, as models are repeatedly trained, performance gains often plateau relatively quickly as diversity decreases (Figure 1) and the self-inprovement loop is often only
"Equal Contribution, Corresponding authors
I Equal advising

---

Published as a conference paper at ICLR 2025
Phi-3
Mistral
LLaMA-3 (8B)
70.0
30
Mulliaget FT (Duns)
Singlesgennt
Accuracy
62.5
24
Mutt FT (Ours)
60.0
95
Single-gent
18-
53.5
Multagent FT (Ours)
Single
50.0
Iterations of finetuning
Iterations of finetuning
Iterations of finetuning
Figure I: Multiagent finetuning improves reasoning performance over multiple rounds of finetuning. Our multiagent finetuming procedure enables models to improve across multiple iterations of finetuing. Results reported on the MATH dataset.
run for two or three rounds (Lu et al., 2023; Song et al., 2024). This limits the applicability of self-improvement autonomously improve language models, models only be improved limited amount above their base performance.
In this paper, propose decreased gains of performance after multiple rounds of fine-tuning. Instead of fine-tuning a single model, our method finetunes a multiagent then independently specializes each model to capture parts of a task of interest. Our key insight that by finetuning multiple models, we can encourage specialization and diversification across responses, which can enable consistent performance gains over many rounds of fine-tuning, To achieve specialization between models, we fine-tune each model repeatedly on independent subsets of the generated data corresponding to responses from the respective particular model.
approach self-improvement that can help mitigate the issue of
of language models from the same base model and
Within our multiagent set of models, we propose to specialize models into distinct functionalities within the output generation procedure. First, we specialize set of models to be generation agents that produce a set of initial responses given queries. Since initial responses can often suboptimal, especially for challenging reasoning tasks, we further propose to specialize set of models critic agents that evaluate and refine the generations of other models. By using this set of distinct models in combination through multiagent debate (Du et al., 2023), are able to construct robust feedback loop for generating final responses, with experiments on other multiagent methods Appendix D.
By training each model on distinct sets of data and roles, approach fosters specialization across models and promotes diversification within the society of models. Consequently, our system can au- tonomously improve over many more rounds of finetuning compared to single-agent self-impprvement methods (Figure 1). We quantitatively demonstrate the effectiveness of our approach across a com- prehensive suite of reasoning tasks, illustrating significant performance gains, as shown Table In our experiments, we illustrate how our proposed method can be directly applied to both open-source LLMs such Phi-3, Mistral, and LLaMA-3 as well proprictary LLMs such as GPT-3.5 to substan- tially improve performance. In addition, the finetuned models can generalize to novel datasets and outperform the baseline methods trained directly on these new datasets.
Overall, our paper has the following contributions: (1) We propose to leverage multiagent interaction as an approach to eeff-improeement with language models. (2) We propose to specialize models with distinct roles to enable detailed feedback between agents and to improve the final output quality. (3) We quantitatively verify the applicability of our approach across a wide suite of reasoning tasks on both open-source and proprictary language models. (4) We demonstrate that the finetuned agents can generalize across different datasets a zero-shot manner.
2 MULTIAGENT FINETUNING OF LANGUAGE MODELS
We provide overview of our approach towards multiagent finetuning of language models, where we learn a multiagent society of models to accomplish a task. Our method involves two components. We first multiagent debate method to construct a finetuning dataset for raining models (though other multiagent generation methods can also be used, see Appendix Section D). We then introduce approach, multiagent finetuning, where specialize each L.LM model by finetuning each model
2

---

Published as a conference paper at ICLR 2025
Multiagent Debate
Finetuning Generation Models
Input x
The "corrcet" output of Model 1 after the first round of
The "correct" output Model
Round
after the first round multiagent debate V'
Model 1
Model n
Model N
multiagent debate 22
Finetuning
Finctuning
V1,
V1n
VI,N
Generation Model 4
Generation Model AG
Summarize the responses from other models
Finetuning Critic Models
Round
A mix of "correct and incorrect" output of Model after the m-th (m > 1) round of multiagent debate 1m.1)
mix of "correct and incorrect" output of Model N after the m-th (m 1) round of mullingent debule (ym,x
Model
Model n
... Model N
Round M
MM
YMa
YM
Finetuning
Finetuning
Critic Model Ah
Critic Model A
Majority Voting Result 9
Figure 2: Overview of Multiagent Finetuning, We first use multiagent debate and majority voting to create the finetuning datasets (left). These datasets are then used to finetune the generation and critic agents (right). When finetuning generation models, we use the majority voted result ("correct" output) select first-round responses from each agent. We then finetune critic models using responses from the final round based on whether responses match the majority voted result (mix of "corect and incorrect" outputs). The finetuned models are combined through multiagent debate to generate more accurate answers. In this figure, we illustrate a single finetuning iteration. Applying multiple rounds of finetuning iterations can significantly boost performance.
on its own generated data. An overview of our approach can be seen in Figure 2. an introduction of our multiagent dehate method in Section 2.1. We then discuss how to fine-tune a single model on generated data in Section 2.2, and the proposed multiagent finetuning in Section 2.3 and Section 2.4. We then show how to apply finetuned models for inference in Section 2.5.
We first provide
2.1 MULTIAGENT DEBATE
Multiagent debate (Du et al., 2023) involves a series of language model eitnts- copies or finetuned versions of the same model-each tasked with generating response to given problem. After the initial responses are generated, a debate round is initiated among the agents. In paper, concatenate and summarize the responses from other agents. Each agent is instructed to construct a new response based on its prior response and the summarized responses from the others. The final result is determined by majority vote based on the outputs from the last round of debate. The multiagent debate is illustrated in Figure 2.
either specific
2.2 FINETUNING MODELS ON GENERATED DATA
We start by considering how to use data generated by multiagent debate data to finetune a single LLM model for self-improvemment Given set of natural language inputs Dusk {x,), we a multiagent debate method (Du et al., 2023), specifically a debate with N agents and M rounds, generate responses for each input in D We obtain the final predicted output S for each o.; through majority voting in the last round of debate. We use this to construct a "ground truth" dataset of {z1 }.)]. In the single LLM model setting, we then finetune the model on the of generated responses y/ which match di given input I;.
While the final debate results ÿ are accurate, they often similar in style and methodology. As result, repeatedly capturing a dataset of {(x,, }i)) pairs for multiple rounds of finetuning often leads to a plateau of cef--improvement performance.
2.3 FINETUNING MULTIPLE GENERATION AND CRITIC MODELS
Our goal in multiagent finetuning is to create datasets that construct of models representing different agents that are diverse and accurately solve problems. Instead of building a single dataset to finetune each model, propose creating different datasets to finetune different models. A set of
3

---

Published as a conference paper at ICLR 2025
Algorithm 1 Multiagent Finetuning of Language Models
Require: A pretrained LLM A; A set of language inputs Dask = (2, }: The number of agents N; The number of dchate rounds M; The number of finetuning iterations L.
1: A A A # Copy the LLM build N generation agents
2: AT AN f- Copy the LLM build critic agents
3: Multippe Iterations of Finetuning
4: for = 1 + L do
5: # Multiagent Debate
6: for in Dink do Iterate over the input tasks m M if m = O then
7:
M rounds debate
8:
9:
y1,1,
VI,N < A9 (x) # Response of each generation agent
10:
else
mm.1
F f- Summarize the responses from other agents round m
12:
1m.1. Un (x A (2 Response of each critic agent
end if end for
14:
15:
Majority Voting {UM,,,. [MM,N # Responses of the final round of debate end for
16:
# Multiagent Finetuning
Initialize datasets for finetuning generation models (D h==1
19:
Initialize datasets for finetuning critic models (DA J for in do Iterate over
agents for x Dusk do # Iterate over the input tasks
21:
22:
D.d D. U ((2, v1,)) I Vı, = 0} # Add pairs
23:
DC + D. {(x, (yi... VM..)) V1.x * U: Y/t.n = /}
# Add pairs
24:
-. - ((x, (yI,,, .BM,,,)) Bi.n û. yM. 5}
# Add pairs
25:
D. + WD + (1 w # Combine the datasets
26:
end for
27:
A + Finetune An,, )
Finetune the generation model
S Finetune (An, D
Finetune critic model
end for Ai.
30:
Ag + AG AN Af
AA f Generation agent for the next finetuning iteration
AP
Critic agent for the next finetuning iteration
end for
models trained generation agents and others critic agents. The generation models produce initial responses to input questions. In contrast, the critic models assess the outputs from all generation agents and then select generate the most effective responses.
The role generation model is to generate accurate responses to input questions. Such models should rely on diverse reasoning chains to promote diversity. Generation agents AÇ are constructed from the N generation models which generate a response to the given input (we omit for simplicity). For each agent, select its outputs yn that match the final debate results ÿ and construct input-output pairs (z, yn). The resulting dataset for agent AC is DC = {(x yr) This approach generates of finetuning datasets {Df
Finetuning Generation Models.
DG } across all N agents. Each dataset contains different outputs, allowing for specialization and diversification of responses. We finetune each generation model with the corresponding dataset to get N correspondingly finetuned agents (AG. A
Finetuning Critic Models. The role of critic model is to further provide accurate critiques to responses from other agents and use these responses to provide an updated answer. Simply finetuning generation models isn't sufficient for achieving optimal results, especially for more challenging tasks, due to the lack of a feedback mechanism on their outputs. Critic agents A are constructed from critic models and evaluate the outputs from all generation agents and then select synthesize the best responses. This additional step ensures that the system continuously improves and adapts, enhancing overall performance.
4

---

Published as a conference paper at ICLR 2025
In the multiagent debate setting, each agent's output in the last round of debates is represented as Vai.ns where M denotes the number debate rounds. We first identify those outputs yMr that align with the final debate results These consistent outputs, together with the previous responses, then used to construct input-output pairs (I, (V1,,.....,UA,..))) -a,,,))
for finetuning the critic models.
To enhance the model's capability to correct incorrect answers generated early in the debate process, we sample a subset of pairs where V1,n differs from V. but VM.n matches ÿ and build a dataset DC = {(x, (BI....... y.M,n) 1y1.. * g. VMM. = 9)- This indicates that the an- swer successfully corrected by the end of the debates. We also construct another dataset DC ((x, (y1.m...... m)) Iy,, = y, JM.a = 5) where both y1.- and yM,a match ÿ, demon- strating the agent's ability maintain the correct answer throughout the debates. We combine these two datasets to create comprehensive finetuning dataset for each critic model to construct updated critic agents A
D =D + (1- w)
(I)
In the above expression, w is tunable hyperparameter representing the proportion of data sampled from the first set, while (1 - w) represents the proportion of data sampled from the second set. This method generates series of datasets (Df,-. ... D for finetuning the critic models, denoted as (Af
AA } after the finetuning process.
2.4 MULTIPLE ITERATIONS OF FINETUNING
The finetuned models are capable of generating responses through multiagent debate. We found that iterative application of the multiagent finetuning allows for continuous learning and adaptation, leading to progressively refined and more accurate responses over time. The finetuned generation agents {Ap Aς and critic agents {AY,.. ................................ .A } are used to gather datasets for the next iteration through multiagent debate. The algorithm for the proposed approach of iterations of finetuning is detailed in Algorithm 1. The steps for collecting data for finetuning the generation models are marked in red, and the finetuning of critic models is shown in blue.
2.5 INFERENCE
At inference time, we have a set of finetuned generation models which represent generation agents (A,...., A& ). and a set of finetuned critic models which represent critic agents {Af, AA ). We conduct a multiagent debate among these agents, where each individual generation agent participates in the first round of the debate, followed by each individual critic agent in subsequent rounds. Each agent takes the responses from all other agents and generates a new response in each round of the debate. We found that summarizing the responses from the other agents helps eliminate redundant information while retaining the most important details, thereby further improving performance. The final result is determined by a majority vote based on the responses from the final round of the debate. We provide pseudocode in Algorithm 2.
EXPERIMENTS
3.1 LANGUAGE REASONING TASKS
We evaluate our method and baselines on three language reasoning tasks.
Arithmetic. consists of 1,000 generated arithmetic problems in the form + b c d Following the generation procedure in (Du et al., 2023), each variable is assigned a random value up to maximum of 30.
Grade School Math (GSM). (Cobbe et al., 2021) consists of math word problems that require multi-step mathematical reasoning. Each example includes problem statement, the numerical answer, and an explanation the answer.
MATH. Hendrycks et al. (2021) consists of competition-leve math problems categorized into five difficulty levels. For our experiments, we sample problems from the first three levels.
For each dataset, randomly select 500 examples for finetuning the language model. Additionally, we select 500 held-out problems for evaluation. We parse the generated answers and evaluate their correctness by comparing them with the ground truth answers. Accuracy is reported based how
5

---

Published as a conference paper at ICLR 2025
frequently the model returns the correct answer. We also report the standard error of each accuracy value to measure the significance of improvement.
3.2 BASELINES
We compare the proposed method with various baselines. In all multiagent settings, we use three agents, and for all debate settings, we conduct two rounds of debates to ensure a fair comparison (additional results with five agents Appendix Section F).
Base utilizes single language model to process input and generate responses.
Majority is a multiagent baseline that selects responses based on majority vote from multiple agents. If no response secures majority, one of the potential answers is chosen at random.
Debate is multiagent debate baseline described in Du al. (2023). The debate structure is outlined in Figure 2.
STaR (Zelikman al., 2022) iteratively finetunes the language agent using a dataset with ground truth answers for each problem. Initially, the LM generates an answer for each problem, and correct responses, as verified by the ground truth, are added to the finetuning dataset. For problems answered incorrectly, the LM is reprompted with hint that includes the ground truth answer. Problems where the generated response includes the correct answer are added to the finetuning dataset. The LM finetuned on the collected dataset. This iterative process of building the dataset and finetuning is repeated until the finetuning loss saturates. The final model then used for evaluation.
Majority FT is a baseline that incorporates both majority voting and finetuning. We prompt the language agents with each problem and conduct a majority vote on their results. We then compile the responses from all agents that align with the majority vote, along with the input, to create finetuning dataset. The language model is finetuned using this dataset. Finally, we apply majority voting to the outputs of the finetuned model to determine the final answer.
3.3 QUANTITATIVE RESULTS
We compare baselines and our method, which was finetuned for only a single iteration (L= = 1), in Table 1. The accuracy and standard error for each dataset are reported. We use three distinct base language models: three open-source models, Phi-3 4B (Abdin et al., 2024), Mistral 7B (Jiang et al., 2023), and LLaMA-3 8B (Dubey et al., 2024); and one proprietary model, GPT-3.5 (OpenAI, 2022).
Our method outperforms all the baselines. Although "STaR" utilizes ground truth labels for data selection and undergoes multiple iterations of finetuning, still performs worse than our method, which uses only single finetuning iteration without access to ground truth. The "Majority", "Debate" and "STaR" methods outperform the "Base" model, demonstrating that majority voting, multiagent debate, and finetuning all contribute to improved performance. "Majority FT enhances the performance of "Majority" by incorporating a finetuning procedure. Our method only finetuned on 500 examples and still shows significant improvement the baselines, particularly more challenging datasets such as GSM and MATH. Additional evaluations on a larger set of problems and datasets can be found in Appendix Section H.
3.4 MULTIPLE ITERATIONS FINETUNING
To verify the effectiveness of multiple iterations of finetuning, as described in Section 2.4, we present the performance of our proposed method "Multiagent FT (Ours)" over five iterations of finetuning in Figure 1. We tested this method on two open-source models, Mistral and Phi-3, using the MATH dataset. The results demonstrate that "Multiagent FT (Ours)" consistently improves performance over time. For example, the accuracy of Phi-3 increased from 58.8% to 66.0%. and the accuracy of Mistral improved from 22.5% to 28.2%. Our method with five rounds of finetuning is 12.6% and 9.31% % more accurate than the best baseline listed in Table using Phi-3 and Mistral, respectively.
In contrast, finetuning a single agent ("Single-agent FT"), as described in Section 2.2, shows that performance saturates after one iteration of finetuning and starts dropping afterward, indicating potential overfitting to generated responses. This issue occurs when the single model, after several finetuning cycles, becomes fixated on small range of responses, which limits its diversity and

---

Published as a conference paper at ICLR 2025
LLM
Methods
Arithmetic
GSM
MATH
Base
81.99 + 0.99
75.60 + 1.36
46,83 2.25
Majority
94.40 = 1.03
81.20 + 1.24
51.40 + 2.23
Debate
98.21 = 0.54
83.30 1.18
55.73
GPT-3.5 (OpenAI, 2022)
STaR
98.38 0.57
83.60 1.17
53.00 :: 2.23
Majority FT
98.40 0.56
83.70 1.17
53.40 2.23
Ours
99.62 0.28
85.60 1.11
60.60 + 2.18
Base
88.30 + 1.44
81.20 + 1.74
45.60 + 2.10
Majority
91.80 = 1.23
81.80 1.72
47.20 1.82
Debate
96.20 + 0.86
84.40 1.58
53.40
2.28
Phi-3 (Abdin et al., 2024)
STaR
94.80 = 0.99
85.80 + 1.21
51.80 2.06
Majority FT
93.80 = 1.08
82.20 + 1.71
48.60 2.16
Ours
99.40 = 0.34
88.60 + 1.42
58,80 + 2.22
Base
10.80 0.51
35.60 1.92
16.60 1.21
Majority
14.80 + 1.17
41.80
0.88
16.80 1.25
Debate
19.60 + 1.12
52.60 + 1.26
18.20 1.37
Mistral (Jiang et al., 2023)
STaR
17.40 0.97
45,50 1.54
17.84 + 1.23
Majority FT
=
44.60
1.65
18.91
1.37
Ours
22.60 0.97
58.40
2.11
22.50
1.87
Base
43.20 +: 2.22
75.00
1.94
46.80
2.23
Majority
45.80 2.23
2.23
76.40 + 1.90
47.20 + 2.23
LLLMA-33 (Dubey et al., 2024)
Dehate
48.40 + 2.24
78.40 + 1.44
51.60
2.23
Majority FT
49.20 2.24
77.20
1.87
52.20 2.23
Ours
52.00 2.24
88.60
1.77
57.40
2.21
Table 1: Quantitative results of the proposed method and baselines. Our method outperforms the baselines
across all datasets, indicated by accuracy (%) + standard error. The highest values highlighted in red, and second-his second-highest values are highlighted blue. results are reported over 500 fixed evaluation problems, expect GSM results for GPT-3.5 which are reported over 1000 fixed evaluation problems (to construct nonoverlapping confidence hars).
Diversity Metric: Likelihood
Diversity Metric: Embedding Dissimilarity
Negative Log Likelihood ood
LLaMA-3
Negabil Likelhhoo
Mistral
Embeddinng ng Dissimiland anrrtytyyy
LLaMA-3
Embedding Dissimilarin Dissimilarity larit
Mistral
Iterations of finetuning
Iterations fifettnning
Iterations finetuning
Iterations of finetuning
Figure 3: Diversity is preserved and can improve across iterations of finetuning. We measure the response diversity of our method and the single-agent finetuning method on the MATH dataset using two diversity measures. The diversity of our method remains consistent over finetuning iterations for one metric and improves for another metric, whereas the diversity of the single-agent method drops significantly.
prevents further enhancement. However, finetuning multiple generation and critic agents using our proposed method increases diversity and consistently improves performance.
4 ANALYSIS
In this section, we aim to answer the following questions: 1) How important is the proposed multiagent finetuning procedure? 2) Will it increase response diversity? 3) Can the finetuned agent generalize to other datasets in a zero-shot setting?
4.1 ABLATION STUDIES
We examine each component of the proposed method, as shown in Table 2. Multiagent FT (Ours) refers to our proposed method with single round of finetuning, L 1.
Multiagent FT w/o summary removes the summarization step from the multiagent debate. Instead of summarizing, the responses from other agents directly concatenated and presented to each agent. Summarization helps by eliminating redundant information and retaining the most critical points; therefore, omitting the summarization step can negatively impact performance.
7

---

Published as a conference paper at ICLR 2025
LLM
Ablations
Arithmetic
GSM
MATH
Multiagent FT (Ours)
99.62 = 0.28
85.60 = 1.67
60.60 + 2.18
Multiagent FT w/o summary
99.20 + 0.40
82.20 + 1.72
51.70 + 2.24
GPT-3.5 (OpenAI, 2022)
Multiagent FT w/o critic
99.20 + 0.40
83,80 = 1.65
50.80 + 2.24
Single-agent FT Single-agent FT w/o dehate
99.00 = 0.45
83.60 + 1.66
56.80 + 2.21
87.20 = 1.49
75.00 + 1.93
48.89 + 2.23
Multiagent FT (Ours)
99.40 + 0.34
88,60 = 1.42
58.80 + 2.22
Multiagent FT w/o summary
98.80 + 0.51
84.40 + 1.68
55.00 + 2.09
Phi-3 (Abdin et al., 2024)
Multiagent FT w/o critic
98.20 = 0.62
86.00 + 1.58
56.60 + 2.22
Single-agent FT
97.40 = 0.71
86.80 = 1.51
56.80 + 2.21
Single-agent FT w/o dehate
92.20 = 1.20
83.60 + 1.66
50.20 + 2.24
Multiagent FT (Ours)
22.60 + 1.87
58.40 2.11
22.50 + 1.87
Multiagent FT w/o summary
21.80 = 1.84
56.00 + 1.56
20.20 + 1.55
Mistral (Jiang et al., 2023)
Multiagent FT w/o critic
21.00 = 1.82
54.80 + 1.60
19.01 + 1.59
Single-agent FT
21.20 = 1.83
55.00 + 2.22
19.21 - 1.69
Single-agent FT w/o debate
17.71 = 1.70
51.20 + 2.24
17.22 + 1.54
Multiagent FT (Ours)
52.00 + 2.24
88.60 + 1.77
57.40 + 2.21
Multiagent FT w/o summary
50.40 = 2.24
83.20 + 1.67
51.60 + 2.23
LLaMA-3 (Dubey et al., 2024)
Multiagent FT w/o critic
48.60 = 2.24
82.20 1: 1.70
50.50 +: 2.23
Single-agent FT
48.00 = 2.23
84,40 + 1.62
52.40 + 2.23
Single-agent FT w/o debate
44.00 = 2.22
81.60 + 1.73
48.80 + 2.24
Table 2: Ablation results. We examine each component of the proposed method and found that summarization, the combination of critic and generation agents, multiagent finetuning, and multiagent debate all contribute to performance improvement. The accuracy (%) + standard error is reported.
Multiagent FT w/o critic: or synthesize the best responses. Removing the critic agents and only finetuning the N generation agents could hurt performance, as the critic agents play a crucial role of refining the final output.
The critic agents evaluate the outputs from all generation agents and select
Single-agent FT involves finetuning only a single LLM as covered in Section 2.2 and using it as an agent in multiagent debate. This approach can easily lead to model collapse, where the agent generates similar responses after finetuning, thereby reducing diversity and hurting performance. Therefore, multiagent finetuning is necessary to maintain high performance in reasoning tasks.
Single-agent FT w/o Debate further eliminates the debate procedure, with the finetuned LLM generating responses directly. As shown in Du et al. (2023), multiagent debate can significantly boost performance, so removing it could lead to performance drop.
These results indicate that summarization, the combination of critic and generation agents, multiagent finetuning, and multiagent debate all contribute to performance improvement, Our proposed method integrates these components into single, unified framework, leveraging their combined benefits.
4.2 AGENT RESPONSE DIVERSITY
By finetuning multiple agents with distinct roles, our approach enables to obtain more diverse responses across rounds of finetuning compared to a single agent. Figure 3 illustrates the diversity of generations from our method and single-agent across rounds of finetuning using two metrics of diversity. We cover one metric of diversity, negative log- likeliiddoood, here and cover the other in Section C.4.
In our first diversity metric, we aim to characterize specialization by tracking the likelihood of responses of other agents using likelihood calculations of specific agent. If we are increasing diversity, then the log-likelihood finetuning. The reasoning used by other agents would be considered less common for the specific agent, indicating a divergence in responses. If accuracy increases while likelihood of responses from other agents decreases, this indicates must specialization.
responses from other agents will decrease across iterations of
We evaluate the negative ligeliool (NLL) of responses from other critic agents using another held-out critic agent and plot this over iterations of finetuning. We do the same with Single-Agent FT, using responses from other agents and evaluate likelihood using held-out agent. Larger NLL values indicate that the model has assigned low likelihood to a sequence and lower NLL values indicate that the model has assigned higher likelihood to a sequence. We measure this over iterations of finetuning for our method as well as Single-Agent FT.
8

---

Published as a conference paper at ICLR 2025
We compute the diversity across all test exam- ples and present the results in Figure 3. For the "Single-agent FT". all agents the same fine- tuned language models, and M = 1. We notice that NLL increases across iterations of finetun- ing for our method, meaning that responses from other critic agents are more diversity accord- ing to our held-out critic agent. Moreover, our responses are more diverse than using Single- Agent FT. This aligns with our previous ob- servation that diverse responses can mitigate model collapse and prevent the model from over- fitting to the finetuning data, Icading to better performance. We also include another metric, embedding dissimilarity, as a further compar- method
PhH-3 Misrral
Single Agent
Multagget (Ours)
MATH Accuracy
40
Multiagent (Ours)
Single-Agent IT
20
0.15
0.25
0.35
0.10 0.55
Diversity Metric: Embodding Dissimlarity
Figure 4: Relationship between accuracy and diver- sity. We visualize the relationship between embedding dissimilarity and MATH accuracy across rounds of finc- tuning. Our multiagent finetuning preserves diversity across rounds of finetuning while improving accuracy.
ison, finding that responses from preserves diversity, where as diversity reduces significantly with Single-agent FT. We provide additional metrics for evaluating diversity in gen- erations in Appendix Section C, and similarly find that multiagent finetuning preserves the final diversity of generations.
We further analyze the relationship between diversity and performance and show this in Figure Specifically, we see that an improvement in the diversity of responses correlates positively with an improvement in performance across rounds of finetuning across both Phi-3 and Mistral models. This suggests that in general, increasing the diversity of responses can be helpful for improvement over multiple rounds fine-tuning. Appendix Section E, we compare our approach with additional approaches to improve the diversity of samples such increasing the temperature which samples are generated, or using unique IDs in a single language to simulate a single agent. We find that our approach outperforms these baselines.
4.3 ZERO- GENERALIZATION
We investigate the zero-shot
generalization of the proposed method across different datasets.
60
$4.00
Specifically, we use generation and critic agents finetuned the MATH dataset and evaluate their performance domly sampled examples from
GSM Accuracy
48.00
45.00
40
100 ran-
20
the GSM dataset. We compare our method to bascline methods used in Table 1. These base- lines are trained on the GSM dataset. All methods use Mistral as the base LLM. Figure 5 shows that our method surpasses all the baseline methods, even though it has never data from the GSM dataset, indicating the strong zero-shot generalization capability of the proposed method. We show further results Section H.3.
Base
Majority
Debate
STaR
Majority-FT Ours (zero-shot)
Figure 5: Zero-shot generalization the proposed method. Our method demonstrates zero-shot generalization capabilities. When trained on the MATH dataset, it can effectively generalize to the GSM dataset. It outperforms all the baselines that trained on the GSM dataset.
5 RELATED WORK
Finetuning methods generally fall into three categories: cumman-in-th-looe distillation, and self-improvement, We briefly cover the first two categories and spend more time on self- self-improvemeet, which is more related to work.
Finetuning with human-in-the-loop and distillation: Several human in-the-loop methods have been introduced for finetuning, most noticeably RLHF (Christiano et al., 2017; Sun et al., 2023) and

---

Published as a conference paper at ICLR 2025
DPO (Rafailov al., 2024). These methods have been employed as part of instruction tuning (Zhang et al., 2023), improving the generated responses to instructions. Several instruction tuning datasets (Wang et al., 2022; Longpre et al., 2023) have been released publicly, some with human-genennneee generated responses. Other datasets have been constructed using the second category of finetuning methods, distillation, whereby much larger, highly performant L.LM is used to generate data that finetunes smaller LLM (Peng et al., 2023; Liu et al., 2024). These approaches have been used to build recent LLMs such as Alpaca (Taori et al., 2023) or Vicuna (Chiang et al., 2023) using responses generated by GPT-3.5 or GPT-4 (Achiam et al., 2023).
improvmmte methods (Huang et al., 2022; Yu et al., 2023; Yuan et al., 2024; Hsich et al., 2023; Welleck et al., 2022) improve the performance of LLMs through the finetuning. Common approaches include iterated learning (Anthony et al., 2017; Vani et al.; Polu et al., 2022; Xu et al., 2024) where solution/methods discovered by optimization on prior data are used to uncover further solutions or, in this context, provide additional finetuning data. Some of the main papers we use for comparison finetune using bootstrapping through rationale generation (Zelikman et al., 2022; Lee et al., 2024; Pang et al., 2024; Zhang et al., 2024; Lu et al., 2023) or use pllylssllf-trainnneen 2024; Chen et al., 2024a). Most methods find that using self- geffgeeeerted rationales leads to significant improvement when finetuning. However, these works and many others rely on access ground truth answer. Overall, existing works often show a plateauing effect with limited boosts in improvement after several rounds of fine-tuning. Our work proposes to use multiagent interaction as an approach to get more consistent gains after multiple rounds of finetuning.
Finetuning with eel--improvement
methods through reinforcement learning (Chen et al., 2024b; Yuan et al.,
Multiagent Interaction: Our work builds on the combination of finetuning and multiagent interaction systems. We primarily incorporate multiagent debate (Du et al., 2023; Chan et al., 2023; Pham et al., 2023; Liang et al., 2023) due to its success in improving factuality and reasoning in LLMs in a variety of tasks at inference time. Several other multiagent interactions could also serve as the basis for this paper. Tree-of-thougee 2024) represent two common multiagent interaction systems over LLMs that incorporate responses across multiple LLMs, which improves reasoning. Other works (Wu et al., 2023) have designed more flexible systems for multiagent conversations built structured program synthesis rather than natural language. Prior work has also focused on incorporating multiagent interaction into domains beyond factuality and reasoning such as strategy and communication games (Abdelnabi et al., 2023). More recently, this has led to multiagent interaction systems over LLMs that have optimized via equilibrium search for factuality and reasoning tasks (Jacob et al., 2023b;a). In contrast to existing works, our work aims to use multiagent interaction as a method to finetune language models.
(Yao et al., 2024; Long, 2023) and graph-of-thougle (Besta et al.,
CONCLUSION AND LIMITATIONS
Limitations. In comparison to existing works in single model finetuning, multiagent finetuning is substantially more expensive at both training and inference time as multiple copies of a model need to be trained and run. To run multiagent finetuning experiments on open source models, we used either four H100 GPUs or four A100 GPUs. Models took between 120GB - 240GB of GPU memory and inference took between 12-24 hours across multiple GPUs. To improve the training time of multiagent models, it may be interesting to instead share weights across different instances of models. To improve inference time in multiagent models, we can directly distill the debate procedure into a single modelor use quantization as part of finetuning.
Conclusion. In this paper, we have introduced a novel multiagent finetuning framework that sig- nificantly enhances the performance and diversity of language models. By employing a society of agents with distinct roles, our method effectively improves the feedback mechanism and overall output quality, mitigating the limitations inherent in single-agent self-improvement methods. This system allows for autonomous self- self-improvement through iterative finetuning, leading to substantial performance gains across a comprehensive suite of reasoning tasks. Importantly, our approach is versatile and can be applied to both open-source and proprietary LLMs, ensuring broad utility and impact. Additionally, our method can be integrated with other finetuning approaches such that incorporate human feedback such as RLHF DPO, which we leave to future work. This work opens new avenues for future research in language model enhancement and sets a foundation for further advancements in the field.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the Center for Brains, Minds, and Machines, NSF STC award CCF- 1231216, the NSF award 2124052, the MIT CSAIL Machine Learning Applications Initiative, the MIT-IBM Watson AI Lab, the CBMM-Siemens Graduate Fellowship, the DARPA Mathematics for the DIscovery of ALgorithms and Architectures (DIAL) program, the DARPA Knowledge Management at Scale and Speed (KMASS) program, the DARPA Machine Common Sense (MCS) program, the Air Force Office of Scientific Research (AFOSR) under award number FA9550-21-1- 0399, the United States Air Force Research Laboratory and the Department of the Air Force Artificial Intelligence Accelerator under Cooperative Agreement Number FA8750-199 1-8800-992--0000 and ONR MURI grant N00014-22 1--00-1---22---- authors and should not be interpreted representing the official policies, either expressed implied, of the Department of the Air Force or the U.S. Government. The U.S. Government authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
The views and conclusions contained in this document are those of the
REFERENCES
Sahar Abdelnabi, Amr Gomaa, Sarath Sivaprasad, Lea Schönherr, and Mario Fritz. Llm-deliberation: Evaluating iims with interactive multi-agent negotiation games. arXiv preprint arXiv:2309. /7234, 2023. 10
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: highly capable language model locally on your phone. arXiv preprint arXiv:2404.1421 14219, 2024.
6, 7,8
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2303::230030877444 2023. 10
Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017. 10
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 2022. 1
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. (1682-16990, 2024. 10
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.0720/1 2023. 10
Zhipeng Chen, Kun Zhou, Wayne Xin Zhao, Junchen Wan, Fuzheng Zhang, Di Zhang, and Ji-Rong Wen. Improving large language models via fine-grained reinforcement learning with minimum editing constraint. arXiv preprint arXiv:2401.06081. 2024a. 10
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fin-tuniig converts weak language models to strong language models. arXiv preprint arXiv:2401.01333 2024b. 1,
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, al. Vicuna: An open-source chatbot
11

---

Published as a conference paper at ICLR 2025
impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2(3);,, 2023. 10
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing systems, 30, 2017. 9
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021. 5
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factual- ity and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325, 2023. 2, 3, 5, 6, 8, 10, 15, 20
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama herd of models. arXiv preprint wxXiv:240.221783 2024. 6, 7, 8, 20, 21, 22
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.0387 2021. 5
Cheng- Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yt Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.02301 2023. 10
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. arXiv preprint 1.X1::2200.11610, 2022. 1, 10
Athul Paul Jacob, Gabriele Farina, and Jacob Andreas. Regularized conventions: Equilibrium computation as a model pragmatic reasoning. arXiv preprint 2rXiv:2311.097/1 2023a. 10
Athul Paul Jacob, Yikang Shen, Gabriele Farina, and Jacob Andreas. The consensus game: Language model generation via equilibrium search. arXiv preprint /rX0v:23/0.09119, 2023b. 10
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06923 2023. 6, 7,
Nicholas Lee, Thanakul Wattanawong, Schoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Anumanchipali, Michael W Mahoney, Kurt Keutzer, and Amir Gholami. Llm2llm: Boosting Ilms with novel iterative data enhancement. arXiv preprint 2rXiv:2403.15042 2024. 10
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang. Yan Wang, Rui Wang. Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305. 19/ 18, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 10
Jieyi Long. Large language model guided tree-of-thought. arXiv preprint arXiv:2305.0829) 2023.
10
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pp. 22631-2222 2661-226448. PMLR, 2023. 10
Jinniia Lu. Wanjun Zhong, Wenyong Huang, Yufei Wang, Fei Mi. Baojun Wang, Weichao Wang, Lifeng Shang, and Qun Liu. Self: Language-driven self-evolution for large language model. arXiv preprint arXiv:2310 005333,, 2023. 2, 10
12

---

Published as a conference paper at ICLR 2025
OpenAI. Chatgpt: Optimizing language models for dialogue, December 2022. URL https:
//openai hwwom//ooo//hhaaappooeeeee 6, 7.7.8
R OpenAl. Gpt-4 technical report. arXiv, pp. 2303 08774, 2023.
Richard Yuanzhe Pang. Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint 2xXiv:240.119733 2024. 10
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, arXiv preprint S.Niv:23004033277, 2023. 10
Chau Pham, Boyi Liu, Yingxiang Yang. Zhengyu Chen, Tianyi Liu, Jianbo Yuan, Bryan A Plum- mer, Zhaoran Wang, and Hongxia Yang. Let models speak ciphers: Multiagent debate through embeddings. arXiv preprint arXiv:2310.06272, 2023. 10
Stanislas Polu, Jesse Michael Han, Kunhao Zheng, Mantas Baksys, Igor Babuschkin, and Ilya Sutskever. Formal mathematics statement curriculum learning. arXiv preprint arXiv:220.003444, 2022. 10
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, al. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21 (1140)):1667, 2020. 19
Yuda Song, Hanlin Zhang, Carson Eisenach, Sham Kakade, Dean Foster, and Udaya Ghai. Mind the gap: Examining the self-improvemmnt capabilities of large language models. arXiv preprint arXiv:2412.02674, 2024. 2
Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu. Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309. 14525, 2023. 9
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-folllwingg model. Stanford Center for Research on Foundation Models. https:/crrf.. stanford. edu/20223033/1//lppaaa html, 36))7, 2023. 10
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023. 1
A Vani, M Schwarzer, Y Lu, E Dhekane, and A Courville. Iterated learning for emergent systematicity in vqa. arxiv 2021. arXiv preprint arXiv:2105.0111 10
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Suppe--naturalinsttuttiosss Generalization via declarative instructions on 1600+ nlp tasks. arXiv preprint arXiv:2204.07705. 2022. 10
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. arXiv preprint 2rXiv:2211.00055. 2022. 10
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen 11m applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08/3 2023. 10
Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402 13/ 16, 2024. 10
13

---

Published as a conference paper at ICLR 2025
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024. 10
Tree of thoughts:
Xiao Yu, Baolin Peng, Michel Galley, Jianfeng Gao, and Zhou Yu. Teaching language models to self-improve through interactive demonstrations. arXiv preprint arXiv:2310. 13522, 2023. 1, 10
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401. 10020, 2024. 1, 10
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D Goodman. Star: Self-taught reasoner bootstrap- ping reasoning with reasoning. In Proceedings of the 36th International Conference Neural Information Processing Systems, pp. 15476-15488, 2022. 1, 6, 10
Shengyu Zhang. Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, al. Instruction tuning for large language models: A survey, arXiv preprint arXiv:2308.1077 2023. 10
Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. Small language models need strong verifiers self-correct rea- soning, arXiv preprint 144142200. 2024. 10
14

---

Published as a conference paper at ICLR 2025
A APPENDIX SUMMARY
add additional details for our methods and experiments as well as additional results to provide more evidence of improvements with multiagent finetuning. In Section B, provide additional details on summarization, inference and training details using multiagent finetuning with debate. In Section C, cover additional metrics for measuring diversity in agent responses based (1) consensus and (2) Kl--divergence (3) likelihood, Both metrics show that diversity maintained or increases while accuracy increase rounds of finetuning. In Section D, introduce cooperative approach for composing agent responses rather than a competitive approach through multiagent debate. We apply multiagent finetuning with the cooperative approach analyze whether our method agnostic to the approach style. We find strong similar improvements when our method is applied to cooperative approach. In Section E, we include an additional baseline based on Single Agent FT where we increase the sampling temperature applied across all agents. This is a proxy for increasing diversity that is complementary to our method. We find that multiagent finetuning significantly outperforms methods that modify temperature to artificially induce diversity. In Section F, we add an additional experiment where we apply multiagent finetuning to responses across 5 agents instead of 3. We see significant improvements in performance when using additional agents. In Section G, present simple mathematical model illustrating how multiagent finetuning can improve diversity. Finally, in Section H, present additional evaluations of multiagent finetuning across wide suite of datasets.
B METHODOLOGY DETAILS
B. SUMMARIZATION DETAILS
As done in Du et al. (2023), we incorporate summarization into the multiagent debate procedure. In summarization, we have an LLM agent take responses from other agents as input and summarize the answers to the responses. During round m of debate, we introduce summarization agent A which takes responses from the other N agents in the last round, (8""1_ .... y 1 3n+1 .... VN and generates a summary of the responses tm,n. This summary sent to the critic agent A to generate new response.
B.2 INFERENCE DETAILS
The pseudocode of our method for inference is shown below.
Algorithm 2 Inference
Require: A set of finetuned generation agents { ... A& ): A set of finetuned critic agents {Af, .A }: A test set of language inputs and ground truth responses Dask {ar, y }; The number of agents N; The number of debate rounds M.
success <-
2: for Ir y in Dus do # Iterate over the input tasks
3: for r in M do # M rounds of debate
4:
if
then
5:
11.1. VI,N (f Ag(e) Response of each generation agent
6:
else
7:
Fm.1 Nn.N t Summarize responses from other generator agents
8:
1m. Vn.NN
Af (*t.)) .A (xt..) Response of each critic agent
9:
end if
10:
end for
11:
Majority Voting (UM.I, ,UM.N) # Responses of the final round of debate success success + I10 V)
end for
14: Accuracy +- sUDcess
15

---

Published as a conference paper at ICLR 2025
B.3 EXPERIMENTAL DETAILS
For all open-source models, we perform finetuning using total of cight 40GB A100 GPUs and four 80GB H100 GPUs. The evaluation of individual inference times for multi-agent finetuning with open-source models took approximately 30 36 hours.
Instruct which has 4 billion tunable parameters. We finetune the entire model end-to-end (no LoRA or memory adaptation) on two 40GB A100 GPUs or one 80GB H100 GPU and run a total of two epochs of finetuning for generation agents and one epoch of finetuning for critic agents. We use batch size of and learning rate of 5e -6 for generation agents and 5-- for critic agents. When applying multiple iterations of finctuning, we use a learning rate of 5e 7 and a weight decay of Le across both generation and critic agents. Models are finetuned with fixed training set of 500 randomly selected questions (where we do not provide answer annotations for the questions) and then evaluated on a separate test set of 500 randomly selected questions.
Phi-3 We ran our results using
Mistral We ran our results using Mistral-70 - Beerall-B-Instiiiie Instructtttt 2, which has 7 billion tunable parameters. We finetune the entire model end-to-end (no LoRA memory adaptation) four 40GB A100 GPUs or two 80GB H100 GPUs and run a total of one epoch of finetuning. We use batch size of and learning rate 5e for generation agents and 5e for critic agents and a weight decay of le When applying multiple iterations of finetuning, we use a learning rate of 5e across both generation and critic agents. Models are finetuned with fixed training set of 500 randomly selected questions (where we do not provide answer annotations for the questions) and then evaluated on separate test set of 500 randomly selected questions.
LLaMA-3 We ran our using Meta-Ll ama - 1ama-----0 ---8B-Instrue which has 8 billion tunable param- eters. We finetune the entire model end-to-end (no LoRA or memory adaptation) on three 80GB H100 GPUs and run total of two epochs of finetuning. We use batch size of I and a learning rate of 5e 7 for generation agents and 2e for critic agents. When applying multiple iterations of weight
a learning rate of 5e-7 across both generation and critic agents as well decay of le 1e-2. Models are finetuned with a fixed training set of 500 randomly selected questions (where we do not provide answer annotations for the questions) and then evaluated on separate test set of 500 randomly selected questions.
finetuning,
results on the 3 5-turbboo66111 0063 model. We use the finetuning API and run a total of two epochs of finetuning, using a batch size of and a learning rate multiplier of Models are finetuned with fixed training set of 500 randomly selected questions (where we do not provide answer annotations for the questions) and then evaluated on a separate test set of 500 randomly selected questions.
GPT-3.5 We
C DIVERSITY METRICS
We cover different metrics for measuring diversity for both Phi-3 and Mistral provide an overview of the diversity of our method in comparison to Single Agent FT.
C.I CONSENSUS
We further analyze the diversity of responses from our method to show that diversity is preserved. Rather than using text embeddings, we further measure the consensus among agents a more interpretable alternative. This is measured as the proportion of agents that have the same final answer in a given round of debate. We take an average of this proportion across 500 problems used for evaluation. To obtain the mean consensus of our single agent finetuning baseline, we prompt the single-agent agent finetuned model times, take a majority vote over generated answers, and find the proportion of agents that had generated answer that was the majority vote. In order convert this to diversity, we take the difference of the mean consensus value from 1, which represents the average number of agents with different response from the consensus answer.
We measure diversity as the inverse of consensus. Specifically, we consider the agent responses in the final round of debate (yM,1,"'* ,yM,N) that match the majority-voted final response ý. The
16

---

Published as a conference paper at ICLR 2025
Phi-3
Mistral
60
60
Multiagent FT (Ours)
Multiagent FT (Ours)
Single-agent
Single-agent
40
40
Diversity
20
20
0
Iterations of Finetuning
Iterations Finetuning
Figure 6: Consensus: Response diversity across finetuning iterations. We measure the response diversity based agent consensus of our method and the angle-ageet finctuming method on the MATH dataset. The diversity our method remains consistent over finetuning itcrations, whereas the diversity of the single-agent method drops significantly.
Phi-3
Mistral
0.10
KL Divergence across agents
0.12
0.08
0.10
0.06
Multiagent (Ours)
0.08
Single-agent FT
0.06
0.04
0.02
2
3
2
5
Iterations of Finetuning
Iterations of Finetuning
Figure 7: KI.-Divergence: Response diversity across finetuning iterations. We measure diversity based on the KI -ivveieeeee hetween the prohabilities of the output tokens between agents. Similar to our likelihood measurement, we find that diversity is preserved across rounds of finetuning.
consensus computed as the percentage of responses [yM,,, ... .MMNN ,,MM,
that match ø/:
Consensus ΣN
IM D)
N
where TI is the indicator function. Diversity is then given by Diversity 1 Consensus.
We show results in Figure 6. As seen with our prior metric, embedding dissimilarity, we can preserve diversity based on the responses given by the agents, rather than based the embeddings of language model.
C.2 RLEDIVERGENCE
We next measure diversity by computing KL divergence between the probability distributions puted based on the final answers from different agents. We estimate the probability distribution of each agent's response using the likelihoods from Gemma-2 (2B) For each test example, compute the KL divergence between the responses of any two agents and then average the values from all pairs of agents to determine the overall KL divergence.
We see results Figure 7. Specifically, we see that diversity is preserved using our method whereby KI KL-divergence is consistently higher than the single-agent finetuning baseline.
17

---

Published as a conference paper at ICLR 2025
LLaMA-3: KL-Divergence between finetuned models and base model
Einple agent/f
0.30
Iuglegeett (ienentiin Agento)
VuhtlgenLFF rarts Agental
KL-Divergence
0.20
0.15
0.10
3
Iterations of Finetuning
Figure 8: KL Diversity between finetuned and unfinetuned LLM. We mcasure the KL-divergence between likelihoods of responses from finetuned agents and base LLM agents for single-agent finctuning and genera- tion/critic agents from multiagent finetuning. Likelihoods are calculated using Gemma-2 (2B). We find that our method diverges from the base LLM probabilities and furthermore, critic agents have better divergence in responses and our method has better diversity metrics than single-agent FT.
Embedding dissimilarts aariaiyyy across agents
Phi-3
Mistral
0.35
0.40
0.25
Multiagent
0.35
Single-agent
0.30
0.20
0.15
0.20
Iterations of Finetuning
Iterations of Finetuning
Figure 9: Emhedding Dissimilarity: Response diversity across finetuning iterations. We measure the response diversity based on embedding dissimilarity between the responses different agents, where embeddings are computed using the T5-3B encoder. We notice that similar to likelihood measurement, we find that diversity is preserved across rounds of finetuning.
C.3 KI-DIVERGENCE ACROSS MODELS
We further analyze diversity by comparing the KL.-divergence generation and critic agents with the likelihood of responses from the base LLM model across iterations of finetuning.
We measure the KL-divergence between each agent responses and responses from a base LLM for 500 MATH examples. We average KL-divergence across all examples for each iteration of finetuning. We apply this measure to agents formed through Single Agent-FT and to generation and critic agents formed through our method. For Single-Agent FT, we find the KL divergence for each finetuned agent and average the KL-divergence across all examples and all agents per iteration of finetuning, For our method, we separate generation and critic agents and find the average KL.-divergence for both. We measure likelihoods using Gemma-2 (2B), similar to Figure 7.
We show results Figure We see that critic agents generally have higher KL-divergences from the base LLM and both critic and generation agents have higher KL-divergences across iterations of finetuning.
C.4 EMBEDDING DISSIMILARITY
Finally, we analyze diversity by measuring the embedding dissimilarity between responses of different agents.
Specifically, we consider agent responses in the final round of debate {UM.1.'... ... .MM.N} that match the majority-voted final majority-voted final response ý. For each response, obtain pretrained
18

---

Published as a conference paper at ICLR 2025
L.M
Methods
Arithmetic
GSM
MATH
Cooperative (Base) Cooperative (FT)
96.60 0.83
81.80
1.73
53.60 + 2.23
GPT-3.5
84.00 1.64
56.40 1 2.21
Table 3: Cooperative Finetuning. Our method supports fine-tuning in cooperative settings, where agents work together (c.g., 3 agents, 2 rounds).
Phi-3
Mistral
30
Multiagent (Ours)
Single-agent [Tamp
65
28
Single agent (Temp 2.01
60
Accuracy
55
50
Multiagent Fl (Durs)
18
45
Single agent FT (Temp 1.0)
Single agent 2.01
16
1
Iterations of finetuning
Iterations of finetuning
Figure 10: Inducing diversity through increasing temperature. We introduce additional baseline where apply Single-Agent FT baselin with temperature 2. By increasing the sampling temperature, we allow the model to generate more diverse responses. We observe that our method out out-performs higher temperature settings, which demonstrates that temperature does not increase diversity in a way that is useful for accuracy.
contextual word embeddings from held-out language model, in this case the T5-3B encoder model (Raffel et al., 2020).
We feed each agent response to the T5 encoder model obtain word embeddings and extract the embedding associated with the classification token [CLS]. As done in prior work, we use this embedding representation of the sequence. We compare the similarity of the agent responses using cosine similarity the [CL5] embeddings. Since cosine similarity measures similarity, to obtain metric for diversity, take the complement of cosine similarity by subtracting the value from
D COOPERATIVE FINETUNING
In this paper, our method mainly builds competitive approach for composing agent responses with multiagent debate. Our approach for multiagent finetuning can applied to both the competitive setting, where critic agents provide feedback generator agents, and cooperative settings, where agents work together in "mixture of cxperts" style to generate answers. Instead of prompting agents to critique responses from other agents, in the second round of conversation, cooperate with other agents. We ask each agent to generate a new response by merging their own response with the responses of other agents, using the prompt "Can you derive new solution by combining your solution with the solutions of other agents?". Under this cooperative setting, the proposed multi-agent finetuning improves the performance, as demonstrated by Cooperative (FT) outperforming Cooperative (Base).
prompt agents to
We show results in Table 3. More specifically, we see that we can finetune with a cooperative method with multiagent finetuning and achieve similar improvements in performance. This demonstrates that our method can be applied other multiagent prompt settings as general finetuning method for LLMss.
E ADDITIONAL COMPARISONS
We compare approach to additional approaches improve the diversity of reasoning chains.
19

---

Published as a conference paper at ICLR 2025
LLM
Methods
Arithmetic
GSM
MATH
Debate Majority FT
99.40 + 0.34
85,40 + 1.58
58.20 + 2.22
GPT-3.5
99.60 + 0.28
86.20 = 1.54
59.00 + 2.19
Ours
100.00 = 0.00
88.20 + 1.44
62.80 + 2.16
Debate Majority FT
97.40 + 0.71
86.00 = 1.55
55.20 + 2.22
Phi-3
95.80 + 0.90
84,80 + 1.61
53.20 + 2.23
(Ours)
99.80 + 0.20
89.40 + 1.38
60.40 + 2.19
Table 4: More agents of debate. With 5 agents and 2 rounds of debate, methods still outperform the baselines and show better results than the 3 agents and rounds of debate results presented in Table of main paper.
LLM
Methods
MATH
Base
46.80 +: 2.23
Debate
51.60 + 2.23
LLaMA-3 (Dubey et al., 2024)
Unique ID
50.80 +: 2.24
Ours
57.40 + 2.21
Table Unique vs Multiagent Finetuning. We introduce an additional comparison to multiagent finetuning where we feed a unique ID token to each agent, corresponding to a generation or critic agent. We find that this is not comparable to improvements multiagent finetuning.
E. MODULATING TEMPERATURES
We first consider inducing diverse responses from LLM agents by increasing the temperature of generation. We add an additional baseline where we vary the temperature of agents finetuned using Single Agent-FT. Higher temperature values may be a proxy for more diverse responses. We show results over rounds of finetuning in Figure 10.
We see that our method surpasses the performance of this baseline. This likely because higher temperature values can reduce accuracy due to increased variability of samples. Our method preserves diversity of responses while increasing accuracy using more carefully designed finetuning method.
E.2 UNIQUE ID FOR AGENTS
We next considering an additional comparison to multiagent finetuning that can preserve diversity while reducing the cost of finetuning. The method involves using unique identifier as part of the prompt fed to each agent. We feed each generation agent an ID given by GEN1, GEN2, etc. Similarly, each critic agent is given an ID CRIT1, CRIT2, etc. Additionally, we provide short description to the agent, explaining what the ID) refers to. For generation agents, we state that the agent is tasked with creating a solution. For critic agents, we state that the agent is tasked with evaluating and improving responses. The ID is presented to the agent at the beginning of each prompt, marked by the string Agent ID: GEN1 (This is a generat ion agent solution.)
tasked with creating
example of the ID fed to generation agent 1.
We compare the unique ID approach on the same 500 MATH examples reported Table 1. Results shown in Table 5. We find that multiagent finetuning performs significantly better and that using unique IDs is fairly similar to debate. This demonstrates that mechanisms for generating solutions and critiquing them unlocked via finetuning.
ADDITIONAL AGENTS IN DEBATE
In Table 4, we show the influence additional agents with finetuning, We use 5 agents and 2 rounds of debate. We find that additional agents improves results noted in prior work (Du et al., 2023) over agents, rounds of debate. This also implics that our method will scale with larger number of finetuned agents.
20

---

Published as a conference paper at ICLR 2025
LLM
Methods
MATH
24.40 1.92
Majonity
25.20 1.94
LLaMA-3 (Dubey al., 2024)
Debate
29.80 + 2.05
Majority
28.00 2.01
Ours
34.20 + 2.12
Table 6: Additional Evaluation of Multiagent Finetuning the baselines difficult tasks including examples from all levels of MATH. This shows the applicability our method more broad settings.
difficult tasks. Our method outperforms
G MATHEMATICAL MODEL OF DIVERSITY OVER ROUNDS OF FINETUNING
We consider simple mathematical model illustrating how diversity can arise by finetuning models only on answers that they are accurate on. Consider a training dataset of problems in three topics, A, B, and C as well as three models we train all initialized from the same base model. For each model, we assign specialization skill score SA, SB. Sc between 0 and 1, representing how accurate the model is at answering questions the specified topic. All three models are initialized to have skill of 0.33 on each topic. The specialization S; for each topic corresponds to the percentage of questions topic the model get accurate, where SA of represents that model would get 0% of questions in topic A correct.
At each iteration, model is trained on all questions it answers correctly each topic. This increases the specialization skill score by fraction of training the model saw for each specific topic. Formally, the updated skill of model A at iteration would be:
St-1 5 S S
S S (1+
(2)
St-
To account for finite amount of capacity in each model, after the above skill update, the skills across all models at iteration are then normalized to have sum of one. Without loss of generality, assume that at iteration S is larger than S and Sy (which happens by random chance, since have finite number of questions). Under the update rule described, the ratio S to S% is given by
) (
(
(1+
S S + 5t)
I
(1+
S St
St
(3)
St
S
St.
(ELA.B.C)
Since 5t greater than or equal to S the above expression greater than or equal to
( Σ (1+
(1+ St 8 + S St.
S 5 5 + St St
S
1,
(4)
VE(A,B,C)
where we use the identity that the sum of 5 is equal to 1 to indicate since they are normalization the scores. We thus have that st+1 will be larger than Sh, with specialization on topic A monotonically increasing over iterations of training.
Since a priori the model has no preference for any particular topic, random sampling each initial base model will lead skill preference eventually result in models specializing in either topic A, B, C, ensuring diversity across models. This mathematical model is similar to the multiagent finetuning procedure in the paper, where selectively train generators and critics on datasets they are accurate on and illustrate how they can then specialize on different portions of data.
different random topic. This repeated procedure will then
H ADDITIONAL EVALUATIONS
H.1 LARGER MATH EVALUATION
To further evaluate multiagent finetuning, evaluate on the MATH dataset across all 5 levels of difficulty, instead of selecting examples from levels 1-3, We extract 500 examples for training and 500 examples for testing and evaluate L.aMA-3.
21

---

Published as a conference paper at ICLR 2025
LLaMA-3 (8B)
Multiagent FT (Ours)
Single-agent FT
40
38
32
30
Iterations of finetuning
Figure 11: Multiple iterations of finetuning over all levels of MATH. We apply multiple iterations of finetuning over 500 examples of MATH sampled from all levels. Even more difficult domain, significant improvements from multiagent finetuning that continue self-improve.
LLM
Methods
MMLU
Base
60.40 2.18
Majority
61.80 2.17
L (Dubey al., 2024)
Debate
65.80 + 2.12
Majority
63.40 2.15
Ours
68.80 2.07
Table 7: MMLU Evaluation We introduce an additional evaluation with the MMLU benchmark, finetuning on 500 MMLU examples and testing 500 different MMLU examples. We find that method performs better than other baselines.
We show results across all baselines in Table 6 and results across multiple rounds of finetuning in Figure 11. We see consistent improvement using LLaMA-3.
H.2 MMLU
We add additional comparison with MMLU to further establish thte improvement of our method on a task related to general factuality and reasoning instead of mathematics.
We finetune 500 MMLU examples randomly sampled from all 57 subjects. We then evaluate different set of 500 randomly sampled examples.
We show results in Table 7. We that method can improve performance task related to factuality.
H.3 ZERO-SHOT GENERALIZATION EVALCATION
We include larger evaluation of zero-shot evaluation of our method in Figure 12, where finetune on 500 MATH problems and test on 1000 GSM problems. We find that our method performs significantly better than all other basclines.
Furthermore, we test another setting to measure zero-shot performance by finetuning on the arithmetic dataset and evaluating on the GSM dataset. We finetune using 500 arithmetic problems and evaluate each method on 1000 GSM problems. See Figure 13. We find that our method also performs significantly better than all other baselines.
22

---

Published as a conference paper at ICLR 2025
60
GSM Accuracy
42,10
I
40
I
20
0
Base
Majority
Debate
STaR
Majority Ours (zera shat)
Figure 12: Testing zero-shot generalization across 1000 GSM problems We test the zero-shot capabilities of our method using models trained MATH dataset. We over 1000 problems GSM, our method performs better than all baselines.
60
53.30
49,70 70
GSM Accuracy
40
20
Base
Majority
Debate
Majority-FT
Ours izero-shoti
Figure 13: Zero-shot generalization after arithmetic finetuning. We evaluate the ability of our method to generalize after finetuning Mistral on the arithmetic task and evaluating GSM. We find that this aids in GSM performance, than finetuning with MATH.
23


Reviews:
[{"review_id": "EO5Bbe6IEs", "reviewer": "Reviewer_Tw8m", "summary": "This paper contributes to the field of self-improvement fine-tuning for LLMs by proposing a multi-agent cooperation approach. It replicates an LLM into multiple generation agents and corresponding critic agents. A multi-agent debate architecture is utilized to generate label responses for generation agents and critic agents. Each agent is then fine-tuned using its unique generated dataset. During the inference phase, the final result is generated following the multi-agent debate. The method demonstrates superiority over the baselines in a series of math-related language reasoning tasks.", "questions": "In addition to my primary concerns, there are also the following questions: \n1. In this paper, is each individual agent fine-tuned using the SFT (a combination of query and label response data) approach? If yes, can this method be integrated with DPO (a combination of query and human feedback preferences data) and PPO (a combination of query and reward signal data) approaches? Please provide a detailed discussion. \n2. How is the Combination of datasets implemented in line 25 of Algorithm 1 pseudocode? What does the hyperparameter denote? The authors need to clearly explain how the two types of data are used respectively in the training of the critic agent and what the role of the weight hyperparameter is.\n3. The abstract mentions “multiagent society of language models”, yet the experiments only utilized 3-5 agents (and the improvement with 5 agents does not seem significant). How do the authors view the relationship between the number of agents and performance improvement, as well as the issue of number versus resource consumption?\n4. The selection of baselines. From the setup of the baselines, this work seems more like an “A+B+C” approach. However, would it be fair to compare “B+C” with B alone, even if the A module is ablated?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is easy to follow and the content is well-organized.\n2. The paper proposes a method for agent self-improvement fine-tuning based on multi-agent collaboration, allowing for multiple rounds of self-improvement fine-tuning, which could be a promising approach.", "weaknesses": "My primary concerns with this paper are centered around the experimental section.\n\n(Major)The first concern is regarding the selection of experimental datasets. The paper exclusively uses mathematical language reasoning tasks, and each task is not particularly challenging. Arithmetic is limited to arithmetic operations, GSM corresponds only to Grade School level difficulty, and MATH selects only the first three levels. If the tasks are not challenging enough, it may lead to questioning the need for incorporating multiple LLMs when a single LLM might be adequate. What’s more, conducting generalization experiments solely within mathematical datasets may not demonstrate the unique advantages in terms of generalizability, given the highly similar nature of the tasks. Including a graduate-level mathematical reasoning set or challenging datasets from other fields would make the experiments more convincing.\n\n(Major)The second concern is that the performance improvement on datasets of modest difficulty is not significant, especially considering the introduction of multiple large models collaborating. It may be worthwhile to quantify other metrics beyond answer accuracy, for example, the KL divergence of each LLM from the original LLM distribution? Introducing multiple large models might allow agents to achieve better results without significant parameter changes.\n\n*Overall, I believe the idea of the paper is commendable. However, as a practice-oriented paper lacking in theoretical explanation, its experimental design is insufficient. Without addressing these major concerns in the experiments, I'm afraid this is the highest score I can give.*\n\n（Minor）The diversity of models: When reading the sentence \" each model to capture parts of a task of interest.\", I am very excited. However, the authors do not take any measures during the implementation to activate the heterogeneous characteristics among the generative agents. In other words, diversity is only brought about by the quantity of agents. The diversity cannot be theoretically guaranteed, nor can it be conceptualized and defined for specific agents after it appears. If the authors could enable different agents to collaboratively complete a complex task and truly capture the sub-tasks they are interested in or excel at, it might greatly enhance the contribution of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ofy0dUVglQ", "reviewer": "Reviewer_TeFj", "summary": "This paper proposes a method to address a key limitation of LLMs: their dependency on static datasets, which limits their ability to self-improve. The authors propose a multiagent approach for LLM finetuning that enables iterative self-improvement through interactions with other models. By using distinct Actor (generation) and Critic (evaluation) agents, their approach improves feedback quality and response diversity, which results in better responses over iterations of finetuning. The authors empirically demonstrate their approach's effectiveness across 3 reasoning benchmarks (Arithmetic, MATH, GSM), showing performance gains of approximately 1-15% across 4 baselines. Additionally, the finetuned agents may demonstrate zero-shot generalization from MATH to the other two benchmarks.", "questions": "See my questions in the \"Major comments\"", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Significance: This paper presents a promising approach to LLM self-improvement and could offer a valuable contribution.\n- Clarity: Most of the Figures in the paper are clear and the paper is generally well-written.", "weaknesses": "There are several comments I would like the authors to address to make some details clearer and the paper more complete.\n\n**Major comments**\n1. Role Specialization: The paper introduces distinct roles for models (generation agents and critic agents). However, it would be helpful to clarify the specific objectives each role optimizes. Additionally, I suggest emphasizing that only two roles are used in this paper (generation and critic) to avoid confusion. \n2. Zero-shot Generalization: In Section 4.3 The authors claim zero-shot generalization on held-out benchmarks. I find the claims made in Section 4.3 not very convincing for three reasons:\n- (a) The authors only evaluate on 100 random samples. Could the authors elaborate on why this specific sample size was chosen, and why not more?  If it is computationally possible to evaluate more samples (e.g. 1000), this would make the evidence for zero-shot transfer more convincing.\n- (b) In Figure 5, the standard errors for performance (accuracy) are absent; could you please report these like was done in the Table 2\n- (c) The choice of finetuning on MATH and evaluating on the GSM dataset is unclear; could you explain the rationale? Why not report all cross-dataset generalizations, such as MATH → GSM or Arithmetic → GSM? If the results are consistent across all cross-evaluations, this would strengthen the findings.\n3. Computational Trade-offs: The limitations section mentions increased computational demands but does not quantify these trade-offs. Details on the additional costs or time requirements would help to evaluate the compute/performance balance.\n4. Limitations Discussion: The current discussion of limitations is very short. Future researchers building on this work would benefit from more detailed insights into any bottlenecks or constraints of the approach. Based on these more detailed suggestions for future research directions would be valuable.\n5. Response diversity: Looking at the slope, Figure 3 seems to indicate that all the benefits could be coming from the initial improvement in diversity. It seems important to mention the initial diversity for each ablation. \n6. Standard error: How exactly is the standard error in Table 2 computed?\n\n**Minor comments** (that did not affect my score)\n- Abstract: The sentence“ A set of language models are initialized from the same base model and then are specialized by independently updating each model using data generated by the model under multiagent interaction with other models” is somewhat lengthy and could be simplified for readability.\n- Citation Formatting: Throughout the paper, `\\cite{}` is used in places where `\\citep{}` might be more appropriate (e.g., lines 137, 244, 448).\n- Figure: In Figure 4, what are the multiple dots in each box for each method? I’m assuming that their order is also the order of performance across iteration but it’s not obvious.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RfgBlrwMH6", "reviewer": "Reviewer_85F3", "summary": "This paper explores the fine-tuning of multiple LLM agents within the framework of multi-agent debate. The authors propose a method where multiple LLMs are fine-tuned and organized into generation and critic agents to facilitate multi-agent debate. Experiment results indicate that the proposed method outperforms baseline methods, and the authors further demonstrate that fine-tuning multiple LLMs helps preserve diversity compared to fine-tuning a single LLM.", "questions": "1.\tLine 77: How does the proposed approach “promotes diversification within the society of models”? \n2.\tWhat precisely does \"Single-Agent\" in \"Fine-tuning Single Agent\" refer to? Is it intended to indicate one generation agent or a single LLM? If the latter, a more fitting term might be “Fine-tuning Single LLM.”\n3.\tHow is a fair comparison with baseline methods established?\n4.\tHow does varying the number of agents affect the performance of the proposed method?\n5.\tIt is pretty costly to train multiple LLMs, especially considering the inference-time compute and resources required to serve N LLMs. A straightforward possible strategy to avoid training multiple LLMs while also maintaining diversity is to include an unique identifier (e.g. an ID) or a special token in the input for each agent. How does this strategy compare to finetuning multiple LLMs?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tJointly optimizing the LLM in the roles of generators and critics appears to be a robust method for enhancing the reasoning ability of LLMs.\n2.\tThe work shows that finetuning multiple LLMs on independent datasets derived from multi-agent debate can preserve diversity, which is a critical challenge for LLM finetuning.\n3.\tThe evaluation results show the strength of the proposed method.", "weaknesses": "1.\tThe title “Multiagent Finetuning of Language Models” may imply a broader scope than the paper addresses. Multi-agent applications of language models can indicate a much broader range of settings besides reasoning tasks and multi-agent debate, such as gaming and social simulation; however, this work focuses solely on multi-agent debate.\n2.\tThe terms  “Single Agent” and “Multi Agent” is vague and unclear in this paper. For example, Sec 2.2 “Fine-tuning Single Agent”discusses scenarios involving multiple agents rather than a true single-agent setting.\n3.\tThis work might not obey the standard training and evaluation procedure on GSM and MATH dataset, as only 500 examples are selected for training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)