Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains

Authors: Vighnesh Subramaniam, Yilun Du, Joshua B. Tenenbaum, Antonio Torralba, Shuang Li, Igor Mordatch
Primary area: foundation or frontier models, including LLMs
Keywords: Language Models, Multi-agent Interaction, Self Improvement

Abstract:
Large language models (LLMs) have achieved remarkable performance in recent years but are fundamentally limited by the underlying training data. To improve models beyond the training data, recent works have explored how LLMs can be used to generate synthetic data for autonomous self-improvement. However, successive steps of self-improvement can reach a point of diminishing returns. In this work, we propose a complementary approach towards self-improvement where finetuning is applied to a multiagent society of language models. A group of language models, all starting from the same base model, are independently specialized by updating each one using data generated through multiagent interactions among the models. By training each model on independent sets of data, we illustrate how this approach enables specialization across models and diversification over the set of models. As a result, our overall system is able to preserve diverse reasoning chains and autonomously improve over many more rounds of fine-tuning than single-agent self-improvement methods. We quantitatively illustrate the efficacy of the approach across a wide suite of reasoning tasks.

Decision: accept (poster)
Reviewer scores: 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730640509555

thread_type:
official_review

<thread_object>
creator: reviewer_85f3

<content>
summary:
This paper explores the fine-tuning of multiple LLM agents within the framework of multi-agent debate. The authors propose a method where multiple LLMs are fine-tuned and organized into generation and critic agents to facilitate multi-agent debate. Experiment results indicate that the proposed method outperforms baseline methods, and the authors further demonstrate that fine-tuning multiple LLMs helps preserve diversity compared to fine-tuning a single LLM.

strengths:
1.	Jointly optimizing the LLM in the roles of generators and critics appears to be a robust method for enhancing the reasoning ability of LLMs.
2.	The work shows that finetuning multiple LLMs on independent datasets derived from multi-agent debate can preserve diversity, which is a critical challenge for LLM finetuning.
3.	The evaluation results show the strength of the proposed method.

weaknesses:
1.	The title “Multiagent Finetuning of Language Models” may imply a broader scope than the paper addresses. Multi-agent applications of language models can indicate a much broader range of settings besides reasoning tasks and multi-agent debate, such as gaming and social simulation; however, this work focuses solely on multi-agent debate.
2.	The terms  “Single Agent” and “Multi Agent” is vague and unclear in this paper. For example, Sec 2.2 “Fine-tuning Single Agent”discusses scenarios involving multiple agents rather than a true single-agent setting.
3.	This work might not obey the standard training and evaluation procedure on GSM and MATH dataset, as only 500 examples are selected for training.

questions:
1.	Line 77: How does the proposed approach “promotes diversification within the society of models”? 
2.	What precisely does "Single-Agent" in "Fine-tuning Single Agent" refer to? Is it intended to indicate one generation agent or a single LLM? If the latter, a more fitting term might be “Fine-tuning Single LLM.”
3.	How is a fair comparison with baseline methods established?
4.	How does varying the number of agents affect the performance of the proposed method?
5.	It is pretty costly to train multiple LLMs, especially considering the inference-time compute and resources required to serve N LLMs. A straightforward possible strategy to avoid training multiple LLMs while also maintaining diversity is to include an unique identifier (e.g. an ID) or a special token in the input for each agent. How does this strategy compare to finetuning multiple LLMs?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730648944687

thread_type:
official_review

<thread_object>
creator: reviewer_tefj

<content>
summary:
This paper proposes a method to address a key limitation of LLMs: their dependency on static datasets, which limits their ability to self-improve. The authors propose a multiagent approach for LLM finetuning that enables iterative self-improvement through interactions with other models. By using distinct Actor (generation) and Critic (evaluation) agents, their approach improves feedback quality and response diversity, which results in better responses over iterations of finetuning. The authors empirically demonstrate their approach's effectiveness across 3 reasoning benchmarks (Arithmetic, MATH, GSM), showing performance gains of approximately 1-15% across 4 baselines. Additionally, the finetuned agents may demonstrate zero-shot generalization from MATH to the other two benchmarks.

strengths:
- Significance: This paper presents a promising approach to LLM self-improvement and could offer a valuable contribution.
- Clarity: Most of the Figures in the paper are clear and the paper is generally well-written.

weaknesses:
There are several comments I would like the authors to address to make some details clearer and the paper more complete.

**Major comments**
1. Role Specialization: The paper introduces distinct roles for models (generation agents and critic agents). However, it would be helpful to clarify the specific objectives each role optimizes. Additionally, I suggest emphasizing that only two roles are used in this paper (generation and critic) to avoid confusion. 
2. Zero-shot Generalization: In Section 4.3 The authors claim zero-shot generalization on held-out benchmarks. I find the claims made in Section 4.3 not very convincing for three reasons:
- (a) The authors only evaluate on 100 random samples. Could the authors elaborate on why this specific sample size was chosen, and why not more?  If it is computationally possible to evaluate more samples (e.g. 1000), this would make the evidence for zero-shot transfer more convincing.
- (b) In Figure 5, the standard errors for performance (accuracy) are absent; could you please report these like was done in the Table 2
- (c) The choice of finetuning on MATH and evaluating on the GSM dataset is unclear; could you explain the rationale? Why not report all cross-dataset generalizations, such as MATH → GSM or Arithmetic → GSM? If the results are consistent across all cross-evaluations, this would strengthen the findings.
3. Computational Trade-offs: The limitations section mentions increased computational demands but does not quantify these trade-offs. Details on the additional costs or time requirements would help to evaluate the compute/performance balance.
4. Limitations Discussion: The current discussion of limitations is very short. Future researchers building on this work would benefit from more detailed insights into any bottlenecks or constraints of the approach. Based on these more detailed suggestions for future research directions would be valuable.
5. Response diversity: Looking at the slope, Figure 3 seems to indicate that all the benefits could be coming from the initial improvement in diversity. It seems important to mention the initial diversity for each ablation. 
6. Standard error: How exactly is the standard error in Table 2 computed?

**Minor comments** (that did not affect my score)
- Abstract: The sentence“ A set of language models are initialized from the same base model and then are specialized by independently updating each model using data generated by the model under multiagent interaction with other models” is somewhat lengthy and could be simplified for readability.
- Citation Formatting: Throughout the paper, `\cite{}` is used in places where `\citep{}` might be more appropriate (e.g., lines 137, 244, 448).
- Figure: In Figure 4, what are the multiple dots in each box for each method? I’m assuming that their order is also the order of performance across iteration but it’s not obvious.

questions:
See my questions in the "Major comments"
</content>
</thread_object>

</thread>
<thread>
creation date:
1730688047503

thread_type:
official_review

<thread_object>
creator: reviewer_tw8m

<content>
summary:
This paper contributes to the field of self-improvement fine-tuning for LLMs by proposing a multi-agent cooperation approach. It replicates an LLM into multiple generation agents and corresponding critic agents. A multi-agent debate architecture is utilized to generate label responses for generation agents and critic agents. Each agent is then fine-tuned using its unique generated dataset. During the inference phase, the final result is generated following the multi-agent debate. The method demonstrates superiority over the baselines in a series of math-related language reasoning tasks.

strengths:
1. The paper is easy to follow and the content is well-organized.
2. The paper proposes a method for agent self-improvement fine-tuning based on multi-agent collaboration, allowing for multiple rounds of self-improvement fine-tuning, which could be a promising approach.

weaknesses:
My primary concerns with this paper are centered around the experimental section.

(Major)The first concern is regarding the selection of experimental datasets. The paper exclusively uses mathematical language reasoning tasks, and each task is not particularly challenging. Arithmetic is limited to arithmetic operations, GSM corresponds only to Grade School level difficulty, and MATH selects only the first three levels. If the tasks are not challenging enough, it may lead to questioning the need for incorporating multiple LLMs when a single LLM might be adequate. What’s more, conducting generalization experiments solely within mathematical datasets may not demonstrate the unique advantages in terms of generalizability, given the highly similar nature of the tasks. Including a graduate-level mathematical reasoning set or challenging datasets from other fields would make the experiments more convincing.

(Major)The second concern is that the performance improvement on datasets of modest difficulty is not significant, especially considering the introduction of multiple large models collaborating. It may be worthwhile to quantify other metrics beyond answer accuracy, for example, the KL divergence of each LLM from the original LLM distribution? Introducing multiple large models might allow agents to achieve better results without significant parameter changes.

*Overall, I believe the idea of the paper is commendable. However, as a practice-oriented paper lacking in theoretical explanation, its experimental design is insufficient. Without addressing these major concerns in the experiments, I'm afraid this is the highest score I can give.*

（Minor）The diversity of models: When reading the sentence " each model to capture parts of a task of interest.", I am very excited. However, the authors do not take any measures during the implementation to activate the heterogeneous characteristics among the generative agents. In other words, diversity is only brought about by the quantity of agents. The diversity cannot be theoretically guaranteed, nor can it be conceptualized and defined for specific agents after it appears. If the authors could enable different agents to collaboratively complete a complex task and truly capture the sub-tasks they are interested in or excel at, it might greatly enhance the contribution of this paper.

questions:
In addition to my primary concerns, there are also the following questions: 
1. In this paper, is each individual agent fine-tuned using the SFT (a combination of query and label response data) approach? If yes, can this method be integrated with DPO (a combination of query and human feedback preferences data) and PPO (a combination of query and reward signal data) approaches? Please provide a detailed discussion. 
2. How is the Combination of datasets implemented in line 25 of Algorithm 1 pseudocode? What does the hyperparameter denote? The authors need to clearly explain how the two types of data are used respectively in the training of the critic agent and what the role of the weight hyperparameter is.
3. The abstract mentions “multiagent society of language models”, yet the experiments only utilized 3-5 agents (and the improvement with 5 agents does not seem significant). How do the authors view the relationship between the number of agents and performance improvement, as well as the issue of number versus resource consumption?
4. The selection of baselines. From the setup of the baselines, this work seems more like an “A+B+C” approach. However, would it be fair to compare “B+C” with B alone, even if the A module is ablated?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732320651719

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank the reviewers for their constructive reviews. We are glad reviewers found the paper well-written and easy to follow (Tw8m, TeFj), found the approach promising in preserving diversity and improving performance (Tw8m, TeFj, 85F3), and found the evaluation robust (85F3). We address some common feedback here as well as provide an overview on what we have updated in the paper. 

* More difficult tasks and datasets; larger scale evaluation (Tw8m, 85F3, TeFj): Reviewers requested further evaluation of multiagent finetuning on more difficult tasks and larger evaluations. We cover some additional experiments we included.
    * MATH Levels 1-5: Instead of evaluating on MATH examples from the first three levels, we evaluate on  examples from all five levels. We find that multiagent finetuning performs significantly better than all baselines in this case. This is included in Appendix Section H and results are visualized here: https://ibb.co/fqppHjW. 
    * MMLU: We add an additional comparison with baselines with MMLU to further address concerns that our evaluation is only on math-related datasets. We include this as Appendix Section K and are visualized here: https://ibb.co/7r722kp. 
    * Zero-Shot evaluation (TeFj): In the paper, we evaluate the zero-shot generalization capabilities of multiagent finetuning over 100 examples, transferring abilities gained from MATH to GSM. We evaluate 1000 examples to further assess capabilities on held-out generalization. We also assess whether we can train on arithmetic and transfer capabilities to GSM over 1000 examples. This is shown in Appendix Section L. 
* Fairness of baselines (Tw8m, 85F3): Two reviewers asked about whether the baselines were fairly established for comparison. Our goal was to use baselines that use methods used for evaluating LLMs currently. Our baselines use different strategies that include (1) different prompting strategies over base, unfinetuned LLMs, (2) multiagent finetuning with simpler data generation methods than debate, and (3) finetuning methods like STaR. 
* Limitations (TeFj): We add a further discussion of limitations to the paper.
* New diversity metric, likelihoods: We added a new diversity metric for measuring response diversity among our finetuned agents based on likelihood. In this metric, we take responses from two of our critic agents and feed them to a third held-out critic agent at a particular iteration. We measure the negative log-likelihood (NLL) of these responses. The same approach is used for our Single-Agent FT ablation but over general finetuned agents instead of critic agents. As agents diversify, we should expect NLL to increase across iterations of finetuning. This is exactly what we find as seen in Appendix Section M and we include the result here:  https://ibb.co/FscDc97. Our method increases the diversity of model responses while improving accuracy.
* Writing changes: We summarize writing changes here and all changes in the paper are highlighted in blue. 
    * Section 2: We make the language clearer around generation agents and critic agents and clarify notation on the parameter “w”.  We make the distinction between single agent and multiple agents clearer.
    *  Appendix Section G: We include a new appendix section to discuss the theoretical intuition of our approach which includes the ideas discussed here. 
    * We have fixed smaller typos with our citations and other writing typos. We thank reviewers for finding these.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a novel and effective approach to enhancing LLM reasoning through jointly optimizing LLMs as generators and critics.  The reviewers appreciate the robustness demonstrated by this method, particularly its ability to preserve diversity during finetuning through independent datasets derived from multi-agent debate. The evaluation results convincingly support the effectiveness of the proposed technique.  While there are some concerns regarding the clarity of terminology and scope implied by the title, as well as the potential deviation from standard dataset usage, these issues do not detract from the core contribution of the work. The strengths of this paper, namely its innovative methodology and strong results, significantly outweigh the weaknesses. The authors are encouraged to revise the title and clarify terminology for the final version. Further clarification on the dataset usage would also be beneficial.  Overall, this paper offers a valuable contribution to the field of LLM reasoning and is well-suited for publication.