Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains
Authors: name: Vighnesh Subramaniam; name: Yilun Du; name: Joshua B. Tenenbaum; name: Antonio Torralba; name: Shuang Li; name: Igor Mordatch
Published: 2025-01-10 04:35:46

Abstract: Large language models (LLMs) have achieved remarkable performance in recent
years but are fundamentally limited by the underlying training data. To improve
models beyond the training data, recent works have explored how LLMs can be
used to generate synthetic data for autonomous self-improvement. However,
successive steps of self-improvement can reach a point of diminishing returns.
In this work, we propose a complementary approach towards self-improvement
where finetuning is applied to a multiagent society of language models. A group
of language models, all starting from the same base model, are independently
specialized by updating each one using data generated through multiagent
interactions among the models. By training each model on independent sets of
data, we illustrate how this approach enables specialization across models and
diversification over the set of models. As a result, our overall system is able
to preserve diverse reasoning chains and autonomously improve over many more
rounds of fine-tuning than single-agent self-improvement methods. We
quantitatively illustrate the efficacy of the approach across a wide suite of
reasoning tasks.