Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent System

Authors: Weize Chen, Jiarui Yuan, Chen Qian, Cheng Yang, Zhiyuan Liu, Maosong Sun
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: llm agent, multi-agent, inference scaling law

Abstract:
Large Language Model (LLM) based multi-agent systems (MAS) show remarkable potential in collaborative problem-solving, yet they still face critical challenges: low communication efficiency, poor scalability, and a lack of effective parameter-updating optimization methods for multi-agent collaboration. We present Optima, a novel framework that addresses these issues by significantly enhancing both communication efficiency and task effectiveness in LLM-based MAS through LLM training. At its core, Optima employs an iterative generate, rank, select, and train paradigm, incorporating a reward function that balances task performance, token efficiency, and communication readability. We explore various RL algorithms, including Supervised Fine-Tuning, Direct Preference Optimization, and their hybrid approaches, providing insights into their effectiveness-efficiency trade-offs for iterative LLM-based MAS training. Additionally, we integrate Monte Carlo Tree Search-inspired techniques for DPO data generation, conceptualizing conversation turns as tree nodes to explore diverse interaction trajectories. We evaluate Optima on common multi-agent tasks, including information-asymmetric question answering and complex reasoning. Our method demonstrates consistent and substantial improvements over single-agent baselines and vanilla MAS based on Llama 3 8B, achieving up to 2.8x performance gain with less than 10\% tokens on tasks requiring heavy multi-agent information exchange. Moreover, Optima's efficiency gains open new possibilities for leveraging inference-compute more effectively, potentially leading to improved inference-time scaling laws. By addressing fundamental challenges in multi-agent collaboration and providing a novel optimization framework, Optima shows the potential towards scalable, efficient, and effective LLM-based MAS.

Decision: reject
Reviewer scores: 6, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1729561234069

thread_type:
official_review

<thread_object>
creator: reviewer_ubkl

<content>
summary:
The paper introduces OPTIMA, a novel framework designed to optimize Large Language Model (LLM)-based multi-agent systems (MAS), which shows the potential towards scalable, efficient, and effective LLM-based MAS.. The key focus is on enhancing communication efficiency and task effectiveness through an iterative generate, rank, select, and train paradigm. OPTIMA utilizes a hybrid of reinforcement learning (RL) techniques, including Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO), along with Monte Carlo Tree Search (MCTS)-inspired data generation. The framework is evaluated on two multi-agent settings: (a) information exchange, including information-asymmetric question answering and  (b) debate, encompassing mathematical and reasoning tasks. OPTIMA’s efficiency gains open new possibilities for leveraging inference-compute more effectively, potentially leading to improved inference-time scaling laws.

strengths:
1, Clarity: The paper is well written and have full proof for their argument, the concepts in this paper are easy to follow and understand.
2, Quality: The performance of their designed pipeline is impressive and outperform the baseline a lot.
3, Originality: The paper introduced a new way to enable the development of MAS that are not only effective and efficient but also maintain interpretable communication patterns, which solves the core issues of communication efficiency and collective optimization.

weaknesses:
1, The method is only evaluated on two tasks: (a) information exchange, including information-asymmetric question answering and  (b) debate, encompassing mathematical and reasoning tasks.  Although the performance of proposed method is great on those two tasks, but not sure how this method will perform on the other more complex tasks such as: StarCraft II, Hanabi and so on. (those two environment are broadly used in reinforcement learning tasks and there are also works of using LLM to play those two games)

2, The scalability of this method. The number of agents in showed tasks are limited,. And because the proposed method are aimed to solve the problem of multi-agent tasks, using more agents in environment(maybe at least 4 or 5 agents?) will be more convincible.

3, They use Llama 3 8B as their base model across all benchmarks. However, can this method still have significant improvement compared to baseline if using other base models still remains as a question and need further proof.

questions:
The method proposed in this paper is great, but still I have following concerns which may need further experiments:
1, More tasks study will make this proposed method more convincible, including using more complex tasks and more agents in a task. (StarCraft II, Hanabi and any other tasks which are broadly used in using LLM in MAS problems)

2, Try at least one more base model on all the tasks such as Llama 3.2 or Llama 3 70B
</content>
</thread_object>

</thread>
<thread>
creation date:
1729755747467

thread_type:
official_review

<thread_object>
creator: reviewer_yhqv

<content>
summary:
This paper investigates inter-agent communication and task inference effectiveness within LLM-based MAS. It introduces a framework based on an iterative generate, rank, select, and train paradigm to address these challenges. The core of this framework is iteration, and build iSFT and iDPO based on the iteration paradigm. The iSFT leverages prompt formats to create a diverse dataset. Subsequently, it removes these formats and fine-tunes the model based on the generated trajectories. The iDPO employs Monte Carlo Tree Search (MCTS) to generate diverse data through multi-agent conversations (MAD), alternating between MCTS-based data generation and model updates using DPO.
The authors evaluate this framework across several benchmarks using Llama 3 8B as the baseline model. They achieve promising results compared to Chain-of-Thought (CoT), SC (n=8), MAD, and AutoForm.

strengths:
The method is written with significant detail, which is easy to follow.

Introducing an effective and efficient LLM-based frameework, which can be treated as a foundational model, seems important and interesting.

Experiments are provided for various environments and demonstrate promising results.

weaknesses:
1. Unfair Comparison: The methods in OPTIMA do show less token consumption and higher accuracy compared to other approaches. However, the comparisons between iSFT, iDPO, or iSFT-DPO and other methods may be unfair. The authors did not clarify whether OPTIMA’s training is conducted online or offline (I assume it is offline). If it is online, the token consumption should not be lower than CoT (since MCTS requires searching across 24 trajectories). If it is offline, then the token consumption comparison is unfair, as the OPTIMA methods fine-tune the model on diverse prompts or multiple sampled data, which would have already consumed a significant amount of tokens during the post-training phase. The paper, however, only compares token consumption during the inference process.

2. Lack of Novelty: The methods iSFT and iDPO seem to lack innovation. iDPO merely combines MCTS from ToT with DPO, while iSFT simply adds a step of supervised fine-tuning (SFT) after removing the prompt. These methods seem incremental rather than novel, similar approaches can already be found, such as [1], [2], [3], [4], [5].

3. Reward Function: The authors mention the reward function multiple times, but it is only briefly defined in line 146. Meanwhile, in line 79, the reward is described as the core element of OPTIMA’s success. This undermines the credibility of the proposed method since the reward function is not thoroughly explained or explored.

4. Scalability: The authors frequently mention the scalability of OPTIMA, which seems exaggerated. There is no detailed discussion of the framework's ability to scale up in the experiments. Additionally, the token consumption during fine-tuning with iSFT and iDPO would increase significantly as the number of agents grows, which could limit the scalability of the framework.

[1]Wu T, Li X, Liu P. Progress or regress? self-improvement reversal in post-training[J]. arXiv preprint arXiv:2407.05013, 2024.

[2]Pang R Y, Yuan W, Cho K, et al. Iterative reasoning preference optimization[J]. arXiv preprint arXiv:2404.19733, 2024.

[3] Mukobi G, Chatain P, Fong S, et al. SuperHF: Supervised Iterative Learning from Human Feedback[J]. arXiv preprint arXiv:2310.16763, 2023.

[4]Xiong W, Dong H, Ye C, et al. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint[C]//Forty-first International Conference on Machine Learning. 2024.

[5] Pang R Y, Yuan W, Cho K, et al. Iterative reasoning preference optimization[J]. arXiv preprint arXiv:2404.19733, 2024.

questions:
As the weakness mentioned.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730684872564

thread_type:
official_review

<thread_object>
creator: reviewer_gssx

<content>
summary:
This work proposes a framework that improve the effectiveness and efficiency of LLMs in multi-agent dialogue systems by iteratively optimizing data and training LLMs using SFT/DPO. For iSFT, it use sampling to generate better data; for iDPO, it use MCTS to generate paired data. Additionally, a carefully designed reward function ensures the system's overall effectiveness and efficiency in task performance.

strengths:
The paper is well-structured and easy to understand, addressing an important problem with a clearly articulated methodology. The data generation mechanism for MAS is interesting and may facilitate further improvements in MAS, including frameworks like AutoGen.

weaknesses:
1.	While this paper offers a well-structured approach to enhancing the effectiveness and efficiency of MAS, iSFT and iDPO have been extensively explored in prior works. Thus, the main contributions here—reward function design and data improvement mechanism in MAS—offer limited novelty within the existing research landscape.

2.	A thorough and fair experimental comparison is very important if the novelty is limited. Another key weakness of this paper is unfair comparison. As this framework has trained LLM to maximize a delicate reward, it’s slightly unfair to compare with prompt-tuning method like CoT, SC, etc.

3.	This paper focuses on LLM-based MAS, but the definition of the problem and the specific methods do not adequately reflect the multi-agent aspect. It appears that the multi-agent scenario is only evaluated in the experiments. The methods are relatively general and do not address the key issues in MAS.

Other trivial weakness in this paper:

1.	Fact error in abstract: iSFT and iDPO with reward filters can be considered as RL, but SFT and DPO are not RL.

2.	The language modeling loss in equation 1 is not defined.

questions:
1.	Figure 3(a) need further elaboration. How does Optima influence MAS’s inference scaling law comparing to baseline?
2.	I would appreciate further discussion on the advantage of optimizing the effectiveness and efficiency of a MAS versus optimizing these aspects of a single LLM using similar reward functions.
3.	Would improvements in effectiveness and efficiency facilitate the design of communication topologies within MAS?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731118834379

thread_type:
official_review

<thread_object>
creator: reviewer_kkwh

<content>
summary:
The paper presents a collaborative training method for systems of LLMs, Optima. Optima features a carefully crafted reward function that incentivizes the communications readibility, task performance, and token efficiency. Additionally, a variant of Optima employs MCTS to sample diverse trajectories. Trajectories are then trained on using SFT or DPO. The authors evaluate their algorithm on diverse complex reasoning and information-asymmetric question answering tasks and observe performance improvements in both tasks performance and token efficiency.

strengths:
The authors tackle an extremely timely problem with significant community interest - how to optimise LLM agents for collaborative tasks.
There are several algorithmic innovations, and the empirical evaluation is extensive.
The observed improvements are significant.
Last but not least, there is a very extensive treatment of related work.

weaknesses:
* lack of theoretical examination of why the reward function used is working 
* lack of evaluation of models other than llama 8bn (other/smaller models would have been interesting to see).

questions:
* what about systemic biases/safety misalignment arising from training?
* can we extend this to systems of more than 2 agents, and how does this scale?
* could you provide assurance that results are not from training on test data (lacks details)
</content>
</thread_object>

</thread>
<thread>
creation date:
1732118199008

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Responses to All Reviewers

comment:
We sincerely thank all the reviewers for their thoughtful feedback and for recognizing various aspects of our work. We would like to take this opportunity to emphasize the novelty and significance of the problem we address, which we feel underpins the contributions of our paper.

---

**Highlighting the Novelty of the Addressed Problem**

Our work is among the **first to highlight the importance of token efficiency** in multi-agent systems (MAS). Communication efficiency is a critical, yet underexplored, aspect of MAS, and addressing it is key to making these systems more practical for real-world applications. Furthermore, we are the **first to leverage training-based methods** to jointly optimize token efficiency and task performance in this context, setting our approach apart from inference-only techniques, such as prompt engineering, which often offer limited improvements.

---

**Importance of Inference Scaling Laws**

Inference scaling laws, which analyze the relationship between inference compute and model performance, are garnering increasing attention in the research community. Unlike adding training compute, which requires substantial resources and time, increasing inference compute has been shown to be a **highly effective alternative** for improving LLM systems. This direction is gaining recognition as an important pathway for achieving better performance without the prohibitive costs of additional training.

Our work makes a significant contribution by demonstrating how reducing inference token usage positively impacts **inference-time scaling laws**, leading to improved efficiency and performance. We believe that the extensive and solid empirical results presented in our paper clearly reveal the benefits of this approach.

---

**Feasibility and Impact of Our Contribution**

Beyond addressing the theoretical significance, we also provide a **feasible and easy-to-understand pipeline** to achieve these benefits, making our methods accessible to the broader research community. By presenting a practical framework for reducing inference tokens while maintaining or improving task performance, our work opens new possibilities for optimizing MAS and paves the way for future research in this critical area.

---

We hope that reviewers will consider the novelty and importance of the problem we address, along with the strong empirical evidence supporting our contributions. We believe that our work represents a meaningful advancement in the field and contributes to the growing interest in optimizing inference efficiency and performance in LLM-based MAS.

Thank you again for your careful review and for engaging with our work. We welcome any additional questions or suggestions and look forward to further improving our manuscript based on your valuable feedback.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732504624697

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Request for Feedback

comment:
Dear reviewers,

We sincerely thank you for your thoughtful comments and valuable feedback on our submission. We have carefully addressed all the concerns and questions raised, providing detailed responses and additional experimental results where necessary. Your insights have significantly helped us refine and strengthen our work.

As the discussion phase is nearing its conclusion, we kindly request your feedback on our responses at your earliest convenience. Your input is invaluable in ensuring that we address any remaining concerns effectively. If there are any aspects of our responses that require further clarification or additional experiments, we would be happy to provide them within the remaining time.

Thank you again for your time and effort in reviewing our paper. We greatly appreciate your engagement and look forward to your thoughts.

Best,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a training method for multi-agent systems of LLMs. Unfortunately reviewers were concerned with novelty and incrementality of the approach.