SiriuS: Self-improving Multi-agent Systems via Bootstrapped Reasoning
Authors: Wanjia Zhao
Mert Yuksekgonul
Shirley Wu
James Zou
Venue: NeurIPS 2025 poster ()
Keywords: Multi-Agent
Self-Improving
LLM Reasoning
Primary area: deep_learning
TL;DR: We introduce SiriuS, a self-improving framework for multi-agent systems that builds and refines an experience library, enhancing the system's ability to tackle complex reasoning problems with minimal supervision.

Abstract: Multi-agent AI systems powered by large language models (LLMs) are increasingly applied to solve complex tasks. However, these systems often rely on fragile, manually designed prompts and heuristics, making optimization difficult. A key challenge in optimizing multi-agent systems is acquiring suitable training data for specialized agents. We introduce SiriuS, a self-improving, reasoning-driven optimization framework for multi-agent systems. Central to our approach is the construction of an experience library: a repository of high-quality reasoning trajectories. The library is built by retaining reasoning steps that lead to successful outcomes, providing a robust training set for optimizing multi-agent system. Additionally, we introduce a library augmentation procedure that refines unsuccessful trajectories, further enriching the library. SiriuS boosts performance by 2.86% to 21.88% on reasoning and biomedical QA and enhances agent negotiation in competitive settings. Our results show that SiriuS enhances multi-agent performance while generating reusable data for self-correction and self-play enhancement in the future.

SiriuS: Self-improving Multi-agent Systems via Bootstrapped Reasoning
Wanjia Zhao, Mert Yuksekgonul, Shirley Wu, James Zou Department of Computer Science Stanford University
Abstract
Multi-agent AI systems powered by large language models (LLMs) are increas- ingly applied to solve complex tasks. However, these systems often rely on fragile, manually designed prompts and heuristics, making optimization difficult. A key challenge in optimizing multi-agent systems is acquiring suitable training data for specialized agents. We introduce SIRIUS, self-improving, reasoning-driven optimization framework for multi-agent systems. Central to our approach is the construction of an experience library: a repository of high-quality reasoning jectories. The library is built by retaining reasoning steps that lead to successful outcomes, providing a robust training set for optimizing multi-agent system. Ad- ditionally, we introduce library augmentation procedure that refines unsuccessful trajectories, further enriching the library. SIRIUS boosts performance by 2.86% 21.88% on reasoning and biomedical QA and enhances agent negotiation in com- petitive settings. Our results show that SIRIUS enhances multi-agent performance while generating reusable data for self-correction and self-play enhancement in the future.
Introduction
Multi-agent Al systems powered by large language models (LLMs), where specialized agents col- laborate to solve complex tasks, are becoming increasingly successful in real-world applications. Recent work has demonstrated their efféctiveness in complex reasoning (Wang al. 2024: Smit
et al. 2024), coding [u et al. 2203), drug discovery (Swanson et al., (2024) and ensuring safety via debate (Chern al. 2024 Irving et al. 2018). These successes arise from specialized agents inte- grating their distinct capabilities through structured interactions, enabling more effective problem- solving than single agents. Moreover, multi-agent scrutiny acts as built-in self-correction mech- anism, where agents refine and verify each other's outputs. This often outperforms single agent setting, particularly on tasks demanding rigorous reasoning or factual validation. Despite these suc- cesses, optimizing multi-agent systems remains fundamental challenge due to (1) the difficulty of acquiring appropriate training signals for each agent and (2) the sensitivity to multiple moving parts that influence overall performance (Smit et al.) 2024). While task-level reward feedback is avail- able, credit assignment across agents remains ambiguous failure to specific intermediate decisions and reasoning steps made by each LLM agent. This chal- lenge parallels the multi-agent credit assignment problem in reinforcement learning (Foerster et al. 2018). However, in language-bar based systems, reasoning unfolds through complex and unstructured interactions, making attribution far more difficult than in traditional RL settings with well-defined action spaces.
unclear how to attribute success or
We present SIRIUS, a framework for learning effective multi-agent behaviors from outcome rewards. Our key insight is that when multiple agents successfully solve a task together, their entire interaction
Correspondence to Sranjizzh@es.stee www.eeetncs.sccccclc.oo and jamesz wttaorssssstaoo.cccuu Code is available here
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(Question) The specal light incile of nunmeggddii diclottric manerial with diclectric cometur 4.0 is A... B.., C... D... Which cormast
Iterations
Base Aultt-Agent System
experience library
Physicist
Summarizer
Trojecerno
eppito
of in Vomee The spont of tich we renonn, doserd in eppro cnfy.
sodio acaides essaga
Flars Soe Alynlia (Barрineii
Fine-
cuarill moulaiin cuterarc pulcttcd. of Medius of dkinst anciiis oikulred...
Eien for repuorr Oue
Arowery
Mathematician
cibiid with de cousen
cletts
experience ougmentation
Digur: EQuestinn How tue recoeso taee piquiis (esssomal)
Ortjett
dim meiles somided play weyly onee mod mestee
The comopopn
X
&oooocio audiaa i.ickiiaa du Kaaven Valaec % cal eauciia luava valus lnto dae faciida
Firsule The
Retali....
Figure 1: General training pipeline of STRIUS.Agemt solve problems sequentially, storing correct responses for fine-tuning and augmenting incorrect ones through feedback, regeneration. and rephrasing. This iterative process improves performance via reward- based evaluation and supervised tine-tuning. The module colors in the figure correspond to those in Algorithm!]
trajectory likely contains useful patterns even if we cannot pinpoint exactly which steps decisions were crucial for success. Drawing inspiration from recent advances in bootstrapping reasoning capabilities (Zelikman al. 2022). we collect and learn from successful agent interactions across many tasks, allowing the system to iteratively discover effective collaboration strategies from self- generated data. This approach sidesteps the need for direct supervision of intermediate steps, instead letting agents learn which interaction patterns tend to lead to successful outcomes. For trajectories that result in failed attempts, we perform trajectory augmentation by resampling original attempts with feedback from an additional agent grounded in the ground truth.
Our experiments demonstrate that SIRIUS significantly enhances multi-agent performance across multiple domains. It improves reasoning and biomedical QA accuracy by 2.86% to 21.88%, while also strengthening agent negotiation in competitive scenarios. Beyond these gains, our approach of- fers a scalable mechanism for seff-improvement, enabling agents to iteratively refine their reasoning and collaboration strategies. More broadly, SIRIUS provides general framework for optimizing multi-agent systems via self-generated synthetic data, offering a principled way to enhance perfor- mance without requiring fine-grained human supervision.
2 Method
2.1
Multi-agent systems with LLMS
We define a multi-agent system by a tuple USSAAT,R,N,, Here, V & (A(")A(2
...., A(N)
is the set of N agents, each agent A uses a policy Ti parameterized by g00) e S is the state of the environment, a E A the joint actions, and A is the joint action space. T S x 4 S the transition function where T(s, a) yields the next state of the environment given the current state and joint actions a. The environment feedback modeled via a payoff function R, which provides rewards for each agent & based on the state-action pairs.
A RN,
The communication structure between agents is modeled as directed graph g (V, E, P), where V represents agents, and F. defines interaction order.
For each edge (i.j) E E,, agent A receives an input derived from the state-action pair (s, a) and
the output of agent A(1) This input determines agent A0i's subsequent action. For each agent ( in topological graph G. its predecessors are the set of agents that influence its output: (r)(
(AG) (AC), A G). Here,
denotes a directed edge in the graph, indicating that the output of agent AD) directly influences the input of agent (
Throughout this paper, the collection of our agents will be based on language models and the primary environment that we use will be natural language. In particular:

---

0; (-(-18t, {az) (PPm((11)))))
AA EN
a, (01, aN) = (s, ar) (sn at)
(I)
where denotes the probability distribution of the i-th language model, Concat is the concatenation of the previous state and the responses, and we will use т {TI...... ................................ AN } to denote the joint policy. Generally, each agent aims to maximize its own reward: max=; E,, [2 (s,,,, ac)]. where R; denotes the i-th component the reward vector R and the expectation is taken under the joint policy
2.2 SIRIUS
The training pipeline of the proposed framework, denoted SIRIUS, is illustrated in Figure | SIRIUS adopts fine-tuning strategy iteratively improve the policy parameters ((m) of each agent A(") over T iterations. The process is initialized with dataset D {z,, y)) where each pair (ma, 81) represents a problem and solution. The core training procedure is outlined in Algorithm
Algorithm 1 SIRIUS
I: Input: A group of agents ( ,.... A AN) An initial dataset problems x with answer D [(r4, 3:))=) total number of fine-tuning Iterations T. Initialize: Initialize policy parameters g(*) for each agent A(m) 1,2...... N.
3: for Fine-tuning Iteration I = 1, ..... do
4: d!")) P, Pa(mn) (-12; a,
Prot A ((m))))))))))))))))) 1.2..... K.
5: for each agent 1,2,
do
6: c(") - [xx, _ li E Dj. A Rs(s, > c)}
Good Trajectory Set of Fach Agent.
Augmentation ((x.)iioon(((()))) (()) R)( < <()))
8: end for
9:
Standard SFT on c(n) 1,....
10:
end for
At each fine-tuning iteration t:
Action Sampling: For each agent A (m), action (nd)) is sampled from its policy,
(m) = Potn (- [ Pro(A(1))
(*)()(")) generated by previous
conditioned on the input problem x; and the action a agents. In scenarios involving multiple interaction rounds, such as the Competitive Setting, Pre(A()) includes outputs from all agents in all preceding rounds.
a
Trajectory Evaluation and Augmentation: The trajectories generated by each agent are evaluated using the payoff function R(s a). Based on a reward threshold high-reward trajectories (R(s, a) > e) are added to the good trajectory set c(²). Since the tasks are challenging, the good trajectory set tends to be small. To leverage more data for fine- tuning, we propose trajectory augmentation pipeline for each task, detailed in the Appendix C Specifically, we first generate feedback to refine the agent's 's original response. The feedback and original response are then combined to prompt the agent to regenerate a new solution, which is then rephrased into direct problem- cobblem-solving solving step. Afterward, we return to the action sampling process to produce the final answer and evaluate
Fine- TuningT The policy parameters g(n) updated via supervised fine-tuning c!*)
This iterative process ensures that each agent's policy progressively refined to maximize perfor- mance based on the joint system dynamics and reward.

---

Table 1: Different settings and tasks. In the corresponding to Communication Structure, nodes denote agents (V), arrows represent edges (E), and color indicates the role agents.
Settlege
Pratim--ohhta
Acme-UUttii
Computitee
Centec
Fower)
Structone 17, 8.79)
Querion
CISL
Oolipp@tanics
PAAMOOON
PaiMoRQA
Receere Saclinng Uliestan
agrat
3 Multi-agent Settings
In this section, we explore several settings where agents with distinct expertise interact to solve challenging tasks. As shown in Table we systematically analyze different agent configurations.
3.1 Problem Solving Settings
Agents with Specific Expertise. In this setting, each agent is assigned coomann-speciff role to fa- cilitate structured and efficient problem-solving process. For instance, the physics and chemistry domains, the problem-solving pipeline begins with domain expert (e.g., physicist chemist) who analyzes the Nomain-specifte problem, followed by mathematician who formalizes the reasoning with quantitative models, and finally, summarizer who consolidates the insights into clear and comprehensive answer. This sequential collaboration ensures that the expertise of each agent is leveraged effectively while maintaining clarity in the solution process.
The sequential dependency between the agents can be described as follows:
Ophy ~ Thy ( (-1g);
@Math Matth (-1g, aphy):
(3)
(Sm
7 Sum ( q Ophy, (Maah);
where q is the input question, aphy is the response generated by the Physicist, aMah is the response generated by the Mathematician based both the question and the Physicist's response,asum is the final answer synthesized by the Summarizer using the question, the Physicist's response, and Mathematician's 's response.
Analyze Long Context and Answer Question. In scenarios involving lengthy and complex con- texts, we consider a common two-agent setup: the Context Analyst and the Problem Solver. The Context Analyst's responsibility to thoroughly examine the context, extract essential information, and provide a concise and accurate summary. The Problem Solver then uses this summary to analyze the question and formulate the final answer. This division of labor not only improves interpretability, but also reduces the cognitive load on each agent.
3.2 Actor-Critic Setting
The popular Actor-Criti framework facilitates iterative agent improvement through a feedback loop: the Actor Agent generates solutions while the critic evaluates and refines them, enhancing both the Actor Agent's 's reasoning and the Critic Agent's 'ss error correction capabilities. In practice, we separate judgment and feedback tasks by introducing a Judgment Agent alongside the Critic Agent, where the Judgment Agent classifies the Actor Agent's solutions as correct or incorrect, and for incorrect solutions, the critic provides feedback guide the Actor Agent in regenerating improved solutions. Reward mechanisms are designed as: the Actor Agent receives rewards for correct solutions, the Judgment Agent for accurate classifications, and the critic for providing actionable feedback that leads to correct regenerations.
3.3 Competitive Settings
Competitive scenarios (Bianchi al. 2024) examine multi-agent interactions under opposing ob- jectives, where agents must balance cooperation and competition to achieve their goals. In this

---

Table Tasks and setups the competitive setting, Each involves two agents with distinct roles, initial resources, and objectives. Rexource Exchange focuses on maximizing total resources through trade. Ultimatum requires negotiating a split of $100. Buy involves price negotiation for an item. Each task follows turn- based structure with fixed maximum number of rounds and ends when agreement is reached.
Task
Resource Exchange
Ultimatum
Sull&Buy
Roles
Player 1
Player
Player 1
Player 2
Seller
Buyer
Initial resources
25Xs, SYs
SXs, 25Ys
$ $000
0
1X Maximize price Selling price 50
100 ZUPs
Goal
Maximize total resources
Negotiate split
Minimize price
Utility
Xs
Xs Ys
Split amount-50 Split amount-50
50-Selling price
Ending condition
When either player accepts rounds of interaction
When either player accepts rounds interaction
W When either player accepts rounds interaction
Max. turns
framework, two agent roles are defined: Player I and Player 2. Each player is initialized with specific amount of resources, which evolve over the course of the game based on their interactions. The game progresses sequence of moves, resulting in trajectory of states:
Player Trajectory: T0 :11 srflaeer
paayer
plaerr
(5)
Player 2 Trajectory: ub uillye player2
2laerr2
The sequence captures the evolution of game states as players compete each timestep
0, 1, T, ultimately determining winner and loser. Our goal is to optimize each player's policy maximize own expected reward based on trajectory data and role-specific context. This can be formulated as:
T
Po (a
player2 Dri #0s--
(6)
1=1
where Player optimizes its policy based on the historical trajectory of both itself and Player 2, and similarly for Player 2. We explore three distinct competitive settings, all of which unfold over multiple rounds:
Resource Exchange Scenario. In this scenario, agents engage in simulated environment where they exchange resources to maximize their individual utility.
Seller and Buyer Scenario. This setting models economic interactions where agent assumes the role of a seller and another the role a buyer. The agents negotiate prices and terms to complete transactions, testing their ability to strategize under asymmetric setting.
Multi-Turn Ultimatum Game. The Multi-Turn Ultimatum Game explores scenarios of fairness, cooperation, and negotiation over multiple rounds. One agent proposes a division of a resource, and the other agent decides whether accept or reject it.
4 Experiments
4.1 Baseline
We compare our SIRIUS against the following baselines:
Single-Agent utilizes single language model process input and generate responses.
STaR (Zelikman et al., 2022), the Self-Taught Reasoner, focuses on enhancing the reasoning ca- pabilities of a single agent by iteratively training it to improve its step-by-step reasoning through self-supervised fine-tuning.
Prompt Multi-Agent System (CoMM) (Chen et al. 2024a) introduces training-free, multi-agent collaborative framework where agents interact and share information to solve tasks collectively.
TextGrad YYuksekgonul et al. 2024) optimizes prompts for each agent in a multi-agent system by backpropagating natural language feedback through each interaction.
DSPy (MIPROv2) (Khattab et al. is a prompt optimizer that jointly optimizes instructions and few-shot examples.
5

---

Table 3: Evaluation results of the proposed method and baselines on accuracy(%)) Best results are in bold numbers and second-bes best results are in underline numbers.
Model
Method
College Physics
College Chemistry
PubMedQA
Single-Agent
25.55 + 1.08
40.00 + 3.95
57.53 + 0.99
STaR
30.84 0.93
45.64 1.72
64.20 + 0.53
GPT-3.5-turbo
COMM
28.97 + 1.62
17.69 + 3.95
72.27 + 0.81 NA 52.67 = 1.29
TextGrad
32.09 + 1.08
41.54 + 0.00
Dspy
29.91 + L.87
39.49 2.45
SIRIUS
33.96 1.43
55.90 3.11
75.33 + 0.70
Single-Agent
38.63 +: 1.95
40.00 2.59
63.87 + 0.12
STaR
42.99 0.93
48,21 2.28
65.93 2.83
GPT-4o-mini
COMM
42.37 1.43
49.23 1.49
70.27 1 0.42
TextGrad
39.25 + 3.37
19.74 7.36
#6.20=2.11
Dspy
46.11 + 1.95
45.65 2.35
55.53 = 0.31
SIRIUS
47.35 1.95
56.41 3.11
73.67 0.31
Single-Agent
26.79 + 1.95
34.46 2.28
54.07 0.31
STaR
27.41 1.95
38.46
54.93 0.12
Llama-34 -.ama-3.2-3B--.. --inttuuucct
COMM
27.73 1.43
37.4
65.80 - 0.87
SIRIUS
29.60 1.43
42.56 2.28
68.27 + 1.03
4.2 Setup and Datasets
Backbone Model. For a fair comparison, we use gpt-3.5-turbo-011 and gpt-4o-mini-2024-0 the backbone model, and set the temperature to 0 in all our experiments. We use OpenAI's Fine-tuning API for supervised fine-tuning.
College Physies/Chemistry.
These two datasets are constructed by combining questions from Understanding (MMLU) (Hendrycks et al.] (020), Graduate-Level
Massive Multitask Language Google-Proof Q&A (GPQA) (Rein et al., 2023). and Theorem-Driven Question Answering (The- oremQA) (Chen et al. 2023). focuses on college-level physies problems, which remain difficult and demonstrate room for improvement in performance with large language models. We split the dataset into training and test sets, with the detailed data distribution provided Appendix D
PubMedQA. This is a biomedical question question-answering answering dataset comprising 1000 open- open-domain ques- tions (Jin et al. 2019), each paired with context from PubMed abstracts and corresponding answers. It focuses on research-driven queries, requiring Nomain-specifie understanding and reasoning over scientific texts. We follow the original split of the dataset for training (500) and testing (500) sets.
4.3
Experimental Result of Problem Solving Setting
4.3.1 Main Result
presentsss performance comparison of various models and methods under the Problem Solv- ing Setting. We observe that the prompted Multi-Agent System (COMM) generally improves per- formance, as agent collaboration enhances the ability to solve complex problems. STaR outperforms the base Anngle-Agent, indicating that fine-tuning contributes improved performance. For smaller and weaker models, and in scenarios with long context lengths such as PubMedQA, TextGrad faces significant challenges in mnstruction-followimn during optimization. TextGrad (GPT-3.5-turbo) could its optimizer failed to parse instructions due to the model's limited
not be applied to PubMedQA capability and the excessive context length of the problem. Similarly. TextGrad (GPT-4o-mini) struggles to generate answers in the required format, requiring manual extraction of answers. Our proposed method, SIRIUS, consistently outperforms across all tasks. By decomposing tasks into manageable sub-tasks assigned to agents and, crucially, fine-tuning each agent to specialize in its designated task, STRIUS maximizes the effectiveness of collaboration, ensuring a more coordinated and efficient overall performance.
4.3.2 Ablation Experiments
To evaluate the contributions of various components in STRIUS, we conducted series of ablation experiments. Each experiment designed to answer key question about the effectiveness of

---

the multi-agent system. All ablations were performed on representative tasks within the Problem Solving Setting (PubMedQA) to ensure consistency in evaluation as shown in Table
Does mixing STRIUS with a base agent degrade performance? To understand the benefits of jointly optimizing a collaborative multi-agent system, we first train all the agents together using SIRIUS. Then we replaced one SIRIUS agent with the original base agent-either SIRIUS Analyst + base Solver or base Analyst + SIRIUS Solver. This substitution hurts performance, demonstrating benefits from joint multi-agent optimization compared to optimizing a single agent.
Should we fine-tune different LLMs for different roles, optimize one LLM for all roles? We explored whether a single LLM fine-tuned on the combined training data of multiple roles could match the performance of separate role- cole-specific models. The results showed notable performance decline, highlighting that different roles require specialized adaptation and that a shared model strug- gles effectively generalize across distinct agent functions.
How useful experience augmentation? To as-
sess the impact of experience augmentation, we removed the augmentation module while keep-
Table 4: Ablation results on PubMcdQA.
ing the rest of the pipeline unchanged. Data aug-
Model
method
PubMed
mentation introduces more diverse and challeng- ing experiences training data, enhancing the model's 's capability; therefore, omitting the aug- mentation module could negatively impact per- formance.
SIRIUS
74.20
SIRIUS + Base
72.00
Base + SIRIUS
73.20
GPT-3.5-turbo
FT One Base LLM
70.40
STRIUS w/o Aug.
73.40
Additional FT Itc
75.00
Does additional fine-tuning improve perfor- mance? We investigated whether increasing the number of fine-tuning iterations leads to further performance gains. Each iteration follows the full optimization pipeline illustrated in Figure [ the previously fine-tuned SIRIUS is used to generate new experience library, which is then used further fine-tune the base model. As expected, an additional iteration yielded marginal performance gains, suggesting that the model can benefit from extended training.
SIRIUS
73.40
SIRIUS + Base
72.80
Base + SIRIUS
71.60
GPT-44-mimi
FT on One Base LLM
72.00
SIRIUS w/o Aug.
72.20
Additional FT Itr
73.60
4.4 Experimental Result of Actor-Critic Setting
Table 5 presents a performance comparison of various models, methods, and ablations under the Actor-Critic Setting on PubMedQA. As mentioned in Section [.2] the Actor Agent first generates a solution, which is then evaluated by the Judgment Agent determine its correctness. For solutions decmed incorrect by the Judgment Agent, the Critic Agent analyzes the original solution and pro- vides feedback without access to the correct answer. The Actor Agent then regenerates the solution based on this feedback.
A key challenge in this setting is the Judgment Agent's 's limited ability to differentiate between correct and incorrect solutions leading to two potential issues: (1) correct solutions may be mistakenly judged as incorrect and potentially modified into incorrect ones during the feedback and regeneration stages; (2) incorrect solutions may be judged as correct, failing to receive the necessary corrections. We report TP (True Positive) Accuracy as the ratio of solutions both correctly generated by the Actor and accurately validated by the Judgment Agent, while Overall Accuracy measures the total correct solutions after regeneration, accounting for the combined contributions of all agents.
We evaluate our method against two representative baselines: (I) Self-Correct, where Actor- generated solutions are refined through direct feedback-guider regeneration, and (2) Prompt, which exclusively employs prompting strategies to coordinate Sonoo--ddmmen----iii interactions without optimization mechanisms. A critical limitation observed in the Self-Correct framework is its sig- nificantly lower TP accuracy. This issue arises from its feedback mechanism, which modifies all generated responses with high probability, potentially leading to erroneous modifications of the ini- tially correct solution. This is a common issue with using out-of-the-bor L.LS for corf-coor with no specialized training (Kumar et al. 2024).
7

---

Table 5: Evaluation results of the proposed method and baselines on accuracy($))
Model
GPT-3.5-Turho
GPT-4o-mini
Method
TP Accuracy
Overall Accuracy
TP Accuracy
Overall Accuracy
Self-Correct
11.80
16,40
24.60
28.80
Prompt
18.40
47.60
51.60
58.20
SIRIUS
35.00
50.60
59.80
66.80
Ablation Study
SIRIUS + BASE Actor Agent
34.20
49.00
54.40
STRIUS + BASE Judgment Agent
20.20
40.20
53.00
59.40
SIRIUS + BASE Critic Agent
35.00
50.40
59.80
64.20
FT' on One Hase I.I.M
33.80
43.60
56.00
59.60
Comparing GPT-3.5-Turbe and GPT-4o-mini, also find that GPT-3.5- struggles more with misjudging correct answers as incorrect, leading to a severe drop in TP Accuracy. Our method, SIR- IUS, achieves notable improvement in TP Accuracy, highlighting the Judgment Agent's enhanced ability to assess whether response requires modification. The overall higher accuracy underscores the effectiveness of SIRIUS'S ' framework, where fine-tuning enhances each agent's task-specific pabilities, and the collaboration of Judgment, Critic, and Actor Agents ensures appropriate revision of incorrect responses while minimizing unnecessary changes to correct answers.
The ablation study further underscores the contribution each agent in SIRIUS. Fine-tuning only a single base LLM leads to a performance drop, highlighting the necessity of specialized agent roles and joint optimization. Notably, replacing the Judgment Agent with a baseline version significantly reduces TP Accuracy, reinforcing its essential role in filtering correct responses before feedback is applied.
4.5 Experimental Result of Competitive Settings
To analyze the effect of training in the competitive setting, we study the performance of agents in scenarios where one player initially had a higher probability of winning, referred to as the "winning player," while the other player was at disadvantage, called the "losing player." In general, when SIRIUS took on the role of the winning player competing against a base agent, it demonstrated an increased win rate and payoff. Additionally, when SIRIUS played the role of the losing player, experienced fewer losses. Similarly, for both GPT-3.5 and GPT-4o-min when they compete with each other, STRIUS-GPT-S and
both demonstrate improved performance.
4.5.1 Resource Exchange
Player
Player Paye
The win rates and average payoffs for the Re- source Exchange game are presented in Fig- ure 2 Overall, the agent going second tends beat the first agent. Furthermore, the fine- tuned SIRIUS demonstrates a significant im- provement in both the win rate and payoff for the current player. To evaluate the generaliza- tion capability of our approach, we conducted additional experiments with models fine-tuned on games featuring Initial Resource configura- tions of 25Xs + 5Ys s and 5Xs + 25Ys, and then tested them on games with different Initial Re- source configurations (35Xs + 15Ys and 15Xs 35Ys). As demonstrated Figure 5 SIRTUS Resource configurations, effectively validating the generalizability of our proposed pipeline.
2.0
GPT3.5 S...3....
0.46
2:43
6.69
2.36
4.72
2:72
Poyprr
2.53
0.00
0.6
0.72
5.52
0.00
044
SilU5 CFT--------i 0.20
2.00
0.00
057
1.22
0.00
2.00
235
Gayaa
Sindi a
GPEA5
Soudh 09:33
selod oye
Player
Figure 2: Resource Exchange Game: Player (25Xs 5Ys), Player (5Xs 25Ys). Win Rate in decisive games Payoff in all games. We show Player 2 Win rate/payoff in all cells.
maintains notable improvements the new Initial
4.5.2 Multi-Turn Ultimatum
In this setting, Player consistently dominates the game. As shown in the Figure 3 SIRIUS fine- tuned Player effectively secure higher share of the split. Generalization experiments show that
8

---

SIRIUS Player I trained in the Resource = 100 setting maintains utility gains in the new Resource 1000 setting (Figure
4.5.3 Buyer-Seller
In this setting, sellers are willing to sell when the price exceeds 40, while buyers are willing to buy when the price is below 60. We plot the final selling price as shown in Figure 4 Notably, is consistently below 50 for most buyer-seller pairs, indicating that the L.M agent performs better as a buyer than as a seller. After fine-tuning, SIRIUS as a seller shows significant improvement, consistently selling at 50, resulting in tie with the buyer. To test the generalization capability and ensure the seller is not overfitting to a price of 50, we adjusted the initial configuration to 30 and 70. Figure shows that the SIRIUS seller trained in the previous setup still demonstrates significant improvement.
GPT-3.5
with swisd- OPTAS
Corepate with GPILA Mayer
Conpett with Sirua--arE-As Buper
sirka war-te-viid viei
uFi-toriin
with sirisi-ori-ta-einn
Poaer
Vayer
29
Sallrr
Figure 3: Player I's payoff in the Ultimatum game with an initial resource of 100, SIRIUS as Player secures a higher share.
Figure 4: Final Selling Price for a Seller&Buyer with object valuations of 40 and 60. A higher number means greater payoff for Seller.
5 Related Work
Enhancing Reasoning in Single-Agent Systems. Building on the reasoning capabilities of state-of-
the-art foundation models (Schulman et al. 2022 OpenAl 2023 Liu et al. (024), recent research
explores approaches beyond scaling model parameters. Chain-of- Chain-of-Thought (Wei et al., 2022) en-
hances reasoning through step-by-step inference, while Tree of Thoughts (Yao et al. 2024). Graph
of Thought (Besta et al., (202)), and Program of Thoughts (Chen et 2002) structure reasoning as tree searches with backtracking. Reasoning with Planning (RAP) (Hao al. 2023) incorporates explicit planning, and Reflexion (Shinn et al., 2024) enables self-evaluation and refinement. Wu al. (2024) introduce contrastive reasoning for instruction generation, while TexIGrad (Yuksek- gonul et al. 2024) applies gradient-basee based optimization refine outputs. These methods enhance reasoning through structured decomposition, search, and planning.
Self-improvementt. Self-improving models (Huang et al. 2022 Yu et al. 2023 Yuan et al. 2024 Zhang et al., 2024, Welleck al., 2022) have garnered increasing attention for their potential to enhance reasoning capabilities through iterative feedback and refinement. Several studies (Zelikman al. 2022 Li al.) 2024a Pang et al. 2024 Lee et al., 2024 employ bootstrapping strategies by
leveraging self-generated rationales, while others (Yuan al. 20244 Chen et al., 2024b Guo et al.
2025) introduces a self-refinement mechanism through reinforcement learning.
Multi-Agent Systems with LLMs. Multi-Agent Systems with LLMs. Recent advancements
in multi-agent systems (Smit al. 2024 Guo al. 2024 Li et al.) [024b) Han et al.) 2024)
highlight the potential of large language models in tackling complex tasks. Society of Minds (Du
2023) enables agents to exchange answers, fostering collaboration. Mixture-of-Agents (Wang al.
et al. 2024) employs a layered architecture where agents refine responses based on prior outputs. CoMM (Chen et al. (024a) enhances problem-solving through structured communication and role
division. Pelto-Persona (Liang et al. 2023) encourages diverse agent behaviors by assigning dis- tinct personas. ChatEval (Chan et al.) (023) explores different multi-agent debate strategies for interaction and response management. Building on advances in multi-agent systems, recent work has explored fine-tuning with independently specialized agents that interact to generate diverse rea- soning chains (Subramaniam et al. 2025). Unlike these approaches, our method prioritizes collab-

---

orative optimization through a shared experience library, enabling agents to collectively learn from and refine successful reasoning trajectories.
Conclusions
We introduced SIRIUS, framework for optimizing multi-agent LLM systems by learning from successful interactions and augmenting failed trajectories with feedback. Our approach enables agents to refine collaboration strategies without explicit supervision. Experiments show that SIRIUS significantly improves performance across college-level reasoning. biomedical QA, and negotiation tasks. More broadly, our work provides a scalable mechanism for multi-agent self-improvement, offering principled approach to optimizing collaborative Al systems.
References
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gian- inazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17682-17690, 2024.
Federico Bianchi, Patrick John Chia, Mert Yuksekgonul, Jacopo Tagliabue. Dan Jurafsky, and James Zou. How well can llms negotiate? negotiationarena platform and analysis. arXiv preprint 2xXi:2442.005863, 2024.
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better Ilm-based evaluators through multi-agent dehate. arXiv preprint arXiv:2308.07201, 2023.
Pei Chen, Boran Han, and Shuai Zhang. Comm: Collaborative multi-agent, multi-reasoning-pee prompting for complex problem solving. arXiv preprint 2rXiv:2404.17729 2024a.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588, 2022.
Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma. Jianyu Xu, Xinyi Wang, and Tony Xia. Theoremqa: A theorem-driven question answering dataset. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 7889-7901, 2023.
Zhipeng Chen, Kun Zhou, Wayne Xin Zhao, Junchen Wan, Fuzheng Zhang, Di Zhang, and Ji-Rong Wen. Improving large language models via ine-grained reinforcement learning with minimum editing constraint. arXiv preprint (XXx:2401.0668) 2024b.
Steffi Chern, Zhen Fan, and Andy Liu. Combating adversarial attacks with multi-agent debate. arXiv preprint M.Xiv:2401.05998, 2024.
Yilun Du, Shuang Li, Antonio Torralba, Joshua Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. arXiv preprint
arXiv:2305. 14325, 2023.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Daya Guo, Dejian Yang. Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: survey of progress and challenges. arXiv preprint arXiv: arXiv:2402.0m 2402.01680, 2024.
10

---

Shanshan Han, Qifan Zhang, Yuhang Yao, Weizhao Jin, Zhaozhuo Xu, and Chaoyang He. Llm multi-agent systems: Challenges and open problems. arXiv preprint arXiv:2402.03557, 2024.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305. 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint 1:Xiv:2009003000 2020.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. arXiv preprint arXiv:2210.1100 2022.
Geoffrey Irving. Paul Christiano, and Dario Amodei. Ai safety via debate. arXiv preprint
htt.1880500099, 2018.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06144 2019.
Omar Khattab, Arna Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language model calls into self-iip pipelines. arXiv preprint arXiv:2310.03714, 2023.
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John Co-Reyes, Avi Singh, Kate Baumli, Shariq lqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409. 12917, 2024.
Nicholas Lee, Thanakul Wattanawong, Sehoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Anumanchipalli, Michael W Mahoney, Kurt Keutzer, and Amir Gholami. Llm2llm: Boosting Iims with novel iterative data enhancement. arXiv preprint arXiv:2403. 15042, 2024.
Siheng Li, Cheng Yang. Zesen Cheng, Lemao Liu, Mo Yu, Yujiu Yang, and Wai Lam. Large language models can self-improve in long-context reasoning, arXiv preprint arX1v:2411.08147, 2024a.
Xinyi Li, Sai Wang, Siqi Zeng, Yu Wu, and Yi Yang. A survey on Ilm-based multi-agent systems: workflow, infrastructure, and challenges. Vicinagearth, 1(1):9, 2024b.
Tian Liang, Zhiwei Hc, Wenxiang Jiao, Xing Wang, Yan Wang. Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi- agent debate. arXiv preprint arXiv:2305. 200.::2.....1188. 2023.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprini arX1v:2412.19437, 2024.
R OpenAL. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2(5), 2023.
Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19733 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. arXiv preprint arXiv:2311. 12022, 2023.
John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Fe- lipe Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, al. Chatgpt: Optimizing language models for dialogue. OpenAl blog, 2(4), 2022.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances Neural Information Processing Systems, 36, 2024.
11

---

Andries Petrus Smit, Nathan Grinsztajn, Paul Duckworth, Thomas D Barrett, and Arnu Pretorius. Should be going mad? a look multi-agent debate strategies for llms. In Forty-fire Interna- lional Conference on Machine Learning, 2024.
Vighnesh Subramaniam, Yilun Du, Joshua B Tenenbaum, Antonio Torralba, Shuang Li, and Igor Mordatch. Multiagent finetuning: Self improvement with diverse reasoning chains. arXiv preprint arXiv:2501.05707, 2025.
Kyle Swanson, Wesley Wu, Nash L Bulaong, John E Pak, and James Zou. The virtual lab: Ai agents design new sars-cov-2 nanobodies with experimental validation. bioRxiv, pp. 2024-11, 2024.
Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-off Mixture-of-agents enhances large language model capabilities. arXiv preprint arXiv:2406. 04692, 2024.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 -44377, 2022.
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning self-correct. arXiv preprint arXiv:2211.0000 2022.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen Imm applications via multi- agent conversation framework. arXiv preprint 2xXiv:2308.08155 2023.
Shirley Wu, Shiyu Zhao, Qian Huang, Kexin Huang. Michihiro Yasunaga, Kaidi Cao, Vassilis N. loannidis, Karthik Subbian, Jure Leskove, and James Zou. Avatar: Optimizing Ilm agents for tool usage via contrastive reasoning. NeurlPS, 2024.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances Neural Information Processing Systems, 36, 2024.
Xiao Yu, Baolin Peng, Michel Galley, Jianfeng Gao, and Zhou Yu. Teaching language models to self-improve through interactive demonstrations. arXiv preprint arXiv:2310. 135222,..33.. 2023.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXi::2401.10020, 2024.
Mert Yuksekgonul, Federico Bianchi. Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and
James Zou. Textgrad: Automatic" differentiation" via text. arXiv preprint arXiv:4406.07996, 2024.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances Neural Information Processing Systems, 15476-1115548,, 2022.
Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. Small language models need strong verifiers to self-correct rea- soning. arXiv preprint 10X1v:2404.17140, 2024.
12

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract claims that SIRIUS is a self-improving, reasoning-driven opti- mization framework for multi-agent systems that constructs an experience library of high- quality reasoning trajectories. also claims performance boosts of 2.86% to 21.88 1.888 on reasoning and biomedical QA, and enhanced agent negotiation. The introduction reiterates these points, highlighting the challenges optimizing multi-agent systems and how SIR- IUS addresses them through learning from successful interactions and augmenting unsuc- cessful ones. The subsequent sections (Methods, Experiments) detail the SIRIUS frame- work, the construction of the experience library, the augmentation process, and present experimental results that support these claims with specific percentage improvements on various tasks (e.g., Table 3 shows improvements on College Physics, College Chemistry, and PubMedQA).
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The paper includes a dedicated "Limitations" section (Appendix B). This section discusses dependence on backbone LLM capabilities, the ongoing challenge of multi-agent credit assignment, and the complexity of designing optimal agent interaction protocols.
Justification:
Guidelines:
The answer NA that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
13

---

If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers penalize honesty concerning limitations.
will be specifically instructed
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
The paper primarily presents an empirical framework and experimental sults. It defines the multi-agent system formally (Section 2.1) but does not focus on deriv- ing new theoretical results with formal proofs.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The paper provides details on the backhone models used (gpt-3.5 -...-..rooo-0122300
and 1p--44--min-2222-----111) temperature setting, and the of OpenAl's Fine-tuning API (Section 4.2). describes the datasets used (College Physics/Chemistry, PubMedQA) and their splits (Section 4.2, Appendix D). The different multi-agent settings (Problem Solving, Actor-Critic, Competitive) and their agent configurations are detailed (Section 3, Table 1, Table 2). Algorithm outlines the SIRIUS training procedure, and Algorithm 2 in the appendix provides more detail. Appendix F provides extensive details on the prompts used for various agents and tasks. This level detail should allow others reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data
14

---

is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Please check the code and data details in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips.
Sutp:/CooeSubbiieccc ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The paper specifies backbone models, temperature, and the use of OpenAl's Fine-tuning API (Section 4.2). Data splits for College Physics/Chemistyy and PubMedQA
15

---

are provided (Section 4.2, Appendix D). The number of fine-tuning iterations (T) is men- tioned as an input to Algorithm I, and additional fine-tuning iterations are explored in ab- lation studies. Specific agent roles and their interaction structures are defined for different settings. Appendix details the prompts. While the choice of every hyperparameter (e.g., learning rate for SFT, reward threshold c) isn't exhaustively detailed, sufficient information is present to understand the setup. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Table 3 and Table 5, which report the main quantitative results, include values with "+" notation (e.g.. "25.55 + 1.08"). This suggests reporting of mean and standard deviation or standard error, indicating variability across runs similar measure of sta- tistical significance. The paper does not explicitly state how these were calculated (e.g., number of runs for averaging) but their presence addresses the reporting of variability.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: The paper mentions the LLMSs used (gpt-3.5-turbo, gpt-4o-mini) and the use of OpenAl's Fine-tuning API. However, specific details about the compute resources (e.g., number of GPUs, type of GPUs, memory, execution time for fine-tuning experiments) is unclear.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g.. preliminary or failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
The research involves LLMs and their fine-tuning for various tasks. The paper discusses broader impacts, including potential negative societal impacts and the inheritance biases from backbone LLMs (Appendix A). It uses established datasets and platforms. There's indication of ethical violations like plagiarism, data fabrication, harm to human subjects. The responsible disclosure potential misuse and biases aligns with ethical considerations.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix A is titled "Broader Impact" and explicitly discusses both positive societal impacts (e.g., applications in biomedical research, scientific discovery) and poten- tial negative societal impacts (e.g., misuse in competitive negotiation, exploitation, unfair ad advantages, inherited biases from LLMs, alignment with human values).
Guidelines:
The answer NA means that there societal impact of the work performed.
If the authors answer NA No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, Iet alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
17

---

11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
The paper focuses on framework (SIRIUS) and its application using existing LLMs provided by OpenAI. It does not appear be releasing new large-scale pretrained models or datasets that would pose a high risk for misuse beyond the risks inherent in the underlying LLMs themselves. The discussion of negative impacts and inherited biases is present, but specific safeguards for "newly released asset* are not applicable as no such new high-risk asset seems be released by this paper itself.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The paper properly credits the creators of existing datasets through citations (e.g.. MMLU Hendrycks al., 2020; GPQA Rein et al., 2023; TheoremQA Chen et al., 2023; PubMedQA Jin et al., 2019).
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the pack- age should be provided. For popular datasets, paperswithcode com datasets has curated licenses for some datasets. Their licensing guide help determine the li- cense of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Question:
Answer: [NA]
18

---

Justification: The primary contribution is the SIRIUS framework. While new fine-tumed models are generated part of the experiments, the paper doesn't explicitly state these models are being released as new assets. The combined datasets (College Physics/Chem- istry) are derived from existing ones. If the framework's code or the specific fine-tuned model weights were to be released, then this question would apply more directly. Based on the text, it doesn't seem to introduce new, distributable assets in that sense.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given participants and sereenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not describe any new research involving crowdsourcing or direct human subject interaction (e.g.. collecting new data from people). The experiments involve LLMs interacting in simulated environments or working on existing datasets.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: As the research does not appear to involve new human subjects directly, IRB approval discussions are not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
The entire paper is centered around the and optimization of LLMs in multi-agent systems. Section 2.1 defines multi-agent systems with LLMs, Section 4.2 specifies the backbone LLMs used (gpt-3.5, gpt-4o-mini) and the fine-tuning API. The core methodology (SIRIUS) about fine-tuning these LLMM-based agents. Appendix provides detailed prompts for the LLM agents. The of LLMs is core, important, and original component of the research described.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy hettap//eeeerriieecccc////eerees for what should or should not be described.
20

---

A Broader Impact
The SIRIUS framework, as self-improving multi-agent system, has the potential for significant positive societal impacts. Its ability to enhance collaboration and reasoning in Al agents can be applied to various complex problem-solving domains such as biomedical research demonstrated by its performance on PubMedQA] Jin et al. (2019)), scientific discovery (like physics and chemistry problem-solving Hendrycks et al. (2020); Rein al. (2023); Chen et al. (2023), and potentially other areas requiring sophisticated reasoning and information synthesis.
However, there are also potential negative societal impacts consider. The celf-improvement capa- bilities and enhanced reasoning of multi-agent systems like SIRIUS could be misused. For instance, in competitive negotiation scenarios, a highly optimized Al agent could potentially exploit human counterparts or create unfair advantages in economic interactions if not implemented with ethi- safeguards. The reliance on LLMs also means that SIRIUS inherits any biases present in the backbone models, which could lead to skewed or unfair outcomes if not carefully monitored and mitigated. As these systems become more autonomous and capable, ensuring their goals remain aligned with human values and preventing unintended consequences will be crucial.
B Limitations
The SIRIUS framework, despite its promising results, has several limitations that should be ac- knowledged:
Dependence Backbone LLM Capabilities: The performance of SIRIUS is inherently tied to the capabilities the underlying LLMs (gpt-3.5-turbo and gpt-4o-mini). Limita- tions in the base models' reasoning, instruction-followin can impact the overall efficacy the multi-agent system.
tendencies for misjudgment
While SIRIUS aims to sidestep direct supervision of intermediate steps by learning from successful trajectories, the fundamental challenge of multi-agent credit assignment - attributing overall success or failure to specific actions of individual agents remains complex issue, especially in language-basse based systems with unstructured interactions.
Credit Assignment:
Complexity of Agent Interaction Protocols:
The current work explores specific commu- Context Analyst-
nication structures and roles (e.g.. Problem Solver). Designing optimal interaction protocols for different types of tasks and numbers of agents can be complex, and suboptimal designs might limit performance.
C Detailed Pipeline
Given the wrong answer problem set W = {(x,, y1) ")) each iteration, we first select the agent to be optimized. For instance, as shown in the diagram, the selected agent is the physicist (A). The external agent provides feedback f. Patewo (-12,, 0,, y4) based on the question I,, the original response à, and the correct answer yt-
The physicist then regenerates the solution by incorporating the feedback: of = PoCAs (-12,, Dis ft).
To ensure clarity and coherence, the regenerated response à; is subsequently rephrased to produce gfinal making it appear as if derived directly through problem-solving without mentioning any mod- iffcctiions feedback. This updated response is then used in subsequent collaborations with other agents to refine the overall solution further.
D Dataset Details
D.I Dataset Split Statistics
In this work, we use three datasets for evaluating the performance of our model: Massive Multitask Language Understanding (MMLU) (Hendrycks et al., 2020), Graduate-L Google-Proof Q&A (GPQA) (Rein et al. 2023), and Theorem-Driven Question Answering (Theorem(A) (Chen et al.
21

---

Algorithm 2 Training Multi-Agent LLM System
1: Input: group of agents A ;... A(K), the system's 's topological graph 9, maximum solution generation tries max.ol. maximum feedback generation tries max,, maximum regeneration tries
maxę- An initial dataset of problems with answer y 'D {(r,, 1=)),= total number of fine-tuning Iterations T.
2: Initialize: Initialize policy parameters ((*) for each agent A A = 1.2 1,2, K. @()) for Critic
Agent Ate)
3: for Fine-tuning Iteration It 1,.,, do
4: while sol S maXx3l do
((()) = Pauel (-1221, a; Pre(a())) ai (K)
5:
6:
7: for each agent = .,,,....A K do
8:
((R) t- [x(x
(1)
/ [1, D] A 0. vi)} (k)
W + (1s; ai
9:
[1, A d: + v1)
for T4 E (((k) do
11:
while 4 I max, do
f/(A) Poron11113 0j 1/1)
(k)
12:
13:
while te I max do
14:
a (k), P (- oj (K) {(())
15:
s, Sus(A n ) j E Sus( (1))
16:
ph Pasn
Pret (1 Ua: (110
(K),re
17:
D.e = d;
18:
if Dr Vi then
(()) + { a, ())re.. = k,. K
19:
break while
21:
end if end while
23:
end while
24:
end for
25:
end for
end while
27:
0
Standard SFT on (k)) k 1,... K
28:
end for
These datasets contain a variety of question types, with a focus on college-level physics and chemistry problems that remain difficult and present room for improvement in performance with large language models.
2023
The dataset was split into training and test sets with 2:1 ratio, and the data distribution for each dataset is shown Table o
Table 6: Dataset Split Statistics.
Task
College Physics
College Chemistry
Dataset
Train Size
Test Size
Train Size
Test Size
MMLU
68
34
66
34
GPQA
57
29
62
31
TheoremQA
87
D.2 Trajectory Augmentation Coverage
Our augmentation module successfully rewrote 32.28% 74.7% of failed trajectories across different tasks and models. We provide the detailed statistics in the table below.
22

---

Table Augmentation statistics across models and datasets.
Dataset
Setting
GPT-335-turbo GPT-4o-mini
LLaMA-332-B-instruuc
College-Physics
Correct
56
106
51
Wrong
156
106
161
Augmented Trajectory
61
42
Augmented Percentage
39.10%
39.62%
42.86%
College-Chemistry
Correct
455
58
48
Wrong
83
70
80
Augmented Trajectory
62
31
32
Augmented Percentage
74.70%
44.29%
40.00%
PubMed
Correct
382
358
Wrong
118
142
158
Augmented Trajectory
50
46
Augmented Percentage
42.37%
32.39%
32.28%
E Additional Experiment Result
In this section, present additional experiments conducted in competitive setting the generalization of SIRIUS. These results demonstrate the adaptability of SIRIUS across various con- figurations.
Player
Player Payoff
1.0
GPT-3.5
0.71
0.70
0.78
1.00
1.24
1.2
1:08
1.36
0.8
Player 1
SiMus-GPT-3.5
0.50
0.93
0.60
1.00
0.6
1.20
4.20
1.20
2:38
GPT-4o-mini-
0.00
0.00
0.00
1.00
-0.40
-0.20
-1.00
2.20
Sifus- MPNN--GP--o-mil-
0.00
0.00
0.00
0.60
-0.12
0.00
0.00
2.36
0.0
G.F-3.5
Strud GPT-3.5
GPF-46 GPi-4o-mini
Swnus WMUS-GPT-4 GPT-4-oT-4-mmii miimii
GFT-3.5
Strud G.P-3.
GiT GPT-4ommn To-mini
Siltus Satus.GPT CRSSS.PT-AA 4o-mini
Player 2
Player
Figure 5: Resource Exchange Game with Initial Resource Player I: 35Xs + 15Ys, Player 2: 15Xs + 35Ys. Win Rate in decisive games and Payoff in all games. We show Player Win rate/payoff in all cells.
Compate OPT335 Bayer
Compate StruS-OPF335 Buper
Computt Nlver
Plger?
8000
bettng mine
400
483
200
203
Compete with GPS4e-ninl
Compate Strud -ii----------iiiin
400
400
200
4974
Figure 7: Player I's payoff in the Ultimatum game with Initial Resource settings of 1000. SIRIUS Player can effectively secure higher share of the split.
Figure 6: Final Selling Price for Seller&Buyer with object valuations of 30 and 70. A higher number means the seller gets a greater payoff.
23

---

Agent Prompts
F.1
Problem Solving Setting
Prompts for College-Physics Task
System -orteptp You part of team with multiple experts from different disciplines. Your team aims to solve a given cross-discipline problem collectively.
The team is composed three experts:
1. The Physicist
Role Definition: You are a physicist with specialization in the field of college-level physics. Your vast knowledge covers multiple aspects of physics including classical mechanics, ther- modynamies, electromagnetism, quantum mechanies, and statistical physics. You under- stand these topics in depth and have the ability to explain them in a way that is easily com- prehensible those less familiar with them.
Responsibility: Focus contributing physics-specific insights and collaborate with the mathematician to help develop and validate mathematical models, **Do not perform cal- culations solve the entire problem**, Your goal is to provide a clear explanation of the physics, leaving calculations to the mathematician. Principles:
Emphasize empirical, systematic, and data-driven approaches while fostering curiosity, innovation, and ethical scientific practices.
2. The Mathematician
Role Definition: You are a mathematician, specializing in the broad and complex field of mathematics at the college level. Your expertise ranges from pure mathematical theory, in- cluding algebra, calculus, geometry. number theory, and statistics, to applied mathematics such optimization and probability theory. You have innate ability abstract and gener- alize problems, solving them with elegance and precision. You excel at creating mathemat- ical models that represent real-world situations and can interpret the implications of those models. You are not only well-versed in complex equations and proof', but also experienced in conveying these concepts others through teaching.
Responsibilities: Apply mathematical reasoning to analyze and address complex, cross- disciplinary problems; Collaborate with the physicist refine mathematical models and validate their conclusions; Convey mathematical insights in a clear manner to facilitate team decision making.
Principles: Foster a culture of analytical thinking and evidence-based decisions; Encourage an atmosphere curiosity, innovation, and continuous learning; Maintain high mathemati- cal integrity and respect for varying perspectives.
The Final Answer Synthesizer
Role Definition: You are the Final Answer Synthesizer, an integrative role in the team sponsible for coalescing the insights provided by the experts. With a clear understanding of the different disciplines, you effectively distill the responses from the physicist and the math- ematician into a coherent, final solution. Your role involves keenly interpreting expert input, synthesizing various problem-solving approaches, and presenting clear, well-rounded an- swer that incorporates the collective wisdom of the team.
Responsibility: summarize the solutions; give final answer. Principles: make sure to give a specific answer to the given task."'
Physicist prompt= = >" Your role the physicist. Here is the given problem: "question" Your task is to explain the relevant physics concepts and principles that apply to this problem.""
Mathematician prompt = "" "YYuur role is the mathematician. Here is the given problem: "question" Here is the response from the physicist: ('agent_I_response Please give your opinion on how to solve the problem in consideration of the response from the physicist." "
tattematiican.ppompt-=
Summarizer prropptpterrppoopt- = ""Your role is the Final Answer Synthesizer. Here is the given problem: "question" Here is the response from the physicist: "(agent.I Here is the response from the mathematician: Please provide final answer to the given problem. (format.prompt)
24

---

Prompts for College-Chemistry Task
System prompt "You part of team with multiple experts from different disciplines. Your team aims to solve a given cross-discipline problem collectively.
The team is composed of three experts:
1. The Chemist
Role Definition: You chemist with specialization in the field of college-level chem- istry. Your vast knowledge covers multiple aspects of chemistry including organic, inorganic, physical, analytical, and biochemistry. You understand these topies in depth and have the ability to explain them in a way that is easily comprehensible to those less familiar with them.
Responsibility: Focus on contributing chemistry-speciffe insights and collaborate with the mathematician help develop and validate mathematical models. ** Do not perform calcu- lations solve the entire problem**. Your goal is to provide clear explanation of the chemistry concepts, leaving calculations the mathematician. Principles:
Emphasize empirical, systematic, and data-driven approaches while fostering curiosity, innovation, and ethical scientific practices.
2. The Mathematician
Role Definition: You are a mathematician,, specializing the broad and complex field of mathematics at the college level. Your expertise ranges from pure mathematical theory, in- cluding algebra, calculus, geometry, number theory, and statistics, to applied mathematics such optimization and probability theory. You have innate ability to abstract and gener- alize problems, solving them with elegance and precision. You excel at creating mathemat- ical models that represent real- world situations and can interpret the implications of those models. You are not only well-versed in complex equations and proofs, but also experienced in conveying these concepts others through teaching.
Responsibilities: Apply mathematical reasoning to analyze and address complex, cross- disciplinary problems; Collaborate with the chemist refine mathematical models and val- idate their conclusions; Convey mathematical insights in a clear manner to facilitate team decision making.
Principles: Foster a culture of analytical thinking and evidence-basee decisions; Encourage an atmosphere curiosity, innovation, and continuous learning; Maintain high mathemati- cal integrity and respect for varying perspectives.
The Final Answer Synthesizer
Role Definition: You are the Final Answer Synthesizer, an integrative role in the team re- sponsible for coalescing the insights provided by the experts. With a clear understanding of the different disciplines, you effectively distill the responses from the chemist and the math- ematician into a coherent, final solution. Your role involves keenly interpreting expert input, synthesizing various problem-solvine approaches, and presenting a clear, well-rounded an- swer that incorporates the collective wisdom of the team.
Responsibility: Summarize the solutions; give final answer. Principles: Make sure to give a specific answer to the given task.""
Chemist.promp "Yoour role is the chemist. Here is the given problem: "question" Your task is **only to explain** the relevant chemistry concepts and principles that apply to this problem. ** **Do ** perform any calculations or try to find the final solution. Your role is to explain the chemical reasoning, such as reactions or principles, but refrain from solving the equations or completing the solution. Leave the mathematical work the mathematician."
Mathematician prompt YYour role is the mathematician. Here is the given problem: "question" " Here is the response from the physicist: "(agent I response)"" Please give your opinion how solve the problem in consideration of the response from the physicist."
Summarizer prompt "Your role is the Final Answer Synthesizer. Here is the given problem: "question" " Here is the response from the physicist: "(asent__ Here the response from the mathematician: Please provide a final answer to the given problem.
25

---

Prompts for PubMedQA Task
System prompt You are part of team of experts working collaboratively to solve science-related yes/no questions using contextual evidence. The goal is to analyze the pro- vided question and context thoroughly to determine the correct answer. The team is composed two roles:
1. The Context Analyst
**Role Definition:** ** You are the Context Analyst, skilled in extracting and summarizing key information from the given context to address the question.
**Responsbblity:** Read the provided question and context carefully, then summarize the most relevant information needed answer the question. Your summary should focus on the evidence directly supporting or refuting the question's ' claim.
**Principles:** Prioritize clarity and relevance. Extract only the essential details from the context that will help guide the next agent in making evidence-based decision.
The Problem Solver
**Role Definition:** You are the Problem Solver, responsible for interpreting the Context Analyst's summary and determining the correct yes/no answer based on evidence.
s*RReponsibiilty:=** Review the question and the Context Analyst's summary, analyze the evidence, and construct a concise final response (yes or no) supported by clear reasoning. the context does not provide sufficient evidence to make confident decision, clearly state that the evidence is inconclusive.
**Principles: ** Ensure logical coherence, accuracy, and completeness. Justify your answer with reasoning directly tied to the summarized evidence.
Analyst prompt "Youur role is the Context Analyst. Here is the provided context: "(context)" Your task carefully read through this context and summarize the main points relevant to the question. Only provide essential information that would help address the question." ""
Solver prompt "YYuu role the Problem Solver.
Here the question: "( "(question)"
Here the summary from the Context Analyst: "(agent response]"
Please analyze the question, using the summary answer the problem. (format_prompt))
F.2 Actor-Critic Setting
Prompts for Actor Agent and Regeneration
System_prompt You are scientist working on solving science-related yes/no questions using contextual evidence. "
Actor- actor-prompt-= "Yoo Yoo are supposed to provide a solution to given problem.
Here is the given context: "(context)"
Problem: " "(question)" Please provide yes, maybe to the given problem. (format prompt)""
Actor regenerate prompt You are supposed to provide solution to given problem.
Here the given context: "(context)"
Problem: "(question)"
Here your original response:
Here is the feedback for your original response: "[feedback)"
Please first consider the feedback and then update your opinion on how to solve the problem.
Please provide final answer to the given problem. (format.prompp]
26

---

Prompts for Judgment Agent
System prompt "Below is yes/no question and prediction. You are critical and creative scientist tasked with evaluating the prediction. Your responsibility is to thoroughly investigate the reasoning behind the prediction. If the original response is entirely correct, output "True." If you identify any errors, inconsistencies, or flaws the reasoning, output "False."
Judgment prompt= "eere is the given context: "(context)" Problem: " "(question)"
Original response: triggnal__esponne
Provide your response in the following format:
1. Analysis: Provide detailed and objective critique of the reasoning in the language model's answer. Discuss whether the logic, assumptions, and conclusions are valid. High- light any errors. alternative perspectives, missing considerations.
Decision: 'Opinion: True or False' (without quotes) where Opinion is your final Decision based your analysis. Your Decision should be either "True" "False". Ensure this conclusion directly reflects the correctness of the reasoning in the language model's 's answer.
Prompts for Critic Agent
System_prompt "'Below is biomedical yes/no question, the context, and a prediction. You are a critical and creative scientist. Your job is to investigate the prediction. Critically go through reasoning steps, and if there is reason why the prediction could be incorrect. Use the Janusian Process, think about whether alternative answers could be true.""
Critic Puttieprommt--i "aampp=""Her"" "eer is the given context: "(context)"
Question: "(question)" Answer by the language model: {original response) ""
Prompts for Rephrasing
System_prooe prommtt = "" ""Rephrase the following solution process to ensure that it appears as though the solution
arrived at directly, with traces of mistakes corrections. Retain all key steps and avoid generating any new content. The focus should be on smoothing the flow and ensuring logical consistency, without altering the meaning or introducing additional information."
Rephrase prompt "'Here the problem and the original solution process: Problem: {question)
Original Solution Process: process:forigen foriginal response) Please output the rephrased solution process""
F.3 Competitive Setting
We use the EEGGOTIATIONAREN Platform Bianchi al. 2024
27

---

Prompts for Resource Exchange
promem "You playing strategic game of trading resources with another
player whose resources you have no knowledge about.
RULES:
1. You can either:
A) Accept the trade by saying:
iplayer answer, ACCEPT :/player answerž
inewly proposed trade; NONE j/newly proposed trade;
B) Reject and propose new trade (you can only trade integer amounts, decimals): iplayer answer; NONE i/player answerž
inewly proposed trade; Player RED Gives iteml: amount, item2: amount, Player
BLUE Gives iteml: amount, item2: amount, j/newly proposed trade;
C) Don't accept or propose anything and wait for new offer:
iplayer answer; NONE i/player answerž
inewly proposed trade; NONE j/newly proposed trade;
Note: the game will end if one of the players accepts. This means that you have to
be careful about both accepting and proposing trade.
You are allowed at most 3 proposals of your own to complete the game, after which you only ACCEPT NONE.
DO NOT propose a new trade after 3 proposals. Your limit for proposals
3. You can reason step by step by using the following format: jreason; [add reasoning] p/reason;
Add as much text as you want. This information will not be sent the other player. is just for you to keep track your reasoning.
At each turn send messages each other by using the following format: imessage, [add message] j/messagej. You decide if you want disclose your resources and goals in the message.
Here what you have access
Resources available in the game: X, Y
jmy resources; X: 25, Y: 5 :/my resources; jmy goals; X: 15, Y: 15 //my goals; jmy name; Player RED j/my name;
All the responses you send should contain the following and in this order:
jmy name; [add here] j/my name; imy resources; [add here] j/my resources; jmy goals; [add here] j/my goals; jreason; [add here] j/reason; iplayer answer; [add here] i/player answer; imessage; [add here] :/message
inewly proposed trade; [add here] :/newly proposed trade;
Please be sure include all.
More resources in general are always better.
This is the last round of trading. There are no future rounds after this one.
28

---

Prompts for Sull&Buy
System-prompt You playing game where you are buying selling object. There is only one object for sale/purcahse.
Player RED is going to sell one object. Player BLUE gives ZUP buy resources. RULES:
1. You must always respond with:
A) Propose trade with (you can only trade in integer amounts, not decimals): iplayer answerz, PROPOSAL AL :/player answery.
inewly proposed trade; Player RED Gives X: amount, ... Player BLUE Gives ZUP: amount j/newly proposed trade;
B) Accept the trade by saying:
iplayer answer; ACCEPT j/player answer;
inewly proposed trade; NONE j/newly proposed trade; C) Reject and end the game:
iplayer answer, REJECT j/player answery
inewyy proposed trade; NONE j/newly proposed trade;
Note: The game will end if one of the players ACCEPT OR REJECT. This means that you have to be careful about both accepting. rejecting and proposing trade.
You are allowed at most 4 proposals of your own to complete the game, after which you only reply with ACCEPT REJECT.
DO NOT propose a new trade after 4 proposals. Your limit for proposals
3. You can reason step by step why you are A) proposing, B) rejecting and C) accepting a trade with:
jreason, [add reasoning] j/reasong add much text you want This information will not sent to the other player. is just for you to keep track of your reasoning.
4. At each turn send messages to each other by using the following format: message,,your message here:/mmsssage You can decide if you want disclose your resources, goals, cost and willingness to pay in the message.
Here what you have access
Object that being bought/sold:
jmy resources; ZUP: 100 //my resources; imy goals; Buy resources with iZUP,. You are willing to pay at most X: 70 ZUP for the resources. j/my goalsū.
All the responses you send should contain the following and in this order:
iproposal count, [add here (inclusive of current)] i/proposal count; jmy resources; [add here] j/my resources;
jmy goals; [add here] j/my goals; jreason; [add here] p/reason; iplayer answer; [add here] i/player answer,
inewly proposed trade; [add here] //newly proposed trade; imessage; [add here] :/message
Please be sure include all.
You are Player BLUE
29

---

Prompts for Ultimatum
System "Yoo are playing game involving split of resources.
Player RED starts with Dollars: 100, Player BLUE has none to trade. RULES:
1. Player RED has 4 moves while Player BLUE has 4 moves.
2. You must either:
A) Accept the current proposal by saying:
iplayer answer; ACCEPT :/player answer; inewly proposed trade; NONE j/newly proposed trade;
B) Reject the current proposal, whereby both players lose all their resources and the game ends immediately:
iplayer answer; REJECT i/player answer;
inewly proposed trade; NONE j/newly proposed trade;
C) Propose new trade:
iplayer answer; NONE j/player answery
inewly proposed trade; Player RED Gives iteml: amount Player BLUE Gives iteml: 0 j/newly proposed tradej,
Player BLUE cannot do (C) move 4/4 and MUST ONLY answer with ACCEPT (A) or REJECT (B) but Player RED is not affected by this condition.
Note: the game ends immediately if you ACCEPT or REJECT.
If either player REJECTS, both players lose all their resources and the ends im- mediately.
There are no future games.
You can reason step by step why you A) proposing, B) rejecting and C) accepting a trade with: jreason; [add reasoning] s/reasong add much text you
This information will not be sent to the other player. It is just for you to keep track of your reasoning.
6. At each turn send messages to each other by using the following format: imessage,.yor message here:/message; You can disclose anything in the message.
Here is what you have access to:
Resources available in the game: Dollars imy resources; Dollars: 100 //my resources;
All the responses you send should contain the following and in this order:
jmy name; [add here] j/my name; imove; [add here] / [add here] i/move; imy resources; [add here] j/my resources; jreason; [add here] j/reason, pllayer answer, [add here] //player answer; message; [add here] :/message
jnwly proposed trade; [add here] //newly proposed trade;
Please be sure to include all.
30


Reviews:
[{"review_id": "6aFPyJ0Vff", "reviewer": "Reviewer_Hkfd", "summary": "The paper introduces SIRIUS, a bootstrapped optimisation loop for multi-agent LLM systems. Starting from GPT-3.5-turbo or GPT-4o-mini, it instantiates domain roles (e.g., Physicist, Mathematician, Summariser), logs complete successful trajectories from datasets such as combined MMLU/GPQA/TheoremQA (Physics & Chemistry) and PubMedQA, rewrites failed ones with agent feedback, and fine-tunes each role on this “experience library.” Across reasoning and three negotiation games, SIRIUS lifts accuracy by 3 – 22 pp and improves bargaining outcomes versus single-agent, STaR, COMM, and TextGrad baselines while requiring no extra human labels.", "questions": "See Strengths And Weaknesses", "limitations": "yes", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper’s chief liabilities all stem from its infrastructure choices. First, every result hinges on proprietary GPT-3.5-turbo-0125 and GPT-4o-mini backbones; the authors themselves note that “the performance of SIRIUS is inherently tied to the capabilities of the underlying LLMs” and that any weaknesses in those models flow straight through the whole pipeline, leaving open-source substitutes untested and uncertain. Second, although SIRIUS learns from full successful trajectories to dodge hand-crafted rewards, it never resolves the classic multi-agent credit-assignment problem—the paper concedes that attributing success to individual actions “remains a complex issue” in unstructured language interactions. Finally, the system demands repeated multi-agent generation, feedback, rewriting, and round-robin fine-tuning; coupled with maintaining large per-role experience libraries and calling the OpenAI fine-tuning API each iteration, this makes the engineering and compute footprint non-trivial, yet the paper offers no cost-benefit analysis.\n\nSIRIUS nonetheless delivers a neat, largely hands-off recipe for bootstrapping specialised agents: it stores whole successful dialogues in an experience library, automatically “cleans” failed trajectories via a feedback agent, and fine-tunes each role on this growing dataset, creating a self-improving loop that needs no manual reward shaping or step-level supervision. This role-specialisation plus joint optimisation yields consistent gains over strong baselines—up to 15.9% on College Chemistry and +17.8% on PubMedQA with GPT-3.5, and solid boosts in negotiation pay-offs and win-rates across three competitive games. The framework is task-agnostic, extending from science Q&A to adversarial bargaining, and its ablations show performance drops whenever a single SIRIUS agent is swapped out, underscoring genuine synergy among the roles", "was_revised": true, "final_justification": "The rebuttal satisfactorily adds open-source Llama-3 results (confirming gains are not GPT-specific) and discloses raw token counts, but it still omits a horizontal cost-effectiveness table that puts Single-Agent, STaR, COMM, TextGrad, and SIRIUS on the same $ scale, and it offers no experiment contrasting whole-trajectory supervision with explicit step-level credit in tasks requiring fine-grained rewards; because these two gaps leave the practical value and robustness of SIRIUS unclear, I maintain a reject (2/5) recommendation.", "rebuttal": "We sincerely thank the reviewer for recognizing that SIRIUS offers an effective approach to bootstrapping and improving specialized agents via role-level fine-tuning. We have addressed your concerns below and hope these clarifications will assist you in re-evaluating and update the score:\n\n### ` W1. Evaluation on Open source model`\n\nWe thank the reviewer for highlighting this important point. We have conducted additional experiments on Llama-3.2-3B-instruct, which show promising improvements across all tasks.\n\n|   Model   |       Method       | College Physics | College Chemistry |   PubMedQA    |\n|:---------:|:------------------:|:----------------:|:------------------:|:-------------:|\n| **Llama-3.2-3B-instruct** | Single-Agent       | 26.79 ± 1.95     | 34.46 ± 2.28       | 54.07 ± 0.31  |\n|           | STaR                    | 27.41 ± 1.95   | 38.46 ± 1.49       | 54.93 ± 0.12  |\n|           | COMM               | 27.73 ± 1.43     | 37.44 ± 3.11       | 65.80 ± 0.87  |\n|           | TextGrad           | *Not supported* | – | – |\n|           | **SIRIUS**         | **29.60** ± 1.43 | **42.56** ± 2.28   | **68.27** ± 1.03 |\n\nWe agree that demonstrating SIRIUS's effectiveness on open-source models is crucial for the community.  We appreciate the reviewer’s comment and hope this addresses the main concern.\n\n### ` W2. Credit Assign Challenge`\n\nWe appreciate the reviewer’s insightful comment. Our goal is not to solve the fundamental credit assignment problem directly, but to provide a practical and scalable approach for training multi-agent systems. By learning from complete successful trajectories, SIRIUS avoids the need for fine-grained rewards or step-level credit attribution, while partially addressing the challenge through targeted agent-level feedback during trajectory augmentation. We will revise the introduction and discussion to better clarify this point, and we aim to more directly explore the open challenge of credit assignment in multi-agent systems in future work.\n\n### `W3. Cost analysis`\n\nThank you for pointing this out. To support reproducibility and cost estimation, we report the approximate token usage at each major stage of the SIRIUS pipeline. Given that API prices for models like GPT‑4o‑mini, GPT‑3.5‑Turbo, and LLaMA‑3 are typically at or below $0.60 per million tokens, the overall cost is affordable. Considering the substantial performance gains (up to 21.88%), we believe the cost-performance trade-off is highly favorable and supports the practical viability of SIRIUS.\n\n| Domain  | Phase     | Category     | M tokens |\n|---------|-----------|--------------|---------|\n| College Physics      | generate  | solve        | 1.42    |\n|         |           | feedback     | 0.34    |\n|         |           | regenerate   | 15.98   |\n|         |           | finetune     | 1.12    |\n|         | eval      | eval         | 4.32    |\n|         |           | **total**    | **23.19** |\n| College Chemistry    | generate  |solve      | 0.80    |\n|         |           | feedback     | 0.16    |\n|         |           | regenerate   | 7.32    |\n|         |           | finetune     | 0.91    |\n|         | eval      | eval         | 2.41    |\n|         |           | **total**    | **11.60** |\n| PubMedQA  | generate  | solve          | 1.13    |\n|         |           | feedback     | 0.22    |\n|         |           | regenerate   | 7.11    |\n|         |           | finetune     | 0.88    |\n|         | eval      | eval         | 7.45    |\n|         |           | **total**    | **16.80** |\n\nThis analysis will be included in the final version for completeness and clarity.\n\n## **Summary**\n\nWe hope our answers address your concerns on (1) open-source evaluation, (2) credit assignment, and (3) cost analysis.  We also sincerely thank the reviewer for recognizing our novel and effective framework, the strength of our empirical evaluation, and its task-agnostic design. Finally, we truly appreciate your reconsideration of our work in light of our responses. Thank you once again for your valuable insights!"}, {"review_id": "vMcxXtEdSY", "reviewer": "Reviewer_AYyk", "summary": "The paper highlights increasing adoption of multi-agent systems, that operate by integrating specialized agents through structured interactions. However, the paper highlights the challenges in optimizing such multi-agent systems, stemming from difficulty in acquiring training signal for individual agents, and sensitivity of the overall system to many moving parts. The issue is exacerbated by difficulty in credit assignment (local rewards) from outcome rewards (global rewards).\nThe paper presents SiriuS, a framework for learning multi-agent behaviour, with the key insight that the entire trajectory contains useful patterns in a multi-agent trajectory when it solves a task successfully. The framework does not need direct supervision for intermediate steps.\nThe framework steps are as follows: For each training step, A trajectory is sampled from the full system, and evaluated (it is replaced with an augmented trajectory if it fails), and the model is then fine-tuned on the successful trajectory.\nThe paper then discusses several common multi-agent arrangements (Experts, Summarizer-Solver, Actor-Critic, Competitive, etc.). SiriuS is evaluated comprehensively on PubmedQA and separate physics and chemistry subsets of MMLU, GPQA and TheoremQA, where SiriuS achieves superior performance against several strong baselines (Single Agent, STaR, CoMM, TextGrad).", "questions": "Can the authors discuss and provide a comparison between SiriuS and BootstrapFT proposed in [1]?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper demonstrates a strong approach for optimizing multi-agent systems towards new tasks. Through a comprehensive evaluation against strong baselines, the paper establishes superior performance achieved by SiriuS. Further, the paper provides ablation studies highlighting interesting insights, specifically, I appreciate the finding that optimizing each individual agent is necessary for optimal performance (Line 194), and that fine-tuning different LLMs for different tasks, instead of a single LLM for each different component is better (Line 199). The paper clearly provides novel evidence for these, along with a comprehensive study across multiple, commonly adapted multi-agent architectures.\n\nWhile the trace-augmentation is novel and the paper provides good insights through baselines, there have been fine-tuning based approaches in the literature, that tackle similar problems as multi-stage LLM pipeline, with the challenge of credit assignment, limiting the novelty of the framework. For example, I find the rest of the SiriuS framework (except trace augmentation) to be similar to the BootstrapFT optimizer for multi-stage LLM programs (which are a generalization of such multi-agent systems studied in this paper) presented in [1].\n\n[1] Khattab et. al, DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. https://arxiv.org/abs/2310.03714", "was_revised": true, "final_justification": "Following the review, the authors have provided further details about the baseline implementations and clarified about the presence of a fine-tuning baseline, sufficiently addressing my concerns.", "rebuttal": "We sincerely thank the reviewer for the positive assessment of our work, especially the recognition of our comprehensive evaluation and strong performance across diverse multi-agent tasks.\n\nWe also appreciate your insightful comparison to DSPy. Similar to TextGrad, DSPy focuses on optimizing declarative, multi-stage LLM pipelines under a centralized optimizer. In contrast, SiriuS is designed for interactive multi-agent systems, where each agent specializes in a distinct role and improves through role-specific fine-tuning and feedback loops. We also have conducted additional experiments comparing SiriuS with DSPy across College Physics, College Chemistry, PubMedQA. The results show that SiriuS consistently outperforms DSPy across tasks, confirming the advantage of role-specialized learning in complex reasoning settings.\n| Model         | Method        | College Physics | College Chemistry | PubMedQA |\n|--------------|--------------|----------------|------------------|--------------------|\n| **GPT-3.5-turbo** | Single-Agent | 25.55 ± 1.08  | 40.00 ± 3.95  | 57.53 ± 0.99  |\n|              | STaR       | 30.84 ± 0.93  | 45.64 ± 1.72  | 64.20 ± 0.53  |\n|              | COMM       | 28.97 ± 1.62  | *47.69 ± 3.95*  |*72.27 ± 0.81*  |\n|              | TextGrad    | *32.09 ± 1.08* | 41.54 ± 0.00  |  NA |\n|              | DSPy          |  29.91± 1.87 |39.49 ± 2.45|52.67± 1.29|\n|              | **SIRIUS**  | **33.96 ± 1.43**  | **55.90 ± 3.11**  | **75.33 ± 0.70**  |\n| **GPT-4o-mini** | Single-Agent| 38.63 ± 1.95  | 40.00 ± 2.59  | 63.87 ± 0.12  |\n|              | STaR        | *42.99 ± 0.93*  | 48.21 ± 2.28  | 65.93 ± 2.83  |\n|              | COMM       | 42.37 ± 1.43  | 49.23 ± 1.49  | *70.27 ± 0.42*  |\n|              | TextGrad   | 39.25 ± 3.37  | *49.74 ± 7.36*  | 66.20 ± 2.11  |\n|              | DSPy          |46.11± 1.95    |45.65± 2.35.       |55.53± 0.31|\n|              | **SIRIUS**   | **47.35 ± 1.95**  | **56.41 ± 3.11**  | **73.67 ± 0.31**  |\n\nWe hope our answers address your concerns, and we will include this baseline comparison in the final version of the paper.  Thank you once again for your valuable insights!"}, {"review_id": "SVdaHuBbLA", "reviewer": "Reviewer_HHDU", "summary": "This paper proposes SIRIUS, a framework for self-improving multi-agent systems. The core method involves iteratively fine-tuning agents on successful interaction trajectories and refining unsuccessful attempts through a guided feedback process. The authors demonstrate the framework's versatility by applying it to various multi-agent designs, including collaborative problem-solving, actor-critic feedback loops, and competitive negotiation games.", "questions": "See weaknesses", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Pros:**\n\n- This paper is well-written and easy to follow.\n- The study of multi-agent systems is a crucial direction that can further unlock the potential of large language models (LLMs).\n- The proposed method leverages negative samples for model improvement rather than discarding them, which enhances data utilization and training efficiency.\n\n**Cons:**\n\n- One major concern lies in the evaluation strategy.\n- The base models used in this study, such as GPT-3.5-Turbo, are somewhat outdated and closed-source. These models do not fully showcase the potential of the proposed system and cannot be easily re-implemented or extended by the research community. Including more recent and open-source models—such as Qwen 3/2.5, Mistral, or state-of-the-art reasoning models like Claude-4-Sonnet—would significantly improve the evaluation.\n- Another issue is the scope of evaluation tasks. Most of the tasks are either knowledge-heavy (e.g., College Physics/Chemistry) or relatively simple. Given the evolution of agent systems toward solving complex, practical problems (e.g., deep research tasks, SWE-agent scenarios), it would be valuable to see whether the proposed method has been evaluated on more sophisticated and realistic benchmarks.\n\n**Question:**\n\n- How many negative trajectories have been successfully augmented by the proposed module?", "was_revised": true, "final_justification": "One major concern was the evaluation strategy. In the rebuttal phase, the authors provided additional details regarding both proprietary models (e.g., GPT-4o-mini) and open-source models (e.g., Llama 3.2 3B Instruct), along with results on \"Trajectory Augmentation Coverage\" across these models. While the evaluation still does not include the most recent reasoning models or highly reasoning-intensive tasks, these additions address most of my concerns and make the paper significantly more comprehensive. Further analysis on more complex reasoning scenarios would strengthen the impact of the work.", "rebuttal": "We sincerely thank the reviewer for recognizing our work on multi-agent systems and our effective use of negative samples to improve training efficiency. We also appreciate the thoughtful comments and address the remaining concerns below.\n\n### ` W1. More Evaluations`\nWe have conducted additional experiments comparing SiriuS with DSPy to make our evaluation more comprehensive. The results show that SiriuS consistently outperforms DSPy across tasks, confirming the advantage of role-specialized learning in complex reasoning settings.\n| Model         | Method        | College Physics | College Chemistry | PubMedQA |\n|--------------|--------------|----------------|------------------|--------------------|\n| **GPT-3.5-turbo** | Single-Agent | 25.55 ± 1.08  | 40.00 ± 3.95  | 57.53 ± 0.99  |\n|              | STaR       | 30.84 ± 0.93  | 45.64 ± 1.72  | 64.20 ± 0.53  |\n|              | COMM       | 28.97 ± 1.62  | *47.69 ± 3.95*  |*72.27 ± 0.81*  |\n|              | TextGrad    | *32.09 ± 1.08* | 41.54 ± 0.00  |  NA |\n|              | Dspy          |  29.91± 1.87 |39.49 ± 2.45|52.67± 1.29|\n|              | **SIRIUS**  | **33.96 ± 1.43**  | **55.90 ± 3.11**  | **75.33 ± 0.70**  |\n| **GPT-4o-mini** | Single-Agent| 38.63 ± 1.95  | 40.00 ± 2.59  | 63.87 ± 0.12  |\n|              | STaR        | *42.99 ± 0.93*  | 48.21 ± 2.28  | 65.93 ± 2.83  |\n|              | COMM       | 42.37 ± 1.43  | 49.23 ± 1.49  | *70.27 ± 0.42*  |\n|              | TextGrad   | 39.25 ± 3.37  | *49.74 ± 7.36*  | 66.20 ± 2.11  |\n|              | Dspy          |46.11± 1.95    |45.65± 2.35.       |55.53± 0.31|\n|              | **SIRIUS**   | **47.35 ± 1.95**  | **56.41 ± 3.11**  | **73.67 ± 0.31**  |\n\n### ` W2. Evaluation on Open source model`\n\nWe have conducted additional experiments on Llama 3.2, which show promising improvements across all tasks.\n\n| Model         | Method        | College Physics | College Chemistry | PubMedQA |\n|--------------|--------------|----------------|------------------|--------------------|\n| **Llama 3.2 3B instruct** | Single-Agent | 26.79 ± 1.95 | 34.46 ± 2.28  | 54.07 ± 0.31|\n|              | STaR       | 27.41 ± 1.95 | 38.46 ± 1.49  | 54.93  ± 0.12|\n|              | COMM       | 27.73 ± 1.43  | 37.44± 3.11  |65.80 ± 0.87|\n|              | TextGrad    | Llama 3.2 3B instruct not supported|  \n|              | **SIRIUS**  | **29.60 ± 1.43**  | **42.56 ± 2.28**  | 68.27 ± 1.03|\n\nWe agree that demonstrating SIRIUS's effectiveness on open-source models is crucial for the community.  We appreciate the reviewer’s comment and hope this addresses the main concern.\n### ` W2. Scope of evaluation tasks `\n\nWe appreciate the reviewer's suggestion to explore more complex, practical problems. Our initial selection of tasks was intended to cover a diverse set of domains (science QA, negotiation games) and to compare with established baselines. We also evaluated SIRIUS in competitive settings (Resource Exchange, Ultimatum, Sell&Buy). These tasks involve different reasoning dynamics, such as negotiation and utility maximization, demonstrating SIRIUS's applicability beyond scientific QA. While we believe the core mechanism of SIRIUS—learning from an iteratively built and augmented library of trajectories —is generalizable, we agree that testing on more tasks would further strengthen the claims of broader applicability, and this is a valuable direction for future work.  \n### ` Q1. Trajectory Augmentation Coverage`\n\nThis is an important point that should indeed be reported. Our augmentation module successfully rewrote  **32.28% ~ 74.7%** of failed trajectories across different tasks and models. We provide the detailed statistics in the table below.\n\nTable: Augmentation Statistics Across Models and Datasets\n| Dataset           | Setting              | GPT-3.5-turbo | GPT-4o-mini | LLaMA-3.2-3B-instruct |\n|-------------------|----------------------|---------------|-------------|------------------------|\n| **College-Physics** | correct              | 56            | 106         | 51                     |\n|                   | wrong                | 156           | 106         | 161                    |\n|                   | augmented Trajectory       | 61            | 42          | 69                     |\n|                   | augmented percentage | 39.10%        | 39.62%      | 42.86%                 |\n| **College-Chemistry** | correct            | 45            | 58          | 48                     |\n|                   | wrong                | 83            | 70          | 80                     |\n|                   | augmented Trajectory       | 62            | 31          | 32                     |\n|                   | augmented percentage | 74.70%        | 44.29%      | 40.00%                 |\n| **PubMed**         | correct              | 382           | 358         | 342                    |\n|                   | wrong                | 118           | 142         | 158                    |\n|                   | augmented Trajectory       | 50            | 46          | 51                     |\n|                   | augmented percentage | 42.37%        | 32.39%      | 32.28%                 |\n\nWe will include this table in the camera-ready version. We thank the reviewer for bringing this to our attention.\n\n## **Summary**\n\nWe hope our answers address your concerns on (1) More Evaluations, (2) open-source evaluation, (3) Scope of evaluation tasks, and (4) Trajectory Augmentation Coverage.  We also sincerely thank the reviewer for recognizing the design and effectiveness of SIRIUS. Finally, we truly appreciate your reconsideration of our work in light of our responses. Thank you once again for your valuable insights!"}, {"review_id": "UoSb3GyfFO", "reviewer": "Reviewer_ko9b", "summary": "This paper introduces SIRIUS, a framework designed to optimize multi-agent systems powered by LLMs. The core problem addressed is the difficulty in training specialized agents within a collaborative system, particularly due to the challenge of credit assignment from a task-level outcome to individual agent actions. SIRIUS tackles this by creating an \"experience library\" composed of entire reasoning trajectories from successful task completions. This library of high-quality examples is then used to fine-tune the individual agents via Supervised Fine-Tuning (SFT). To further enrich the training data, the framework includes a trajectory augmentation procedure that refines unsuccessful attempts by generating feedback (using the ground truth) and prompting the agent to regenerate a correct reasoning path. The authors demonstrate the effectiveness of SIRIUS across three distinct settings: collaborative problem-solving (e.g., college-level science and biomedical QA), an actor-critic setup, and competitive negotiation games. The results show significant performance improvements over baseline methods, including single-agent and prompt-based multi-agent systems.", "questions": "1.In the Actor-Critic setting, the Critic Agent provides feedback to the Actor. How is this Critic Agent trained? Is it also fine-tuned using the SIRIUS framework? If so, what constitutes the reward signal for generating \"good\" feedback, especially since it operates without access to the correct answer?\n\n2.For the benefit of practitioners, could you provide an estimate of the computational resources (e.g., API costs, GPU hours, wall-clock time) required for one full fine-tuning iteration (as in Algorithm 1) on a dataset like PubMedQA? Understanding the practical costs is crucial for assessing the feasibility of applying SIRIUS in real-world projects.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1.The paper addresses a highly relevant and challenging problem in the field of LLMs: the effective optimization of multi-agent systems (Lines 26-31). As research moves from single-agent prompting to more complex collaborative frameworks, developing principled methods for training and improving these systems is crucial.\n\n2.The experimental validation is a major strength of this work. The authors evaluate SIRIUS across three diverse and well-chosen settings:\n- Collaborative Problem Solving: Demonstrates the ability to decompose complex reasoning tasks.\n- Actor-Critic: Shows how the framework can be used to improve self-correction and evaluation capabilities.\n- Competitive Settings: Tests the framework's ability to enhance strategic reasoning.\nThe inclusion of generalization experiments in the competitive setting (e.g., Figure 5, 6, 7), where agents trained on one set of initial conditions are tested on another, provides strong evidence that the models are learning robust strategies rather than overfitting.\n- Thorough Ablation Studies: The paper includes detailed ablation experiments that systematically validate the core design choices of SIRIUS. These studies effectively demonstrate the benefits of:\n\n3.The paper is generally well-written, clearly structured, and easy to follow. \n\n-----\n\nWeaknesses\n\n1.A significant weakness lies in the trajectory augmentation process, which is a key component for learning from failures. This process relies on an \"external agent\" that is provided with the ground-truth answer $y_i$ to generate corrective feedback. This dependency on an oracle or ground-truth signal undermines the \"self-improving\" narrative. It limits the framework's applicability to domains where such ground-truth answers are readily available for every failed instance, which is not the case for many complex, real-world problems. This critical detail should be discussed more prominently as a limitation in the main paper.\n\n2.Some key details of the methodology are ambiguous or omitted from the main text. For instance, the reward threshold $\\epsilon$ used in Algorithm 1 (line 6) to identify successful trajectories is a critical hyperparameter, but its selection process is not discussed. The framework's sensitivity to this value is unknown.\n\n3.The paper motivates the work by highlighting the multi-agent credit assignment problem (lines 29-31) but explicitly states that the proposed method \"sidesteps\" it (line 41). While learning from successful trajectories is a valid and effective engineering solution, it does not fundamentally solve the problem of attributing success to specific, crucial decisions within a long interaction history. This is also acknowledged in the paper's limitations (lines 853-857).", "was_revised": true, "final_justification": "The authors' response addressed most of my concerns, but I still believe the method is limited in its applicability, especially in Actor-Critic scenarios.", "rebuttal": "We sincerely thank the reviewer for the positive assessment and valuable feedback. We address your concerns below:\n### ` W1. GT for trajectory augmentation`\n\nWe appreciate the reviewer’s observation and agree that relying on ground-truth signals for feedback may limit applicability in certain domains. However, for a broad class of tasks—such as question answering, math problems, and many educational applications—ground truth is readily available, and leveraging it offers an effective and scalable way to generate corrective feedback. To address settings where ground truth is not accessible, our second task (Actor-Critic) explicitly explores a feedback mechanism that operates without ground-truth answers, demonstrating that SIRIUS remains effective under weaker supervision. We will make the discussion of this limitation and its applicability more explicit in the main paper.\n\n### ` W2. Reward Threshold`\n\nThank you for pointing out this ambiguity. In our experiments, a trajectory was considered successful if the final answer exactly matched the ground truth, meaning ε was effectively set to 1 for a binary reward. We will clarify this in the final version and apologize for the confusion.\n### ` W3. Credit assignment`\n\nWe appreciate the reviewer’s insightful comment. Our goal is not to solve the fundamental credit assignment problem directly, but to provide a practical and scalable approach for training multi-agent systems. By learning from complete successful trajectories, SIRIUS avoids the need for fine-grained rewards or step-level credit attribution, while partially addressing the challenge through targeted agent-level feedback during trajectory augmentation. We will revise the introduction and discussion to better clarify this point, and we aim to more directly explore the open challenge of credit assignment in multi-agent systems in future work.\n### ` Q1. Clarification for  Actor-Critic setting `\n\nIn the Actor-Critic setting, the system operates without ground truth (GT) during inference: the Judgment Agent evaluates the Actor’s initial solution to determine if it is correct or requires revision. If deemed incorrect, the Critic Agent provides feedback (without GT access), then the Actor regenerates the solution. This mimics real-world scenarios where GT is unavailable during problem-solving, and the system must self-correct iteratively. GT is only used in the final evaluation. We fine-tune each agent using its input-output pair. Ablation studies reveal that using base (unfine-tuned) components degrades performance:  a base Actor generates lower-quality initial responses, reducing the pool of correctable answers (e.g., TP drops by 14.8% with GPT-3.5), while a base Judgment Agent misclassifies correct solutions. Similarly, a base Critic produces less actionable feedback, lowering post-correction accuracy. These results underscore the necessity of fine-tuning all agents—Actor (generates/refines solutions), Judgment (accurately filters responses), and Critic (provides targeted feedback)—to ensure self-correction.\n### ` Q2. Cost analysis`\n\nThank you for pointing this out. To support reproducibility and cost estimation, we report the approximate token usage at each major stage of the SIRIUS pipeline. Given that API prices for models like GPT‑4o‑mini, GPT‑3.5‑Turbo, and LLaMA‑3 are typically at or below $0.60 per million tokens, the overall cost remains negligible. Considering the substantial performance gains (up to 21.88%), we believe the cost-performance trade-off is highly favorable and supports the practical viability of SIRIUS.\n| Domain  | Phase     | Category     | M tokens |\n|---------|-----------|--------------|---------|\n| phy     | generate  | solve          | 1.42    |\n|         |           | feedback     | 0.34    |\n|         |           | regenerate   | 15.98   |\n|         |           | finetune     | 1.12    |\n|         | eval      | eval         | 4.32    |\n|         |           | **total**    | **23.19** |\n| chem    | generate  | solve         | 0.80    |\n|         |           | feedback     | 0.16    |\n|         |           | regenerate   | 7.32    |\n|         |           | finetune     | 0.91    |\n|         | eval      | eval         | 2.41    |\n|         |           | **total**    | **11.60** |\n| Pubmed  | generate  | sol          | 1.13    |\n|         |           | feedback     | 0.22    |\n|         |           | regenerate   | 7.11    |\n|         |           | finetune     | 0.88    |\n|         | eval      | eval         | 7.45    |\n|         |           | **total**    | **16.80** |\n\nThis analysis will be included in the final version for completeness and clarity.\n## **Summary**\nWe hope our responses address the reviewer’s concerns regarding (1) reliance on a ground-truth signal, (2) reward threshold, (3) credit assignment, (4) clarification of the actor-critic setting, and (5) cost analysis. We also sincerely thank you for acknowledging the motivation and inherent difficulty of the problem we address, as well as the strength and comprehensiveness of our experimental validation across diverse multi-agent settings.  Finally, we truly appreciate your reconsideration of our work in light of our responses. Thank you once again for your valuable insights!"}]

Author rebuttal:

Meta-review: The paper proposes SiriuS, a self-improving framework for multi-agent LLM systems. It builds an experience library by retaining successful reasoning trajectories and augmenting failed ones with corrective feedback, then fine-tunes specialized agents on this evolving dataset. SiriuS is evaluated across scientific QA (MMLU, GPQA, TheoremQA, PubMedQA) and negotiation games, showing consistent improvements of 2.9%–21.9% over strong baselines such as Single-Agent, STaR, COMM, and TextGrad.

### Strengths of the paper:

- The reviewers (AYyk, HHDU, ko9b) highlight the novelty and practicality of the trajectory-based self-improvement loop, noting its broad applicability across collaborative problem solving, actor–critic feedback, and competitive multi-agent negotiation.

- The work is technically sound and well-structured, with ablations demonstrating the necessity of each design choice (role specialization, trajectory augmentation, multi-agent fine-tuning).

- The reviewers appreciate the effective use of failed trajectories to enrich training data, which improves data efficiency and stability.

- The empirical results are strong, consistently outperforming baselines across multiple tasks and model backbones, with meaningful improvements in reasoning and negotiation performance.

- The paper is generally well-written, easy to follow, and offers clear insights into both opportunities and limitations of multi-agent optimization.

### Weaknesses of the paper:

- The reviewer Hkfd emphasizes the lack of a fundamental solution to the credit assignment problem: SiriuS sidesteps it rather than resolving attribution of success to individual actions.

- Reviewers note the dependency on ground-truth signals for trajectory augmentation, limiting applicability to domains with labeled supervision.

- Cost and scalability concerns remain: the iterative process of multi-agent generation, regeneration, and fine-tuning requires substantial compute, and while cost–benefit analyses were later provided, questions of long-term scalability persist.

- Evaluation scope is somewhat narrow, omitting more challenging or practical agent settings (e.g., deep research tasks, real-world planning). Some baselines (e.g., BootstrapFT in DSPy) required clarification.

- Presentation could be more precise on methodological details (e.g., success thresholds, oracle reliance in augmentation).

### Primary reasons for Accept (Poster)

The primary reasons for recommending Accept (Poster) are that SiriuS introduces a general and practical bootstrapped learning loop for multi-agent systems, demonstrating clear novelty in systematically transforming both successful and failed trajectories into reusable training data. The framework is well-motivated, empirically validated across diverse tasks, and provides consistent gains over strong baselines. While the reliance on ground-truth signals and unresolved credit assignment limit broader applicability, the rebuttal addressed major reviewer concerns by adding open-source evaluations (LLaMA-3.2), cost–benefit comparisons, and clarifying Actor–Critic feedback. Overall, the paper makes a timely and meaningful contribution to multi-agent LLM optimization, justifying acceptance as a poster.

### Summary of the discussion and rebuttal

The authors provided detailed and constructive responses. For R-Hkfd, they added open-source results (LLaMA-3.2), disclosed token-level cost analyses, and clarified retrospective credit attribution strategies and Actor–Critic supervision, partially addressing concerns about cost and credit assignment. For R-AYyk, they compared SiriuS against DSPy’s BootstrapFT, showing consistent improvements, and clarified the distinction between pipeline optimization and interactive agent training. For R-HHDU, they reported augmentation coverage (32%–74%) across datasets and demonstrated improvements with newer models like GPT-4o-mini, leading to an upgraded score. For R-ko9b, they clarified the Actor–Critic setup and acknowledged the limitation of oracle-assisted augmentation, while highlighting broader applicability in negotiation settings. Overall, the rebuttal strengthened empirical support and methodological clarity, and several reviewers raised their scores, leading to consensus on poster acceptance.
Decision: Accept (poster)