Reflective Multi-Agent Collaboration based on Large Language Models
Authors: Xiaohe Bo
Zeyu Zhang
Quanyu Dai
Xueyang Feng
Lei Wang
Rui Li
Xu Chen
Ji-Rong Wen
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
Multi-Agent Systems
Reflection Mechanism
Primary area: natural_language_processing

Abstract: Benefiting from the powerful language expression and planning capabilities of Large Language Models (LLMs), LLM-based autonomous agents have achieved promising performance in various downstream tasks. Recently, based on the development of single-agent systems, researchers propose to construct LLM-based multi-agent systems to tackle more complicated tasks. In this paper, we propose a novel framework, named COPPER, to enhance the collaborative capabilities of LLM-based agents with the self-reflection mechanism. To improve the quality of reflections, we propose to fine-tune a shared reflector, which automatically tunes the prompts of actor models using our counterfactual PPO mechanism. On the one hand, we propose counterfactual rewards to assess the contribution of a single agent’s reflection within the system, alleviating the credit assignment problem. On the other hand, we propose to train a shared reflector, which enables the reflector to generate personalized reflections according to agent roles, while reducing the computational resource requirements and improving training stability. We conduct experiments on three datasets to evaluate the performance of our model in multi-hop question answering, mathematics, and chess scenarios. Experimental results show that COPPER possesses stronger reflection capabilities and exhibits excellent generalization performance across different actor models.

Reflective Multi-Agent Collaboration based on Large Language Models
Xiaohe Bo', Zeyu Zhang', Quanyu Dai Xueyang Feng',
Lei Wang', Rui Li", Xu Chen : Ji-Rong Wen'
Gaoling School of Artificial Intelligence, Renmin University of China
Huawei Noah's Ark Lab
(xiache, zeyuzhang, xueyangfeng, wanglei154, 1ir.uii1,,11 xu, chen, jrwen) Gruc edu, cn,
daiquanyuQuuavei.com
Abstract
Benefiting from the powerful language expression and planning capabilities of Large Language Models (LLMs), LLM-based autonomous agents have achieved promising performance in various downstream tasks. Recently, based the development of single-agent systems, researchers propose to construct LLM-based multi-agent systems to tackle more complicated tasks. In this paper, we propose a novel framework, named COPPER, to enhance the collaborative capabilities of LLM-based agents with the self-reflection mechanism. To improve the quality of reflections, we propose to fine-tune shared reflector, which automatically tunes the prompts of actor models using our counterfactual PPO mechanism. On the one hand, we propose counterfactual rewards to assess the contribution of single agent's 's reflection within the system, alleviating the credit assignment problem. On the other hand, we propose to train a shared reflector, which enables the reflector to generate personalized reflections according to agent roles, while reducing the computational resource requirements and improving training stability. We conduct experiments on three datasets to evaluate the performance of our model in multi- hop question answering, mathematics, and chess scenarios. Experimental results show that COPPER possesses stronger reflection capabilities and exhibits excellent generalization performance across different actor models.
Introduction
With the emergence of Large Language Models, LLM--bsse autonomous agents are becoming research hotspot in the field of artificial intelligence. Leveraging the impressive planning and reasoning ability of LLMs, these agents can understand and generate human-like instructions, engage in sophisticated interactions, and make decisions in a wide range of contexts, leading to remarkable
success various downstream tasks 26 24 25 5 22 Recently, based on the development of single-agent systems, researchers propose to construct multi-agent systems in response to the growing task complexity. Prior works BI
12 suggest that multiple agents can help improve factuality and reasoning, encourage divergent thinking, and effectively facilitate task completion.
To improve the collaborative performance of multi-agent systems, various cooperation frame- 39 have been developed, which generally encode intricately crafted agent profiles and cooperation mechanisms into prompts. However, hindered by the contextual understanding ability of LLMs, such frameworks fall short of fully exploiting the collaborative capacities of agents. To tackle this challenge, one natural idea is to gather extensive collaborative data for agents' fine-tuning. Yet this strategy risks diminishing the model's general abilities [35), contradicting the aspiration to
works 31 23
"Correponniing Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

HotPetQA
GSMBK
Checkmate in Move
Wat Nakct
Rabi cot
50
+4
Shg Reftndar
Sing tuact
90
Hutt Burtiertedin
Singl-cof
Grgie-rdtrz.o
34
-4-
-6-
Malt kndrodnn
Segle: out
Shaft Refendr
30
Succass
26
73
35
18
0.0
0.5
1.0
1.5
2.0
2.5
3.0
355
4.0
70
0.0
0.5
1.0
1.5
2.5 NialNumber
2.0
3.0
3.5
4.0
14
0.0
0.5
1.0
1.5
2,0
25
3.0
255
4.0
Thiil Number
Thial Number
(a) HotpotQA.
(b) GSM8K.
(c) Cheekmate in One Move.
Figure I: Performance comparison between single-agent and multi-agent systems on three datasets. We use GPT-3.5 as the base model and prompts are shown in the Appendix H
attain artificial general intelligence (AGI). Considering that, in this paper, we propose to optimize the collaboration process through the self-reflection mechanism [27], with which binary or scalar rewards from environments can be converted into verbal reflections, providing additional context to improve task performance. To accommodate the characteristics multi-agent systems, we addition- ally incorporate agent profiles in reflector prompts for agent-specific reflections and adopt fully observable setting to facilitate agents' error detection.
Although the self-reflection mechanism enables iterative refinement, deriving useful reflections from pre-trained frozen LLM is challenging. In dealing with that, Retroformer (38) suggests approximating reflection rewards with the difference between two consecutive episode returns and training a plug-in reflector through policy optimization. However, extending the method to multi-agent systems is not straightforward. On the one hand, the episode difference score from the environment can only capture the overall contribution from all agents' reflections, while the credit to each agent's reflection unknown and nontrivial to obtain. Employing the overall score directly and uniformly for reflector training of multiple different agents could lead to lazy reflectors, and thus the credit assignment problem becomes critical. On the other hand, to achieve personalized reflections of intelligent agents, the number of reflectors to be fine-tuned will expand proportionally with the number of agents in the system, posing challenges for practical applications in real- world scenarios.
To address the challenges above, we propose reflective framework: COunterfactual PPO Enhanced Shared Reflector for LL.M-based Multi-Agen, Collaboration, named COPPER. For the first challenge, we propose counterfactual rewards as supervision signals for individual agent reflections. Specifically, we first integrate the reflections of all agents into the corresponding actor model prompts and utilize the episode return difference score as the overall reward. Then we sequentially marginalize out the reflection of one agent, repeat the interaction process, and attain new task score. The resulting episode return difference is referred to as marginal reward. The counterfactual reward then calculated by subtracting the marginal reward from the overall reward to reflect the contribution of the removed reflection. For the second challenge, considering the homogeneity between different reflectors, which means their action space (reflection) is consistent and the optimization objectives are aligned (to assist in solving the overall task), we propose to train a shared reflector for agents in the collaboration system. With carefully designed prompts, the shared reflector grasp the role information of each agent, simultaneously reducing the demand for computational resources and enriching the training data pool for stable training. The counterfactual reflection data from all agents are then collected and utilized to train a shared reflector through proximal policy optimization (PPO).
Our contributions can be summarized as follows:
We propose a novel reflection framework, named COPPER, to improve the multi-agent collaboration. We incorporate agent profiles into reflector prompts for agent-specific reflections and adopt fully observable setting to assist in error detection.
We propose to train a shared reflector using our counterfactual PPO mechanism. To alleviate the credit assignment problem in multi-agen systems, we design counterfactual rewards rate each agent's 's reflection. Besides, we propose to train a shared reflector, which could generate personalized reflections while reducing computational resource demands and improving training stability.
Experimental results on three open-source datasets demonstrate that COPPER possesses stronger reflection capabilities against basclines. More concretely, compared to the initial success rate,

---

COPPER brought improvements of 31.8%, 18.5%, and 86.4% on the HotPorQA, GSM8K, and Checkmate in One Move datasets, respectively.
2
Related Work
2.1 LLM-based Multi-Agent Systems
Based on the development of ingle-agent systems, LLM-based multi-agent systems have been rapidly studied and achieved significant progress in complex task resolution and world simulation. Within the task resolution domain, various agents, each with specialized expertise are developed to collaborate
on complex problems. For instance, [ 32 29 17
suggest improving the accuracy of scientific
question-answering tasks through multi-agent debates. [23] 13] suggest constructing multi-agent
systems for software development following the waterfall Standardized Operating Procedures (SOPs) workflow. Another mainstream application scenario of LLM -baeed multi-agent systems is the world simulation, which mainly leverages the role-playing abilities of agents to represent different roles and perspectives within a simulated environment. Research in this area is advancing quickly and encompasses a wide variety of fields, including social sciences 442 [ gaming
[3) 34 20
psychology (2), economics [66 41, policy making [15]. etc. In this paper, we focus on improving the complex problem-solving abilities of multi-agent systems.
2.2 Self-Reflection of Large Language Models
Various studies on reflection mechanisms have been proposed, which play a crucial role in enabling LLM-based agents learn from the environment and improve themselves autonomously. Early works primarily focus on refining responses based on a single feedback [19 6 or contrast between multiple models [TO 40) and fail to form comprehensive understanding of the task based on past experiences, Recently, Reflexion (27] involves prior trajectories and environmental rewards to generate reflections. which are further incorporated into the context of subsequent episodes. Although it enables iterative enhancements, the effectiveness of reflections heavily relies on the model's ' inherent reflective capabilities. In light of that, Retroformer [3 proposes to fine-tune the reflector using environmental rewards with standard RLHF 21 process. However, optimizing reflection in multi-agent systems remains challenging, which is crucial for improving agents' cooperation capacity and task performance.
3 Preliminary
In this paper, we use a tuple ( SN,S, S,A,A,A.7 Pε., R) to denote the LLM-based multi-agent cooperation system, where N stands for the number of agents, S = S₁ x S2 ... SN is the joint space of environment states, A = A₁ X A2 x .... x An is the joint action space and Pe. S x A - S is the state transition function. Here, we denote the randomness associated with the state transition using & according to (38). In cooperative settings, all agents share an aligning goal, and the reward function R: (S, R is typically designed to promote collaboration. One major challenge in cooperation settings is credit assignment, which means we need to decompose R into R1 x Rg x x RN and evaluate the agent contribution with respect to their objective in the form of scalar value. The multi-agent systems complete the target task through interactions with the environment. Here we use trajectory { ag. $1, aj. , ST, 3 to denote the process and describe the accumulative reward using R(T). where R(T) = E R(st, at) and T is the length of the trajectory. In most of the situations, rewards from the environment are sparse, which means R (tt, az) are mostly zero except very few states, such as the terminal state for indicating task success or failure.
Specifically, for each agent i, we consider its actor model as function M : X, A,, where X, is the space of the prompts and & represents the random variables involved in the sampling process. To maintain the general abilities of agents, in this paper, we select LLMSs with frozen parameters such as ChatGPT and GPT-4 as actor models. Current environment states are incorporated into prompts in the form of natural language and actions are selected based on the contextual learning ability of LLMs. Meanwhile, we propose to improve collaboration through self-reflection and introduce a reflector model M IR + X₁ for each agent, where T denotes the space of trajectories, &,- represents the randomness in the reflector model and denotes the learnable parameters. The reflector model takes the prior trajectory and the reward signal from the environment as input, outputting reflections
3

---

to refine the prompt of the corresponding actor model. We fine-tune reflector models through policy optimization for better task performance specific environments.
4
Method
4.1
Multi-Agent Collaboration
LLM-based multi-agent systems aim to deliver advanced capabilities by leveraging collective intel- ligence and specializing LLMs into agents with distinct capabilities. In this paper, each agent in the system operates in predetermined sequence, taking turns to produce responses, while shared message pool is maintained to facilitate efficient communication. We illustrate the details of the collaboration settings in Appendix A
Specifically, in dealing with problem & at time t, agent (i mod N) first subscribes the preceding interaction records [5k,i, ix.,i from the message pool, and then acquires the current environment state tk,r. The decision-making process can be expressed
= Actor' (p', [SR,5, a sk,e),
where p¹ is the profile of the current agent, which encompasses its role, action space, and additional constraints in the form of natural language. Once reaching a decision, the agent publishes new message {5R,e, } J to the message pool.
However, during implementation, the interaction history could potentially exceed the token limit of LLMs, given the large number of agents and decision steps. To address this challenge, we introduce a context model recursively update the interaction history from each agent's 's perspective, serving as its short-term memory. New messages since the agent's 's last action and the profile will be integrated form new short-term memory based on the previous one. The process can be written as:
smi = Context' (p', sm {st, a; m))==mx(0.)-+++))
where sm represent the short-term memory of agent when solving problem at time
We then replace the interaction history with the agent's short-term memory. Therefore, the decision- making process can be further rewritten as:
0 = Actor (p', e.
4.2 Multi-Agent Reflection Framework
To bolster the collaborative performance of multi-agent systems in specific scenarios, while preserving the general capabilities of agents, we introduce a self-reflection mechanism to multi-agent systems, of which the details are shown on the left part of Figure [ Using environmental rewards as guidance, the generated reflections could act as semantic gradient signals by providing concrete direction for improvement, thereby helping the agent learn from prior errors and perform better on the task.
Different from reflections in single-agent systems, we integrate agent profiles into the multi-agent reflection process obtain role-specific reflections, and take fully observable setting assist the agent in error detection by offering interaction histories from each agent's 's perspective. The reflection process of the agent can be defined as:
vi. Reflector (p', [sm Tk,a):
where i represents the problem, À indicates X-th trial of answer to question k. T the length of the trajectory Tæ. and Tk, is the environmental rewards. Due to the iterative updating nature of the short-term memory, smi,A,7 contains the complete action information of agent i in trajectory TR
We store all previous reflections of agent in its long-term memory. which are then added as additional context of the actor model. The decision making process in Equation 3 can be further defined as:
0k.A. Actor' (p', Im, sm sk,A.),
where we additionally incorporate subscript due to the introduction of the reflection mechanism.

---

Action Module
Memory
Counterfactual Reward Construction
Trill A
Trial
Ubserve
Verray
xla
Shatler mensy
Ali
Rutfocilie
Plass Thans
Step Get arita Actiot
Seep Ager SURiCiC
raccay
BI
Environment
Action
(
Overall Reward G2 Jasss -7a
Countorfactual Roward
Statler
Reward
B C2,, -22
Long tenn
PA
Reflection Module
Trial
Hu
Spnn Reflecto
Rotrcctoo yE.
Yaans fur
Larpiam
Retlcction 20.2
Autto
Faxes Parrs
LLM based Agents
Agent Profile
Frior Trajestory
Reward
The overview of our proposed COPPER. The left side illustrates the multi-agent reflection framework. The system first computes the identifier i of the agent to respond at the current time (Step 1). Then, agent updates its memory, including reflections of previous trials and the current trial's s historical interactions (Step 2), perceives the environmental state such the question and current task scores (Step 3), and generates the action (Step 4). After several rounds of interaction, the task trajectory and the reward score are fed into reflectors along with agent profiles generate reflections, which are then stored in long-term memories and serve as additional context for the continuous optimization of actor prompts. On the right side, we depict the construction of counterfactual rewards, which are further employed for fine-tuning the shared reflector.
Figure 2:
4.3 Optimization of the Shared Reflector
Generating useful reflective feedback with frozen LLMs in multi-agent systems proves to be chal- lenging, since it demands a profound grasp of agent characteristics and collaborative environments. Hence, in this paper, we propose to fine-tune a shared reflector using open-source LLMs (such as Llama) with our counterfactual enhanced proximal policy optimization mechanism.
4.3.1 Instruction and Response Collection
In the episode À of problem k the multi-agent system first interacts with the environment to produce trajectory TR,A, after which the reward function returns a score Ти.x. Agents in the system then reflect on the prior failed trajectory and generate verbal feedback to refine the corresponding actor prompt. In the process, Reflector' takes {p', (,m,,,, | A as the instruction =x,x> and is prompted to produce reflection response V Considering the homogeneity between agent reflectors well as the training efficiency, we gather reflection data from all agents across tasks and trials to train shared reflector. The offline training data can be defined as follows:
D = { v < i < N, 1 < X A. S k KKK,
where A is the maximum trial count and K is the total number of problems.
4.3.2 Counterfactual Reward
In this paper, to alleviate the credit assignment issue, we propose the counterfactual reward to achieve agent-specific reflection ratings for multi-agent collaboration. The construction of counterfactual rewards shown on the right side of Figure 2
Specifically, we first calculate an overall reward of the multi-agent system Gj following Retro-
former (38 i.e. Gk.A T4, Tk, Then, sequentially marginalize out piece of reflection from agent (which means we do not add the reflection to the actor model's prompt in the subsequent trial), while keeping other agents' reflections fixed. new reward score Áx,A+1 then returned
after an interaction trajectory, based on which we calculate a marginal reward Gj,A = fk,A+1 - Fk.A- Finally, the counterfactual reward of reflection pair (I vk.)) is calculated by subtracting the
5

---

marginal reward from the overall reward:
G GR,A Gk
(7)
Our counterfactual dataset Der can be further denoted as:
Der (1rk GR
(8)
4.3.3 Counterfactual Proximal Policy Optimization
Following previous works that tackle Reinforcement Learning from Human Feedback (RLHF) 2T. we adopt similar three-step approach to fine-tune the shared refeector with counterfactual rewards.
For the first step, take the reflections with positive supervised reflector SFT with Supervised Fine-Tuning (SFT), which can be written as:
demonstration data and train
SSFT -R( Der log ( [r, yc*)],
where is the reflection prompt, and y represents the generated reflection.
For the second step, taking construction expenses into account, instead collecting pairwise responses for each input, train regression model to assess prompt and reflection pairs. We optimize the reward model RCF, with counterfactual dataset Der by minimizing the Mean Square Error (MSE) loss:
Caa (Φ) = E (ecti)DDDe(eeee (RCF, (x.y) r)
(10)
For the third step, utilize the counterfactual reward model optimize the supervised reflector via PPO. We begin by initializing TSF which is used to produce predictions for randomly chosen samples from the entire dataset DEF. Subsequently, the counterfactual reward model RcF, assigns reward to each response. Our goal is to optimize the reflector model by maximizing the total reward, which can be accomplished by minimizing the following loss objective:
RL (y xc)
(P)o( -Ex~ By By (B) (2, (2,8) - B log SFT (y/xr
(11)
5 Experiments
5.1 Datasets
We choose HotPotQA 33). GSM8K 77. and Checkmate in One Move (28 to evaluate the collabora- tive abilitics of multi-agent systems multi-hop question answering, mathematics and chess.
HotPotQA HotPotQA is multi-hop question -antiionnnnwwringg dataset designed to evaluate models' complex reasoning ability. It contains 90,447 question-anm answer pairs that generally require multiple reasoning steps across documents to arrive at an answer.
GSM8K GSM8K is collection of 8.5K diverse and high-quality math word problems for grade school students. Each problem requires between 2 to 8 steps to solve, with solutions mainly involving a series of fundamental calculations with basic arithmetic operations.
Checkmate in One Move Checkmate in One Move is a dataset from The Beyond the Imitation Game Benchmark (BIG-bench), featuring 3,500 games to assess language models' proficiency in playing chess using standard algebraic notation (SAN). When presented with a move sequence leading potential checkmate, the model is tasked with identifying the move that achieves checkmate.
5.2 Baselines
We compare the following baseline models to verify the effectiveness of COPPER: 1) CoT 30 CoT suggests bridging the gap between question and answer by generating intermediate reasoning and useful for simple questions without tool needs. We adopt CoT in math and chess environments

---

following 8 to represent the initial success rate of the system. 2) ReAct [37. This is the state- of-the-art frozen language agent architecture, which mainly relies on the reasoning and planning ability of LLMss. It serves as a baseline in HotPotQA to denote how the agent performs without using environmental feedback. 3) Reflexion 27]] This a classic framework learn from environment signals and generate verbal feedback to improve task performance. We extend the method to multi- agent systems and respectively employ GPT-3.5 and LongChat as reflectors to reflect on multi-agent ReAct or CoT trajectories, without fine-tuning the reflectors. 4) Retroformer (38). The paper proposes an effective method for enhancing the reflective capability of agents in single-agent systems. Here, treat the agents in multi-agent environment mutually independent and fine-tune the reflector of each agent following Retroformer as a baseline.
5.3 Implementation Details
Model We use GPT-3.5 (model: gpt-3.5-turbo) the frozen actor models as well as the context models of agents and fine-tune LongChat (model: longchat-765 7b-16k) as the shared reflector. We choose gpt-2 as the regression reward model for counterfactual PPO training.
Collaboration Settings We adopt a cooperative debate paradigm GSM8K and Checkmate in One Move following [8) while on HotPoiQA, in alignment with [27 38 we design a teacher-student paradigm enable agents to call the retrieval tool.
Data Collection We randomly select 2,000 tasks collect reflection data on HotPotQA and Checkmate in One Move, while on the GSM8K dataset, due to the higher initial success rate and fewer reflections, we randomly select 3,000 instances. We set the maximum number of trials to 5, the temperature of GPT-3.5 to O, and the temperature of LongChat to 0.9. We use the FI score as the reward function of HotPotQA following (8] and exact match score in other environments. Comprehensive details regarding the quantity of collected datasets can be found in Appendix B
Training We use LoRA 14 for efficient fine-tuning of the shared reflector and implement RLHF through the trl package of HuggingFace. For SFT training, we tune the epoch in {1, 2, 3, 4). batch size in {64, 128, 256]. and learning rate in (le-4, 2e-4, 3e-4, 5e-4) through grid search on validation set with 100 instances, while for counterfactual PPO, we change the search range of learning rate to (le-5, 2e-5, 3e-5, 5e-5). As for the reward model, we set learning rate to 5e-5, training epoch to 3 and batch size to 16. We conduct all experiments on four NVIDIA A800-80G GPUs.
Evaluation In alignment with constraints imposed by computational resources and following precedents set by earlier research [38 (27), we randomly sample 100 instances the test set. We set the temperature of both GPT-3.5 and LongChat to 0 during the test phase to ensure reproducibility. We measure the performance of the system in exact match accuracy during the test phase.
5.4 Main Results
We compare the performance of COPPER against different baselines on HotPotQA, GSM8K, and Checkmate in One Move after 5 trials as main results, which are shown in Figure 3 By observing the results, we find that the results of different methods on the three datasets show roughly the same pattern: (1) Contrasted with the outcomes of multi-agent ReAct or CoT, employing the multi-agent reflection framework outlined in Section 4.2 can notably enhance the performance of multi-agent systems in specific tasks. For instance, in HotPotQA environment, the inclusion of LongChat and GPT-3.5 as reflectors leads to improvements of 15.9% and 22.7%, respectively, over the initial success rate, (2) Compared to the original LongChat and GPT-3.5, COPPER demonstrates stronger reflective abilities. The fine-tuned reflector is proficient in identifying the cause of task failure and devising personalized improvement strategies for diverse intelligent agents. Compared to the initial success rate, COPPER brought improvements of 31.8%, 18.5%, and 86.4% on the HotPotQA, GSM8K, and Checkmate One Move datasets, respectively. (3) Compared to Retroformer, COPPER can improve the performance of multi-agent collaboration faster. We speculate that the improved performance is brought by our special designs for multi-agent settings, such as the counterfactual reward and the shared reflector.
7

---

HotPotQA
GSMBK
Checkmate One Move
Rokit
CKT
Cul
55
Antelsarer DURI
Retrotamer peatt
COPFER:
- COPRC4
52
30
24
035
155 2.0
3.
0. 0.5
155
255
3.0
4.0
0.
10
1.5
2/5
4.
Trial Number
Thall Number
Tal Number
(a) HotpotQA.
(b) GSM8K.
(c) Checkmate in One Move.
Figure 3: Performance of COPPER against baselines on three datasets.
5.5
Ablation Study
We conduct ablation study on three datasets explore the effectiveness of each component of COPPER. We exclude the counterfactual reward (w/o CF) and proximal policy optimization (w/o PPO) individually and illustrate the outcome of Reflexion (LongChat) for comparison purposes (equivalent to eliminating the entire fine-tuning process), Experimental results are shown in Figure ! From the results, we can conclude that both counterfactual reward and PPO fine-tuning are crucial for COPPER, and removing any part will lead to a decrease in performance. On the one hand, substituting counterfactual rewards with episode return difference rewards will lead to uniform rewards for all agents' reflections, meaning the contribution of reflection by each agent is equal. This could elevate the reward score for reflections that offer little assistance in enhancing collaboration performance, presenting a challenge in refining the reflector. On the other hand, fine-tuning PPO on the basis of SFT can further enhance the reflective ability of the shared reflector. This indicates that by maximizing environmental rewards, PPO can refine the model's 's output to better suit human preferences. For HotPotQA and GSM8K, we notice that the enhancement from COPPER during the initial two rounds is comparatively lower than solely fine-tuned with SFT. However, COPPER exhibits the highest success rate after five trials. This may be due to the fact that during the PPO training process, the reflector learns to sacrifice early performance for greater ultimate benefits.
HotPotQA
COPFER
CDAP2R
Colliii
COP-E Jall 1901
(wto (H-0)
38
54
Astotan Zaep(hei)
Reficcon Lisrethul)
Rellooan (lamgOhio
52
50
24
035
2.0 Trial Number
0.5
2.
30
43
00
15
1.5
Trial Namber
Tial Number
(a) HotpotQA.
(b) GSM8K.
(c) Checkmate in One Move.
Figure 4: Ablation study.
5.6 Generalizability of the Shared Reflector
We conduct experiments on three datasets investigate the generalizability of COPPER, with the outcomes visualized in 5 Specifically, we implement COPPER trained in multi-agent systems with GPT-3.5 actors to systems with GPT-4 (model: gpt-4-turbo) actors. We compare generalized COPPER against two baselines: one featuring GPT-4 as the reflector and the other utilizing LongChat as the reflector. We conclude that COPPER remains proficient in reflection capabilities within the systems featuring GPT-4 actors. Compared to the initial success rate, COPPER demonstrates improvements of 27.7%, 9.0%, and 53.3% in HotPotQA, GSM8K, and Checkmate in One Move respectively, and achieves comparable performance to GPT-4 reflectors after 5 trials.
5.7 Generality of Counterfactual Rewards
To tackle the credit assignment challenge in multi-agent systems, the paper suggests deriving scores for individual agent reflections using counterfactual rewards, which is essentially a data augmentation approach. Hence, in this section, we delve into the suitability of counterfactual rewards for L.M fine- tuning techniques beyond RLHF. Specifically, evaluate the performance of CF SFT (employing
8

---

HotPotQA
GSMBK
Checkmate in One Move
52
Cell
Artreson Loen
Aefiodin unrt
Adfoution regiioo
Artroiin Eleeehorr
-8- PeEndantinnffa
42
34
0.5 1.0
20
3.5
0.5
1.5
2.0
3.0
00
1.5 2.5
Numle
(a) HotpotQA.
(b) GSM8K.
(c) Checkmate in One Move.
Figure 5: Apply the shared reflector trained for GPT-3.5 to GPT-4.
counterfactual rewards screen positive examples) against that of typical SFT fine-tuning (utilizing episode difference rewards to filter positive examples), as illustrated in Figure 6 Analysis of the results reveals that CF SFT outperforms regular SFT across all three scenarios. This underscores the effectiveness of counterfactual rewards in offering more objective score based on model reflection contributions, thereby ensuring the selection of positive examples of higher quality.
HotPotQA
Checkmate
contnn ISFT (F) comnn I971!
CD (EPS1-5PTIN
COPPER.GTT
34
9 18
24
Triil Number
Namber
Thal Number
(a) HotpotQA.
(b) GSM8K.
(c) Checkmate in One Move.
Figure 6: Applying counterfactual rewards to SFT.
5.8 Effect of the Shared Reflector
In multi-agent systems, the quantity of reflectors will increase with the number of agents. This will lead to an excessive search space of Therefore, suggest training shared reflector that employs carefully designed prompts to enhance the training efficiency and stability, without compromising personalized reflective abilities. In this section, we explore the effectiveness of shared reflector and present results in Figure 7 During the implementation of non-sh reflectors, given the uniformity among agents, we streamline the hyper-parameter search by aligning the syper-parameters of each reflector. Experiments indicate that shared reflector can deliver better reflection effects, possibly because it can access more training data, leading to superior training outcomes.
posing challenges for practical applications.
HotPatQA
GSMBK
Checkmate in One Mave
5.3
56
CDPER (SFf!
CDPER inlistaro
90 tndo sharei
CD (5P))
- confee out stard
COFFic (3P))
1/4
22
3.0 0'5
20
3.
4.0
0.0 6.5
1:5
2.0
3.0
4.0
0.0
110
1.5
30
4'0
(a) HotpotQA.
(b) GSMSK.
(c) Checkmate in One Move.
Figure 7: Exploring the effectiveness of shared reflector.
5.9 Effectiveness of Agents' Profiles
order to reduce training costs while generating personalized reflections multi-agent systems, we propose to add agent profiles to the input of reflectors and train shared reflector. In this section, we further verify the necessity role information in multi-agent reflection scenarios. Specifically, we remove the agents' profiles from the input of the reflector, and the experimental results are shown

---

in Figure By comparing Figure and Figure g we can observe that when using pre-trained LMs (LongChat and GPT-3.5) reflect, the removal of the agent profile has greater impact on the GPT-3.5 reflector. This may be due to GPT-3.5's better contextual understanding ability. Our COPPER can further improve the model's reflection ability under no-profile setting. However, the results are slightly worse than the setting with agent profiles.
HotPatQA
GSMBK
Checkmate One Move
Rokit
RoME
Adflstan Cang
langchall
Rellidan (angOOra))
CO7NER
33.0
-6-
COPPER
32.5
3..0
27.5
4.-
44
03
2.0
20
35
03 0.5
1.3 2.0 235
4.
00
1,5
23
30
40
Trail Number
Thal Number
That Number
(a) HotpotQA.
(b) GSMSK.
(c) Checkmate One Move.
Figure Performance under no-profile setting.
5.10 Different LLMs as Base Reflectors
CCT
In this section, we replace the base reflector from LongChat with Llama-3 (model: llama-3-8b- 16k) and explore the applicability of COPPER for different base models the GSM8K dataset. Exper- imental results shown Figure 9 demonstrate that our proposed COPPER has good performance across different base models. When comparing to the initial success rate, fine-tuning Llama-3 with coun- terfactual PPO shows 17.3% enhancement, surpassing the per- formance of the GPT-3.5 reflector after 5 trials. Additionally, we include the outcome from fine-tuning Llama-3 exclusively with SFT. From the results, we find that PPO can further improve the reflective capabilities of the shared reflector.
Success mreikl
16
Thtal Number
Figure 9: Fine-tuning Llama-3 as the shared reflector.
Limitations
While counterfactual rewards can mitigate the credit assignment issue in multi-agent collaboration, constructing such rewards with L.MSs imposes additional data requirements. Though our proposal in- volves training a shared reflector and updating the reward model's loss function to MSE, investigating more efficient data collection approaches is still needed. Besides, in this study, restrict long-term memory to a sliding window with a maximum capacity. We believe extending the agent's 's memory to more advanced structures such vector embeddings presents a promising direction for development.
7 Conclusion
In this paper, we consider leveraging the self-reflection mechanism to improve multi-agent collabo- ration, and propose an elegant framework COPPER. Towards more efficient reflection, we train shared reflector using the counterfactual PPO mechanism. The counterfactual reward can be evaluated according the impact of each agent reflection on enhancing task performance. To enhance the training efficiency and stability, we gather reflection data across agents and train shared reflector. Experiments on three datasets indicate that our COPPER exhibits superior reflective ability and effective generalization across various actor models.
Acknowledgements
This work is supported in part by National Key R&D Program China (2023YFF0905402) National Natural Science Foundation of China (No.62102420), Intelligent Social Governance Platform, Major Innovation & Planning Interdisciplinary Platform for the "DoubleFirst Class" Initiative, Renmin University of China, Public Computing Cloud, Renmin University of China, fund for building world- class universities (disciplines) of Renmin University of China, Intelligent Social Governance Platform. The work is also supported by Huawei Innovation Research Programs. We gratefully acknowledge
10

---

the support from Mindspore CANN (Compute Architecture for Neural Networks) and Ascend AI Processor used for this research.
References
[1] Saaket Agashe, Yue Fan, and Xin Eric Wang. Evaluating multi-agent coordination abilities in large language models. CoRR, abs/2310.03903, 2023.
[2] Gati V. Aher, Rosa I. Arraga, and Adam Tauman Kalai. Using large language models to simulate multiple humans and replicate human subject studies. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, pages 337-371, 2023.
[3] Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang. Jie Fu, and Zhiyuan Liu. Chateval: Towards better Ilm-based evaluators through multi agget debate. CoRR, fbs/2308.00200. 2023.
[4] Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje Karlsson, Jie Fu, and Yemin Shi. Autoagents: A framework for automatic agent generation. CoRR, abs/2309. 17288, 2023.
[5] Xinshi Chen, Shuang Li, Hui Li, Shaohua Jiang, Yuan Qi. and Le Song. Generative adversarial user model for reinforcement learning based recommendation system. In Proceedings of the 36th International Conference on Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97
of Proceedings of Machine Learning Research, pages 1052- 1061, 2019.
[6] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models self-debug. CoRR, abs/2304.051 2023.
[7] Karl Cobbe, Vineet Kosaraju. Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/21 10. 2021.
[8] Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. CoRR, abs/2305.14333 2023.
[9] Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. CoRR, abs/2305. 14325, 2023.
[10] Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. CoRR, abs/2305.14325 2023.
[11] Chen Gao, Xiaochong Lan, Zhihong Lu, Jinzhu Mao, Jinghua Piao, Huandong Wang, Depeng Jin, and Yong Li. $3: Soeial-network simulation system with large language model-empowered agents. CoRR, Abs/2307.149983 2023.
[12] Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. CoRR, Abs/2402.01680 2024.
[13] Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, and Chenglin Wu. Metagpt: Meta programming for multi-agent collaborative framework. CoRR, abs/2308 003552, 2023.
[14] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022.
[15] Wenyue Hua, Lizhou Fan, Lingyao Li, Kai Mei. Jianchao Ji, Yingqiang Ge, Libby Hemphill, and Yongfeng Zhang. War and peace (waragent): Large language model-based multi-agent simulation of world wars. CoRR, abs/231 17227, 2023.
[16] Nian L., Chen Gao, Yong Li, and Qingmin I Liao. Large language model-empoweredd agents for simulating macroeconomic activities. CoRR, abs/2310. 10436, 2023.
[17] Ruosen Li, Teerth Patel, and Xinya Du. PRD: peer rank and discussion improve large language model based evaluations. Trans. Mach. Learn. Res., 2024, 2024.
https: ://ww mindspore.com

---

[18] Jonathan Light, Min Cai, Sheng Shen, and Ziniu Hu. From text to tactic: Evaluating Ilms playing the game of avalon. CoRR, abs/2310.05036, 2023.
[19] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye. Yiming Yang. Shashank Gupta. Bodhisattwa Prasad Majumder. Katherine Hermann, Sean Welleck, Amir Yazzanbakhsh,, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[20] Gabriel Mukobi, Hannah Erlebach, Niklas Lauffer, Lewis Hammond, Alan Chan, and Jesse Clifton. Welfare diplomacy: Benchmarking language model cooperation. CoRR, abs/23 10.08901 2023.
[21] Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almcida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA. November 28- December 9, 2022, 2022.
[22] Boci Peng. Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang. Graph retrieval -augmeeee generation: A survey, 2024.
[23] Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. CoRR, abs/2307.072 079944, 2023.
[24] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong. Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real- woal-world apis. CoRR, abs/2307.16789, 2023.
[25] Timo Schick. Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[26] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving Al tasks with chatgpt and its friends in huggingface. CoRR, abs/2303. 17580, 2023.
[27] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[28] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Ahu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonse Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray. Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang. Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew K. Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakas, and et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. CoRR, abs/2206.044 044615. 2022.
[29] Xiangru Tang, Anni Zou, Zhuosheng Zhang, Yilun Zhao, Xingyao Zhang, Arman Cohan, and Mark Gerstein. Medagents: Large language models as collaborators for zero-shot medical reasoning. CORR, 055/2311.105533 2023.
[30] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NewrlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
[31] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang. Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen LLM applications via multi agent conversation framework. CoRR, abs/2308. 08 155, 2023.
12

---

[32] Kai Xiong, Xiao Ding, Yixin Cao. Ting Liu, and Bing Qin. Examining inter-consistency of large language models collaboration: An in-depth analysis via debate. Findings the Association for Computational Linguisticx: EMNLP 2023, Singapore, December 6-10, 2023. pages 7572-7590, 2023.
[33] Lin Xu, Zhiyuan Hu, Daquan Zhou, Hongyu Ren, Zhen Dong, Kurt Keutzer, See- See-Kiong Ng. and Jiashi Feng. Magic: Investigation of large language model powered multi-agent in cognition, adaptability, rationality and collaboration. CoRR, 0bs/2311.0885e 2023.
[34] Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Exploring large language models for communication games: An empirical study on werewolf. CoRR, abs/2309,0460 04665, 2023.
[35] Haoran Yang, Yumeng Zhang, Jiaqi Xu, Hongyuan Lu, Pheng-An Heng, and Wai Lam. Unveiling the gencralization power of fine-tuned large language models. CoRR, abs/2403.09 0966 2024.
[36] Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpoiqa: A dataset for diverse. explainable multi hop question answering. In Proceedings the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pages 2369-2380, 2018
[37] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023.
[38] Weiran Yao, Shelby Heinecke, Juan Carlos Niebles, Zhiwei Liu, Yihao Feng. Le Xue, Rithesh Murthy, Zeyuan Chen, Jianguo Zhang, Devansh Arpit, Ran Xu, Phil Mui, Huan Wang. Caiming Xiong, and Silvio Savarese. Retroformer: Retrospective large language agents with policy gradient optimization. CoRR, abs/2308.02151, 2023.
[39] Yifan Zeng, Yiran Wu, Xiao Zhang, Huazheng Wang, and Qingyun Wu. Autodefense: Multi-agent LLM defense against jailbreak attacks. CoRR, abs/2403.04 04778333,,,3, 2024.
[40] Wenqi Zhang, Yongliang Shen, Linjuan Wu, Qiuying Peng, Jun Wang, Yueting Zhuang, and Weiming Lu. Self-contrast: Better reflection through inconsistent solving perspectives. CoRR, ahs/2401 .00009, 2024.
[41] Qinlin Zhao, Jindong Wang. Yixuan Zhang, Yiqiao Jin, Kaijie Zhu, Hao Chen, and Xing Xic. Competcai: Understanding the competition behaviors in large language model-based agents. CoRR, (tos/2310.115//
2023.
[42] Xuhui Zhou, Hao Zhu, Lenna Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis quis--hilippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, and Maarten Sap. SOTOPIA: interactive evaluation for social intelligence in language agents. CoRR, abs/2310. 1667, 2023.
13

---

A Collaboration Settings
In HotPotQA scenario, we adopt a teacher-student collaboration paradigm, while on GSM8K and Checkmate in One Move datasets, we employ collaborative debate setting follow (8]. We illustrate the process of the collaborations in Figure I0 and detailed introductions are as follows.
HotPotQA In the HotPotQA scenario, both teacher and student agents adopt the ReAct method for action selection. Among them, student agents can call retrieval tools to search for relevant text segments provided by the HotPotQA dataset. The retrieval tool constructed using SimCSE (model: musup-simcse-robertt-bbee)
The action space of student agent includes: (1) Search(entityl, which invokes local searcher provide relevant information. (2) Finishlanswerr,, which returns the answer and finishes the task.
The action space of teacher agent includes: (D)Continue). which means the student made good decision and should continue the process. (Rethhik,,,, which means the student's previous step is wrong and should consider another step.
GSM8K the GSM8K scenario, we set three debater agents to engage in two rounds of debate, and the final answer of the system is determined by voting the last round answers. During the debate, intelligent agents utilize CoT to analyze and answer questions, or update the answer based on solutions of other agents.
Checkmate in One Move In the Checkmate One Move scenario, we adopt the same col dehate approach construct multi-agent system GSMSK. The only difference is that employ three intelligent agents for three rounds of debate.
Environment
Environment
Debater (ide1)
Message Pool
swar
Message
Student Agent
Teacher Agent
Deboter (id:0)
Debater (dd:2)
Beacher-Studeree Paradigm
Cooperative Debate Paradigm
Figure 10: Multi-agent collaboration framework.
B Details of Training Data
We first use the original LongChat to construct counterfactual training data and select positive examples for supervised fine-tuning of the reflector model. The detailed data information generated in this stage is shown in Table
Table Statistics of training data generated by original LongChat. We show the total data volume on the left, the number of positive examples the middle, and the number of negative examples on the right side.
Episode Difference Reward
Counterfactual Reward
Dotaset
aggrnt.
agent
apent.2
ageni.
apont
apunt
HotPorQA
9714/196/13300
3714/11371190
4857/172/644
-P---
1049/2840
Checkmate.
17466/90610
5822/302/0
5822/30200
5822/30200
17466/682/161
5822/23/165
5822/223/377
5822/2281559
Afterwards, we employ the reflector model fine-tuned using counterfactual SFT to generate training data of the PPO stage. We use the collected counterfactual reward data to train reward model using
14

---

MSE loss and score the reflections predicted during the PPO training process. The data collected in this stage is shown in Table
Table 2: Statistics of training data generated by LongChat fine-tuned with SFT. We show the total data volume on the left, the number of positive examples on the middle, and the number of negative examples on the right side.
Episode Difference
Counterfactual
Datese
agent U
agont
agent 2
agont 0
agoni
agann
HotPotQA
8822/2218/1410
4411/I 1OAYROS
14
--4-
88:2// 1679/985
441 11855/11777
4411/824958
-1---
GSMKK
32044/8900
1068/26330
1068/26300
106R/26300
3204//84/409
1068/129148
1068/128/132
1069/127/134
Chockmate:
17892/59180
5964/19970
5964/19700
5964/19700
17892/412/444
5964/1 38/147
5963/1 38/169
5966/1 96/128
C Experimental Results on ALFWorld
ALFWorld a classical dataset designed for training and evaluating Al agents in interactive envi- ronments. To further improve our study, we additionally conduct experiments on ALFWorld. We follow the same multi-agent collaboration setting as (31] and test the model with 134 instances. The experiment results are presented in Figure 11 From the results, achieves better reflection performance than the original LongChat and GPT-3.5. Besides, compared to the initial success rate, COPPER brings an improvement of 37.2% with 4 times of reflections.
can observe that COPPER
ALFWorld
CST
Refiexlon (CPT 3.5)
Bellasion (Longthee)
ratel())
COPPER (win PPO)
COPPER
75
65
2.5
4.0
Number
Figure 11: Experimental results on ALFWorld.
D Comparison between SFT and RLHF under No-profile Setting
We also provide experimental results comparison of different fine-tuning methods under no profile setting. The results are shown Figure [2 We can find that even without agent profiles, the PPO method still provides enhancement to the fine-tuning of the shared reflector, further validating the effectiveness of our method.
HotPatQA
GSMBK
Checkmate
Move
corfeit
30
4.4
44
3.0
055 1.0
2.0 30 0.
0.5
1.5 2.0
3.0
4.5
00
1.5
30
40
Trail Number
Tnaal Number
Nuneer
(a) HotpotQA.
(b) GSM8K.
(c) Checkmate One Move.
Figure 12: Comparison between SFT and RLHF under no-profile setting.
E Experiments under Partial Information Settings
To improve our paper, we conduct more experiments to investigate the settings with partial information. In specific, we introduce two models. For the first one, remove the information of the other agents.
15

---

For the second one, we use proxy model to predict the information of the other agents. We present the experiment results in Figure T3 and Figure [4 respectively. We can find from the results that our COPPER still achieves better reflection performance under these partial settings. Checkmate in One Move
HotPotQA
CSMBK
46-
Aufludion JongChutt
Aufostin J0P12.51
Jefecin (iPEAS) Apticiin fangchat
keBlidann (anggChv
Aediodan (G
37 44
CO7PE5
42
%5
13
92
3.0
0.5
1.0
13
2.0 Number
23
3.
33 4.0
0.0
0.5
3.0
1.5
2.0
2.5
3.0
35
4.
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Number
HotpotQA.
(b)
GSM8K.
(c) Checkmate in One Move.
Figure 13: Main results without other agents' trajectories.
HotPotOA
GSMBK
Checkmate in One Move
Rokiz
Cel
50
Aufliston J0PP
Aptiecion (SPtasi liorechult
38-
kefledan (GST-35) 5:
Adttcin ZoeeChell
-B- Pelicimn (loegOha))
4.-
60
54
23
23
20
0.5
0.0
40
Trial Number
Tial Nambee
Number
(a) HotpotQA.
(b) GSM8K.
(c) Checkmate in One Move.
Figure 14: Main results when predicting other agents' information with a proxy model.
F Case Study
We conduct case studies on three datasets to visually verify the effectiveness of our method, as shown in Figure 15 16 and
Prior Failed Trial
Reflection of GPT 3.5
Duection in Person ina 1935 fime souring in artwes bor shatyas??
tralinog lect effuc en lindng the onttertte rimm YIe Farm onf to diseriin the jetess's yoar oe speciiiilly entamnaiin abeat lin iitakie High lord plarced hocss on castel thr Perzai lind the actess's ' bett yearled Raliain firm seaeriing for ane event wnwmmnttmotioon atsut tim.
The Lillre
Thoaght rese the aue same the octres who sarre in the iim ihss aanant 2005 bethoe can det vine him year
stadont sholk llvve ineead
Searrits
arness)
Technicolo makial corn ody in aturiig Baity Thoughe The stadentts search for the Pemon 2935 fimm actreee da net pres the retermation rocded shoels focas on fixcin act esd sarrel 2935 Fessoll? They Ackors|Batiin
Search Beaut The furmer Takes
ohe 1935 afe
buted to the ind rrsfection soarch crreney. Thes froord in Findiig fommaatioo shout (in the et deunt apprect, the teacher sha rnvthvobee the importance of friiiing ng the releveri actrr or actrussss bytthorlly fim the teacher abould enccurage the studant, is than criti eity and in previous malaave..
quiston comecity
Thouge seart the arowes theth in the Partant not provite relecant Artent Swvetpemees in the firi Pareegy 995)
desi
Seach Resin Conumation danl s 1835 iim exiiing Rose Hoborn and thats lin Steim sovard the Thres Staages. The move was di rudtdd by Lutter
bead repeatig same
Troughh
costel (T1 Partant detervike bith your a'the actisso The stot search for the caetel Pamani to tine aethern's beth yoar.
Reflection of COPPER
Thzughh need to focue anding cxcrattee Par dememin the your Ariss Severi) Benson cardf
(clb)
fecusing the connec elbrrraion The stado mn have seurched the cast rimi Parszette the birn and ceserrine ine her sith year below ar arowening the quesitin.
Saach kesuln in Perceels as 1905 iin stuning dllgge a aagers tradde of 2000 it's
ibeut Muss carel Certes, beautiid move staras miseuus her fume. that are goss
sill fontrnac ea Riccing the comettiitooattnnn The esctent stoule fawe sparched cas bey ensuld
Thogle The stakorseartee for infammation about the din Ua Pessa if and found to Goyyeer caned
Whi resoot find guded vaudentit looss on Inding the correct into melton suppoisiig a sppclli for castal the fim Panarrt ta find actessa beit.
Parscn (f The and step ned anet Sayeere bith yearte
Actori M Contnnee
sing
ow catet seuching burgehs You must anooee the quetion the sto..
Thougte: You mattas Finsh action snower quaston contact and volion Feihiil 995]
Studenn Agent
Teache" Agant
Environment Proxy
Figure 15: Case study on HotPotQA dataset.
For example, Figure 15 shows a prior failed trajectory in HotPotQA scenario, as well as the reflections generated by COPPER and GPT-3.5. During interactions with the environment, the student agent
16

---

Prior Failed Trial
Questio: jes colletss stamp cares. Fowry week, he gets 6 cards. Butevery two weeks, 2 cands to his friends. II eedd stirted with cards, Hiter how many wesks he have a tbtal of cands?
Round 1
Sure, lets break kdowntttee jd statts with 20 cards. i-- Every week, he gets 6 cards. So 1 VJ weeks, te will have 0j - 6i 10 cards-in- But every two weeks. he gives 2 cards away. So fer every bid weeks, he effectvlly onty get extra cands (s cards gaited minus 2 cants given awaylin ii- Therefire, after 01 lime (422 ni the weet periods, he will have - 20s2mmcc0h2 y 2x oi cards intnkoos we want find aut how many weeks the have total aicestmrnl Bx 20 (DrtsDDvide beth sides Bunini However, the premle asks the weewer temms ofweols, can round up the nearest whole number a rince not poss to have weet Wibaeeggy (X)
cands-ttint 29 fh 2x map VartyComiiee like temms (meey) iix +20 40 Wdrinsubtree 70 from
StninSo 2.5 waeks, or weok pertode. ed have total cartis.
Lets break le'ss card collection precess very week.] es gaiee cards cards, so up the equationtt200 4x 420nix =2Unx =iirihhheerelor alter weeks
pain of cards intiter x Jeeke. ed have 20
cards winkiee e want find when he jes
cards, vibbovedb))
Lets break down the problem into smaller parts, vntai rsst we know that] es starts with 20 cards, very week hhe pess 6 cards, So after 9i15 weeks, he wil have $20 ows cends. inint very two wes 1s, he gives 2 cards to his fhiends. So atter 9n.5 weeks, he will have given away stiftorr (x))))) Inttorr ludder 25 cards. Uninin these bwo equattons together, we can creaile equustion for the total number of di ed has after $wi wreels
Round 2
effectivly gains cards because he lotal cards carcss 12/116 ubract 20 from destevvtt 4w 20 Divdd 41mirt0 10rinnS after wooks.
Absokitety Lets bidil the problem enttti e know evate week gains cards, butevery away
weekseentt Wheiifittil
by
loti cards (bexed5)1)
Based the explene provided the other cards given as Wileen 20 + 400ehotttrc 2wpp Jrignt 20 + Kjninkow, ow, we setths equal 40 and solve for worknil 20 +2 -40 [[nnl] 2 =0 (2172 10
that ed ettectively cands, 'nsst he comectequaton represent the totwl number canen je has after weeks writte
carts every week pertod cands gained minus
Lets reitt problen the information provised
agents, leyt starts with cards and gaiis cards every However every have W220 4wii cardsinind
esuation effectively 40 invi yd
cards he dives total 40 cands-itunii23 401gmiini)4 20/2murtii Sttriinkeerr weeks, have lota
So, after
Reflection of COPPER
Reflection of GPT-3.5
provious trial I made mistake thtt od glves cards friends Ishouls weeks, which weeks. should have used his infommation to set equation and solve the prob em more accurately
In fatur problem-colving ensure clarity defiring variables and steps, while double-checiing checking calcultioons for sccurscy. Aim a comprehensive approach to avoid amolguity and ensure a precise solutiss.
he loses cards
Focus on cons lorrin the net gain and loes cards each intersal rather than sale relying or equations. Break down the prodlem m into simpler steps and carefully track the changes in card count to antive at the conect lio efficientty.
In the previous tria, i made mistace in solving the problem by not consadering the cards g vive to friends every two wzees. Ishoud have considered the cards given away in the equation and made sure to include in my calulaters.
To avoid ambiuuiy, ensure consistant appliiation of conditiong ns the proslem and thorough exe minelloo e each step. Clarty assumptins and volidate results meticulausly to reach a grecise solution
the previous by considering cards jes gives away every weeks, have equation. avo this account to carefully ronaider constaints wher soling problems.
mistate in sclving sroblem
I will up and considerant any
Agent 0
Agent_
Agent
Environment Proxy
Figure 16: Case study on GSM8K dataset.
needs constantly modify keywords of the retriever API obtain information related to the problem. However, in the previous trajectory, the student agent underwent multiple modifications to complete the first step of inference, which results in reaching the pre-defined maximum search step limit. Therefore, the agent is forced to answer the question directly based on search result of the first reasoning step, leading to an incorrect answer. By analyzing the reflections of COPPER and GPT-3.5, we can observe that: (1) Compared to GPT-3.5. COPPER has a deeper understanding of agent roles and thus generate corresponding reflections for each agent. For example, in the reflection of the teacher agent, the reflection generated by GPT-3.5 contains lots of descriptions of problems related to the student agent, while COPPER focuses more on reflecting on the role of the teacher. (2) Compared to GPT-3.5. COPPER can conduct more effective and comprehensive root cause analysis to identify the cause of failure. In the reflection of the student agent, both GPT-3.5 and COPPER mention that "the case of In Person" should be used as the search keyword, but COPPER additionally suggests that the relevant information of the actress should be retrieved before answering the question.
As for the case studies on GSM8K (Figure 16) and Checkmate in One Move (Figure 17) datasets, we can find that in the context of the multi-agent debate, compared to GPT-3.5. COPPER can accurately identify key issues where mistakes occurred in previous trials and generate more specific reflections to guide agents improving their responses. For example, the case of GSM8K, COPPER identified that previous errors mainly stemmed from an incorrect analysis of equal relations and emphasized
17

---

Prior Failed Trial
Qussioom curre st sequence moves in chess game: Bi2 03 NO Nc5 B55 Bacde 16 Ne5 8. oho-ge a Codh 31. thath agg 13.ct Radk should next? Glve single move suggesiion ohe 29, and
es 23, X.gg Rgb. What move
ge+
urrent
Round 1
move to contiuue your advantagg &s Re.. This move In the current chess positio, conside fing the board stall and petertaal thnals, the mave threatens the black bishop on g3 while also preas ring to double rooks on the e-ile, which can putsigr sutsigriican pressure on lock's pos
Looking at the cumeent boa ri stale, itseems you have strong postion with your pieces well -olv loped and your opponennts king exposed trint onsidering this, move casttaize on the exposed king and the reakened squa couid be/Win88. Qa3evivihis move attacks the black ling and threatenss immed checkmate with Qh3a it abs keops up the pressure on your ooponents position.
Antr carefull ana jyling the poskion, the most for you would be/Wi RxoBeisTThis move threaters B Black's rook on and puts adoitional pressure on her position.
Round 2
Given the curre ntaard state and cors idering the suggestons from other agants, as well as sour strakggs reflecting en potentall weaknesses, the best move is continue your advantage ie- GgiimmeThi move aftic the king on and fhriaters immediate checiemate with Qh3 w. also maintains pressure on your opporents postiion cap ailliiiing the expe king and wesiknred dark squanes.
Given the current board stase and considering the sungestion provided believe the move 28, Re5 would be stre ng cont nuation of your position. int move sttacks the block bishop 95, puttig pressure and potentally forting to retrest Additonall,, itpresanes couble rooks on the e file which can further intensily the pressure on Blackts pas ton trins 28, Reb seeme like pro chokcc keeping the pres sure and mainaliinng control of the game.
After revlow ng the sugaestions from other agents and cons ids fing the current sait move that maintains pressue on Blackcs position and castalizes the weakinesses their sotup woule beimin2 Q The move attacts the black king on hts and threatons immeclate checkmat te wit Qh5a, aloo keeps up the pressure on your opporents pos Bon wi le expliling the vuinerab liles their delennse.
Round 3
Gleen the current beard stare and cons idering the sugpestore srovied, selil believe 29 0gaif i the most It intains pressure on Bladk's position capitaizie on the woaknesses in ther setup and threalars inneddille checkmate with Q haa This mave keeps up the i ressura on your oppanents position whlll explo kng the vulnerat Bio their defonse.
Core dering the suggestions from ogher agents and curment beard state. believe that move 29, Op3mls the mostporent
immediata checkrals Qh3 soponentss position exgloiting their vunnerabilites. executing Ogga weakened dirccty the maintainin advantage cotentaaly lead favorable
seems like dacisee
wel strategy reflecting tental weainesses, potentialyy it e-fle, when can furthrr intensily pressure Black's position. int8 iies maintsirs pressure and control the game, while also exp olting the weaknesses Black's setuo.
Given the current board and consider ing supgestions from 29, Rei addioonlly
prudent choice. Wintis sttacle the bishop gs, puttigg double
Reflection COPPER
Reflection of GPT-3.5
In the previous trib. it led to consider the potertal weaknesses my own pos binn will now carefully examine my own posiiion and consider the potantal wesk esses befbre making move.
previous trial overooked stategic importance ntaining
control. ensu rng a more praative
priottiee
aspicts sppoach.
trial, billed consider potantial position. should structure the podential
waakresses coutous about weakened for counerplay the queenside in future, will tike more time analyne the patential ses my own position zelore making move.
futrre debotes, ensure to consider moves that maintain intiaaiive
sngg apponentts integratin ig varied perspectives enitance logit depth.
In future debatos, and the consequences, foousing
critically evallzte candidate
previuus trial. biledd consider ider the cotantal weatne-sues my spponent's salely mainraiing
focused structue Instaed advartage the next murd,
stategic advanttgges while considering factica loprortunties.
explei
alowed opponent will carefully examine bon corsidera potertaal weaknesses before making more..
Agant
Agent_ 1
Agent
Envinnment P Proxy
Figure 17: Case study on Checkmate in One Move dataset.
the need to pay attention to "every two weeks". In the case of Checkmate in One Move, COPPER pointed out the importance of focusing on the weakness in the own position.
G Ethical Consideration
In this paper, we propose using the self-reflection mechanism to enhance the collaborative ability of multi-agent systems, and explore the effectiveness of our method in three scenarios: question answering, mathematics, and chess. Experimental results show that our method can effectively enhance the ability of multi-agent systems to solve complex tasks, which helps to enhance the application of multi-agent systems in real-world scenarios, such as disaster response, intelligent transportation systems, and other scenarios. However, in the task of guestion-answering the agent can call API to utilize retrieval tools, which may pose potential risks, such as tampering with the information in Wikipedia during the searching process. However, during the implementation, we limit the text content that intelligent agents can access, thus avoiding this issuc.
18

---

H Prompts
H.1
Single-Agent Prompts
H.1.1 HotPotQA
For single-agent setting on HotPotQA, we adopt the same few-shot examples as
The actor prompt for single-agent ReAct.
Solve a question answering task with interleaving Thought, Action, Observation steps. Thought can reason about the current situation, and Action can be three types:
(1) Search[entity), which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, will return some similar entities to search. (2) Finish(answer), which returns the answer and finishes the task. You may take as many steps as necessary.
Here are some examples: fexamples]
(END OF EXAMPLES)
Question: (question)
(scratchpad)
The actor prompt for single-agent Reflexion.
Solve a question answering task with interleaving Thought, Action, Observation steps. Thought can reason about the current situation, and Action can be two types:
(1) Search[entityl, which searches the exact entity on Wikipedia and returns the first paragraph if it exists. If not, it will return some similar entities to search.
(2) Finish(answer], which returns the answer and finishes the task. You may take as many steps as necessary.
Here are some examples: (examples)
(END OF EXAMPLES)
[reflections]
Question: (question) ) (scratchpad)
19

---

The reflector prompt for single-agent Reflexion.
You advanced reasoning agent that improve based self refection. You will be given a previous reasoning trial in which you were given access to an Docstore API environment and question answer. You either because you guessed the wrong answer with Finish[<answers <answer>], or you used up your number of reasoning steps. In few sentences, Diagnose possible reason for failure and devise new, concise, high level plan that aims to mitigate the same failure. Use complete sentences.
unsuccessful answering the question
Here are some examples: (examples)
Previous trial: Question: [question) (scratchpad)
Reflection:
H.1.2 GSM8K
We adopt a zero-shot setting on GSM8K and below are the prompts.
The actor prompt for single-agent COT.
Can you solve the following math problem? {question]
Explain your reasoning. Your final answer should be single numerical number, in the form Wboxed (answer)) J, at the end your response.
The actor prompt for single-agent Reflexion.
Reflections of previous trials: [reflections)
Can you solve the following math problem? {question]
Explain your reasoning. Your final answer should be single numerical number, in the form Wboxed (answer) J. at the end your response.
The reflector prompt for single-agent Reflexion.
You are an advanced reasoning agent that can improve based on self refection. In previous trial, your task is to solve math problem. However, you were unsuccessful in the previous trial. Now given previous interactions, you need to carefully examine the problem-sol solving ideas and calculation results, and form reflection to avoid these problems in the next round. The reflection should be less than 50 words.
Previous Question:
{question]
Previous interactions:
{context]
Reflection:
20

---

H.1.3 Checkmate in One Move
We also adopt zero-shot setting Checkmate in One Move and below the prompts.
The actor prompt for single-agent CoT.
Here current sequence of moves chess game: {question]
What is the best chess move should execute next?
Give single move suggestion of the form (answer_stee sterrrsteppppttt xtep).xxXX> and make sure the chess move is valid in the current board state.
The actor prompt for single-agent Reflexion.
Reflections of previous trials: [reflections]
Here is the current sequence of moves in chess game: {question]
What is the best chess move should execute next?
Give single move suggestion of the form (esseer_step.<XXte and make sure the chess move is valid in the current board state.
The reflector prompt for single-agent Reflexion.
You are an advanced reasoning agent that can improve based on self refection. In previous trial, your task is to give the best next chess move. However, you were unsuccessful in the previous trial. Now given previous interactions, you need carefully examine the problem-solving ideas and calculation results, and form reflection to avoid these problems the next round. The reflection should be less than 50 words.
Previous Game:
{question]
Previous interactions:
{context)
Reflection:
21

---

H.2
Multi-Agent Prompts
H.2.1
HotPotQA
We follow
38
to design the prompts this scenario.
The actor prompt of the student agent when deploying ReAct.
You are student agent. Your task to answer the question under the guidance the teacher agent. You should make reasonable plan at first. Solve the task below with interleaving Thought, Action, Context steps. Context is the summary of historical interactions. Thought can reason about the current situation.
Your action can be two types:
(1) Search[entity], which invokes local searcher to provide you with relevant information. (2) Finish[ answer], which returns the answer and finishes the task.
Please note: You only need to complete the thought step and output Search [Entity] in the action step, and will return the relevant content in "Observation" for you. If you find an answer, submit it via "Finish [answer]". Identical searches will only return similar content. If the returned content has no relevant information, please actively try searching for different keywords. When submitting your answer, please try to submit the full answer if you think ambiguous, e.g. "movie director" is better than "director"! The answer to the question should be as accurate and concise as possible, i.e. try to answer the question with phrases instead of long sentences. Please answer yes-no question with either "yes" "no".
Examples:
Context: Searched for Arthur's Magazine and found it was started in 1844. Teacher agreed with previous action and suggested finding the founding date of First for Women. Searched for First for Women. Question: Which magazine was started first Arthur's Magazine First Women?
Observation: Search Result: Search[First for Women] First for Women is woman's magazine published by Bauer Media Group in the USA.[1] The magazine was started in 1989. Teacher's Suggestion: [Continue] Now you should answer the question. Thought: First for Women was started in 1989. 1844 (Arthur's Magazine) < 1989 (First for Women), Arthur's Magazine started first.
Action: Finish[ Arthur's Magazine]
END OF EXAMPLES
Context: (context)
Question: [question)
Observation: (observation)
22

---

The actor prompt of the teacher agent when deploying ReAct.
You teacher agent, and your task is guide the student agent to answer the questions. You should analyze whether the student's step is logically helpful, provide an analysis, and give your final action. You can also give advice on the student's future step. Solve the task below with interleaving Thought, Action, Context steps. Context is the summary of historical interactions. Thought can reason about the current situation.
Your action can be two types:
(1) [Continue], which means the student made good decision and should continue the process.
(2) [Rethink], which means the student's previous step is wrong and should consider another step.
Please note: You can analyze the correctness of student agent's action, summarize the useful information student found, or provide suggestions for subsequent steps. Your action step only have two types: [Continue] [Rethink]. You can only take one of the above actions. Most the time, please be an encouraging teacher, that is, unless the student is completely wrong, [Continue] action more often.
Examples:
Context: Searched for Arthur's Magazine and found it was started in 1844, Teacher's advice
continue.
Question: Which magazine was started first Arthur's Magazine or First for Women?
Observation: Previous advice: [Continue] Finding the start time of Arthur's Magazine is helpful. Next the student should search First for Women and find its founding date. Student's action: Search[First for Women] First for Women is woman's magazine published by Bauer Media Group in the USA.[1] The magazine was started in 1989.
Thought: The founding date of First for Women helpful. The student should take the Finish step next.
Action: [Continue]
END OF EXAMPLES
Context: (context)
Question: {question]
Observation: (observation)
23

---

The prompt of the student agent when deploying Reflexion.
You student agent. Your task is answer the question under the guidance of the teacher agent. You should make a reasonable plan at first. Solve the task below with interleaving Thought, Action, Context steps. Context is the summary of historical interactions. Thought can reason about the current situation.
Your action can be types:
(1) Search(entity], which invokes local searcher to provide you with relevant information. (2) Finish[answert which returns the answer and finishes the task.
Please note: You only need complete the thought step and output Search [Entity] in the action step, and we will return the relevant content in "Observation" for you. If you find an answer, submit it via "Finish [answer]". Identical searches will only return similar content, the returned content has relevant information, please actively try searching for different keywords. When submitting your answer, please try to submit the full answer if you think is ambiguous, e.g. "movie director" is better than "director"! The answer to the question should be as accurate and concise as possible, i.e. try to answer the question with phrases instead of long sentences. Please answer yes-no no question with either "yes" "no".
Reflections of previous trials: [reflections]
Examples:
Searched for Arthur's Magazine and found it started in 1844. Teacher agreed with previous action and suggested finding the founding date of First for Women. Searched for First for Women. Question: Which magazine was started first Arthur's Magazine First for Women?
Context:
Observation: Search Result: Search[First for Women] First for Women is woman's magazine published by Bauer Media Group in the USA.[1] The magazine was started in 1989. Teacher's Suggestion: [Continue] Now you should answer the question. Thought: First for Women was started in 1989. 1844 (Arthur's Magazine) < 1989 (First for Women), so Arthur's Magazine was started first.
Action: Frthur' Magazine]
END OF EXAMPLES
Context: (context)
Question: (question]
Observation: [observation]
24

---

The actor prompts of the teacher agent when deploying Reflexion.
You teacher agent, and your task is guide the student agent to answer the questions. You should analyze whether the student's step is logically helpful, provide an analysis, and give your final action. You can also give advice on the student's future step. Solve the task below with interleaving Thought, Action, Context steps. Context is the summary of historical interactions. Thought can reason about the current situation.
Your action can be two types:
(1) [Continue], which means the student made good decision and should continue the process.
(2) [Rethink], which means the student's previous step is wrong and should consider another step.
Please note: You can analyze the correctness of student agent's action, summarize the useful information student found, or provide suggestions for subsequent steps. Your action step only have two types: [Continue] [Rethink]. You can only take one of the above actions. Most the time, please be an encouraging teacher, that is, unless the student is completely wrong, [Continue] action more often.
Reflections of previous trials: [reflections]
Examples:
Context: Searched for Arthur's Magazine and found it was started in 1844, Teacher's advice is continue.
Question: Which magazine was started first Arthur's Magazine First for Women? Observation: Previous advice: [Continue] Finding the start time of Arthur's Magazine is helpful. Next the student should search First for Women and find its founding date. Student's action: Search[First for Women] First for Women is woman's magazine published by Bauer Media Group the USA.[1] The magazine was started in 1989.
Thought: The founding date of First for Women is helpful. The student should take the Finish step next.
Action: [Continue]
END OF EXAMPLES
Context: (context)
Question: {question]
Observation: [observation]
25

---

The reflector prompt of the student agent when deploying Reflexion.
You advanced reasoning agent that improve based self refection. In previous trial, you are a student agent, and your task is to answer the question under the guidance of the teacher agent. The teacher agent provides guidance on your step and explains the reasons. You were unsuccessful in answering the question either because you guessed the wrong answer with Finish(answer), you used up the number of reasoning steps. Now given previous interactions from student's 's and teacher's 's perspective, you should diagnose possible reason for failure and devise a new, concise, high level plan that aims mitigate the same failure in a few sentences.
Please note: If you believe that the previous searching and collaboration process as well as the answer were correct, please try answering the question in different way, e.g. try to provide more concise answers, or using the same words as the question itself.
Previous trial:
Question: (question)
Interaction:
Student: (student c
Teacher: [teacher_contexl
Reflection:
The reflector prompt of the teacher agent when deploying Reflexion.
You are an advanced reasoning agent that can improve based on self refection. In previous trial, you are a teacher agent, and your task is to guide the student agent answer the questions. The student is unsuccessful in answering the question either because he guessed the wrong answer, or he used up the set number of reasoning steps. Now given previous interactions from student's and teacher's perspective, you should diagnose a possible reason for failure and devise new, concise, high level guiding plan.
Please note: If you believe that the previous searching and collaboration process as well as the answer were correct, please try answering the question a different way, c.g. try to provide more concise answers, using the same words the question itself.
Previous trial:
Question: (question)
Interaction:
Student: (student_contexy
Teacher: [teacher_context]
Reflection:
26

---

The prompt of the context model of the student agent.
You student agent. Your task is answer the question under the guidance of the teacher agent. Now you are provided with previous summary, as well as new messages that were not included the original summary. Your summary should encapsulate the main points of the new messages and integrate them into the existing summary to create a comprehensive recap. Highlight the key issues discussed, decisions made, and any actions assigned. Record the helpful factual information given by the search engine.
Please ensure that the final summary does not exceed [char limit) characters.
Examples:
Question: Which magazine was started first Arthur's Magazine or First for Women? Previous Summary: Searched for Arthur's 's Magazine.
New Observation: Search Result: Search[Arthur Magazine] Arthur's Magazine (1844-1840 was an American literary periodical published Philadelphia the 19th century. Teacher's Suggestion: [Continue] Finding the start time of Arthur's 's Magazine is helpful. Next the student should search First for Women and find its founding date.
started in 1844. I need to search First for Women next. New Action: Search[First for Women]
New Thought: Arthur's Magazine
Summary: Searched for Arthur's Magazine and found it was started in 1844, Teacher agreed with previous action and suggested finding the founding date of First for Women. Searched for First for Women.
END OF EXAMPLES
Question: (question)
Previous Summary: [context]
New Observation: {observation
New Thought: (thought)
New Action: faction]
Summary:
27

---

The prompt of the context model of the teacher agent.
You teacher agent, and your task is guide the student agent to answer the questions. Now you are provided with a previous summary, as well as new messages that were not included in the original summary. Your summary should encapsulate the main points of the new messages and integrate them into the existing summary to create a comprehensive recap. Highlight the key issues discussed, decisions made, and any actions assigned. Record the helpful factual information given by the search engine.
Please ensure that the final summary does not exceed [char limit) characters.
Examples:
Question: Which magazine was started first Arthur's Magazine or First for Women? Previous Summary: The student searched for Arthur's Magazine and found it was started in 1844. The action is helpful and the next step is finding the founding date of First for Women. New Observation: Previous advice: [Continue] Finding the start time of Arthur's Magazine is helpful. Next the student should search First for Women and find its founding date. Student's action: Search[Firg First for Women] First for Women is woman's magazine published by Bauer Media Group in the USA.[1] The magazine was started in 1989.
New Thought: The founding date First for Women helpful. The student should take the Finish step next.
New Action: [Continue]
Summary: The student searched Arthur's Magazine and found it started in 1844. The student then searched First for women and found was started 1989. Previous actions are helpful. The student should give the answer in the step. END OF EXAMPLES
Question: (question)
Previous Summary: [context]
New Observation: (observation)
New Thought: (thought)
New Action: (action)
Summary:
H.2.2 GSMSK
We follow g to design the prompts of debaters in this scenario.
The actor prompt for the agent to generate initial answers when deploying Co'.
Can you solve the following math problem? {question]
Explain your reasoning. Your final answer should be single numerical number, in the form Wbooxdd( (answer)), J, at the end your response.
The actor prompt for the agent to generate updated answers when deploying CoT.
These are the solutions to the problem from other agents: {solutions]
Using the solutions from other agents as additional information, can you provide your answer to the math problem? The original math problem is (question). Your final answer should be a single numerical number, in the form Wboxed (answer)) ), at the end of your response.
28

---

The actor prompt for the agent generate initial answers when deploying Relfexion.
Reflections of previous trials: [reflections]
Can you solve the following math problem? {question]
Explain your reasoning. Your final answer should be single numerical number, in the form Wboxed (answer) ), at the end of your response.
The actor prompt for the agent to generate updated answers when deploying Reflexion.
Reflections of previous trials: [reflections]
These the solutions to the problem from other agents: {solutions]
Using the solutions from other agents as additional information, can you provide your answer to the math problem? The original math problem is (question). Your final answer should be single numerical number, in the form Wboxed [answer) ), the end of your response.
The reflector prompt of each debater agent.
You advanced reasoning agent that improve based self refection. In previous trial, you are [role) and you are supposed to solve math problem through debating with other agents. However, you were unsuccessful in the previous trial. Now given previous interactions, you need to carefully examine the problem-solving ideas and calculation results, and form reflection to avoid these problems in the next round. The reflection should be less than 50 words.
Previous Question: {question)
Previous interactions: {context]
Reflection:
The prompt of the context model of each agent.
Please summarize the following process in concise language, including the opinions of all agents.
Please note: Please summarize the viewpoints of each agent and retain the role of the agent the summary, such as agent_0, agent_1, agent_2.
[scratchpad)
Summary:
H.2.3
Checkmate in One Move
We follow 8 to design the prompts of debaters in this scenario.
29

---

The prompt for the agent generate initial answers when deploying CoT.
Here is the current sequence of moves in chess game: {question]
What is the best chess move I should execute next?
Give a single move suggestion of the form (answer_step).<xXx and make sure the chess move valid in the current board state.
The actor prompt for the agent to generate updated answers when deploying CoT.
Here are other chess move suggestions from other agents: {solutions}
Using the chess suggestions from other agents additional advice, you give your updated thoughts on the best next chess move should play given chess sequence?
The current sequence of moves in chess game is: [self.question)
Give a single move suggestion of the form [answer_stey] sttp_sttppp. <XXX> and make sure the chess move is valid in the current board state.
The actor prompt for the agent to generate initial answers when deploying Reflexion.
Reflections of previous trials: [reflections)
Here the current sequence of moves chess game: {question]
What is the best chess move I should execute next?
Give a single move suggestion of the form (etpwer_steppxxXXXX and make sure the chess move valid in the current board state.
The actor prompt for the agent to generate updated answers when deploying Reflexion.
Reflections of previous trials: [reflections]
Here are other chess move suggestions from other agents: {solutions]
Using the chess suggestions from other agents as additional advice, can you give me your updated thoughts on the best next chess move should play given the chess sequence?
The current sequence of moves a chess game is: [self.question)
Give single move suggestion of the form (answer_ste))xxXXXX <XXX> and make the chess move is valid in the current board state.
30

---

The reflector prompt of each debater agent.
You advanced reasoning agent that improve based self refection. In previous trial, you are [role] you are supposed to give the best next chess move through debating with other agents. However, you were unsuccessful in the previous trial. Now given previous interactions, you need to carefully examine the problem-solving ideas and calculation results, and form reflection avoid these problems in the round. The reflection should be less than 50 words.
Previous Game: {question)
Previous interactions:
{context]
Reflection:
The prompt of the context model of each agent.
Please summarize the following process in concise language, including the opinions of all agents.
Please note: Please summarize the viewpoints of each agent and retain the role of the agent in the summary, such as agent_0, O, agent_ 1, agent 2.
(scratchpad)
Summary:
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In this paper, we suggest improving multi-agent collaboration through self- reflection and propose COPPER, which fine-tunes shared reflector model with counterfac- tual PPO mechanism. We accurately introduce COPPER in abstract and introduction and highlight contribution at end of introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
32

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We focus on the application of multi-agent systems on downstream tasks and do not include theoretical results in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
We provide all the information for reproducibility including data and coding details in Section 5 and Appendix
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
33

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide all the information in Section 5 and Appendix found at: https: //ponymunno 4open. wwwpi/enceer//oppe--////
A Our code can be
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
publicc/guide///deeSttmt www.aimmmiiioooiccii.ooo.. for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips /nipsscc/puut
guides/ PwoddSSumiissiooooo.....ooo
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We provide this information in Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: We do not provide ertor bars because using GPT-3.5 or GPT-4 for test is costly. Previous relevant works 38 27 do not provide error bars, either.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
34

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the information in Section 5.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our paper conforms with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential societal impacts in Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
35

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification:
We discuss the question in Appendix G
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite them in the paper properly.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
36

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The details are well introduced in the paper and our code can be found at https://anonyy: 4open sciencce/t/copper-///
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do involve research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
37


Reviews:
[{"review_id": "2TzrguAfuL", "reviewer": "Reviewer_gdf9", "summary": "This paper proposes COPPER to enhance the collaboration ability of multi-agent systems through a learnable self-reflection mechanism. It involves reflections from different agent-specific profiles. The contribution of each agent-specific reflector is measured based on their marginal reward. This reflector is shared among agents and generates personalized reflections according to agents' roles. Experimental results on several datasets demonstrate its effectiveness.", "questions": "* The left part of Figure 2 is a little confusing due to the position of Step 1 to 4. The execution order is unclear. The text in Figure 2 is too small to be seen, especially the right part.  \n* Will there be any negative counterfactual reward? For example, the removal of a specific reflection will improve the performance.  \n* What is the impact of the agent profile on the reflector? Will a personalized reflector be better than a general reflector?", "limitations": "The authors discuss their potential limitations in their paper", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper explores the reflection on multi-agent collaboration. Previous work on reflection mainly focuses on a single LLM, ignoring the complex environment and interaction in the multi-agent system.\n2. The introduction of the counterfactual reward in PPO training assigns the reward to rate each agent's reflection, helping the credit assignment problem.\n3. The comprehensive analysis of the counterfactual reward, the shared reflector, and different LLMs for reflectors provide a deep insight into the proposed method.", "weaknesses": "1. Including the Retroformer under the multi-agent setting as one of the baselines would be better.\n2. When the environment provides a sparse reward such as the credit for the reflection of different agents may become very similar. For example, the removal of all reflections may result in a counterfactual reward of 0 because both trials fail. Then the counterfactual reward may degrade to the episode reward in Retroformer.\n3. With the complex PPO training, COPPER's performance is not very impressive, especially when the trial is small (in GSM8k and Checkmate in One Move  of Figure 4 and Figure 8)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "To Reviewer gdf9:\n\nThanks so much for your positive comments on our manuscript. In the following, we try to alleviate your concerns in detail (we combine all the questions in the weaknesses and questions).\n\n**Q1: Including the Retroformer under the multi-agent setting as one of the baselines would be better.**\n\nFollowing your suggestion, we have implemented Retroformer under multi-agent setting, and compared it with our model. The experiment results are presented in Figure 2 of the PDF.\n\nFrom the results, we can see that our model can outperform Multi-Retroformer on all the datasets. We speculate that the improved performance is brought by our special designs for multi-agent settings, such as the counterfactual reward and so on.\n\n\n**Q2: When the environment provides a sparse reward such as the credit for the reflection of different agents may become very similar. For example, the removal of all reflections may result in a counterfactual reward of 0 because both trials fail. Then the counterfactual reward may degrade to the episode reward in Retroformer.**\n\nTo begin with, the counterfactual reward aims to evaluate \"the effectiveness of conducting reflection on each agent\". For agent $i$, we compare the final rewards when agent $i$ uses and does not use reflections. The reward change is regarded as the real effect of the reflector on agent $i$. Basically, we decompose the overall reward improvement into sub-rewards, which are more tailored to different agent reflectors.\n\nAccording to the above explanations on the counterfactual reward, we can not remove all the reflections, for each agent, we only remove its corresponding reflection to obtain the counterfactual reward, while all the other reflections are still valid. This corresponds to the basic meaning of counterfactual effect: \"What is the effect of one variable when all the other variables remain the same\". \n\nHere, we present a toy example, suppose we have two agents $A$ and $B$, and their reflectors are $X$ and $Y$, respectively. For $X$, we first run a task with $(A+X, B+Y)$ and obtain the reward $r(A+X, B+Y)$, and then we run the same task with $(A, B+Y)$ to get the reward $r(A, B+Y)$, then the counterfactual reward for $X$ is $r_X = r(A+X, B+Y) - r(A, B+Y)$. Similarly, the counterfactual reward for $Y$ is $r_Y = r(A+X, B+Y) - r(A+X, B)$.\n\nSince $r(A, B+Y)$ and $r(A+X, B)$ are very different, the counterfactual rewards for different agents are various. For example, in the HotPotQA dataset, after a failed trial, the student agent $A$ generates a reflection $X$: \"The student did not find the correct answer due to incomplete search scope and insufficient specificity...\", and the teacher agent $B$ generates a reflection $Y$: \"I think the main reason for failure was the unclear pronoun usage...\". We first add both the reflections to the corresponding agent's memory, and run the next trial to calculate the episode difference score $r(A+X, B+Y)=1.0$. Then we run the same task with $(A, B+Y)$ and $(A+X, B)$ respectively and obtain $r(A, B+Y)=0$ and $r(A+X, B)=0.211$. The counterfactual reward of $X$ and $Y$ can be calculated as follows:\n$$\nr_X = r(A+X, B+Y) - r(A, B+Y) = 1.0 \n\\notag\n$$\n\n$$\nr_Y = r(A+X, B+Y) - r(A+X, B) = 0.789\n\\notag\n$$\n\n\n**Q3: With the complex PPO training, COPPER's performance is not very impressive, especially when the trial is small (in GSM8k and Checkmate in One Move of Figure 4 and Figure 8).**\n\nThe trial number in these figures means that, for the same task, how many times the agents reflect on their behaviors to achieve better task success rates. \nWe believe this could be a character of the RLHF method, that is, when using RLHF, we need to let the agents reflect more times to achieve better performances. \n\n\n**Q4: The left part of Figure 2 is a little confusing due to the position of Step 1 to 4. The execution order is unclear. The text in Figure 2 is too small to be seen, especially the right part.**\n\nThe Step 1 to 4 on the left part of Figure 2 describes the process where the multi-agent system generates actions in response to problem $k$  at time $t$, which corresponds to lines 136-138 in Section 4.1. First, the system needs to compute the identifier $i$ of the agent to respond at the current time (Step 1). Then, agent $i$ updates its memory, including reflections of previous trials and the current trial's historical interactions (Step 2), and perceives the environmental state such as the question and current task scores (Step 3). Finally, the agent $i$ generates the action based on its own memory and the environment state (Step 4).\n\nTo make Figure 2 more clear and accurate,  we will add explanations of steps 1 to 4, and replot the figure in the final version. The reploted version can be found in Figure 1 of the PDF.\n\n\n**Q5: Will there be any negative counterfactual reward? For example, the removal of a specific reflection will improve the performance.**\n\nYes, sometimes removing the reflection of a certain agent can lead to an improvement in the task performance. To verify this, we further count the number of negative reflections in the constructed training data. The statistics are shown in Table 1 and Table 2 of the PDF.\n\n**Q6: What is the impact of the agent profile on the reflector? Will a personalized reflector be better than a general reflector?**\n\nThanks for this very interesting question. Inspired by your comments, we conducted additional experiments by removing the personalized profiles. The results are presented in Figure 5 of the PDF.\n\nFrom the results, we can observe that when using pre-trained LMs (LongChat and GPT-3.5) to reflect, the removal of the agent profile has a greater impact on the GPT-3.5 reflector. This may be due to GPT-3.5's better contextual understanding ability. Our COPPER can further improve the model's reflection ability under non-profile settings. However, the results are slightly worse than the setting with agent profiles.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FVvgwkRb1k", "reviewer": "Reviewer_ogh3", "summary": "The paper proposes a multi-agent reflection framework COPPER to solve reasoning tasks on several datasets such as HotPotQA, GSM8K, and Checkmate in One Move. The two main contributions are:\n1. designing counterfactual rewards to alleviate the credit assignment problem;\n2. training a shared reflector to personalize the reflection for each agent.", "questions": "1. How to determine the number of agents for each task?\n2. Can you present the computational cost? Including training and inference stages.\n3. Can you provide more case studies, especially for the other two datasets?\n4. Does the shared reflector take all the agents' trajectories together to reflect?", "limitations": "The efficiency of data collection and the length of reflection memory may limit the application of the method.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Novelty: The paper introduces counterfactual rewards from RL to LLM agents, to deal with the credit assignment problem in multi-agent cooperation.\n2. Soundness: The authors conducted extensive experiments to thoroughly analyze the proposed mechanism.", "weaknesses": "1. The motivation of the shared reflector may not align with reality. Embodied scenarios do not allow complete information sharing with a central reflector.\n2. The computation of counterfactual rewards can be very high. Every agent demands two times of simulation to calculate the rewards, and the computational costs could be much higher when the number of agents increases.\n3. The claims of personalized reflection may not be completely conducted. For the Cooperative Debate Paradigm, there are no roles for the debaters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "To Reviewer ogh3:\n\nThanks for your comments. In the following, we try to alleviate your concerns one by one.\n\n**Q1: The motivation of the shared reflector may not align with reality. Embodied scenarios do not allow complete information sharing with a central reflector.**\n\nThanks for this comment. As an initial study for multi-agent reflection, we focus on the settings where the agents' information can be fully observed, which we believe indeed simplifies the real scenarios.\n\nHowever, we believe our study is also meaningful since it formally proposes the direction of considering agent reflection under multi-agent settings. Based on our study, one can easily extend to the settings where the information between agents is not fully observable.\n\nTo improve our paper, we conduct more experiments to investigate the settings with partial information. In specific, we introduce two models. For the first one, we remove the information of the other agents. For the second one, we use a proxy model to predict the information of the other agents. We present the experiment results in Figure 6 and Figure 7 of the PDF, respectively. We can find from the results that our COPPER still achieves better reflection performance under these partial settings. \n\n\n\n**Q2: The computation of counterfactual rewards can be very high. Every agent demands two times of simulation to calculate the rewards, and the computational costs could be much higher when the number of agents increases. Can you present the computational cost? Including training and inference stages.**\n\nFollowing your suggestion, we have incorporated the computational costs of the training and inference stages.\n\nIn the training stage, we first collect training reflection data and then train the reflector model via RLHF offline. So although constructing counterfactual rewards requires multiple times of simulation, the process does not incur additional computational cost. For example, when training reflector on HotPotQA dataset with one NVIDIA A800-80G, both SFT and reward model training stages can be finished in about 1 hour, while the PPO training requires about 4 hours (For reference only. Training time may vary with different hyperparameters).\n\nIn the inference stage, we load the fine-tuned reflector model on GPU and call GPT API as the actor model. The duration of the stage mainly depends on the speed of calling GPT.\n\n\n\n**Q3: The claims of personalized reflection may not be completely conducted. For the Cooperative Debate Paradigm, there are no roles for the debaters.**\n\nThanks for this comment. We believe personalization should be considered as a general term, any differences between the agents can be actually regarded as \"personalization\" and written into the profile. For the Cooperative Debate Paradigm, the stances, the goal of each agent, and the knowledge owned by different agents can be regarded as personalized.\n\n\n\n**Q4: How to determine the number of agents for each task?**\n\nFor the HotPotQA dataset, we design a teacher-student collaboration framework, in which an intuitive number of agents is 2 (with one student agent and one teacher agent). In GSM8K and Checkmate in One Move datasets, we follow [1] to set the number of agents and debating rounds. The paper explores the best number setting in each scenario through empirical experiments.\n\n\n\n**Q5: Can you provide more case studies, especially for the other two datasets?**\n\nFollowing your suggestion, we have added more case studies in the PDF（Figure 8 and Figure 9）, which will be incorporated into the final version.\n\nFrom the cases, we can find that in the context of the multi-agent debate, compared to GPT-3.5, COPPER can accurately identify key issues where mistakes occurred in previous trials and generate more specific reflections to guide agents in improving their responses. For example, in the case of GSM8K, COPPER identified that previous errors mainly stemmed from an incorrect analysis of equal relations and emphasized the need to pay attention to \"every two weeks\". In the case of Checkmate in One Move, COPPER pointed out the importance of focusing on the weakness in the own position.\n\n\n\n**Q6: Does the shared reflector take all the agents' trajectories together to reflect?**\n\nYes, in our model, the reflector takes all the agents' trajectories to reflect. To further improve our study, we also conduct more experiments on the settings where the agent only takes its own trajectories to reflect, see the answers to Q1 for more details.\n\n\n\n**References:**\n\n[1] Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. *CoRR*, abs/2305.14325, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jObQIo3keU", "reviewer": "Reviewer_tdnV", "summary": "The paper introduces COPPER, a novel framework designed to enhance collaboration in multi-agent systems using a learnable self-reflection mechanism. COPPER utilizes a shared reflector fine-tuned to adjust actor model prompts via a counterfactual PPO mechanism. This approach includes counterfactual rewards to address the credit assignment problem and enables the reflector to customize reflections based on agent roles, optimizing computational resources and training stability. The framework's efficacy is validated through experiments in multi-hop question answering, mathematics, and chess, demonstrating improved reflection capabilities and generalization across various actor models.", "questions": "See Weakness.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper is clearly written and explores a new setting — multiagent reflection. It also show improved performances on all three tasks. Using counterfactual rewards to perform PPO training sounds straightforward.", "weaknesses": "- My main concern is this paper involves a combination of various components and I could not clearly infer from the paper which part is most important. This makes the improvement for each part look marginal. Generally, this paper proposes a novel training method to enhance reflection, as well as use reflection-based multi-agent discussion to improve agent reasoning. I believe the method could be directly applicable to single agent scenario as reward for each agent is updated independently. Could you perform ablation in terms of single-agent?\n \n- The test scenario focuses on single-step tasks, can this framework be applied to multi-step agent tasks like AlfWorld?\n\n- How is the performance of COPPER compared to shared parameter & loss training for all LLMs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "To Reviewer tdnV:\n\nThanks for your comments. We will try to alleviate your concerns one by one in the following.\n\n**Q1: My main concern is this paper involves a combination of various components and I could not clearly infer from the paper which part is most important. This makes the improvement for each part look marginal. Generally, this paper proposes a novel training method to enhance reflection, as well as use reflection-based multi-agent discussion to improve agent reasoning. I believe the method could be directly applicable to single agent scenario as reward for each agent is updated independently. Could you perform ablation in terms of single-agent?**\n\n\nThanks for this comment. In our paper, we propose the problem of multi-agent reflection, which, to the best of our knowledge, is the first time in the area of LLM-based agents. For this problem, we have actually designed several tailored techniques to solve its special challenges, for example, designing the counterfactual rewards to obtain the real effect of different agent reflections, and so on.\n\nIn our setting, the reward is determined by the trajectory composed of all the agents' actions, and the reflector also needs to consider all the agent actions to reflect. As a result, one agent's action may influence the final reward and reflection, and further impact the other agents' updates. \n\nTo follow your suggestion, we use the episode difference reward as the reward for all the agents, and update the agent independently. We refer to the method as Multi-Retroformer and present the experiment results in Figure 2 of the PDF.\n\n\n\n**Q2: The test scenario focuses on single-step tasks, can this framework be applied to multi-step agent tasks like AlfWorld?**\n\nThanks for the question. Actually, HotPotQA is a multi-step agent task, where agents can retrieve relevant knowledge by invoking search engines multiple times to ultimately answer questions. Each agent in the environment follows the ReAct framework and generates actions sequentially. The main experiment results on HotPotQA can be found in Section 5.4.\n\nTo further improve our study, we additionally conduct experiments on ALFWorld. We follow the same multi-agent collaboration setting as [1] and test the model with 134 instances. The experiment results are presented in Figure 4 of the PDF. \n\nFrom the results, we can observe that COPPER achieves better reflection performance than the original LongChat and GPT-3.5. Besides, compared to the initial success rate, COPPER brings an improvement of 37.2% with 4 times of reflections.\n\n\n\n**Q3: How is the performance of COPPER compared to shared parameter & loss training for all LLMs?**\n\nActually, we have compared the shared and non-shared versions of our reflector in Section 5.8. To further improve this part, we have added more experiments on the other datasets, which are shown in Figure 3 of the PDF.\n\nThe experiment results indicate that in multi-agent collaboration scenarios, training a shared reflector enables higher-quality reflections, both when fine-tuned with SFT and RLHF techniques.\n\n\n\n**References:**\n\n[1] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen LLM applications via multi-agent conversation framework. *CoRR*, abs/2308.08155, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers:

Thanks for your detailed reviews. Additional tables and figures mentioned in the rebuttals are shown in the submitted one-page pdf.

Meta-review: The paper proposes COPPER, a framework designed to enhance collaboration in multi-agent systems through a learnable self-reflection mechanism. The method leverages counterfactual rewards to address the credit assignment problem and uses a shared reflector to generate personalized reflections tailored to each agent's role. The approach has been evaluated across various tasks, including multi-hop question answering, mathematics, and chess, and is reported to show improvements over existing models.

The reviewers generally acknowledge the novelty of applying counterfactual rewards and the comprehensive experimental analysis provided in the paper. However, concerns were raised about the practical effectiveness of COPPER, particularly in scenarios where complete information sharing is unrealistic or where computational costs are a concern. The introduction of Retroformer as a baseline under a multi-agent setting was requested by one reviewer, and while the authors addressed this in their rebuttal, the improvement over Retroformer was deemed marginal, especially in early trials.

Another significant point is the motivation behind the shared reflector. The potential misalignment with real-world scenarios, where full information sharing is not feasible, was highlighted. The authors attempted to mitigate this concern by presenting additional experiments under partial information settings. The paper's focus on combining various computationally intensive methods also drew criticism, with one reviewer questioning whether the resulting performance gains justify the increased costs.

Overall, multi-agent collaboration is an emerging topic, and this paper can contribute to the field by providing certain insights on applying counterfactual rewards, and cost could be optimized in the future work.
Decision: Accept (poster)