360◦REA: Towards A Reusable Experience Accumulation with 360◦ Assessment for Multi-Agent System (2024)

Abstract: Large language model agents have demonstrated remarkable advancements across various complex tasks. Recent works focus on optimizing the agent team or employing self-reflection to iteratively solve complex tasks. Since these agents are all based on the same LLM, only conducting self-evaluation or removing underperforming agents does not substantively enhance the capability of the agents. We argue that a comprehensive evaluation and accumulating experience from evaluation feedback is an effective approach to improving system performance. In this paper, we propose Reusable Experience Accumulation with 360◦ Assessment (360◦REA), a hierarchical multi-agent framework inspired by corporate organizational practices. The framework employs a novel 360◦ performance assessment method for multi-perspective performance evaluation with fine-grained assessment. To enhance the capability of agents in addressing complex tasks, we introduce dual-level experience pool for agents to accumulate experience through fine-grained assessment. Extensive experiments on complex task datasets demonstrate the effectiveness of 360◦REA1.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 13149–13162 August 11-16, 2024 ©2024 Association for Computational Linguistics
360◦REA: Towards A Reusable Experience Accumulation with 360◦ Assessment for Multi-Agent System
Shen Gao1†, Hao Li2†, Chengrui Huang1, Quan Tu3, Zhiliang Tian4*, Minlie Huang5, Shuo Shang1 1 University of Electronic Science and Technology of China,
2 Shandong University, 3 Renmin University of China, 4 National University of Defense Technology, 5 Tsinghua University
shengao@pku.edu.cn, lihaohan@mail.sdu.edu.cn, {yunrongyuxi, jedi.shang}@gmail.com, quantu@ruc.edu.cn,
tianzhiliang@nudt.edu.cn, aihuang@tsinghua.edu.cn Abstract
Large language model agents have demonstrated remarkable advancements across various complex tasks. Recent works focus on optimizing the agent team or employing self-reflection to iteratively solve complex tasks. Since these agents are all based on the same LLM, only conducting self-evaluation or removing underperforming agents does not substantively enhance the capability of the agents. We argue that a comprehensive evaluation and accumulating experience from evaluation feedback is an effective approach to improving system performance. In this paper, we propose Reusable Experience Accumulation with 360◦ Assessment (360◦REA), a hierarchical multi-agent framework inspired by corporate organizational practices. The framework employs a novel 360◦ performance assessment method for multi-perspective performance evaluation with fine-grained assessment. To enhance the capability of agents in addressing complex tasks, we introduce dual-level experience pool for agents to accumulate experience through fine-grained assessment. Extensive experiments on complex task datasets demonstrate the effectiveness of 360◦REA1.

heading: 1 Introduction
text: A variety of tasks, spanning from reasoning (Besta et al., 2023; Wang et al., 2023b; Wei et al., 2023; Yao et al., 2023b,a; Zhou et al., 2023) to code generation (Anishka et al., 2023; Chen et al., 2022, 2021; Dong et al., 2023; Hong et al., 2023; Wu et al., 2023), as well as video gaming (Wang et al., 2023a; Zhu et al., 2023) and autopilot (Dong et al.,
2These authors contributed equally to this work. 1Code is available at https://anonymous.4open.
science/r/360-REA-0FBD
2023; Lykourentzou et al., 2021; Nascimento et al., 2023), have seen promising advancements through the utilization of Large Language Model (LLM) agents. Recent works have explored the effectiveness of combining multiple LLM agents to collaborate on complex tasks (Chen et al., 2023c; Liu et al., 2023a; Packer et al., 2023; Ren et al., 2024; Wang et al., 2024). These methods can generally be classified into two main categories. One category involves pre-defining the roles and specific tasks of each agent within a multi-agent system, such as software development (Hong et al., 2023; Qian et al., 2023). The other category of methods allows multiple agents to decide their tasks and roles autonomously, e.g., travel planning (Zheng et al., 2023). In these methods, they usually focus on how to allocate tasks among agents (Cheng et al., 2023; Wang et al., 2023d) or how to score agents to subsequently remove those with low scores (Liu et al., 2023b). While some methods (Xi et al., 2023; Shinn et al., 2023; Wang et al., 2023c) aim to correct the result of the current task by employing self-reflection in the agent, these approaches are limited to improving current specific task performance. The capability of the agent to address similar tasks is not enhanced.
In a company, the ultimate goal of company management is not merely to score employee performance, to promote or fire them. The best practice is to use the performance assessment to assist employees in purposefully reflecting on their work, thereby enhancing their capability to fulfill their roles better. 360-degree assessment (Bracken et al., 2016; Alimo-Metcalfe, 1998; Peiperl, 2001) is a comprehensive process that involves collecting assessment on an employee’s behavior from various perspectives, including colleagues, supervisors, and sometimes external sources like customers. Business organizations are increasingly incorporating 360-degree assessment into performance evaluations and employment
13149
decisions, such as pay and promotions (Bytedance, 2024). Motivated by this, in multi-agent systems, the design principle should be to assist agents in accumulating experience based on their assessment results and enabling them to perform better in subsequent tasks. This principle emphasizes the importance of helping agents to improve rather than solely evaluating or removing underperforming agents from the system. Therefore, designing a mechanism for agent assessment and capabilities learning poses a challenge in the design of multiagent systems.
In this paper, we propose Reusable Experience Accumulation with 360◦ Assessment (360◦REA) which is a hierarchical multi-agent interaction framework. The objective is to enhance performance in handling complex tasks by emulating organizational structures, employee performance assessments, and the accumulation of experience within a company. We employ a hierarchical structure for organizing agents, where the leader agent allocates tasks and roles to crew agents, and the crew agents collaborate to accomplish the given task. To evaluate the performance of each crew agent, instead of using only self-reflection, we introduce a novel 360◦ performance assessment method that employs a multidimensional evaluation in crew-crew and leader-crew dimensions. It provides agents with more valuable assessment by evaluating from both the peer level and supervisory level.
As mentioned above, the accumulation of experience is essential to achieve better agent performance (Madaan et al., 2023; Miao et al., 2023; Shinn et al., 2023; Wang et al., 2023c). In this paper, we propose a dual-level experience pool to help agents collect useful experiences when tackling complex tasks. Specifically, within the local experience pool, agents utilize fine-grained insights from the 360◦ performance assessment to reflect on their current outputs and form experiences related to the sub-tasks. To effectively address a complex task, optimizing individual subtasks separately is sub-optimal, and it is essential to a comprehensive understanding and analysis of the complex task. To construct a high-level experience, leader agents use the evaluation of the final result to comprehensively summarize the task, forming a global experience pool. Finally, the team accumulates sufficient experience in handling such tasks, leading to improved performance on similar tasks. Extensive experiments conducted on two
widely used complex task datasets demonstrate the effectiveness of our proposed 360◦REA.
Our contributions of this work are as follows: • We propose 360◦REA, which is a multi-agent interaction framework inspired by contemporary corporate organizational governance methods. • We propose 360◦ performance assessment to evaluate the performance of agents from multiple dimensions. • We introduce dual-level experience pool to help agents collect useful experiences based on the evaluation result. • Experimental results on benchmark datasets illustrate the advantage of 360◦REA on several complex tasks.

heading: 2 Related Work
text: Large-scale language models (LLMs) such as ChatGPT have demonstrated outstanding performance across various natural language tasks (OpenAI, 2023). In recent years, several methods have emerged to enhance the ability of LLM to tackle complex tasks, such as Chain of Thought (Wei et al., 2023), Tree of Thoughts (Yao et al., 2023a), and Graph of Thoughts (Besta et al., 2023). These methods aim to improve reasoning by simulating human thinking processes within LLMs. ReAct (Yao et al., 2023b) enables LLMs to generate interpretable action plans before taking actions, thus facilitating the completion of more complex tasks. Reflection (Shinn et al., 2023) enhances task performance by encouraging LLM to engage in self-reflection. However, some complex real-world tasks typically require collaboration among multiple people to accomplish. Thus, relying solely on prompting methods often yields suboptimal performance (Xi et al., 2023; Wei et al., 2023; Yin et al., 2023).
Recent work has begun exploring frameworks involving multiple LLM-based agents, which significantly outperform direct prompting methods
in natural language understanding and reasoning tasks (Chen et al., 2023c; Hong et al., 2023; Chen et al., 2023b). Some work involves conducting several rounds of debate among multiple LLMbased agents to enhance their authenticity and reasoning abilities (Liang et al., 2023). Other works propose dividing tasks into subtasks and employing multiple LLM-based agents to address complex tasks through a divide-and-conquer approach (Chen et al., 2023c; Hong et al., 2023; Chen et al., 2023a). Furthermore, some works employ scoring the performance of multiple LLMbased agents and terminating those with poor performance to better accomplish tasks (Liu et al., 2023b).
However, existing methods for evaluating agent performance often rely on the self-reflections of agents, neglecting comprehensive considerations within the entire multi-agent framework. And solely evaluating and terminating low-scoring agents does not substantially enhance their tasksolving capabilities.

heading: 3 360◦REA Methodology

heading: 3.1 Overview
text: In this section, we detail the Reusable Experience Accumulation with 360◦ Assessment (360◦REA). An overview of 360◦REA is shown in Figure 1. We will detail the 360◦REA in three parts: (i) We first illustrate the main workflow of the 360◦REA which uses the user query x as input and generates the response y. (ii) Next, we introduce the 360◦ performance assessment which prompts the agents to give a review of each other and update their response according to others’ reviews. (iii) After finishing the response generation of each agent, each agent will summarize their experience and construct a dual-level pool that contains reusable task-solving experiences.

heading: 3.2 Main Workflow of 360◦REA
text: In this paper, we propose the 360◦REA which is a general multi-agent learning framework and can be applied to the hierarchical LLM-based multi-agent frameworks (Zheng et al., 2023). In this LLM-based multi-agents framework, it consists of a leader agent Al and N crew agents {Ac1, Ac2, . . . , AcN} to solve the given task defined by the user input query x. Given an input query x, the leader agent Al first generates the instruction of each crew agent, which first divides the task into
several sub-tasks:
{Ic1, Ic2, . . . , IcN} = Al(x), (1)
where Ici is the task instruction for the i-th crew agent.
To enhance the crew agent with task-solving capabilities, we propose dual-level experience pool (introduced in § 3.4), which contains to subexperience pool: global experience Eg and local experience El,i. The experiences in dual-level experience pool describe (1) the general skill of tackling the task and (2) the specific skill of the sub-task for agent Aci respectively. Then the crew agents generate their responses according to the instruction and useful context to solve the sub-task.
Hti = A c i (Ii, Eg, El,i, R t i), (2)
where Hti is the response of the i-th agents in tth turn. In this paper, we propose a multi-turn response generation paradigm for the crew agents. After each turn of the agent generating the response Hti , we employ the 360
◦ performance assessment (introduced in § 3.3) to evaluate the performance of agent Aci from multiple perspectives and generate a review set Rti. In Equation 2, we also incorporate the reviews from multiple sources Rti when generating the response of the agent. By this mechanism, agent Aci can iteratively revise the response in multiple turns.
After T turns revising, we can obtain the final responses {HT1 , HT2 , . . . ,HTN} from each crew agent. To form the final answer y to user query x, the leader agent summarizes all the responses by incorporating the global experiences Eg and its review Rtl :
y = Al({HT1 , HT2 , . . . ,HTN}, Eg, Rtl). (3)
Next, we employ an LLM-based evaluator agent P to evaluate the quality of the response y:
m = P (x, y), (4)
where m is an overall review which not only contains the judgment of the quality but also illustrates the reason why P gives this evaluation result.
You will be given a multi-day travel plan. The task is {total_task} Your task is to evaluate the travel plan on specific criteria. Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed. Evaluation Criteria: 1. Plan Customization (1-20): [Evaluation Criteria of P.Cu.] 2. Plan Novelty (1-20): [Evaluation Criteria of P.N.] 3. Plan Correctness (1-20): [Evaluation Criteria of P.Co.] Evaluation Steps: 1. Review the travel plan in detail. 2. Compare the plan against each criterion. 3. Assign a score for each criterion on a scale of 1 to 20, where 1 is the lowest and 20 is the highest, based on the Evaluation Criteria. Travel Plan: {Travel_Plan}

heading: 3.3 360◦ Performance Assessment
text: In this paper, inspired by the performance assessment of employees in large companies, we propose an agent feedback method, 360◦ performance assessment. This method comprehensively evaluates the shortcomings of agents in completing tasks through three types of evaluations: self-level, peer-level, and supervisor-level. Different from the agent evaluation methods only based on the agent selfreflection (Madaan et al., 2023), 360◦ performance assessment boosts the mutual collaboration among
agents in multi-agent systems. It enables the evaluation of the agent capabilities from the perspectives of different agents (a.k.a., sub-tasks), thereby encouraging each agent to consider and solve the problem from a global perspective.
After each agent responds to their own sub-task, we will conduct a 360◦ performance assessment to assess whether their responses effectively complete the sub-task and whether they are helpful to other agents in solving their tasks. Subsequently, this review will be provided to the agent, and the agent will then revise their response to the sub-task based on the review from other agents. Through several turns of such iterative evaluations, agents can continuously refine their responses by incorporating feedback from others, thereby improving the quality of their responses. Since our 360◦ performance assessment is a multiturn generation and assessment paradigm, in this section, we illustrate the details of 360◦ performance assessment method by using the agent action in t-th turn as an example. Self-level Assessment. Similar to the previous work (Madaan et al., 2023), we also employ selfreflection on the response of the agents:
Rts,i = A c i (H t i , I c i ), (5)
where Rts,i is the self-reflection of i-th agent A c i according to its response Hti and task instruction Ici . Peer-level Assessment. In LLM-based multiagent systems, cooperation among crew agents is prevalent, thus facilitating mutual assistance among crew agents can enhance the overall system performance. Therefore, we introduce peer-level assessment among crew agents, which allows for assessing the performance of intelligent agents from the perspectives of other tasks, thereby obtaining a more comprehensive evaluation. Additionally, it can also boost collaboration among the agents. Specifically, a crew agent reviews the responses of other crew agents and generates feedback for others:
Rti,j = A c i (H t i , H t j , I c j ), (6)
where Rti,j is the feedback for the response H t j generated by j-th agent, which is reviewed by the i-th agent Aci . Supervisory-level Assessment. On the other hand, assessment from the leader agent Al can evaluate the completion of sub-task by agents, and also assist them in better problem-solving from a comprehensive perspective. Therefore, we employ the leader agents Al to evaluate the performance of the crew agent according to its response Hti and instruction Ici :
Rtl,i = A l(Hti , I c i ), (7)
where Rtl,i is the review for the i-th agent written by the leader agent Al.
After conducting the 360◦ performance assessment from three different sources, in the t-th turn, the crew agent Aci will receive a review set Rti which contains three types of review:
Rti = {Rt∗,i, Rtl,i, Rts,i}. (8)
For the leader agent, it only has the self-level assessment review Rtl .

heading: 3.4 Dual-level Experience Pool
text: For company managers, performance evaluation of employees is not the ultimate goal. Instead, it is essential to enable employees to accumulate experience from evaluation assessment and perform better in subsequent tasks. Inspired by this, we propose the dual-level experience pool, which involves constructing dual-level experience pools:
local and global experience pools, to help each agent and the entire multi-agent system accumulate reusable of problem-solving experience during the task completion process.

heading: 3.4.1 Local-level Experience Construction
text: Local experience pertains to each agent itself in completing the current sub-task, summarized from previous multiple turns of 360◦ performance assessment in the current task, thereby helping the agent achieve better results in the next turn. After obtaining the reviews Rti, the agent A c i will reflect their local experience e′l:
e′l = A c i (H t i , R t i), (9)
where the crew agent Aci is prompted by the following instruction:
Based on feedback from others, past experiences, and from the perspective of one’s own role, summarize some experiences that you may use in the future. Your answer should be as concise as possible. Do not output other non output required content. Your role: {role}. Feedback from others: {peer_feedback} Rti Previous experience: {pre_exp} El,i Structured output requirements: Role: [Your Role] Experience: [Short experience description]
Then we append the e′l into the local experience pool El,i for the i-th crew agent Aci .

heading: 3.4.2 Global-level Experience Construction
text: Global experience refers to reusable experiences for the entire multi-agent system, and we build these experiences from the final response of each crew agents and the feedback m of the evaluator P after completing the task:
e′g = A l(m, {HT1 , HT2 , . . . ,HTN}), (10)
where the leader agent Al is prompted by the following instruction:
Obtain new experiences based on previous experiences and current scores. The new experience should be summarized while presenting the experience gained from this task. Here are the final results: {Final_Res} H Here are the evaluation of the task: {evaluation} m As a leader, you need to learn experience from this task, and the format should be "Where did I do well this time... why didn’t I do well this time... next time I should...".Note that these experiences are for this type of task. The answer should be as concise as possible.
Finally, we can accumulate the global experience pool by appending the e′g into the local experience pool Eg.

heading: 4 Experimental Setup

heading: 4.1 Tasks and Datasets
text: Following existing works (Wang et al., 2023d; Xie et al., 2024), we verify our 360◦REA performance on two tasks, creative writing, and travel plan making. (i) Creative Writing. We use Trivia Creative Writing (Wang et al., 2023d) dataset, which contains 100 samples. Each sample is required to generate a story to cover the required input topic and input questions. (ii) Travel Plan Making. We use Trip Planning (Xie et al., 2024) dataset that comprises travel plans for 50 world-famous attractions, with specific requirements generated based on the characteristics of the destinations. It includes the following fields: destination, basic requirements, duration of travel, season, type of travelers, number of travelers, and budget.

heading: 4.2 Evaluation Metrics
text: Since the objectives of travel planning tasks and creative writing tasks are not the same, we employ different evaluation perspectives in our experiments to assess the quality of the model-generated results.
For the story generation task, we employ the metric matched answer rate Match% (M%) used in Wang et al. (2023d), to measure the percentage of generated story cover the ground truth target answers for each question. We also propose two semantic evaluation metrics, which prompt the LLM to evaluate whether the generated results satisfy the specific aspects including: (1) Emotional Engagement (E.E.), whether the generated story evokes the reader’s emotion and empathy; (2) Insightfulness (Ins), whether the story carries the insightful plot and cause a profound impact on readers.
For the travel planning task, we also employ three semantic metrics for evaluating three aspects: (1) Plan Correctness (P.Co.), whether the plan covers all required details and the plan is reasonable; (2) Plan Novelty (P.N.), whether the plan is novel and creative; (3) Plan Customization (P.Cu.), whether the plan is customized to the personal interests and preferences of traveler.
For the logic grid puzzle task, we use one evaluation metric: Accuracy (Acc), which measures the percentage of correctly predicted house numbers compared to the ground truth targets.

heading: 4.3 Baselines
text: We compare our method with agent-based methods, including: (i) GPT-4 (OpenAI, 2023) is a closed-source LLM from OpenAI. We conduct task solving as an in-context learning that uses a single instruction with the same input data as our 360◦REA. (ii) Mistral (Jiang et al., 2023) is an open-source LLM and we use the mixture-of-experts version named Mistral-medium with 46 billion parameters, and use the same prompt as GPT-4. (iii) OKR (Zheng et al., 2023) is a hierarchical multi-agents collaboration framework. They propose the hierarchical OKR generation to decompose objects into multiple sub-objects and assign new agents based on key results and agent responsibilities. (iv) AgentVerse (Chen et al., 2023c) is a multi-agent framework that enhances collaborative problem-solving by employing specialized agents working together through structured interactions. (v) SPP (Wang et al., 2023d) propose the Solo Performance Prompting (SPP), which transforms a single LLM into a cognitive synergist by engaging in multi-turn self-collaboration with multiple personas.
We implement the OKR,AgentVerse and SPP with the backbone LLM GPT-4. We also employ two variants of 360◦REA: 360◦REA-G and 360◦REA-M with GPT-4 and Mistral as the backbone respectively.

heading: 4.4 Implementation Details
text: We conduct 2 turns of 360◦ assessment. We use 1 leader agent and set the number of crews is range from 3 to 5, we ask the LLM to follow that range by stating the range in prompt text. The number of experiences selected by the leader from the globallevel experience pool is 10. We use GPT-4 version of 1106-preview with a temperature of 1.0. Our anonymized source code is publicly released 2.

heading: 5 Experimental Results

heading: 5.1 Overall Performance
text: Table 2 shows the performance of our proposed 360◦REA and baselines in terms of three metrics on two benchmark datasets. We can find that 360◦REA shows superior performance in terms of all metrics compared to the state-of-the-art method SPP which is also a multi-agent hierarchical
2https://anonymous.4open.science/r/360-REA-0FBD
collaboration framework with agent discussion. This phenomenon indicates that our proposed 360◦ assessment and dual-level experience pool can help the agents to generate better responses in a higher quality. And we can also find that each variant of our 360◦REA (e.g., 360◦REA-G and 360◦REA-M) achieves better performance than simply prompting its LLM backbone, it demonstrates our proposed framework can adapt to any LLM backbones.

heading: 5.2 Human Evaluation
text: As using LLM as the evaluator is sometimes not robust, we also employ human evaluation to further validate the effectiveness of the model-generated answers. For creative writing tasks, we evaluate the quality of the generated response in terms of: (1) Coherence, the score indicating the story’s coherence; (2) Creative, the score indicating the story’s creativeness. For travel plan making, we evaluate the response in two aspects: (1) Correct, the score showing whether the plan is correct and complete, and (2) Customized, the score measuring whether the plan is customized. We hire 3 educated annotators with Bachelor degree to conduct double-blind annotation on randomly shuffled results. The annotating scores range from 1 to 100. From Table 3, we observed that our 360◦REA excels in all aspects compared to the strong multi-agent baseline SPP and GPT-4, which are the strongest baseline in the two tasks (Table 2). We also conclude that the automatic metrics in Table 2 are also consistent with the human evaluation.

heading: 5.3 Ablation Study
text: To verify the effectiveness of each module of our 360◦REA, we conduct ablation studies with model 360◦REA-G with several ablation variants: (i) 360◦REA-G w/o ExpPool: We remove both local and global experience from 360◦REA and directly use the assessment from other agents as the prompt of each agent. (ii) 360◦REA-G w/o 360◦F: We directly use the output of each crew agent without conducting all levels of assessment.
As the results shown in Table 2, we found that the 360◦REA-G w/o ExpPool method achieved lower scores compared to our full model at most metrics, indicating the necessity of using experience pool to recall reusable experience. And 360◦REA-G w/o 360◦F mainly underperforms 360◦REA-G, showing the effectiveness of conducting all level assessments among multi-agents.

heading: 5.4 Case Study
text: Table 6 shows an example of creative writing generated by SPP and 360◦REA-G. The bold text in the response indicates the correct answer to the input question. 360◦REA-G’s results contain the answers to all five questions, whereas SPP only correctly answered two of them. The text in green and red indicates the strengths and weaknesses in the generated story respectively. The green parts show 360◦REA-G’s output has a rich plot and illustrates the cause and ending of the story. However, the red parts show SPP’s story plot lacks a brief introduction and the use of “Valerie” is abrupt and weird. From this case, we observed that 360◦REA-G obtains more reasonable, logical, and fluent stories rather than SPP. We also show an example with the corresponding experiences in the Appendix A

heading: 6 Discussion

heading: 6.1 Analysis on Different Level Experiences
text: In this paper, we propose a novel dual-level experience pool. In this section, we employ two
ablation models to explore which layer of the experience pool is more effective and conduct experiments on the creative writing dataset. The results in Table 4 indicate that local experience is the most helpful. One possible reason for this phenomenon is that local experience contains lower-level experiences, including more specific task-completion skills. From Table 4, we also observe that compared to the 360◦REA, the model without global experience has a performance decrease of 5.9% in terms of Ins, indicating that the higher-level experience for task-solving in global experience also plays a crucial role in improving model performance.

heading: 6.2 Analysis on Different Level Assessment
text: We also investigate the effectiveness of different levels of assessment on the creative writing task. From Table 5, we find that all the ablation models achieve worse performance than 360◦REAG which demonstrates the effectiveness of each level of assessment in 360◦REA. We also find that the variant model without the peer-level assessment achieves the worst among all the ablation models. This phenomenon indicates that peer-level assessment provides the greatest improvement in overall performance, confirming our hypothesis that mutual evaluation among crew members helps each agent not only in completing their task but also in assisting other agents as much as possible.

heading: 7 Conclusion
text: In this paper, we propose a hierarchical multi-agent framework, Reusable Experience Accumulation with 360◦ Assessment (360◦REA), which represents a significant step towards enhancing the capabilities of LLM-based agents in tackling complex tasks. Unlike existing approaches that primarily focus on team optimization or selfreflection, 360◦REA incorporates a comprehensive agent performance evaluation strategy through the novel 360◦ performance assessment method. This method provides multi-perspective and finegrained feedback to agents, in which agents receive valuable insights from self, peers, and supervisory levels. Furthermore, to substantively improve agent task-solving capabilities, we introduce the dual-level experience pool for the accumulation of experiences based on evaluation feedback. Our experiments on complex task datasets demonstrate
the effectiveness of 360◦REA in achieving superior performance compared to the state-of-the-art baselines. By emphasizing the importance of comprehensive evaluation and experience accumulation, 360◦REA establishes a promising direction for advancing the field of multi-agent systems built on LLMs, offering a practical and impactful framework for addressing intricate tasks.
Limitations
In this paper, we only use the text as the input. However, multi-modal data (e.g., images and videos) is commonly used in many tasks. Since existing multi-modal LLMs are capable of understanding both textual and multi-modal information, our multi-agent framework can adapt to leverage multi-modal inputs in many complex tasks. In the future, we plan to incorporate multimodal information into our framework.
Ethics Statement
The multi-agent collaboration framework proposed in this paper is built upon LLM which is trained by reinforcement learning with human feedback (RLHF) (Lambert et al., 2022). Since LLM has already been aligned with human values and preferences during the RLHF stage, the likelihood of generating unethical content is low. However, existing LLMs still cannot completely prevent generating harmful content. Therefore, the answers generated by the multi-agent collaboration framework still require further human expert verification before use.

heading: Acknowledgements
text: This work was supported by the Natural Science Foundation of Shandong Province (ZR2023QF159), the National Natural Science Foundation of China (U2001212, 62032001, 61932004, 62306330, and WDZC20235250103), and Young Elite Scientist Sponsorship Program by CAST (2023QNRC001) under Grant No. YESS20230367.

heading: A Example of Generated Response
text: In this section, we show an example of the generated response when training our 360◦REA in the early training stage (with few experiences in the global experience pool). We show the output of several modules in 360◦REA including the review m of the evaluator (shown in Equation 4), and the global experience e′g (shown in Equation 10) learned from this example.
In the review m of the evaluator, we can see that our generated travel plan still has some flaws, with a lower score in terms of feasibility, and the evaluator has provided corresponding reasons. The text in blue indicates the flaw mentioned in the evaluation.
In the global experience e′g, the text in blue indicates that 360◦REA summarizes some detailed skills from the review m. The text in green indicates the experiences that are summarized from input experiences when generating this response (a.k.a., experiences obtained from previous samples). It demonstrates that our 360◦REA can not only summarize new experiences from the current task-solving process, but also keep previous experiences in mind and form new experiences from them,
Task description: destination:Barcelona, Spain days:6 description:A 6-day adventure in Barcelona, experiencing its unique architecture by Gaudí, vibrant food markets, and beautiful beaches.
season:Summer month:June interests:Architecture, Food Markets, Beaches members:adults:4, children:0 preferences:Cultural Explorers Budget Range:Mid-range The generated travel plan by 360◦REA: Day 1: Arrival and Architectural Introduction - Arrive at Barcelona and check-in at Hotel Midmost. - Afternoon: Private walking tour with a local architect to explore lesser-known Modernista sites. - Evening: Guided visit to Recinte Modernista de Sant Pau.
Day 2: Iconic Gaudí and Culinary Delights - Morning: Guided tour of La Sagrada Familia.
- Afternoon: La Boqueria market tour followed by a cooking class with a local gastronomy expert.
- Evening: Free time for relaxation or independent exploration.
Day 3: Park Güell Wonders and Coastal Activities
- Morning: Park Güell visit including a creative workshop on Gaudí’s nature inspirations.
- Afternoon: Barceloneta Beach with sunbed reservation and option for a Kite-surfing lesson or beach volleyball.
- Evening: Attend a local jazz festival by the sea or a Festa Major street festival.
Day 4: Traditional Markets and Twilight Art - Morning: Local market tour at Mercat de la Libertat. - Afternoon: Casa Vicens visit and leisurely stroll through La Ribera district to see Basilica de Santa Maria del Mar.
- Evening: Visit to lesser-known architectural wonders with a rooftop concert featuring Spanish guitar at Torre Bellesguard.
Day 5: Artistic Immersion and Seaside Farewell - Morning: Beach clean-up activity followed by a picnic or if weather permits, a visit to Torre Bellesguard including a rooftop concert.
- Afternoon: Artistic workshop inspired by Gaudí.
- Evening: Passeig de Gràcia tour with Casa Batlló & La Pedrera and reserved seating at a seaside restaurant for a farewell dinner.
Day 6: Leisure and Departure - Morning: Guided beach yoga session or another family-friendly activity such as a visit to Hospital de Sant Pau’s pavilions.
- Afternoon: Free for last-minute explorations or shopping.
- Airport transfer via pre-booked minivan service.
Emergency Strategy and Contingencies: - Daily travel insurance briefings, weather advisories, and emergency point of contact sharing. - Accessibility to VIP access to indoor venues like the Picasso Museum in case of hot weather or heavy rain.
- Communication protocols set to inform the group of any changes.
Logistics and Budget Management: - Hola BCN Travel Card and bicycle rentals for daily commutes. - Real-time budget tracking with contingency fund for spontaneous decisions. - Comprehensive local interactions from curated market tours to hands-on workshops, ensuring rich cultural exposure.
Concluding Remarks: This plan strikes a balance between gourmet experiences, art and architecture, and coastal leisure, aptly tailored for adults and takes into account Barcelona’s summer climate. Each activity is selected to enhance the group’s travel experience, seasoned with local insights and flexibility for a safe and memorable journey in Barcelona.
Review from evaluator: Alignment with Interests and Preferences: 20 The travel plan meticulously aligns with the group’s interests in architecture, food markets, and beaches, as well as their preference for cultural exploration. Activities such as private architectural tours, cooking classes, and creative workshops directly cater to these interests. The inclusion of iconic Gaudí sites, local markets, and beach activities ensures a tailored experience for the group’s cultural and leisure preferences.
Variety and Balance: 19 This itinerary offers a commendable balance of activities, from architectural tours and artistic workshops to leisure time at the beach and culinary experiences. The diversity of experiences ensures engagement throughout the trip, catering to both cultural interests and relaxation needs. The inclusion of evening activities like rooftop concerts and a farewell dinner at a seaside restaurant adds a delightful variety to the travel experience.
Feasibility and Logistics: 17 Overall, the plan appears feasible with a coherent schedule and consideration for logistics, including the use of the Hola BCN Travel Card and bicycle rentals for daily commutes. However, the packed itinerary might challenge the group’s energy levels, especially with multiple activities scheduled on the same day. Ensuring sufficient rest and travel time between activities would enhance the feasibility of this ambitious schedule.
Local Insights and Unique Experiences: 20 The plan excels in offering unique, local experiences, from a market tour followed by
a cooking class with a local expert to creative workshops inspired by Gaudí. These activities not only provide an immersive cultural experience but also allow for meaningful interaction with the local community. The thoughtful inclusion of lesserknown sites and local festivals further enriches the travel experience with authentic insights into Barcelona’s culture and lifestyle.
Overall Evaluation: This travel plan is excellently crafted, demonstrating a deep understanding of the group’s preferences and a strong alignment with their cultural and leisure interests. While there is a minor consideration for the feasibility of the packed schedule, the variety of activities and the emphasis on local, unique experiences promise a memorable and enriching visit to Barcelona. The plan’s thoughtful contingencies and logistics management also underscore its practicality and attentiveness to the group’s comfort and experience.
Generated Global Experience:
Where did I do well this time: I excelled in aligning the travel plan closely with the group’s specified interests and preferences, especially in architecture, food markets, and beaches. My success in integrating unique and authentic local experiences, such as creative workshops and cooking classes with local experts, significantly enriched the cultural exploration aspect of the trip.
Why didn’t I do well this time: I underestimated the importance of ensuring adequate rest periods and manageable schedules within the plan. The feasibility and logistics score reflected this oversight, indicating that the itinerary might be too packed, potentially leading to fatigue and limiting the group’s ability to fully enjoy each activity.
Next time I should: Focus on creating a more balanced itinerary that not only aligns with the travelers’ interests but also accounts for practical aspects such as rest, travel times between activities, and the overall pace of the trip. This means potentially reducing the number of activities per day or incorporating more flexible, free time slots to ensure a comfortable and enjoyable experience for all travelers. Additionally, I should consider more deeply the seasonal appropriateness of each activity, ensuring that the group is comfortable and able to enjoy the trip regardless of weather conditions.

heading: Future Work
text: In the future, we plan to incorporate multimodal information into our framework