MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
Authors: Sirui Hong
Mingchen Zhuge
Jonathan Chen
Xiawu Zheng
Yuheng Cheng
Jinlin Wang
Ceyao Zhang
Zili Wang
Steven Ka Shing Yau
Zijuan Lin
Liyang Zhou
Chenyu Ran
Lingfeng Xiao
Chenglin Wu
Jürgen Schmidhuber
Venue: ICLR 2024 oral (2024)
Keywords: Autonomous Agent
Meta Programming
Multi-Agent Society
Group Intelligence
Primary area: applications to robotics, autonomy, planning
TL;DR: This paper introduces MetaGPT, an innovative meta-programming framework for LLM-based multi-agent collaborations.

Abstract: Recently, remarkable progress has been made on automated problem solving through societies of agents based on large language models (LLMs). Previous LLM-based multi-agent systems can already solve simple dialogue tasks. More complex tasks, however, face challenges through logic inconsistencies due to cascading hallucinations caused by naively chaining LLMs. Here we introduce MetaGPT, an innovative meta-programming framework incorporating efficient human workflows into LLM-based multi-agent collaborations. MetaGPT encodes Standardized Operating Procedures (SOPs) into prompt sequences for more streamlined workflows, thus allowing agents with human-like domain expertise to verify intermediate results and reduce errors.  MetaGPT utilizes an assembly line paradigm to assign diverse roles to various agents, efficiently breaking down complex tasks into subtasks involving many agents working together.  On collaborative software engineering benchmarks, MetaGPT generates more coherent solutions than previous chat-based multi-agent systems.

Preprint
METAGPT: META PROGRAMMING FOR A MULTI-AGENT COLLABORATIVE FRAMEWORK
Sirui Hong'; Mingehen Zhuge²; Jonathan Chen', Xiawu Zheng', Yuheng Cheng', Ceyao Zhang'. Jinlin Wang', Zili Wang. Steven Ka Shing Yau", Zijuan Lin", Liyang Zhou', Chenyu Ran', Lingfeng Xiao???, Chenglin Wu'! Jürgen Schmidhuber
'DeepWisdons AI Initiative, King Abdullah University of Science and Technology.
"Xiamen University, "The Chinese University of Hong Kong, Shenzhen, Nanjing University, University of Pennsylvania,
University of California, Berkeley, 8The Swiss AI Lab IDSIA/USI/SUPSI
ABSTRACT
Remarkable progress has been made on automated problem solving through so- cieties of agents based on large language models (LLMs). Existing LLM-based multi-agent systems can already solve simple dialogue tasks. Solutions to more complex tasks, however, are complicated through logic inconsistencies due cascading hallucinations caused by naively chaining L.Ms.. Here we introduce MetaGPT, an innovative mett-prrogammming framework incorporating efficient human workflows into LLM-based multi-agent collaborations. MetaGPT en- codes Standardized Operating Procedures (SOPs) into prompt sequences for more streamlined workflows, thus allowing agents with human-like domain expertise to verify intermediate results and reduce errors. MetaGPT utilizes an assembly line paradigm to assign diverse roles various agents, efficiently breaking down complex tasks into subtasks involving many agents working together. On col- laborative software engineering benchmarks, MetaGPT generates more coherent solutions than previous chat- based multi-agent systems. Our project can be found at
INTRODUCTION
Autonomous agents utilizing Large Language Models (LLMs) offer promising opportunities to en- hance and replicate human workflows. In real-world applications, however, existing systems (Park et al., 2023; Zhuge et al., 2023; Cai et al., 2023; Wang et al., 2023c; Li et al., 2023; Du et al., 2023; Liang et al., 2023; Hao et al., 2023) tend to oversimplify the complexities. They struggle achieve effective, coherent, and accurate problem-solving processes, particularly when there is a need for meaningful collaborative interaction (Chen et al., 2024; Zhang et al., 2023; Dong et al., 2023; Zhou et al., 2023; Qian et al., 2023).
Through extensive collaborative practice, humans have developed widely accepted Standardized Operating Procedures (SOPs) across various domains (Belbin, 2012: Manifesto, 2001; DeMarco & Lister, 2013). These SOPs play a critical role supporting task decomposition and effective coor- dination. Furthermore, SOPs outline the responsibilities of each team member, while establishing standards for intermediate outputs. Well-defined SOPs improve the consistent and accurate exe- cution of tasks that align with defined roles and quality standards (Belbin, 2012; Manifesto, 2001; DeMarco & Lister, 2013; Wooldridge & Jennings, 1998). For instance, in a software company, Product Managers analyze competition and user needs to create Product Requirements Documents (PRDs) using a standardized structure, to guide the developmental process.
Inspired by such ideas, we design promising GPT-based Meta- Prataprrorammnngggggg framework called MetaGPT that significantly benefits from SOPs. Unlike other works (Li et al., 2023; Qian et al., 2023), MetaGPT requires agents to generate structured outputs, such high-quality requirements
"These authors contributed equally to this work. 1 Chenglin Wu (alexanderwu @ fuzhi.ai) the corresponding author, affiliated with DeepWisdom.

---

Preprint
MetaGPT Agents Collaboration with Developing SOP
Human interaction
Requirement document
1/5 Define
Oue-line requirement
Planning
Write classic and simple Plappy Bird game.
Product Manuger
Requirement Analysis
System design
Meta Programming
2/5 Design
Architect
Architectural Design
Tasks
Boss makes acceptance check and payment
System Design
Project Manager
3/5
Human Developing SOP
Code
Plan&Code
Pretty good directty interface and keyboard to play Flappy Bird.
Coding
Engineer
4/5
Testing
QA Engineer
Acceptance Check
5/5
Figure 1: The software development SOPs between MetaGPT and real-world human teams. In software engineering, SOPs promote collaboration among various roles. MetaGPT showcases its ability decompose complex tasks into specific actionable procedures assigned to various roles (e.g., Product Manager, Architect, Engineer, etc.).
documents, design artifacts, flowcharts, and interface specifications. The use of intermediate struc- tured outputs significantly increases the success rate of target code generation. Because it helps maintain consistency in communication, minimizing ambiguities and errors during collaboration. More graphically, in company simulated by MetaGPT, all employees follow a strict and stream- lined workflow, and all their handovers must comply with certain established standards. This reduces the risk of hallucinations caused by idle chatter between LLMs, particularly in role-playing frame- works, like: "Hi, hello and how are you?" Alice (Product Manager); "Great! Have you had lunch?" Bob (Architect).
Benefiting from SOPs, MetaGPT offers promising approach to meta- pottaprrggammingg In this context, we adopt meta prograarggammnnn as "programming to program", in contrast the broader fields of meta learning and "learning to learn" (Schnidhuber, 1987; 1993a; Hochreiter et al., 2001; Schmidhuber, 2006; Finn al., 2017).
This notion of meta- programming also encompasses earlier efforts like CodeBERT (Feng et al., 2020) and recent projects such as Codel (Rozière al., 2023) and WizardCoder (Luo et al., 2023). However, MetaGPT stands out unique solution that allows for efficient meta- programming through well-organized group of specialized agents. Each agent has a specific role and expertise, following some established standards. This allows for automatic requirement analysis, system design, code generation, modification, execution, and debugging during runtime, highlight- ing how agent-based techniques enhance meta-programmmig.
To validate the design of MetaGPT, we use publicly available HumanEval (Chen et al., 2021a) and MBPP (Austin et al., 2021) for evaluations. Notably, in code generation benchmarks, MetaGPT achieves a new state-of-the-art (SoTA) with 85.9% and 87.7% in Pass@1 1. When compared to other popular frameworks for creating complex software projects, such as AutoGPT (Torantulino al., 2023), LangChain (Chase, 2022), AgentVerse (Chen et al., 2023), and ChatDev (Qian et al., 2023). MetaGPT also stands out in handling higher levels of software complexity and offering extensive functionality. Remarkably, in our experimental evaluations, MetaGPT achieves 100% task com- pletion rate, demonstrating the robustness and efficiency (time and token costs) our design.
We summarize our contributions as follows:
hhwps://een http:///wwewiiiiliaaooooooooiioe
2

---

Preprint
We introduce MetaGPT, prograanggn framework for multi-agent collaboration based on LLMs. It is highly convenient and flexible, with well-defined functions like role definition and message sharing, making it a useful platform for developing LLM-based multi-agent systems.
Our innovative integration of human-like SOPs throughout MetaGPT's design significantly en- hances its robustness, reducing unproductive collaboration among LLM-based agents. Furthermore, we introduce a novel executive feedback mechanism that debugs and executes code during runtime, significantly elevating code generation quality (e.g.. 5.4% absolute improvement on MBPP).
We achieve state-of-the-art performance on HumanEval (Chen et al., 2021a) and MBPP (Austin al., 2021). Extensive results convincingly validate MetaGPT, suggesting that it is promising meta-programming framework for developing LLM-based multi-agent systems.
2
RELATED WORK
Automatic Programming The roots of automatic programming reach back deep into the previ- ous century. In 1969, Waldinger & Lee (1969) introduced "PROW," system designed to accept program specifications written in predicate calculus, generate algorithms, and create LISP imple- mentations (McCarthy, 1978). Balzer (1985) and Soloway (1986) made efforts to advance auto- matic programming and identified potential methods to achieve it. Recent approaches use natural language processing (NLP) techniques (Ni et al., 2023; Skreta et al., 2023; Feng et al., 2020; Li et al., 2022; Chen et al., 2018; b02 Zhang et al., 2023). Automatic programming has grown into an industry delivering paid functions such as Microsoft Copilot. Lately, LLMs-based agents (Yao et al., 2022; Shinn et al., 2023; Lin et al., 2023) have advanced automatic programming develop- ment. Among them, ReAct (Yao et al., 2022) and Reflexion (Shinn et al., 2023) utilize a chain of thought prompts (Wei et al., 2022) to generate reasoning trajectories and action plans with LLMs. Both works demonstrate the effectiveness of the ReAct style loop of reasoning as a design paradigm for empowering automatic programming. Additionally, ToolFormer (Schick et al., 2023) can learn how to use external tools through simple APIs The research most closely aligned with our work by Li al. (2023) proposes a straightforward role-play framework for programming that involves communication between agents playing different roles. Qian et al. (2023) utilizes multiple agents for software development. Although existing papers (Li al., 2023; Qian et al., 2023) have improved productivity, they have not fully tapped into effective workflows with structured output formats. This makes it harder to deal with complex software engineering issues.
LLM-Based Multi-Agent Frameworks Recently, LLM-based autonomous agents have gained tremendous interest in both industry and academia (Wang et al., 2023b). Many works (Chen et al., 2024: Wang et al., 2023c; Du et al., 2023; Zhuge al., 2023; Hao et al., 2023; Akata et al., 2023) have improved the problem-solving abilities of LLMs by integrating discussions among multiple agents. Stable-Alignment (Liu et al., 2023) creates instruction datasets by deriving consensus on value judgments through interactions across a sandbox with LLM agents. Other works focus on sociological phenomena. For example, Generative Agents (Park et al., 2023) creates a "town" of 25 agents to study language interaction, social understanding, and collective memory. In the Natural Language-Bar -Basd Society of Mind (NLSOM) (Zhuge al., 2023), agents with different functions interact to solve complex tasks through multiple rounds of "mindstorms." Cai et al. (2023) propose a model for cost reduction by combining large models as tool makers and small models as tool users.
Some works emphasize cooperation and competition related to planning and strategy (Bakhtin et al., 2022); others propose I.L.- economies (Zhuge et al., 2023). These works focus on open- world human behavior simulation, while MetaGPT aims to introduce human practice into multi- agents frameworks. Besides, LLM-based agents face the challenges of "assistant repeated instruc- tion" or "infinite loop of message" (Talebirad & Nadiri, 2023; Li et al., 2023). These challenges become more urgent in task orientede collaborations, which require consistent and mutually benefi- cial interactions (Elazar et al., 2021; Wang et al., 2022; Jiang et al., 2023). This motivates focus on applying advanced concepts such as Standard Operating Procedures in software development to multi-agent frameworks.
3

---

Preprint
Name
Agent Profile
Profie
Engineer
Write olegani, cestibl,, efficient code
The code you conform to code PEPS, be modalar, easy to read and maintain
Architect diagram
Project Managee
diagram
Memory
Memory Retriezal
mayA
content: (Archieee:: implementaiion appro..)
subscription
Engineer
#F Codo: gonte.p
Iterative Programming
msgli
by: WaiteTasks
content structures
S
+
import random
game.ay
Ilhoory Message
megC
SITK, from: ProjesMamaper
from:
Eagineer
PRU tbeaument
megD
to: Engineer
publish
Werte
cleres Gormen
Isetf, sizenslt
Code
Shared Message Pool
rangetsizell
Product Manager
Engincer
search tool
Debrur
Excentable Feedhack
debugzing diagram
Structured Messagel...)
An example of the communication protocol (left) and iterative programming with exe- cutable feedback (right). Left: Agents use a shared message pool to publish structured messages. They can also subscribe to relevant messages based on their profiles. Right: After generating the initial code, the Engineer agent runs and checks for errors. If errors occur, the agent checks past messages stored in memory and compares them with the PRD, system design, and code files.
Figure 2:
METAGPT: A META-PROGRAMMING FRAMEWORK
MetaGPT is a meta-programming framework for LLM-based multi-agent systems. Sec. 3.1 pro- vides an explanation role specialization, workflow and structured communication in this frame- work, and illustrates how to organize multi-agent system within the context of SOPs. Sec. 3,2 presents a communication protocol that enhances role communication efficiency. We also imple- ment structured communication interfaces and an effective publish-subseribe mechanism. These methods enable agents obtain directional information from other roles and public information from the environment. Finally, we introduce executable feedback self-correction mechanism for further enhancing code generation quality during run-time Sec. 3.3.
3.1 AGENTS IN STANDARD OPERATING PROCEDURES
Specialization of Roles Unambiguous role specialization enables the breakdown of complex work into smaller and more specific tasks. Solving complex tasks or problems often requires the collab- oration of agents with diverse skills and expertise, each contributing specialized outputs tailored to specific issues.
In software company, Product Manager typically conducts business-oriented analysis and derives insights, while software engineer is responsible for programming. We define five roles in our software company: Product Manager, Architect, Project Manager, Engineer, and QA Engineer, as shown in Figure 1. In MetaGPT, we specify the agent's profile, which includes their name, profile, goal, and constraints for each role. We also initialize the specific context and skills for each role. For instance, a Product Manager can use web search tools, while an Engineer can execute code, as shown in Figure 2. All agents adhere to the React-style behavior described in Yao et (2022).
Every agent monitors the environment (i.e., the message pool in MetaGPT) to spot important ob- servations (e.g.. messages from other agents). These messages can either directly trigger actions or assist in finishing the job.
Workflow across Agents
By defining the agents' roles and operational skills, establish basic workflows. our work, we follow SOP in software development, which enables all agents to work in sequential manner.

---

Preprint
Make 2048 aliding number puzzle game
Human Input
MotaGPP
Product
NctinnWrineRR
Ardhieet
Think: requires creation sliding number
Think: For this project, will Pygame, designed making with Python.
library
od Prodact Goals
Progrum
"Coste web hase 2008 gone flat is play"
fraure echonce exgageeent retratoo
"A suc pame be aupure
It Competitve Analysis
Ht Compeciive Quadrant Chart
by
original 2048 game, simple and may...
Al File list
("muirayy" "prmpp""""app"".. "aaateap"" "se hhd datituncpp"!
2008 Keschopp: Offers
variety of gane moko...
Cadia
by Andhobsby ineefaaee and light soore taakiing...
Action Wirtee ede
Think: For this project, we will use Pygams, an upen-souree library designed for making video games with Python.
5t Requirement Analysis
The prodect aboald be 2046 sisding liks canhar puack Jane which sar-diondly.
Code: gome.y
Repulcement
gamw-ay Import random
"arpienarr eapaging paneçlay zeckaanks" "PV.
closs 430-43
Project Menager Pic
sey.stbe
smyf-score =
Think: List out the task list that movts the requirements and design. ty Shared Knowledge
Caciel
sey/boord [[O"siae for_. rangesizell
"pane.ry cooriss Grard class, includes gioee logit..
AU Tack List
dlff stortsaetf::
"Sttrris the game by adding two numbers the board. sey add. new tile))
["ooings" "gronpp" "gripp"" "testoos py"
Logic Analysis
"rols py "Cossiis main grae toep and hucilles user iepu."
pams-yy
"pancey" "Impleeent du gare ligit, inclaling ihe exes, and pars ona conditses"
Cadier
Caskeso
QA Enpiscer
gimeplay.
(25]
project, Pygane, an open-source library
designed making viden gumes with Python.
Coda qualiy sivee
Figure 3: A diagram showing the software development process in MetaGPT, emphasizing its sig- nificant dependence on SOPs. The more detailed demonstration can be found in Appendix B.
Specifically, as shown in Figure 1, upon obtaining user requirements, the Product Manager under- takes thorough analysis, formulating a detailed PRD that includes User Stories and Requirement Pool. This serves as preliminary functional breakdown. The structured PRD is then passed to the Architect, who translates the requirements into system design components, such as File Lists, Data Structures, and Interface Definitions. Once captured in the system design, the information is directed towards the Project Manager for task distribution. Engineers proceed to execute the des- ignated classes and functions outlined (detailed in Figure 2). In the following stage, the QA Engineer formulates test cases to enforce stringent code quality. In the final step, MetaGPT pro- duces meticulously crafted software solution. We provide detailed schematic (Figure 3) and concrete instance (Appendix B) the SOP workflow in MetaGPT.
3.2 COMMUNICATION PROTOCOL
Structured Communication Interfaces Most current LLM-based multi-agent frameworks (Li et al., 2023; Zhuge et al., 2023; Zhang et al., 2023; Park et al., 2023) utilize unconstrained natural language communication interface.
However, despite the versatility of natural language, a question arises: does pure natural language communication suffice for solving complex tasks? For example, in the telephone game (or Chinese
5

---

Preprint
whispers)", after several rounds of communication, the original information may be quite distorted. Inspired by human social structures, we propose using structured communication to formulate the communication of agents. We establish a schema and format for each role and request that individ- uals provide the necessary outputs based on their specific role and context.
As shown in Figure 3, the Architect agent generates two outputs: the system interface design and sequence flow diagram. These contain system module design and interaction sequences, which serve as important deliverables for Engineers. Unlike ChatDev (Zhao al., 2023), agents in MetaGPT communicate through documents and diagrams (structured outputs) rather than dialogue. These documents contain all necessary information, preventing irrelevant or missing content.
Sublish-Subscribe Mechanism Sharing information is critical in collaboration. For instance, Architects and Engineers often need reference PRDs. However, communicating this information each time in a one-to-one manner, indicated by previous work (Li et al., 2023; Zhao et al., 2023; Zhang et al., 2023), can complicate the communication topology, resulting inefficiencies.
To address this challenge, a viable approach is to store information in a global message pool. As shown in Figure 2 (left), we introduce shared message pool that allows agents to exchange messages directly. These agents not only publish their structured messages in the pool but also access messages from other entities transparently. Any agent can directly retrieve required information from the shared pool, eliminating the need to inquire about other agents and await their responses. This enhances communication efficiency.
Sharing all information with every agent lead to information overload. During task execution, an agent typically prefers to receive only task-related information and avoid distractions through irrelevant details. Effective management and dissemination of this information play crucial role. We offer a simple and effective solution-suussciition mechanism (in Figure 2 (left)). Instead of relying on dialogue, agents utilize role-specific interests to extract relevant information. They can select information to follow based on their role profiles. In practical implementations, an agent activates its action only after receiving all its prerequisite dependencies. As illustrated Figure 3, the Architect mainly focuses on PRDs provided by the Product Manager, while documents from roles such the QA Engineer might be of lesser concern.
3.3 ITERATIVE PROGRAMMING WITH EXECUTABLE FEEDBACK
In daily programming tasks, the processes of debugging and optimization play important roles. However, existing methods often lack self-correction mechanism, which leads to unsuccessful code generation. Previous work introduced non-executable code review and self-reflection (Zhao et al., 2023; Yao et al., 2022: Shinn et al., 2023; Dong et al., 2023). However, they still face challenges in ensuring code executability and runtime correctness.
Our first MetaGPT implementations overlooked certain errors during the review process, due to LLM hallucinations (Manakul et al., 2023). To overcome this, after initial code generation, introduce an executable feedback mechanism to improve the code iteratively. More specifically, as shown in Figure the Engineer is asked to write code based the original product requirements and design.
This enables the Engineer continuously improve code using its historical execution and debugging memory. To obtain additional information, the Engineer writes and executes the corre- sponding unit test cases, and subsequently receives the test results. If satisfactory, additional devel- opment tasks are initiated. Otherwise the Engineer debugs the code before resuming programming. This iterative testing process continues until the test passed or a maximum of retries is reached.
EXPERIMENTS
4.1 EXPERIMENTAL SETTING
Datasets We use two public benchmarks, HumanEval (Chen et al., 2021a) and MBPP (Austin al., 2021), and self generaede,,,,d, more challenging software development benchmark named Soft-
hhtp:://wwwwiinnniiiiinnhihhiiet whispers

---

Preprint
wareDev: (1) HumanEval includes 164 handwritten programming tasks. These tasks encompass function specifications, descriptions, reference codes, and tests. (2) MBPP consists of 427 Python tasks. These tasks cover core concepts and standard library features and include descriptions, ref- erence codes, and automated tests. (3) Our SoftwareDev dataset is a collection of 70 representa- tive examples of software development tasks, each with its own task prompt (see Table 8). These tasks have diverse scopes (See Figure 5), such as mini-games, image processing algorithms, data visualization. They offer robust testbed for authentic development tasks. Contrary to previous datasets (Chen et al., 2021a; la; Austin et al., 2021), SoftwareDev focuses on the engineering aspects. In the comparisons, we randomly select seven representative tasks for evaluation.
For HuamnEval and MBPP, we follow the unbiased version of Pass @k as presented by (Chen et al., 2021a; Dong et al., 2023), to evaluate the functional accuracy of the top-k
Evaluation Metrics
generated codes: Pass @k
Erroblems
For SoftwareDev, we prioritize practical use and evaluate performance through human evaluations (A, E) or statistical analysis (B, C, D): (A) Executability: this metric rates code from (failure/non- functional) to 4 (flawless). '1' is for funccucttionl, '2' for runnable but imperfect, '3' for nearly perfect, and '4' for flawless code. (B) Cost: the cost evaluations here include the (1) running time, (2) token usage, and (3) expenses. (C) Code Statistics: this includes (1) code files, (2) lines of code per file, and (3) total code lines. (D) Productivity: basically,
defined as the number of token usage divided by the number of lines of code, which refers to the consumption of tokens per code line. (E) Human Revision Cost: refers to times of manual code corrections, which tackle problems like package import errors, incorrect class names, incomplete reference paths. Typically, each correction involves up to 3 lines of code.
Baselines We compare our method with recent domain-specific LLMs in the code generation field, including AlphaCode (Li et al., 2022), Incoder (Fried et al., 2022), CodeGeeX (Zheng et al., 2023), CodeGen (Nijkamp et al., 2023), CodeX (Chen et al., 2021a), and CodeT (Chen et al., 2022) and general domain LLMs such as PaLM (Chowdhery et al., 2022), and GPT-4 (OpenAI, 2023). Several results of baselines (such as Incoder, CodeGeeX) are provided by Dong et al. (2023). In HumanEval and MBPP, we slightly modified the prompts align with response format requirements. These modifications aim to address format-specins issues (i.e., Python problems). With the SoftwareDev benchmark, we provide comprehensive comparison between MetaGPT, AutoGPT (Torantulino et al., 2023), LangChain (Chase, 2022) with Python Gead-Eval-Prins Loop (REPL. tool?, Agent- Verse (Chen et al., 2023), and ChatDev (Qian et al., 2023).
4.2
MAIN RESULT
HumanEval MBPP
Pass@1 1 of MBPP and HumanEval (%)
82.3
87.7
08.8
67.7
58.1
38.6
35.0
26.9
17.1
152
17.6
AlphaCode
Incoder (8.7
CodeGeex (138)
CodeGeeX-Mono
PAlM Coder (5408)
Godex (1758)
Codex CodeT
Q
MetaGPT (wo Faedback)
MetaGPT
(1. 1B)
(16. 18)
Figure Pass rates on the MBPP and HvmanEval with a single attempt.
Performance Figure 4 demonstrates that MetaGPT outperforms all preceding approaches in both HumanEyal and MBPP benchmarks. When MetaGPT collaborates with GPT-4, significantly im- proves the Pass @k in the HumanEval benchmark compared to GPT-4. It achieves 85.9% and 87.7%
hhhps://en http:///ewwwiinppiiiinnnnniiiiia loop

---

Preprint
Music Website
META GPT
Excel Processor
Pooreer
Frenet Inest econamic China
Figure 5: Demo softwares developed by MetaGPT.
in these two public benchmarks. Moreover, as shown in Table 1, MetaGPT outperforms ChatDev on the challenging SoftwareDev dataset in nearly all metrics. For example, considering the executabil- ity, MetaGPT achieves a score 3.75, which is very close to 4 (flawless). Besides, it takes less time (503 seconds), clearly less than ChatDev. Considering the code statistic and the cost of human revi- sion, also significantly outperforms ChatDev. Although MetaGPT requires more tokens (24,613 or 31,255 compared to 19,292), it needs only 126.5/124.3 tokens to generate one line of code. In contrast, ChatDev uses 248.9 tokens. These results highlight the benefits of SOPs in collabora- tions between multiple agents. Additionally, we demonstrate the autonomous software generation capabilities of MetaGPT through visualization samples (Figure 5). For additional experiments and analysis, please refer to Appendix
Table 1: The statistical analysis on SoftwareDev.
Statistical Index
ChatDev
MetaGPT w/o Feedback
MctaGPT
(A) Executability
2.25
3.67
3.75
(B) Cost#1: I: Running Times (s)
762
503
541
(B) Cost#2: Token Usage
19,292
24,613
31,255
(C) Code Statistic#1: Code Files
1.9
4.6
5.1
(C) Code Statistic#2: Lines of Code per File
40.8
42.3
49.3
(C) Code Statistic/33 Total Code Lines
77.5
194.6
251.4
(D) Productivity
248.9
126.5
124.3
(E) Human Revision Cost
2.5
2.25
0.83
4.3 CAPABILITIES ANALYSIS
Compared open-source baseline methods such as AutoGPT and autonomous agents such as Veent and ChatDev, MetaGPT offers functions for software engineering tasks. As presented in Table 2, our framework encompasses wide range of abilities handle complex and specialized development tasks efficiently. Incorporating SOPs (e.g.. role-play expertise, structured communi- cation, streamlined workflow) significantly improve code generation. Other baseline methods
8

---

Preprint
Table 2: Comparison of capabilities for MetaGPT and other approaches.
indicates the
presence of specific feature in the corresponding framework, * its absence.
Framework Capabiliy
AutoGPT LangChain AgentVerse ChatDev MetaGPT
PRD generation
Tenical design genenration
X
X
API interface generation
X
X
X
Code generation
/ /
/
Precompilation execution
X
X
X
Role-based task management
X
X
Code review
Table 3: Ablation study roles. "#' denotes 'The number of', 'Product' denotes 'Product man-
ager', and 'Project' denotes Projecct manager'. 'V indicates the addition of specific role. 'Revi- sions' refers to 'Human Revision Cost'. Engineer Product Architect Project #Agents #Lines
Expense
Revisions
Executability
83.0
$ 0,915
10
1.0
2
112.0
$ 1.059
6.5
2.0
3
143.0
$ 1.204
4.0
2.5
X
205.0
1.251
3.5
191.0
$ 1.385
2.5
4.0
can easily integrate SOP-like designs improve their performance, similar to injecting chain-of- thought (Wei et al., 2022) in LLMs.
4.4 ABLATION STUDY
The Effectiveness of Roles To understand the impact of different roles on the final results, we perform two tasks that involve generating effective code and calculating average statisties. When exclude certain roles, unworkable codes are generated. As indicated by Table 3, the addition of roles different from just the Engineer consistently improves both revisions and executability. While more roles slightly increase the expenses, the overall performance improves noticeably. demonstrating the effectiveness of the various roles.
The Effectiveness of Executable Feedback Mechanism As shown in Figure 4, adding executable feedback into MetaGPT leads significant improvement of 4.2% and 5.4% in Pass @1 on Hu- manEval and MBPP, respectively. Besides, Table shows that the feedback mechanism improves feasibility (3.67 to 3.75) and reduces the cost of human revisions (2.25 0.83). These results illustrate how our designed feedback mechanism can produce higher-quality code. Additional quan- titative results of MetaGPT and MetaGPT without executable feedback are shown in Table 4 and Table 9.
5 CONCLUSION
This work introduces MetaGPT, a novel meta-programming framework that leverages SOPs to en- hance the problem-solving capabilities of multi-agent systems based on Large Language Models (LLMs). MetaGPT models a group of agents as a simulated software company, analogous to simu- lated towns (Park al., 2023) and the Minecraft Sandbox in Voyager (Wang al., 2023a). MetaGPT leverages role specialization, workflow management, and efficient sharing mechanisms such as mes- sage pools and subscriptions, rendering it a flexible and portable platform for autonomous agents and multi-agent frameworks. It uses an executable feedback mechanism to enhance code generation quality during runtime. extensive experiments, MetaGPT achieves state-of-ther on multiple benchmarks. The successful integration of human-like SOPs inspires future research on numan-inspired techniques for artificial multi-agent systems. We also view our work as an early attempt to regulate M.LM-based multi-agent frameworks. See also the outlook (Appendix A).
performance

---

Preprint
Acknowledgement
We thank Sarah Salhi, the Exccutive Secretary of KAUST AI Initiative, and Yuhui Wang, Postdoc- toral Fellow at the KAUST AI Initiative, for helping to polish some of the text. We would like to express our gratitude to Wenyi Wang. a PhD student at the KAUST AI Initiative, for providing com- prehensive feedback on the paper and for helping draft the outlook (Appendix A) with Mingchen. We also thank Zongze Xu, the vice president of DeepWisdom, for providing illustrative materials for AgentStore.
Author Contributions
Sirui Hong conducted most of the experiments and designed the executable feedback module. She also led the initial version of the write-up, supported by Ceyao Zhang, and also by Jinlin Wang and Zili Wang. Mingchen Zhuge designed the self-improvemmnt module, discussed additional experi- ments, and led the current write-up. Jonathan Chen helped with the MBPP experiments, outlined the methods section, and contributed to the current write-up. Xiawu Zheng provided valuable guid- ance, reviewed and edited the paper. Yuheng Cheng contributed to the evaluation metric design and HumanEval experiments. Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Lingfeng Xiao helped with the MBPP experiments and comparisons open-source baseline methods. Chenyu Ran cre- ated most of the illustrative figures. Chenglin Wu is the CEO of DeepWisdom, initiated MetaGPT, made the most significant code contributions to it, and advised this project. Jürgen Schmidhuber, Director the Al Initiative at KAUST and Scientific Director of IDSIA, advised this project and helped with the write-up.
REFERENCES
Elif Akata, Lion Schulz, Julian Coda-Forno, Scong Joon Oh, Matthias Bethge, and Eric Schulz. Playing repeated games with large language models. arXiv preprint, 2023.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carric Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021.
Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, al. Human-level play in the game of diplomacy by com- bining language models with strategic reasoning. Science, 2022.
Robert Balzer. A 15 year perspective on automatic programming. TSE, 1985.
R.M. Belbin. Team Roles Work. Routledge, 2012. URL WtLLht:p:///okk www.ss.oooooooo......oooooo.oo.. co..c...u.. .o.uk/
books? www.HHIMIIAAAA MHIQAAAAQBAU,
Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as tool makers. arXiv preprint, 2023.
Harrison Chase. LangChain. https: https://githhb.t....
2022.
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests, 2022.
Jiaqi Chen, Yuxian Jiang. Jiachen Lu, and Li Zhang. S-agents: self-organizing agents in open-ended environment. arXiv preprint, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards. Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings. Mathiis Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code, 2021a
10

---

Preprint
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang. Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Agentverse: Facili- tating multi-agent collaboration and exploring emergent behaviors in agents, 2023.
Xinyun Chen, Chang Liu, and Dawn Song. Execution-guided neural program synthesis. In ICLR, 2018.
Xinyun Chen, Dawn Song, and Yuandong Tian. Latent execution for neural program synthesis beyond domain-specinf languages. NeurlPS, 2021b.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov. Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.
Douglas
DeMarco and T.R. Lister. Peopleware: Productive Projects and Teams. Addison Wesley,, 2013. URL ptttps:/bboos. googglooooooooooooooooooooooo e. co. .o. uk/ookk i d=DV1 SAQAAQBAJ.
Yihong Dong, Xue Jiang. Zhi Jin, and Ge Li. Self-collaboration code generation via chatgpt. arXiv preprint, 2023.
Yilun Du Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate, 2023.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy. Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language mod- els. TACL, 2021.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjum Shou, Bing Qin. Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint, 2020.
Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint, 2023.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- reta-learning for fast adaptation of deep networks. In ICML, 2017.
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. arXiv preprint, 2022.
Irving John Good. Speculations concerning the first ultraintelligent machine. Adv Comput., 1965.
Rui Hao, Linmei Hu, Weijian Qi, Qingliu Wu, Yiru Zhang, and Liqiang Nie. Chatllm network: More brains, more intelligence. arXiv preprint, 2023.
S. Hochreiter, S. Younger, and P. R. Conwell. Learning to learn using gradient descent. In Lecture Notes on Comp. Sei. 2 2130, Proc. Intl. Conf. on Artificial Newral Networks (ICANN-2001), pp. 87- 94. Springer: Berlin, Heidelberg, 2001.
Xue Jiang. Yihong Dong, Lecheng Wang, Qiwei Shang, and Ge Li. Self plafnillnng code generation with large language model. arXiv preprint, 2023.

---

Preprint
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large scale language model society. arXiv preprint, 2023.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 2022.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang. Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi- agent debate. arXiv preprint, 2023.
Bill Yuchen Lin, Yicheng Fu, Karina Yang, Prithviraj Ammanabrolu, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks. arXiv preprint, 2023.
Ruibo Liu, Ruixin Yang, Chenyan Jia, Ge Zhang, Denny Zhou, Andrew M Dai, Diyi Yang, and Soroush Vosoughi. Training socially aligned language models in simulated human society. arXiv preprint, 2023.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint, 2023.
Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hal- lucination detection for generative large language models. arXiv preprint, 2023.
Agile Manifesto. Manifesto for agile software development. Snowbird, UT, 2001.
John McCarthy, History of lisp. In History of programming languages. 1978.
Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang. and Xi Victoria Lin. Lever: Learning to verify landuage-to-code generation with execution. In /CML, 2023.
Erik Nijkamp. Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis, 2023.
OpenAL. Gpt-4 technical report, 2023.
Joon Sung Park. Joseph C O' Orien, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint, 2023.
Chen Qian, Xin Cong, Cheng Yang. Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development, 2023.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi. Jingyu Liu. Tal Remez, Jérémy Rapin, al. Code llama: Open foundation models for code. arXiv preprint, 2023.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprinr, 2023.
Schmidhuber. A self-referential weight matrix. In Proceedings of the International Conference Artificial Neural Networks, Amsterdam, pp. 446-451. Springer, 1993a.
J. Schmidhuber. Gödel machines: self-referential universal problem solvers making provably optimal self-imprrvemments Technical Report IDSIA-19-03 arXivics.LO003309448 v3, IDSIA, Manno-Lugano, Switzerland, December 2003.
J. Schmidhuber. Gödel machines: Fully self-referential optimal universal self-improvers. In B. Go- ertzel and C. Pennachin (eds.). Artificial General Intelligence, pp. 199-226. Springer Verlag, 2006. Variant available 1000399..
12

---

Preprint
J. Schmidhuber. Ultimate cognition à la Gödel. Cognirive Computation, 1(2):177-193, 2009.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, learning how to learn: the meta-meta-... hook. PhD thesis, 1987.
Jürgen Schmidhuber: A 'self-referential' weight matrix. ICANN'93: Proceedings of the Interna- tional Conference Artificial Neural Networks Amsterdam, The Netherlands 13-16 September 1993 3, 1993b.
Jürgen Schmidhuber. On learning to think: Algorithmic information theory for novel combinations of reinforcement learning controllers and recurrent neural world models. arXiv preprint, 2015.
Jürgen Schmidhuber, Jieyu Zhao, and Nicol N Schraudolph. Reinforcement learning with self-
modifying policies. In Learning to learn. 1998.
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint, 2023.
Marta Skreta, Naruki Yoshikawa, Sebastian Arellano-Rubach, Zhi Ji, Lasse Bjørn Kristensen, Kourosh Darvish, Alán Aspuru-Guzik, Florian Shkurti, and Animesh Garg. Errors are useful prompts: Instruction guided task programming with verifier-assisted iterative prompting. arXiv preprint, 2023.
Elliot Soloway. Learning to program learning to construct mechanisms and explanations. Com- mumications of the ACM, 1986.
Yashar Talebirad and Amirhossein Nadiri. Multi-agent collaboration: Hamesssing the power of intelligent I1m agents, 2023.
Torantulino et al. Auto-gpt. https://github.. wttp:/thww...commmi.ii.go. Gravitts/ Auto-GPT, 2023.
R. J. Waldinger and R. T. PROW: and M. Norton (eds.), Proceedings of the International Joint Conference on Artificial Intel- ligence (ICCAI), 1969.
a step toward automatic program writing. In D. E. Walker
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint, 2023a.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. arXiv preprint, 2023b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- cry, and Denny Zhou, Self-consistnncy improves chain thought reasoning in language models. arXiv preprint, 2022.
Zhenhailong Wang. Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing cognitive synergy in large language models: A tosk-solving agent through collaboration. arXiv preprint, 2023c.
self-
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi. Quoe V Le, Denny Zhou, et al. Chain-of-thor of-ttuughtt prompting elicits reasoning in large language models. NeurIPS, 2022.
Michael Wooldridge and Nicholas R. Jennings. Pitfalls of agent-oriented development. In Pro- ceedings of the Second International Conference on Autonomous Agents, 1998. URL https: //doi.org/11 10. 111.//...111//165.11 280886.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint, 2022.
Eric Zelikman, Eliana Lorch, Lester Mackey, and Adam Tauman Kalai. Self-taught optimizer (stop): Recursively self-improving code generation. arXiv preprint, 2023.
13

---

Preprint
Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tian- min Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models. arXiv preprint, 2023.
Xufeng Zhao, Mengdi Li, Comeelus Weber, Muhammad Burhan Hafez, and Stefan Wermter. Chat with the environment: Interactive multimodal perception using large language models. arXiv preprint, 2023.
Qinkai Zheng. Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x, 2023.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint, 2023.
Mingehen Zhuge, Haozhe Liu, Francesco Faccio, Dylan R Ashley, Róbert Csordás, Anand Gopalakrishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Hermann, Kazuki Irie, et al. Mindstorms in natural language-bas based societies of mind. arXiv preprint, 2023.
14

---

Preprint
A OUTLOOK
A. IELF-EMPROV MECHANISMS
One limitation of the MetaGPT version in the main text of this paper that each software project executed independently. However, through active teamwork, software development team should learn from the experience gained by developing each project, thus becoming more compatible and successful over time.
This somewhat related to the idea of recursive self-improvement, first informally proposed in 1965 (Good, 1965), with first concrete implementations since 1987 (Schmidhuber, 1987; 1993b; Schmidhuber et al., 1998), culminating in the concept of mathematically optimal self-referential self-improvers (Schmidhuber, 2003; 2009). Generally speaking, system should learn from experi- ence in the real world, and meta-learn better learning algorithms from experiences of learning, and meta-meta-leaam better meta-learing algorithms from experiences of meta-learning, etc., without any limitations except those of computability and physics.
More recent, somewhat related work leverages the reasoning ability of Large Language Models (LLMs) and recursively improves prompts of L.MMs, to improve performance on certain downstream tasks (Fernando et al., 2023; Zelikman et al., 2023), analogous the adaptive prompt engineer of 2015 (Schmidhuber, 2015) where one neural network learns to generate sequence of queries or prompts for another pre-trained neural network whose answers may help the first network to learn new tasks more quickly.
In our present work, we also explore a self-referential mechanism that recursively modifies the con- straint prompts of agents based on information they observe during software development. Our initial implementation works as follows. Prior to each project, every agent in the software company reviews previous feedback and makes necessary adjustments to their constraint prompts. This en- ables them to continuously learn from past project experiences and enhance the overall multi-agent system by improving each individual in the company. We first establish a handover feedback action for each agent. This action is responsible for critically summarizing the information received dur- ing the development of previous projects and integrating this information in an updated constraint prompt. The summarized information is stored long-term memory such that can be inherited by future constraint prompt updates. When initiating new project, each agent starts with react action. Each agent evaluates the received feedback and summarizes how they can improve in a constraint prompt.
One current limitation is that these summary-basee optimizations only modify constraints in the specialization of roles (See. 3.1) rather than structured communication interfaces in communication protocols (Sec. 3.2). Future advancements are yet to be explored.
A.2 MULTI-AGENT ECONOMIES
In real-world teamwork, the interaction processes are often not hardcoded. For example, in a soft- ware company, the collaboration SOP may change dynamically.
One implementation of such erf-orgaanzttionn is discussed in the paper on a "Natural Language- Based Society of Mind" (NLSOM) (Zhuge et al., 2023), which introduced the idea of an "Economy of Minds" (HOM), a Reinforcement Learning (RL) framework for societies of LLMs and other agents. Instead of using standard RI. techniques to optimize the total reward of the system through modifications of neural network parameters, EOMs use the principles of supply and demand in free markets assign credit (money) those agents that contribute to economic success (reward).
The recent agent-based platform of DeepWisdom (AgentStore") is compatible with the credit as- signment concept of EOMs. Each agent in AgentStore provides a list of services with corresponding costs. A convenient API is provided so that human users or agents in the platform can easily pur- chase services from other agents to accomplish their services. Figure 6 displays the User Interface (UT) of AgentStore, where various agents with different skills are showcased. Besides, individual developers can participate in building new agents and enable collaborative development within the community. Specifically, AgentStore allows users to subscribe to agents according to their demands
hhap://tettadeeppiiddomai
15

---

Preprint
and pay according to their usage. Moreover, users purchase additional capabilities expand the plug-and and--aadyyyy functions of their existing agents. This allows users to gradually upgrade their agents. support the collaboration of various agents. Users
Within the MetaGPT framework, AgentStore can collect several agents together to carry out more complex tasks or projects, and all the agents share and comply with development and communication protocols defined in MetaGPT.
Mieva
Wedeeslay
Pate
Belweer Com
Contourthh
Senal
Aali
Sourch
Offid leaiiin
Wester ceabirt
HusoicaFige guce
the ceriirn photo
relo un canc mes
wac hevs pheee
Lamvii relerngg moder
seber dooer Coersnning sbase
emock
Mends
Coffer
Vedicoator Adders
Albett Diirten affer
Figure 6: AgentStore is platform dedicated to serving users in the creation and development of agents within the MetaGPT framework. This platform provides users with an operational interface, allowing users to easily manage variety of agents with different emotions, personalities, and capa- bilities for specific tasks.
16

---

Preprint
B A DEMO OF THE EXECUTION
In this section, we outline the complete process of software development using MetaGPT. It begins with user's imput command (as shown in Appendix B.1) and ends with software designed according to the user's specifications.
B.
USER INPUT
Upon receiving an instruction from the user, MetaGPT collaborates with a professional development team fulfill the task. Here demo of user input:
User input
python3 app you imago
B.2 METAGPT DEVELOPMENT PROCESS
Now we provide step-by-sted explanation the standardized output process for each agent.
Product Manager The Product Manager generates a Product Requirement Document (PRD), as detailed in the specified documentation. This document encompasses goals, user stories, competitive analysis, requirement analysis and requirement pool. Additionally, a competitive quadrant chart is produced (see Figure 7). Subsequently, these documents and charts are handed over to the architect for system design.
Product Requirement Document
Requirements
The basd requices a Python3 GUI color metor that cutputs the RGB values whon it noves.
Product Goals "sython
"Create a user-triendly GUI color mecor", the coler "Ensure coler neter apdatea BGB values real moves"
outguts values",
10 1
11
12
Stories
14 "python
15
16
*as usery
R0B uswsy want Lhe valuea update seal cine muve oolos melec, can get, accurate color incormation
weleen any
17
18
*is
want single and intuit fow Interface, that can esaily the colos netwot
19
21
22
#4 Corpet lve Analyaia
23
"yyyhhon
25
*Calor Cups à populas colos pickus for Rindowa. ILfa simple and easy Lo uaw, bus lacka real- < time RCB value updates""
26
"Just Color Picker: offers real color pleking and miltiple color Formetss
interCace cluttered and vary user friendly",
27
*Iastant Fyedroppers A sinp'e e color picker that lives in the systen tray. It'ss convent ent but <a lacks advanced featurest,
*balor2illas color picker excension for browaera, It'ss convenient
design bus *allorPis offwew colar palallls and mixer luols, IL's Feature rich but can be oversheliing simple tacks"
suiteble for desktop appl lost lone",
29
17

---

Preprint
Product Requirement Document
Requisement Analysis
Tae product should ze EythorJ Gu: application that serves as 3 coler meter, It should allow usezs to select any color on the acrsen and display the cocraspsading RG2 values, The aG2 should update real-tine color aimple usez user-feiene feiendly.
The inzerface should
FE Requizement Pcol
"python
("Design uscr-triendly GIZ for the color noter", "20"),
("Inolement color selection functionality', 1834), ("Display 10
solected color", "20"1, ("UpdaLe veluwa seal Line cular neles noces", *p0*),
12
("Test the application for accuracy and pes portaanne"", *p1*)
13 ]
14 ...
15
16 #t UI Design droft
17 The OI should be ninimaliatic and intuilive. It should prinacily consial. of colos ae which of the could sacnified
usez around. The values current ly selected oolor should proninently displayed, updating real time the colos meser noves. The layeat abould be clean and atteed, with a Zocus on functionality and case of use.
18
Anything
20 There aze no uncleaz points.
Reach and engagement of campaigns
Need to promote
We should expand
Corel Painter
High Engagement
Adobe Photoshop
SkatchBook
Our Target ProdBinI
Ne
May improved
Low Engagement
MS
Low Reach
High Reach
Figure 7: The quadrant chart for the "Drawing App" generated by the product manager agent.
18

---

Preprint
Architect Based on the requirements in PRD, the Architect agent devises technical specifications including system architecture diagrams and interface definitions. Initially, the Architect defines the overarching technical trajectory. Subsequently, the project's architecture, including files, classes (Figure 8) and the sequence flow chart (Figure 12), is designed. The Architect's documentation is then given to the project manager for task allocation and execution.
System design
Implcmentation appzeach
We will use Pythen' a Tkinter library to create the GUI for the zolor moter, Exinter is a standard + Python interrace to the TK BUI tooikit and is included with most Pyshon installations It powarful
is simple
The color se celection tunctionality will be inglenenn emplemented using the PIL (PLLlow) library, which alox us to capture the scrcen's o pixel data. By noving the neuse, can gez the RGB values of real-time.
pixel
Ihe R22 values will be displayed on the eur asing Tkintor's tabsl widget. Wo will use Python's threading library so update the RE3 values in real-time ae the color meter noves.
application
tested accuracy performance using Python's unictest 1ibrary.
10
#t Python packags name
11
** 'python
"coior_nster"
13
14
list
"python
17
18
"main.pp",
19
"color_pcckzz_pr
"qui-p",,
21
"Lasta.py"
22
25
88 loything UECLEAR
26
The eequenteent in clear
ColorPicker
+get_rgb reb_rrgbbbttt at(x: int, y: int) Tuplefint, int, int]
uses
GUI
Soriit__oooooppaiiiieeeeeeee poolorrpickkr: ColorPicker)
+update rgb. lab_labb(rrgb:: Tuple(int, int, int]) +start()
Figure 8: Data structures and interface definitions for the "Drawing App" generated by the architect agent.
Project Manager The Project Manager breaks down the project into task list. Furthermore, each code file is analyzed based on its intended functionality and then treated as a separate task assigned to Engineers.
19

---

Preprint
main
ColorPicker
GUI
Instantiate
Instantiate(CP)
start(
User moves mouse
getrgb_aat ro__rg__a(xx, at(x, y)
rgb
update_ rgbb label(rgb)
main
ColorPicker
GUI
Figure 9: The program call flow for the "Drawing App" generated by the architect agent.
Tasks
Requized Python third-party
Mpython
L2in.er=e8.6
PilloxeeB.3.
Requized lanquage chird-pazty packages
11 "python
12
third party packages reguired in languages.
15
16
Full apec
"gython
19
20 No APIa used in this projec..
*e*
23
24 Togio Ana yys'
25 "gyython
27
P"eaingpy", "Contain ina the mein appl loa loop and initiallisss all the other compornnts "I, responds Lo nouse eventay and use the selected myy"" *fngi enents the tool sel selection interface,"),
28
("canses-py", 'Ipplmmnna the dowwing
tool color. f),
30
(Poolar picker.py', "Iaplementa the color select fon Inte:faas."l, le menager.py", "Implements the tile caving opening tunct ioneltoyyy
31
20

---

Preprint
Product Requirement Document
Task list "sython
"main.py",
"canvas.pp
"Loola.9y"
pe_korccck py*
#rsia_nnnager.po py"
11
12 68 Shared Knowledge
13 "yyython
15 The 'Tiinter' library is used foz creating the 6oI of the apolination, It provides various vidgets by 1ike frames and canvases which are used in this project.
16 'Pillow"
the (ille_anaggee.pp fos 17 Ihe 'main.py' file is the entry point of the applicztion. It initializes all the other components
ls
files.
opening image ciles.
< and stazta the application loup.
18 ***
20
21 #8 Anything UNCLEAR
project requicetents implementatioo approach However, need
Lean mebbera faniliar with Tkiater and libraries. If not, they may by need sens tine to learn these libraries bafore starting the develognenz.
21

---

Preprint
Engineer Given the provided file structure and function definitions, Engineer agent requires only fundamental development skills to complete the development tasks. Due to the large number of files, present only auto-generated code file here.
Generated code (color picker.py)
import ee
From tkinter inport colorchomer
class ColorPickers
der Init (selt, tk.Tk, color: "black"1; self.fremm th.Fram [ropt]
welf.color woloe
self-co'er crtt-oolor_uuttteeee but ck.Rctton/sell: S.frate, text Coloe', commandesel selelett _wwt__oooolee wtt.cco.cc.oottoo.pccc.c
11
def select_.moor (self):
12
color colerchosse. (rrchooses.asccooor))
13
selfr color
15
16
def pack Ise f, askxarge) celt.rromo.pack [x
18
19
def get col.oooorreoo (sel))
20
sclt.color
QA Engineer Upon receiving the code output from the Engineer, the QA Engineer generates unit test code and reviews it to identify and fix any bugs, ensuring high-quality software.
Generated Unit tests (test.py)
inport unittest
import
You night wane novn ColorPicker class andchas file, way pieker.y
from collor prockr import Coll ColorFioker
TesiColosPicker funilteat. (eaca
def set spt
selc.root (k.7k()
self.ep RolorPicker root)
12
det tearDown(seltt::
13
This la ingortant to close the t' window after tests sulf. dest
15
16
def test in'tial poll posloriaelf
17
Teot the defaull color dusing iniliel asset.ssert (selt.op.get coloooo,,,,, *slackf1
20
det tost_sott_nde prw__an_ppt_ccoooreeee
21
Test setting coler and retrieving
22
"4fffeFF* white color
23
selt.cp.coler nev_coloz
24
sel ass sp.get...eet................e..e color1,, new rev_color)
25
27
1f
pame
28
unictost.mainn
Output Ultimately, shown in Figure 10, MetaGPT generates functional application named "Drawing App".
22

---

Preprint
Drawing App
Galers
Save
Color
Pencil
Open
Brush
Eraser
Cancel
OK
Figure 10: The "Drawing App" generated by MetaGPT.
C EXPERIMENTS
C.I DETAILS THE SOFTWAREDEV DATASET
The SoftwareDev dataset includes 70 diverse software development tasks. Table displays the names and detailed prompts of 11 tasks within the dataset. Note that the first seven tasks listed are used in the main experiments of this paper.
C.2 ADDITIONAL RESULTS
Quantitative results of MetaGPT As shown in Table 4, MetaGPT achieves an average score of 3.9, surpassing ChatDev's 's score of 2.1 Zhao et al. (2023), which is based on the Chat chain. Compare the scores of general intelligent algorithms, including AutoGPT Torantulino et al. (2023), which all score 1.0, failing generate executable code. We observe that the generated code is often short, lacks comprehensive logic, and tends to fail to handle cross-file dependencies correctly.
While models such as AutoGPT (Torantulino et al., 2023), Langchain (Chase, 2022), and Agent- Verse (Chen et al., 2023) display robust general problem-solving capabilities, they lack an essential element for developing complex systems: systematically deconstructing requirements. Conversely, MetaGPT simplifies the process of transforming abstract requirements into detailed class and func- tion designs through specialized division of labor and SOPs workflow. When compared to Chat- Dev (Zhao et al., 2023), MetaGPT's structured messaging and feedback mechanisms not only reduce loss of communication information but also improve the execution of code.
Quantitative results of MetaGPT w/o executable feedback Table 9 presents the performance of MetaGPT with GPT-4 32K on tasks within the SoftwareDev dataset. also shows the average performance across all 70 tasks (in the last line). Note that the version of MetaGPT used here is the basic version without the executable feedback mechanism.
Quantitative results of MetaGPT with different LLMs on different LLMs, we randomly selected SoftwareDev tasks and conducted experiments using GPT-3.5 and Deepseek Coder 33B as backends. As shown in Table 5, the results indicate that although MetaGPT can complete tasks with these LLMs, using GPT-4 the backend yields superior performance.
To verify the performance of MetaGPT
hhap::/ddeppscccoodeeggihubbo
23

---

Preprint
Table Executability comparison. The executability scores are on grading system ranging from 'I' to '4'. A score of 'I' signifies complete failure, '2' denotes executable code, '3' represents
largely satisfying expected workflow, and '4' indicates a perfect match with expectations.
Task
AutoGPT
LangChain
AgentVerse
ChatDev
MetaGPT
Flappy bird
2
Tank battle game
2
4
2048 game
1
1
4
Snake game
4
Brick breaker game
Excel data process
4
4
CRUD manage
2
4
Average score
1.0
1.0
1.0
2.1
3.9
Table 5: Performance of MetaGPT on SoftwareDey using different LLMs as agent backends.
Model
Open source Time(/s) # Lines Executahility Revisions
MetaGPT (w/ GPT-3.5)
75.18
161.6
2.8
2.4
MetaGPT (w/ GPT-4)
X
552.94
178.2
3.8
1.2
MetaGPT (w/ Deepseek Coder 33B)
1186.20
120.2
1.4
2.6
Impact of Instruction Levels (High-level v.s. Detailed Instructions) Does the variation in the level of initial input from humans significantly influence performance outcomes? For examples:
1. High-level prompt: Create a brick breaker game.
2. Detailed prompt: Creating a brick breaker game. In a brick breaker game, the player typically controls paddle at the bottom of the screen to bounce ball towards wall of bricks. The goal is to break all the bricks by hitting them with the ball.
Additional experiments were conducted to investigate this aspect: we selected 5 tasks from Soft- wareDev, and constructed detailed prompts for them. Here are the experimental results:
Table 6: Impact of Instruction Levels. The executability is scored grading system ranging
from '1' to $4', A score of '1' signifies complete failure, '2' denotes runnable code, '3' represents
largely expected workflow, and '4' indicates perfect match to expectations.
Model
# Word Time(/s) Token usage Lines Executability Productivity Reversions
High-level
13.2
552.9
28384.2
178.2
3.8
163.8
1.2
Detailed
42.2
567.8
29657.0
257.0
4.0
118.0
1.6
We observe that: detailed prompts lead to better software projects with lower productivity ratios because of clearer requirements and functions, while simple inputs can still generate good enough software using MetaGPT with executability rating of 3.8, which is comparable to the detailed prompt scenario. (Note that, Productivity = Token usage / Total Code Lines. The lower this ratio, the better.)
The performance of GPT variants in HumanEval benchmark We use the GPT-4's 67% Hu- manEval score (OpenAl, 2023) as our baseline, acknowledging its acceptance in the HvmanEval benchmark. We further extend to experiments(five times) with GPT-4 (gpt-4-0613) and GPT-3.5- under various conditions to assess performance. (A) We directly called the OpenAl API with the prompt in HumanEval.
Turbo
(B) We called the OpenAl API and parsed the code with regex in the response. (C) We added an additional system prompt, then called the OpenAI API. The prompt is "You Al that only responds with Python code, NOT ENGLISH. You will
24

---

Preprint
be given a function signature and its docstring by the user. Write your full implementation (restate the function signature)." As shown in Table GPT-4 is more sensitive to prompt, code parser, and post-processing results the HumanEval data set. It is difficult for GPT-3.5-Turbo to return the correct completion code without prompt words.
Table 7: Performance of GPT models on HumanEval. Experiments were conducted five times using gpt-4-0613 and gpt-3.5-turb-06113 3 with different settings.
Settings
Model
1
2
3
4
5
Avg.
Std.
A
gpt-4-0613
0.732
0.707
0.732
0.713
0.738
0.724
0.013
A
gpt-3.5-turbo-6613
0.360
0.366
0.360
0.348
0.354
0.357
0.007
B
gpt-4-0613
0.787
0.811
0.817
0.829
0.817
0.812
0.016
B
gpt-3.5-turbo-613
0.348
0.354
0.348
0.335
0.348
0.346
0.007
C
gpt-4-0613 0.805
0.805
0.817
0.793
0.780
0.800
0.014
C
gpt-3.5-turbo-6613
0.585
0.567
0.573
0.579
0.579
0.577
0.007
Qualitative results Figure 11 and Figure 12 illustrate the outcomes of the Architect agent's ef- forts to design complex recommender system. These figures showcase comprehensive system interface design and program call flow. The latter is essential for creating sophisticated automated system. is crucial to emphasize the importance this division of labor developing an automated software framework.
D LIMITATION AND ETHICS CONCERNS
D.1 LIMITATION
System side At present, our system cannot fully cater to specific scenarios, such as UI and front- end, we have yet to incorporate such agents and multimodal tools. Furthermore, despite gen- erating the most amount of code among comparable frameworks, remains challenging to fulfill real-world applications' diverse and complex requirements.
Human user side A key challenge for users to interrupt the running process of each agent, or set the starting running point (checkpoint) for each agent.
D.2 ETHICS CONCERNS
Unemployment and Skill Obsolescence MetaGPT enables more people to program in natural languages, thereby making easier for engineers get started. Over the years, programming languages have evolved from punched cards to assembly. C, Java, Python, and now natural lan- guage. As result, humans have become more proficient programming, increasing the demand for programming-rrlatte positions. Furthermore, programming with natural language may offer significantly easier learning curve, making programming more accessible to broader audience.
Transparency and Accountability MetaGPT is an open-source framework that facilitates inter- active communication between multiple agents through natural language. Humans initiate, ob- serve, and stop running with the highest level of control. It provides real-time interpretation and op- eration of the natural language, displayed the screen and logs, ensuring transparency. MetaGPT enhances "natural language programming" capabilities, and human engineers are the users and re- sponsible for the outcomes.
Privacy and Data Security MetaGPT operates locally, ensuring user data privacy and security. does not collect user data. For interactions with thiry-party LLMs, such those by OpenAl, users are encouraged to refer to the respective privacy policies (e.g.. OpenAl Privacy Policy). However, provide the option of open-source I.LMss backends.
25

---

Preprint
ContentBesedFitteing Model
Advertising
Privacy
-DatsFrame
Jype
Iiit Juser: Userl None ronoure, privooyO None
Juser lnt, user. profie UserProfii, of model Sokibbooikss--hhengggeee cbt, modol Contenibogodiiiiiiggggee None
Jnit Lodvorttor jd int, target, audience stri --x None
User
Content
eine user.id
thte deccription
content_Jo
vetr emall
+DateeTime
+CateTime publiished..
IJHM stromall
desoription ot,, category:
Profile
Collaborative FilteringModll
Feedback
-int user, 3d
-dict preferencee
Cataframe cata eatr model type
+int user
-dien harory
+int content +int rating
Hit luser
spredictluser
proferencce: hissonn: dict) None
nusoomeendotooo Ustint)
intl
Recommender
elleeerrrtte protte
Callaborative Fitteing et model
coontentilaeedFFie Model model
Irii Juser id int, user profie:: UserProfte,
model CollatocatheFFitrrigggge obt modet ContomiiicodFiiirrngggee inghldell None
Experimentation Platform
Optimization
Monitoring
Init (nccommerder:: Jec-ecommedder] -- None
*.nit /mcommmmneer Recommender): +s None :oprimizel): -> None
Inft Inscommendden Recommender): None
inn, experimentuuorr Inc incommendaiions: Dietor, Licknnl
generste, (eppt/eepmmm -> None
Figure 11: The system interface design for "recommendation engine development" is generated by the architect agent (zoom in for better view).
E MORE DISCUSSIONS
E.1 DEEP-SEATED CHALLENGES
MetaGPT also alleviates or solves these challenges with its unique designs:
Use Context Efficiently Two sub-challenges are present. First, unfolding short natural language descriptions accurately to eliminate ambiguity. Second, maintaining information validity lengthy contexts, enables LLMs concentrate relevant data without distraction.
Reduce Hallucinations Using LLMs generate entire software programs faces code halluci- nation wroblems-includnng incomplete implementation of functions, missing dependencies, and potential undiscovered bugs, which may be more serious. L.MMs often struggle with software gen- eration due to vague task definitions. Focusing granular tasks like requirement analysis and package selection offers guided thinking, which LLMSs lack in broad task solving.
E.2 INFORMATION OVERLOAD
In MetaGPT, we use global message pool and a subscription mechanism to address "information overload," which refers to the problem of receiving excessive or irrelevant information. This issue dependent specific applications. MetaGPT employs message pool streamline communi- cation, ensuring efficiency. Additionally, a subscription mechanism filters out irrelevant contexts, enhancing the relevance and utility of the information. This design is particularly crucial in soft-
26

---

Preprint
User
UserProfile
Recommender
Collaboracive FiteringMode
ContentBased FittringMedel
Platform
Experimentation
Feedback
Optimization
Monitoring
Priwacy
Advertaiing
create UserProftl
Recommender
nt model
unn_epprrment))
predict))
submit feedback
update models
nt model
optimizel)
updatt models
pereratee_epartt
ensurn_priracyd
ensuro_priracy)
User
UserPrafte
Recommender
FitoringModal
Collaborative
FitaringModal
ContentBased
Platform
Experfmentation
Feedback
Optimization
Monttoring
Privay
Advertiingg
The program call flow for "recommendation engine development" generated by the architect agent (zoom in for a better view).
Figure 12:
ware design scenarios and standard operating procedures (SOPs) where effective communication is essential.
27

---

Preprint
Table 8: Examples of SoftwareDev dataset.
Task ID Task
Prompt
Snake game
Create a snake game.
1
Brick breaker game
Create a brick breaker game.
2
2048 game
Create a 2048 game the web.
3
Flappy bird game
Write p5.js code for Flappy Bird where you control yellow bird continu- ously flying between a series of green pipes. The bird flaps every time you left click the mouse. If it falls to the ground or hits pipe, you lose. This game goes indefinitely until you lose; you get points further you go. Create a tank battle game.
4
Tank battle game
5
Excel data process
Write excel data processing program based streamlit and pandas. The screen first shows an excel file upload button. After the excel file is uploaded, use pandas display its data content. The program is required to be concise. easy maintain, and over-designed. It streamlit process web screen displays. and pandas is sufficient to process excel reading and display. Please make sure others can execute directly without introducing additional packages.
6
CRUD manage
Write management program based the crud addition, deletion, modifi- cation and query processing of the customer business entity. The customer needs to save this information: name, birthday, age, sex, and phone. The data is stored client.db, and there a judgement whether the customer table ists. it doesn't, 't, it needs to be created first. Querying is done by name; same for deleting. The program is required to be concise, easy to maintain, and not over-designed. The screen realized through streamlit and sqlite-no need introduce other additional packages.
7
Music transcriber
Develop program to transcribe sheet music into a digital format; provid- ing error-free transcribed symbolized sheet music intelligence from audio through signal processing involving pitch and time slicing then training a neural net to run Onset Detected CWT transforming scalograms to chroma- grams decoded with Recursive Neural Network focused networks.
8
Custom releases
Create custom
releases; develop Python script that extracts rele- information about company news from external sources, such social media; extract update interval database for recent changes. The program should create press releases with customizable options and export writings PDFs, NYTimes API JSONs, media format styled with interlink internal fixed character-lenglt metadata.
9
Gomoku game
Implement Gomoku game using Python, incorporating an Al opponent with varying difficulty levels.
10
Weather dashboard
Create Python program to develop an interactive weather dashboard.
28

---

Preprint
Code exccutability
3.36
3
4
Table 9: Additional results of pure MetaGPT w/o feedback on SoftwareDev; Averages (Avg.) of 70 tasks are calculated and 10 randomly selected tasks are included. denotes The number of", while 'ID' is "Task ID'
lack
1. PNG file miss
ing 2. Compile bug fixes
PNG file missing 2. Com-
pile bug fixes 3. pygume. weressurffce not initialize
1. dependency er- ror 2. ModuleNot- FoundError
1. tensorflow ver-
sion error 2. model
ror 2. URL 403 er- ror dependency er-
1. dependency er- ror missing main func.
0.51 (only consider item scored 2, 3 or 4)
Cost of revision
training method implement
@ @app.route(")
TypeError
TypeError
1.
0
0
#oode files filines of code #liees per code file fddo files filines of doe #linees per doc file #prompt tokens ficompletion commplettonn tokens time costs money costs
$ 1..0
$ 1.35
1.04
S 1.00
$ 1.30
$ 1.51
$ 0.90
0.92
$ 1.27
$ 0.92
$1.12
$
Cost statistics
582.04
566.30
481.34
599.58
585. 10
398.83
435.83
621.73
550.88
438.50
516.71
553.
48
6157.00
6238.00 00
6316.00
5312.00
6567.00
7336.00
5832.00 00
7104.00 100
6499.00 .00000
5734,00
5363.00
6218 8.000
24488.000 1000
32517.00
21934.00
22951.00 1.00
30087.00
35590.00
25673.00 00
29139.00
29372.00
24799.00
24109.00
26626.86
70.00
76.67
78.33
67.33
70.00
966.00
87.00
110.67
100.33 33
90.00
84.67
80.00
Doc statistics
210.00
230.00
235.00
202.00
210.00
294.00
261.00
332.00
301.00
270.00
254.00
240.00
261
3.00
3.00
3.00
3.00
3.00
3.00
3.00
3.00
3.00
3.00
3
39.20
31.83
66.00
32.80
33.83
36.50
18.25
79.00
43.00
31.00
42.98
Code statistics
164
203.00 ,000
219.00 199.00
73.00
316.00
215.00 .000
215.00 .00
93.00
191.57
196.
191
198.
5.00
6.00
3,00
5.00
6.00
6.00
4.00
4.00
5.00
5.00
3.00
4.71
Avg.
ID
4
29


Reviews:
[{"review_id": "haoCfWsayk", "reviewer": "Reviewer_WkUQ", "summary": "MetaGPT is a framework designed for programming in multi-agent collaborative environments, specifically utilizing the Large Language Model (LLM). Its main contribution is the integration of human workflows into LLM-based multi-agent collaboration, which is effective for complex software engineering tasks. The framework assigns different roles to the GPT model, similar to a pipeline in software development, including roles such as product manager, architect, and engineer. Its main advantage lies in generating coherent solutions and decomposing complex tasks through the collaboration of multiple AI agents.", "questions": "1. My primary concern is that MetaGPT lacks methodological and theoretical innovation. To gain a more innovative edge, MetaGPT could deeply explore what deep-seated challenges in integrating newer AI techniques or unique collaboration strategies make it more clearly distinguishable from other LLM-based frameworks. (Honestly, I think MetaGPT is more suited for system demonstration conferences or tracks)\n\n2. In the Introduction, \"MetaGPT stands out as a special solution that allows efficient meta-programming through a well-organized and specialized group of agents\". How does the author define these professional concepts? For example, \"efficient\" and \"well-organized\".1) I haven't seen any evidence or convincing analysis of why it can be \"efficient\", nor have I seen experimental validation.2) How do you define \"specialized group\"? This seems trivial and can be varied.\n\n3. In the Introduction, \"MetaGPT achieves a 100% task completion rate, further demonstrating the robustness and efficiency of our design\", and \"Our innovative integration of human-like SOPs throughout MetaGPT’s design significantly enhances its robustness\". 1) What is the definition of \"robustness\" here? I didn't see any discussion of \"robustness\" except in the Introduction section. 2) What are the deeper methodological challenges of integrating human-like SOPs?\n\n4. In the Related Work, \"In this paper, we identify the main challenges in multi-agent cooperation: maintaining consistency, avoiding unproductive cycles, and controlling beneficial interactions\". There seems to be inconsistency in the description of the research challenges between the \"Related Work\", the \"Introduction\" and the rest of the paper. Furthermore, is it trivial to maintain consistency in MetaGPT?\" What are the professional definitions of \"unproductive cycle\" and \"beneficial\"? Are there experimental results to validate these advantages?\n\n5. In the Experiments, \"We compare recent domain-specific LLMs in the code generation field ...\" Is it fair to compare with these LLMs?\n\n6. In the Experiments, \"We modified certain role-based prompts in order to instruct the MetaGPT framework to generate individual functions instead of entire classes\" 1) Is it fair to compare with these frameworks? LangChain and AutoGPT, for example, which are not specific to the tasks involved in this commit. 2) How can we ensure that \" prompt modification\" does not lead to unfair comparisons?\n\n7. Figure 4. What are the results of multiple attempts and are they stable? Were there any problems, like robustness?\n\n8. In Section 4.2, what is \"the cost of human revision\" and how can it be rigorously measured?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Problem-solving: MetaGPT can solve complex tasks by using different AI agents to handle different parts of the problem.\n\n2. Clear solutions: MetaGPT provides sensible solutions that favor tasks that require different elements to work together seamlessly.\n\n3. Flexibility: MetaGPT can be used for different tasks, and provide multiple outputs ranging from project plans to technical documentation.", "weaknesses": "1. Methodological innovation: A key issue with MetaGPT is the lack of innovation in its methodology. While it effectively utilizes Large Language Models (LLMs) in multi-agent systems, this approach may not be significantly different from existing approaches. Or MetaGPT is different from other methods in a trivial way, I don't really see their differences as being significant and requiring exploration with greater depth.\n\n2. Fairness of experimental comparisons: The comparison methods used to assess the effectiveness of MetaGPT do not appear to be fully equivalent to what MetaGPT offers. Such differences may lead to biased or misleading conclusions about their superiority or efficiency.\n\n\n3. Experimental validation of statements: Current experiments may not adequately validate the authors' claims about the efficiency and robustness of MetaGPT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qUr42RO1IH", "reviewer": "Reviewer_itRX", "summary": "The paper introduces MetaGPT, a meta-programming framework designed to enhance the efficiency and accuracy of large language model (LLM)-based multi-agent systems. By incorporating Standardized Operating Procedures (SOPs) into prompt sequences and adopting an assembly line approach, MetaGPT enables agents to specialize in diverse roles and collaboratively solve complex tasks with reduced errors. \n\nI believe this paper is interesting and makes a contribution by introducing an instrumental tool for automated code generation. I have several questions that I believe need to be addressed before publication. Aside from the questions below, my biggest concern is the relevance of the paper to Learning Representations. This paper mostly seems to be a nice fit for a journal focused on systems and applications.\n\n- It appears that this system needs existing expertise and strategy, correct? How primal/high-level the initial instructions can be? For instance, is a novice user going to have a hard time using this because they don’t have enough knowledge about breaking an initial task into sub-steps?\n\n- Does the level of initial input from human affect the performance? For instance, if the human gives a very high-level input vs a more detailed, structured input instruction?\n\n- The human user side of the system is never discussed, although I believe it’s very critical to understand the requirements, skills, knowledge, etc. the system imposes on or requires from a human user. The initial task or input instructions from the user are also never discussed.\n\n- The paper uses the term collaborative multi-agent systems, but later mentions that they define multi-agent frameworks as LLM-based multi-agent systems. I think the term collaborative multi-agent systems is very broad and comprehensive and it could be misleading to readers. The ‘LLM-based’ attribute of the multi-agent systems must be reflected in Title. Also, this leads to some misleading sentences such as “Most of current multi-agent frameworks utilize natural language as a communication interface.” Without using the term LLM-based multi-agent frameworks, this sentence is incorrect.\n\n- Is the shared communication/message pool interpretable to humans? Is it accessible by humans during or after the process?\n\n- Would the system ever need human interaction, or further input from human to address its potential questions?\n\n- There are multiple agents introduced in the system each with different roles, however, it is never discussed how are these specialized agents built or trained, algorithmically. Are they using specifically trained models?\n\n- Can there be multiple agents with the same role? i.e., multiple engineers that would work on different parts of implementations in parallel? How’s the parallelization process performed? For example, if multiple engineers will be working different parts of the problem designated by the architect, how will they choose which part to work on? Is there priority assigned, or is there a decision-making problem being solved? Sometimes the order by which a problem is solved could significantly affect its efficiency.\n\n- What are some of the limitations of the system at its current stage? Limitations, both in the system and on the human user side must be discussed.\n\nAt current states I vote weak reject (mostly due to relevance), although the system seems to be sound and working. I need to see more discussions and revisions as suggested above, as well as suggested by my fellow reviewers to make this an ICLR-ready paper. I’d be happy to increase my score when authors satisfactorily addressed the questions and comments.", "questions": "See above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "See above.", "weaknesses": "See above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a2Wdn7SSsA", "reviewer": "Reviewer_uj5i", "summary": "This paper introduces MetaGPT, an innovative meta-programming framework for multi-agent collaborations based on LLM, which encodes Standardized Operating Procedures (SOPs) into prompt sequences for more streamlined workflows. It selects a group of agents as a simulated software company, to generate a variety of code-based softwares. Through extensive experiments, MetaGPT achieves state-of-art performance on multiple code benchmarks HumanEva, MBPP and a software development benchmark SoftwareDev.", "questions": "1. Does the framework have strong generalizability when switched to GPT 3.5 or other open-source models? how far can the framework or practical use go with open-source models?\n2. In table 3, In this multi-agent collaboration with different roles, do you have some qualitive analysis or case study to show where each role contributes to the final performance, except for an executability score? Can you provide more detailed analysis?\n3. Whether the framework can also solve problems other than code, such as math and QA in Autogen? Is there other challenges for these kinds of problems for the framework?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The idea of encodes SOPs of software development into LLM-based multi-agent systems is very interesting and also pracitical to use.\n2. The framework is very sound and solid, with Specialization of Roles, PRD workflow across Agents, Structured Communication for complex tasks, and a compute-efficient Message Pool mechanism with both global memory and Subscription Mechanism. It also introduces an executive feedback mechanism to enhance code generation quality during runtime. \n3. MetaGPT achieves state-of-art performance on multiple benchmarks such as HumanEva, MBPP and a software development benchmark SoftwareDev. It opens new possibility for the software development paradigm.", "weaknesses": "1. Most of the experiment are on GPT4, which is expensive to access, how is the performance on the benchmarks or real development demands when used with open-source LLMs? Can you share some insight on which abilities of the LLMs matters most for the success of using this multi-agent framework and how to choose proper LLMs for use?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)