Self-Evolving Multi-Agent Collaboration Networks for Software Development
Authors: Yue Hu
Yuzhu Cai
Yaxin Du
Xinyu Zhu
Xiangrui Liu
Zijie Yu
Yuchen Hou
Shuo Tang
Siheng Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Software development
LLM
Multi-agent collaboration
Primary area: foundation or frontier models, including LLMs

Abstract: LLM-driven multi-agent collaboration (MAC) systems have demonstrated impressive capabilities in automatic software development at the function level. However, their heavy reliance on human design limits their adaptability to the diverse demands of real-world software development.
To address this limitation, we introduce EvoMAC, a novel self-evolving paradigm for MAC networks. Inspired by traditional neural network training, EvoMAC obtains text-based environmental feedback by verifying the MAC network's output against a target proxy and leverages a novel textual backpropagation to update the network.
To extend coding capabilities beyond function-level tasks to more challenging software-level development, we further propose RSD-Bench, a requirement-oriented software development benchmark, which features complex and diverse software requirements along with automatic evaluation of requirement correctness.
Our experiments show that:
i) The automatic requirement-aware evaluation in RSD-Bench closely aligns with human evaluations, validating its reliability as a software-level coding benchmark.
ii) EvoMAC outperforms previous SOTA methods on both the software-level RSD-Bench and the function-level HumanEval benchmarks, reflecting its superior coding capabilities.

Published as a conference paper at ICLR 2025
SELF-EVOLVING MULTI-AGENT COLLABORATION NET- WORKS FOR SOFTWARE DEVELOPMENT
Yue Hu', Yuzhu Cai²a Yaxin Du', Xinyu Zhu', Xiangrui Liu', Zijie Yu',
Yuchen Hou', Shuo Tang', Siheng Chen 1.3 Shanghai Jiao Tong University, 2 Beihang University, Shanghai Al Laboratory (1867112931, sihenge) ) és jtu, edu. en, ccaiyuzuubbua edu. en
ABSTRACT
LLM-driven multi-agent collaboration (MAC) systems have demonstrated impres- sive capabilities in automatic software development the function level. How- ever, their heavy reliance on human design limits their adaptability to the diverse demands of real-world software development, To address this limitation, we in- troduce EvoMAC, a novel self-evolving paradigm for MAC networks. Inspired by traditional neural network training, EvoMAC obtains text-based environmen- tal feedback by verifying the MAC network's output against target proxy and leverages a novel textual backpropagation to update the network. To extend cod- ing capabilities beyond function-level tasks to more challenging software-level development, we further propose DDecBench, a requirement-orientt velopment benchmark, which features complex and diverse software requirements along with automatic evaluation of requirement correctness. Our experiments show that: i) The automatic requirement-awar evaluation in rSDE-Bench closely aligns with human evaluations, validating its reliability as a software-level cod- ing benchmark. ii) EvoMAC outperforms previous SOTA methods on both the software-level rSDE-Bench Bench and the function-level HumanEval benchmarks, re- flecting superior coding capabilities. The benchmark can be downloaded https//yuuhuucaa.ggghub..o//Sii w.t.gihh.b.o/SSDDBBeneehhh
software de-
INTRODUCTION
Automatic software development focuses on generating code from natural language requirements. Code is a universal problem-solving tool, and this automation presents significant potential to provide substantial benefits across all areas of our lives Li et al. (2022a). Recently, the industry has introduced several large language model (dLM))driven coding assistants, including Microsoft's Copilot Microsoft (023), Amazon's 's Code Whisperer Amazon (2022), and Google's 's Codey Google (2023). These coding assistants significantly advance human efficiency and yield considerable commercial benefits. Despite the initial success of LLMS in assisting with line-level coding, they struggle to tackle more complex coding tasks. This limitation stems from the restricted reasoning abilities single LLMs and their lack capacity for long-context understanding Wang al. (2024a);
Li et al. 2024a); Wang et al. 2024b).
To handle function-level coding tasks, numerous multiple language agent collaboration (MAC) systems have been proposed Li et al. (2023); Hong et al. (023);; Chan et al. (2024); Islam et al. (2024): Yang et al. (2024b): [ et al. (2022b); Osika (2023). These MAC systems function as LLM- driven agentic workflow. They follow human- desiand standardized operating procedures to divide the complex coding tasks into simpler subtasks within the workflow, allowing each agent to conquer specific subtasks. These MAC systems significantly advance coding capabilities from line-level to function-level tasks. However, current MAC systems rely on heuristic designs. These human-crafted static systems have two inherent limitations: i) their performance is confined to human initialization. Given the diversity of real-world coding tasks, human design cannot fully address the specific needs of each task; and ii) they lack the flexibility to adapt to new tasks. This rigidity necessitates that researchers and developers manually decompose tasks and create prompts. The complexity this process inhibits effective human optimization for adapting new challenges.

---

Published as a conference paper at ICLR 2025
To address these limitations, we present EvoMAC, novel self-evolving paradigm for MAC networks. EvoMAC's key feature is its ability to iteratively adapt both agents and their connections during test time for each task. Inspired from the standard neural network training, the core idea of self- evolution is to obtain text-based environmental feedback by verifying the MAC network's generation against target proxy, then leverage novel textual back-propagation to update the MAC network. Following this general paradigm, we specify EvoMAC for software development, which comprises three essential components: i) an adaptable MAC network-based coding team that generates code through forward: ii) specifically designed testing team that creates unit test cases serving as the target proxy and verifies the generated code in the compiler to produce objective feedback; and iii) an updating team that uses the textual pack-propagation algorithm to update the coding team. By cycling these three components, the coding team can iteratively evolve and generate codes that are better aligned with the unit test cases, eventually fulfilling more requirements of the coding task.
Our self- self-evolving MAC network has the potential to further advance coding capabilities from function- level to more complex software-level tasks. As it can iteratively address lengthier task requirements and cater to realistic software development demands, However, existing benchmarks typically focus on specific individual functions Chen et al. (202T); Austin et al. (2021); |; Yang et al. (2024a) Khan iet al. (2023) or lug-fiing Jimenez et al. (2023). leaving a significant gap in providing comprehensive requirements for software development. This gap makes it difficult to fully assess the potential of our
self-evolving MAC network.
To support the development of software-level coding capabilities, propose rSDE- novel requirement-oriented software development benchmark. It is the first benchmark that features both complex and diverse software requirements, as well the automatic evaluation of requirement correctness. rSDE-Bench involves 53 coding tasks with 616 requirements, covering two typical software types, Website, and Game, and two requirement difficulty levels, Basic and Advanced. Each coding task consists of two components: i) multiple requirements that clearly outline measurable software functionalities, item by item, and ii) paired black-box test cases that automatically verify the correctness of each requirement. rSDE-Bench can achieve automatic evaluation with these synchronized pairs of requirements and test cases. The rSDE-Bench introduces new software-level challenges, including lengthy requirement analysis and long-contex, coding, which are essential in real-world software development but are absent in existing benchmarks.
To validate the effectiveness of proposed EvoMAC and DeDeBBench, we conduct three key evaluations. First, we compare our automatic evaluation in ISDE-Bench with human evaluation, achieving coherence score of 99.22%, demonstrating its reliability. Second, we compare EvoMAC against five multi-agent and three single-agent baselines. EvoMAC significantly outperforms previous SOTAs by 26.48% 34.78% and 6. 1000% Website Basic, Game Basic, and HumanEval, respectively, underscoring its effectiveness. Third, we evaluate EvoMAC with varying evolving times and two different driving LLMs. The results indicate that EvoMAC consistently improves with more evolving times and shows convincing enhancements regardless of the driving LLM used, further demonstrating the effectiveness of our self self-evolving design.
To sum up, our contributions are:
We propose EvoMAC, novel self-evolving MAC network, and apply it to software development. EvoMAC can iteratively adapt both agents and their connections during test time for each task.
novel requirement-orien orientee software development benchmark. It is the first benchmark that features both complex and diverse software requirements, as well as the automatic evaluation of requirement correctness.
We propose
We conduct comprehensive experiments and validate that: automatic evaluation in rSDE-Bende is highly aligned with human evaluation; EvoMAC outperforms previous SOTAs, and self-evolving promises continuous improvement with evolving times.
RELATED WORKS
LLM- based aauti--geett collaboration. LLM-driven multi-agent collaboration (MAC) systems Xu et al. (2023); ; Hua et al. (202);; Ziems et al. (0024): Wu et al. (2023); Hong et al. (002): |: Chan et al.
(2024): y Mandi al. (202g)) enable multiple agents to share information and collaboratively complete
the overall task. These MAC systems function as agentic workflows. They have demonstrated enhanced problem-solving capabilities in various domains, such as mathematics Islam al. (0224).
2

---

Published as a conference paper at ICLR 2025
software development Qian et al. (2023) Hong et al. (023). embodied task Mandi et al.
(2024b)
and social simulation Ziems et al. (2024); Pang et al. (2024); Li et al. (2024b). However, these systems Wu et al. (2023); Chen al. (2223) heavily rely on manually designed workflows, which lack generalizability and the labor-intensive nature of manual design poses significant limitations. To address this issue, we propose novel self-evolving paradigm, which allows agents to update and improve through external feedback, enabling dynamic adaptation and more advanced performance
across varied tasks.
Software development benchmarks. Software development benchmarks aim to evaluate models in the task of generating code from natural language descriptions Zheng et al. (2023). These benchmarks typically include task definitions and evaluation criteria. Existing benchmarks can be categorized into three types: i) function completion (HumanEval Chen al. (021), MBPP|Austin et al. (2021) ,
EvalPlus Liu et al. (2023), xCodeEval Khan et al. (2023): ii) bug repair JSWB-eenhhJimente
et al.
Hoong (2023); ); and iii) software generation (SRDD Qian et al. (2023), SoftwareDev Hong et al. (2023).
Function completion and bug repair benchmarks are confined to function-level task definitions, missing the diverse realistic software requirements. Software generation benchmarks often depend on expensive human evaluations or indirect similarity- measurements, unable to automatically and accurately verify the requirement correctness. To address these limitations, we introduce rSDE- Bench, the first benchmark contains both diverse software requirements and automatic evaluation of requirement correctness. It can support the development of more realistic software-level coding capabilities.
EVOMAC: SELF-EVOLVING MULTI-AGENT COLLABORATION NETWORK
This section presents EvoMAC, a novel self-evolving multi-agent collaboration network and its application to software development. The key feature of EvoMAC is its ability to iteratively adapt both agents and their connections during test-time for cach task, mimicking the pack-propagation process, a core algorithm in neural network training. We first formulate a general self-evolving paradigm in Sec. 3.1 and then describe its application software development in Sec. 3.2
3.1 A GENERAL SELF-EVOLVING PARADIGM VIA TEXTUAL BACKPROPAGATION
Multi-agent collaboration network. A multi-agent collaboration (MAC) network a computational graph representing agentic workflows, where multiple agents empowered by LLMs interact as interconnected nodes to coordinate and share information for complex task-solving. The intuition behind to divide the complex task into more specific and manageable subtasks for each agent, allowing the overall task to gradually conquered through the agentic workflow. Mathematically, we represent MAC network with N autonomous agents as directed acyclic graph A = (V, E", where V= [ is the set of N nodes, and & {et, M..............M..... is the set of directed edges with no circles. The i-th node Y represents the i-th autonomous agent with the prompt pi, which specifies its subtask. The edge Cs,j represents the task dependency between the i-th agent and the j-th agent, indicating that the j-th agent's subtask should be executed after the i-th agent's subtask in the agentic workflow. The overall graph topology specifies the agentic workflow. Analogy to traditional neural networks, agents function similarly to neurons, with agent prompts serving as neurons" weights and the agentic workflow as the layers and connections.
The fced-forward pass of MAC network is the execution of the agentic workflow. In this process, each agent is given two inputs: the initial task requirement and the output from the previous agent. Using these, each agent produces an output that fulfills its specific subtask. Eventually, the last agent's ' generation constitutes the final output, integrating all completed subtasks. Note that the initial task requirement is input to each agent as context, providing supplementary details to aid in the implementation of each subtask.
Recently, various MAC networks have been designed using human expertise to assign fixed agent prompts and workflows Hong et al. )0 Chan al. (2024), resembling untrained neural networks. However, these designs solely rely on human priors and lack adaptability, causing limited performance improvement over single agent. To overcome this, inspired by neural network training, we propose a self-evolving paradigm for multi-agent collaboration networks, enabling both agents and their connections to dynamically evolve during tist-time for each given task.
Optimization problem. Here we consider a general generation task. During test-time, given a task, the MAC network performs a feed-forward pass generate the final output without knowing its
3

---

Published as a conference paper at ICLR 2025
Self-Evolution Paradiagm
Gradient Analysis
Network Update
Textual
Agent
Agent
Agent
Agent
Backpropagation
MAC Neework
MAC Network
Update Agent
A!"
A
Agent
Agent
odd
Vrvvse
Agene
Agent
VL
Settevolving Times
Agant
Gradiem
Update
VL
Gradient Agent
Environment Tool
VL
agene
MAC Neowork
Figure 1: The general self-evolving paradigm.
quality. The key to evolution during test-time is to set up target proxy for the MAC network to guide its improvements in the generated output. Here we consider this target proxy as the conditions for task completion, such as unit tests in coding, and we can produce such a target proxy by another group of autonomous agents based on the same task description. Then, the quality of cach generated output can be verified according to the target proxy. This approach relies on two key assumptions: (i) generating a target proxy significantly simpler than completing the original generation task, and (ii) the generated output be correctly verified against the target proxy through an objective environment. These assumptions are practical in many applications. For example, in code generation, producing unit tests, the expected input-output pairs, is much easier than generating the entire code; meanwhile, a code compiler naturally acts as the objective environment to check the correctness of the generated code against the unit test, providing objective and informative feedback.
Mathematically. let X be the textual description of a task. Given the MAC network Ag. the generated output is G ((X, where -(:--) is the general feed-forward operator that executes the agentic workflow, processing the input text through the MAC network. Similarly, the target proxy is T = ( At). where Ar is another MAC network designed for producing the target proxy. Note that we aim to evolve and optimize Ag. while keep Ar predefined and fixed. The optimization of our self-evolution is formulated as,
A = min Ag),
T) subject to: T 1 AX,
where (.-)) g is an objective environment executor that receives the generated output and the target proxy inputs and outputs a text-based environmental feedback. Akin to the loss function in traditional neural network training, which quantifies the difference between the generated output and the ground-truth, the objective ( evaluates whether the generated output meets the conditions of the task completion using the environment, subsequently producing exccution reports as the text-based environmental feedback. Here the minimization operation min is defined to reduce the failures during execution. With the guidance of the target proxy and the objective feedback given by the environment, the MAC network can improve its success rate of task completion during test time.
Note that, another straightforward way to enable the MAC network's 's evolution is through the self- critique strategy Zhou et al. (2024); Valmeekam et al. (2023); [Xu et al. (2024); Asai et al. (2023). which employs critique agent to assess the generated output directly. This approach has two inherent limitations: i) the critique may be subjective and biased, and ii) the critique agent can have hallucinations, causing inconsistencies and errors. These limitations can cause the MAC network to become entrenched in its own preferences or evolve in the wrong direction, especially iterating comparison, our approach leverages an
multiple times; see our experimental validations in Tab. environment executor to provide objective feedback, preventing bias and hallucinations.
While we use the analogy between our self-evolution process and neural network training for motivating, they are significantly different in three key aspects: (i) our self-evolution occurs at test time without dedicated training phase; (ii) evolves for each specific task individually rather than batch of samples; and (iii) the environmental feedback are usually texts, not be numerical values, which cannot be optimized by the standard backpropagation. This motivates us to propose textual backpropagation.
Solution based on textual backpropagation. The self-evolution solution iteratively updates the MAC network using textual backpropagation algorithm. guided by the environmental feedback. The core idea is to analyze the influence of each agent in the MAC network A, to the final environmental feedback and use these analyses to update the agent prompts and the agentic workflow in Ag- This
4

---

Published as a conference paper at ICLR 2025
Task Roquirements
XK Times
Codling Team
Tusting
Updating Team
Updated Coding Team
Eocarquse and
Cruss sun sar and gare ralms
Cadlggg Organitr
Agees Dondap log
Toiig Organier.
Agenn
Agen Andou
Upraar the sgrn teee
Coding Agune 2:
I po ralrad
Codlin Ageess 12 Develop
Develop GLI
Cocling At Deveip give rudo
Teitig Agert Text
Tept Taxs gara rule
sgre rome
anbie agrin bota salion
deparder
Develop mol:
Implement
Gradiend Analy eisc
Code:
Codec
That Cane:
def nuvetpsddoon)
tett noveju
Source Deciled aualpii mice requirennce gare un unsimplaraatde sised
par
Bium polttio
Task Requirements
Envirenment Feedbacks
Compller
Activs sdi and agaan
Sakkask aepleneet crowe function, the pooliiee dhould
Updasting Suggection:
I.G.l
5.5w pars casse the
reciimemnce
fhll sor,
Verdhow roding 3
Gara E'Accape afber
ogail feetiinnliyy
Figure 2: EvoMAC takes task requirements input and iteratively updates the coding team to generate code that better fulfills the requirements.
is achieved by two collaborative agents, each responsible for one of the two key steps: (i) textual gradient analysis and (ii) network update. The overall algorithm can refer to Alg. T in the appendix.
First, the gradient agent takes the environmental feedback as the input and outputs textual gradients
that describe the impact of each agent in the MAC network. Let AAA and (() be the MAC network and the environment feedback the k-th iteration. The textual gradient is then (k)
G(A4)) (()), where ((:-.) is the gradient analysis operator managed by the gradient agent; see its prompt in Appendix. The textual gradient details three-fold fold information for each agent inside A!*), 1) whether this agent's subtask is fulfilled; 2) whether this agent introduces errors; and 3) whether any subtask is missed in the current MAC network.
Second, based on the textual gradients, the updating agent iterates the MAC network as Ag (1+1)_ u(A(a)) (*)), where (((--)) is the updating operator managed by the updating agent. This operator guides the updates from three-folds: 1) removing the agents whose subtasks have been completed; 2) revising the erroneous agent's 's prompts by adding potential solutions provided in the gradient analysis; and 3) adding new agents for missing subtasks and restructuring the workflows based on the subtask dependencies noted in the gradient analysis; see the prompt details in Appendix. These adjustments address existing issues and fulfill unmet requirements in the current generation of the MAC network, promising improvements in the updated version.
Note that, the key of the textual backpropagation is the prompt designs for both gradient analysis and network updates. The design must i) thoroughly evaluate the subtask of each agent in the MAC network according to the objective environment feedback and determine necessary adjustments to the MAC network to address existing issues, fulfilling the unmet requirements; and ii) maintain coherence, ensuring that issues identified by the gradient agent can be effectively resolved by the updating agent's modifications to the MAC network.
3.2 SELF-EVOLUTION FOR SOFTWARE DEVELOPMENT
In this section, we apply the self-evolving paradigm to the task of software development. The overall architecture of the proposed self-evolving multi-agent collaboration network for software development illustrated in Fig. D Given coding task, the coding team, corresponding to the MAC network Ag. generates all the codes through its forward-pass pass; the testing team, associated with the MAC network Ar, responsible for creating the target proxy; that is, unit tests of the coding task; and the objective environment tool is realized through the compiler. The identified bugs during execution form the textual environmental feedback. The updating team, consisting of two collaborative agents, manages the textual backpropagation. By continuously eycling through collection, and textual backpropagation processes, the coding team is iteratively refined to align more closely with the test cases.
feedback
Since unit test generation is much easier than the original logical code generation, the testing team usually can produce high-quality test cases, which are closely aligned with the task requirements.
5

---

Published as a conference paper at ICLR 2025
Software Instruction
Software Requirement
Generated Codes jAcemooml cods criuee)
Objective
coopl leggin
AEP-PY
DatyHoesithi Tis where
webete cslled
Develop a website (Dulyticuahiiips)
Lang-age
<Bt ctare Torn-graphh
login pere
PESE
daily beatsh rip log their accounts, ops, submit feedback with all dats stored text website sho offers persoraliaed heaks advice based profiles
Page: Design
Login Page
Evaluation Metrics
Elnowate:
Usemame Fleck D-useenans passwene
Test Reports (RSD-Bench)
PASSED toxt, Logincctamnns FASLEP Sogin Tenciionally Lity Park
Schware
not fnd TTDDO)
Daily Tips Page
ArpsArctive Page
onaaree......................o..
Completeness
Data Storage
(wdvvnnedd : 9, "total"1 13
Senlery
User Dars Scored SESES.
Erecurabilty
Doupli::
Accuracy (Ours):2 13
Consistency 0.69
Quality 0.89 11 0.89
Figure 3: Comparison between instruction-oriented and Bench accurately reflects requirement fulfillment with the proposed accuracy score of 2/13, while the indrection evaluation misjudges with high scores (0.89), failing to detect missing functionality.
evaluations. rSDE-
Then, improving alignment with the unit tests through MAC network updates ensures better adherence to the actual task requirements.
Coding team for weeddforward. In the feed-forward process, the coding team synthesizes code according to the given coding task. To handle the extensive software requirements, the coding team is implemented as MAC network. It divides the comprehensive requirements into sequence of smaller, more specific function implementation subtasks, and progressively conquers them through the agentic workflow. Unlike existing MAC systems that heuristically decompose coding tasks and define the agentic workflow, we initialize the MAC network using a novel self-organizing approach. coding organizer agent automatically and flexibly decomposes the task requirements into subtasks and assembles the coding agent team accordingly. The number of coding agents is dynamic, adjusting in response to the task requirements. Note that, the quality of the generated code unknown during the forward pass, which necessitates the self self-evolving paradigm to iteratively refine the generation.
Testing team and compiler for feedback collection. To verify whether the generated code meets the requirements of the coding task, we employ unit tests as the target proxy. These test cases consist of input-output pairs tailored to specific requirements. For example, a test case for keyboard control requirement would detail the type of control as the input and describe the expected behavior as the output. To create flexible and comprehensive unit tests, we set up the testing team as a MAC network and also initialize it in a self-organized way. A testing organizer agent automatically decomposes specified key testing criteria into subtasks and accordingly forms the testing agent team
Once the test cases and generated code are ready, they executed in the compiler, which functions as the environmental tool, producing execution logs. These logs clearly point out the gap between the generated code and the test cases. It shows satisfied testing requirements, existing function errors, and unmet testing requirements. This feedback information can be used to verify whether each agent's subtask is accomplished and guide the MAC network update.
Updating team for textual back-propagation. The updating team consists of two collaborative agents: the gradient agent and the updating agent, adjusting the MAC network based on the execution logs, including the agent prompts and workflows. This process consists of two steps. First, the gradient agent summarizes the textual gradient by identifying accomplished subtasks for satisfied requirements, appending new subtasks for unmet requirements, and analyzing errors to detail their originating agents and revising suggestions. Second, the updating agent modifies the coding agent team by removing agents that have completed their subtasks, adding new agents for the new subtasks, and revising agent prompts to address issues identified in the previous generation. The agent workflow is updated once the agent team is revised, based the dependencies among the subtasks.
CENTIEEMETT-RRENTEE SOFTWARE DEVELOPMENT
ENGINEERING BENCHMARK
This section introduces sDE Bench, requirementt-rriented benchmark designed to assess the ability of models to handle software- coding tasks. Each coding task involves multiple detailed software requirements. These requirements specify each functionality and constraint of the software, item by item, serving as measurable benchmarks for assessing the software's effectiveness. As shown in Fig. unlike previous instruction-oriented approaches Qian et al. 2023); Hong et al. (2023) which

---

and Gemini Published as a conference paper at ICLR 2025
rely brief instructions input, =SDE-Bench uses comprehensive software requirements input, complemented by unit test cases to automatically evaluate the correctness. This benchmark provides software-level coding tasks and automatic evaluation, aligning more closely with real-world software development practices.
4.1
BENCHMARK CONSTRUCTION
iSDE-Bench involves two typical real-world software types: game and website. They can reflect different coding capacities demanded in realistic software development. Game often requires handling dynamic interactions, real-time state changes, and user-driven operations, focusing on elements like logic execution, initialization, and game state transitions. Website emphasizes static and dynamic content management, user interaction through forms and buttons, and ensuring page elements are displayed and functional.
iSDE-Bench involves diverse requirements, each paired with a test case. Specifically, rSDE-Bente provides 53 unique coding tasks and 616 test cases. iSDE-Bench introduces two requirement difficulty levels, including basic and advanced, to reflect the varying complexity of real-works software development tasks. For details on the benchmark construction, software statistics, software requirements, and test case examples, see Sec. 7 in the Appendix.
4.2 AUTOMATIC EVALUATION
rSDE-Bench supports automatic evaluation of requirement correctness. It achieves this by pairing a specifically designed black-box test case with each requirement. The test case can directly verify whether the generated code achieved the requirement. Its evaluation metric is the accuracy, which quantifies the proportion of correctly passed test cases. is similar to the passel metric in HvmanEval Chen et al. (2021), which evaluates the pass ratio of correctly achieved functions against the total functions via unit test verification. It a fully automated evaluation process, eliminating the need for human involvement while still providing accurate and reliable assessments.
Previous benchmarks for software code generation mainly rely on two evaluation methods. One method is human evaluation Hong al. (203)), which is time-consuming and not scalable for large datasets. The other method is indirect evaluations Qian et al.] (2023), which defines metrics like consistency, completeness, and quality. Consistency measures how closely the generated software aligns with the original requirement description by comparing the cosine similarity between the two. Completeness is determined by detecting the presence of placeholder (such as pass or TODO), which results in a binary value of 0 or 1. Quality is then calculated as the product of several factors: consistency, completeness, and executability. As illustrated in Fig. 3 they could not measure the correctness of the generated code in fulfilling requirements. In contrast, =SDE -BeeEcheehhh 's test cases- based evaluation is more rigorous and precise. These test cases can accurately verify the correctness of generated code in fulfilling the requirements. rSDE-Bench promises reliable and scalable automatic evaluation. We have validated the significant advantages the proposed automatic evaluation over the previous metrics, including consistency and quality; see Fig.
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Baselines. To validate the effectiveness of our EvoMAC, we conducted comparisons against both single-agent and multi-agent baselines. The single-agent baselines involve three prominent large models: GPT-4o-Mini (gpt-4o-mini). Claude-3.5-Somme
(gemini-1.5 flashh. For multi-agent baselines, we included five sfate-of-the-arr (SOTA) methods: MetaGPT Hong et al. (2023). Autogen Wu et al. (2023), Mapcoder Islam et al. (2024). Agent- verse Chen verse(Chen et al. (2023), and ChatDev Qian et al. (2023). To ensure a fair comparison, all multi-agent baselines, including our EvoMAC, are powered by the efficient and powerful GPT-4o-Min model. Additionally, to demonstrate the adaptability and robustness of our EvoMAC, we developed two EvoMAC variants using GPT-4o-Mini and Claude-3.5- 3.5de-..--Sonne....
Datasets. Our experiments cover both the proposed rSDE- Bench and the standard coding benchmark HumanEval Chen et al. (2021). HumanEval comprises 164 Python function completion problems, where the task is to generate code from a single function description.
5.2 EFFECTIVENESS OF RSDE -ODE-BENH'S ' EVALUATION AND EvoMAC
rSDE-Bench's 's automatic evaluation metric (accuracy) is highly aligned with human evaluation. Our primary goal is to validate the effectiveness of the proposed automatic evaluation in rSDE-Bench
7

---

Published as a conference paper at ICLR 2025
GPT-4o-Minl
MetaGPT
ChatDev
EvoMAC
Human
Accuracy
Consistency
Quality
Performance
75
Performanc
75
Performance
75
Performon
75
50
50
50
25
25
25
25
0-
WB
WA
GB GA
WB
WA
GB
GA
WB
WA
GB
GA
WB
WA GB
GA
Dataset
Dataset
Dataset
Dataset
Figure 4: Performance of four methods in terms of four evaluation metrics, including human evaluation, automatic evaluation (accuracy), consistency, and quality. WB/GB and WA/GA
represent Web/Game Basic and Web/Game Advanced respectively. Our accuracy metric is highly aligned with human evaluation across four dataset settings.
Table 1: Comparison of EvoMAC with five multi-agent and three single-agent SOTA baselines, all powered by GPT-44-Mini. Red values represent the percentage improvement of EvoMAC, shade in pink, over the single-agent baselines, shade in grey.
ISDE-Bench
HumanEval
Method
Model
Website( %)
Game %)
(%)
Basic
Advanced
Basic
Advanced
Pass@1
Seemini-I.5-Flashh
29.79=1.00
11.61=2.34
21.74#6.39
6,45#6.97
73.17
Single-Agent
Claude-3.5-Somner
58.90=1.48
37.11#1.06
44.20#5.41
18.29
89.02
GPT-4o-Mini
62.90+2.52
44,40+4.21
42.76=15.50
30.10#11.87
88.41
MctaGPT
10.41=0.00
0,0020.00
16.67=2.7T
0.00=0000
88.41
Autogen
25.68+4.14
5.40=3.34
17.39
0.00+0.00
85.36
MapCoder
34.70&1.59
14.57+0.66
29.71
7.52x6.10 10
90.85
Multi-Agent
Agentverse
15.4140.00
0.00+0.00
37.6748.20
16. 13:4,55
90.85
ChatDev
62.67+0.28
43.4520.77
53.63+5.70
32.26:44.55
70.73
89.38+1.01
65.05+1.56
77.54+2.04
51.6024.54
94.51
EvoMAC
+26.48
+20.65
+34.78
+21.50
+6.10
by comparing with two existing evaluation metries: consistency and quality, both from SRDD Qian et al. (2023). For a fair comparison, our golden standard is human evaluation, conducted by two expert code engineers who manually verify the fulfillment of requirements by interacting with the developed software. This process is tedious, taking around four hours per expert to evaluate the entire benchmark. The effectiveness of an evaluation metric depends on how closely it aligns with human evaluation.
Fig. 4 presents the performance four methods in terms of four evaluation metrics, including human evaluation, our automatic evaluation, consistency, and quality. We see that: i) our automatic evaluation is highly aligned with human evaluation across two software types (Website and Game), four methods, (GPT-4o-Mini, MetaGPT, ChatDev, and our EvoMAC), and two requirement difficulties (Basic and Advanced). The correlation coefficient between human evaluation and our accuracy metric 0.9922, demonstrating the effectiveness of the proposed automatic evaluation in rSDE-Bench; ii) Consistency and quality metrics differ significantly from human evaluation, with correlation coefficients of 0,2583 and 0,3041, respectively. This discrepancy occurs because consistency in SRDD measures similarity, and quality in SRDD focuses on executability, which does not guarantee that all requirements are met. This highlights the need for SDDE-Bench, the SRDD benchmark does not support eggurrement-oriented software development.
EvoMAC outperforms previous SOTAs on both software-level and function-level coding bench- marks: rSDE-Bench and HumanEval. Tab. T compares EvoMAC with five multi-agent and three single-agent SOTA baselines, all powered by GPT-4o-Mini for fair comparison. We see that EvoMAC significantly outperforms previous SOTAs across all datasets. EvoMAC outperforms single-agent methods by 26.48% on the rSDE-Bench Website Basic and 34.78% on the -SDE-Bench Game Basic, well surpassing existing multi-agent methods by over 20%. This highlights the effectiveness of multi-agent collaboration and the power of EvoMAC.
5.3 EFFECTIVENESS OF EVOLVING
Fig. shows the accuracy of EvoMAC over multiple evolving iterations the rSDE-Bench and HumanEval. Each figure presents two curves: one for EvoMAC powered by GPT-4o-Min (red) and the other by Claude-3.5 (blue). We have the following findings:
8

---

Published as a conference paper at ICLR 2025
Basic
Advanced
Advanced
199
HumanEval
50
60
(C000 K. soveet
Evolving Times
Evolving Times
Evalving Times
Evolving Times
Evolving Times
(a) Website
(b) Game
(c) HumanEval
Figure 6: Effect of EvoMAC performance across evolving times empowered by GPT-4o-Mini and Sonude-3.5-Somit on Website, Game, and HumanEval datasets. The figure shows EvoMAC continuously improves with the evolving times on both LLM drives.
Website
Game
Table 2: Ablation study about coding/testing team with 250 single/multi-agen with/without evolving, and with/without environment tool. Best performances are bolded.
Dispay
80
Contient Menage
Community
User Manage
60
Website( %)
Game %)
Count
Count
Coding Testing
Evol.
Env.
Basic
Advanced
Basic
Advanced
Single
63,70
41.70
42.76
30.10
Logic
b) Multi
c)
Single Single
v
67.47
80.82
60.32
39.27
68.10
71.73
41.93
41.93
50
20
Initalization
Log
Operation
d)
Multi Single
/
83.90
60.72
76.08
41.93
Symtax
c)
Single Multi
v
83.56
61.94
73.91
45.16
Multi Multi
78.08
52.23
55.80
33.32
Evolving Times
Evolving Times
Figure 5: Failure distribution across
Multi
Multi
v
90.75
67.20
77.54
51.60
evolving times on Website and Game.
EvoMAC continuously improves with the evolving times. Fig. 6 shows that evolving iterations increase, performance consistently improves across all five dataset settings, covering two difficulty levels. two software types, and both requirement-orienter and function complement benchmarks. This highlights the effectiveness, generalizability. and robustness of the self-evolving approach, encouraging EvoMAC evolve whenever possible.
EvoMAC indistinguishably improves with different driving LLM. From Fig. we see that: i) both EvoMAC variants continuously improve with evolving iterations, demonstrating the robustness of the self-evolving design; ii) the two curves do not intersect, indicating that the EvoMAC variant powered by a more powerful single model consistently outperforms the other, highlighting the advantage of using stronger model. Success builds on success.
Failure case analysis. Fig. 5 shows the failure case statistics across iterations for Website and Game, showing a general decrease in errors iterations progress. We see that: i) the most common errors are page display issues in Website and logic errors in Game; ii) page errors are resolved more quickly, while logic errors persist, suggesting that more isolated issues easier to fix during the evolution process. This results in a sharp initial performance improvement as sipler problems are addressed early, followed by plateau as more complex issues remain unresolved, shown in Fig.
5.4 ABLATION STUDY
To assess the effectiveness of each component, Tab. 2 details an ablation study featuring seven EvoMAC variants.
Effectiveness of objective environment feedback. Environment feedback, such as code execution logs, essential for software development. Variant omits this tool, instead using an LLM--riven agent to critique the code. Comparing Variant g) with Variant f) shows a notable performance drop: Website tasks decrease by 12.67% and 14.97%, and Game tasks by 21.74% and 18.28% for Basic and Advanced levels, respectively. This underscores the importance of objective environmental feedback, as agent-driven critiques may introduce bias and fail to guide the evolution effectively.
Effectiveness of multi-agent collaboration in coding team and testing team. Comparing Variant g) to Variant e), we observe performance decrease of 7.19% and 5.26% on Website Basic and Advanced respectively, when the coding team is reduced to single agent. Similarly, comparing Variant g) to Variant d), there is a performance drop 6.85% and 6.48% on Website Basic and Advanced respectively, also when the team is reduced single agent. These results demonstrate

---

Published as a conference paper at ICLR 2025
GPT-4o-Mini
ChatDev
EvoMAC
Website Basic
Website Advanced
Game Basic
Game Advanced
Accurscy
Accuracy
Accuracy
68
320
920-33 3240
1240-1543
600-023
Tokens
Task Tokens
Task Takens
Task Takens
Figure 7: EvoMAC outperforms previous multi-agent and single-agent systems across all the context lengths across the four dataset settings on rSDE-Bende
Requirermentsi Pless doii veredee for Tecpethhh, wwlll conetcc the levinng caper weer Legie Page Home Pace Page
Mini
ChatDev
EvoMAC
Vermeen vestter. Gesauiee maseel"))
Updating Team
berston
Analysis
Cremarte
The anstisst Festlaet
Source
Deciled Aralyasc The loge reqpineneer in grannts 13. unteparareed
Picaes comren persucte of cerro
Updacing Suggestions
setr sneer de 201 riper
parsert Ligust
coding apee
sperl2
Mudiy
the ke kensonig aoopp requsio aue with antarniutioe, with arrile iosmare lrpus
Worldoow afher coding spent
L-1 acsico 20th Nusto ctasic Loan
Figure 8: We show the generated code of single-agent, GPT-4o-Mini, and multi-agent systems, ChatDev, and our EvoMAC (iteration =0/1) given the Website task (RecipeHub). After evolving.
EvoMAC can revise previous issues and fulfill the task requirement.
the necessary for involving multi-agent collaboration, highlighting that multi-agent setups offer more flexible adjustments and enhanced capabilities for evolution.
Effectiveness in handling varied task token lengths. Fig. 7 shows a comparison of task token lengths and performance across GPT-44-Mini, ChatDev, and EvoMAC. We see that: i) EvoMAC consistently outperforms ChatDev and GPT-4o-Mini across all context lengths, with its self-evolving mechanism enabling the identification and correction of missed contexts and errors during iterations; ii) EvoMAC experiences less performance degradation on the rSDE-Bench Website than on the Game, Website tasks are more modular and can be broken into subtasks, whereas Game tasks require more coordinated management, making them more challenging.
5.5
CASE STUDY
presents the generated code by a single agent, GPT-4o-Mini, multi-agent systems, ChatDev, and EvoMAC before and after evolving (iteration=O/1). We see that: i) EvoMAC after evolving can correct issues from previous iterations and successfully fulfill the task requirements; ii) multi-agent systems tend better comprehend the task requirements and produce more well-structured code. More generated software can refer to Sec. 10 in the Appendix.
Fig.
CONCLUSION
We propose EvoMAC, novel self-evolving paradigm for MAC networks. EvoMAC iteratively adapts agents and their connections during the testing phase of each task. It achieves this with a novel textual
algorithm. EvoMAC can push coding capabilities beyond function-level tasks and into more complex, software-level development. Furthermore, we propose -SDE-BBenhh, a novel requirement-orreetted software development benchmark. rSDE-Bench features both complex and diverse software requirements, as well as the automatic evaluation of requirement correctness. Comprehensive experiments validate that the automatic requirement-an aware evaluation in rSDE-Bench aligns closely with human evaluation. EvoMAC outperforms previous SOTAs in both software-level iSDE-Bench and function-level HumanEval benchmarks.
Future works. In the future, we plan to introduce a reward model to enhance the self-evolving paradigm's 's ability to learn from feedback and extend the rSDE-Bench more software types.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Amazon.
CodeWhisperer.
2022.
URL https::/platformm. qa. www.
amazon cot.er/...eeeeeee..eaaaaaaaaooon
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. ArXiv, abs/2310.11511, 2023.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewsk,, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, al. Program synthesis with large language models. arXiv preprint arXiv:2108.07733 2021.
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better blsed evaluators through multi aulti- debate. In The Twelfth International Conference on Learning Representations, 2024.
Mark Chen, Jenyy Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards. Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray. Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethca Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Marthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. 2021.
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang. Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu. Ruobing Xie, et al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents. arXiv preprint arXiv:2308. 10848, 2(4):6, 2023.
Google. Codey. In wttp:///ww.oooteeoo////////////e 2023. URL
model ga waa.den/codechattttte
Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Mctagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00 00352, 2023.
Wenyue Hua, Lizhou Fan, Lingyao Li, Kai Mei, Jianchao Ji, Yingqiang Ge, Libby Hemphill, and Yongfeng Zhang. War and peace (waragent): Large language model-based multi-agent simulation of world wars. arXiv preprint arXi::2311.17227 2023.
Md. Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. Mapcoder: Multi-agent code generation for competitive problem solving, 2024.
Carlos Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? The Twelfth International Conference on Learning Representations, 2023.
Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do Weishi Wang. Md Rizwan Parvez, and Shafiq Joty. xcodeeval: large scale multilingual multitask benchmark for code understanding, generation. translation and retrieval, 2023. JtL/tttss://// //atpps://axxvvv http///wxwvi../../a////////oo/// abs/ 03004]
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration large language model society. Thirty sevent Conference on Neural Information Processing Systems, 2023.
Jiaqi Li, Mengmeng Wang. Zilong Zheng, and Muhan Zhang. Loogle: Can long context language models understand long contexts?, 2024a. UtR.hhttss:/arxii..oor hbtprrgg/b///31111/ 04939
Junkai Li, Siyu Wang, Meng Zhang, Weitao Li, Yunghwei Lai, Xinhui Kang. Weizhi Ma, and Yang Liu. Agent hospital: Á simulacrum of hospital with evolvable medical agents, 2024b. URL https: / ///xxiiv/////// org/ abs/2403 02957
11

---

Published as a conference paper at ICLR 2025
Yujia Li, David Choi, Junyoung Chung. Nate Kushman, Julian Schrittieser,, Rémi Leblond, Tom, Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de, Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey, Cherepanov, James Molloy, Danicl Jaymin Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de, Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. Science, 378:1092 1097, 2022a.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Lehlond, Tom Becles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition- code generation with alphacode. Science, 378 37866624)10092-1999 December 2022b. ISSN 1095 9203, doi: 10. 126 158.
1126/science abqi158
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and LINGMING ZHANG. your code generated by chatGPT rcally correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL http::://ppenrrvviee..to////
Lqvx610cu7
Zhao Mandi, Shreeya Jain, and Shuran Song. Roco:
Dialectic multi-robot collaboration with large language models. In 2024 IEEF International Conference on Robotics and Automation (ICRA), pages 286-299. IERF, 2024a.
Zhao Mandi, Shreeya Jain, and Shuran Song. Roco: Dialectic multi-robot collaboration with large language models. In 2024 IEEE International Conference Robotics and Automation (ICRA), pages 286-299, IEEE, 2024b.
Microsoft. Copilot. In http::/wwwwwwooooooooooooooeoeoe 2023, URL https 7/www. wtw.a//www......eeo......ooooooa
Anton Osika. GPT Gnn-nnnineer. In http:://giluabooomm//oooooooeeee
2023. uttpp:s::///itttt
Com/ Attp//tttnnnakaakaaaaaaaa gpt. engi neer
Xianghe Pang. Shuo Tang, Rui Ye, Yuxin Xiong, Bolun Zhang, Yanfeng Wang, and Siheng Chen. Self-alignment of large language models via monopolylogue-base social scene simulation. In Forty-first International Conference on Machine Learning, 2024.
Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang. Jiahao Li. Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. Chatdev: Communicative agents for software development. arXiv preprint ar arXiv:2307.0799/ 2023. URL httpp: wttp://www.wo//ab///..0.0000 07924
org absg
Karthik Valmeekam, Matthew Marquez, and Subbarao Kambhampati. Can large language models really improve by self-critiquing their own plans? ArXiv, abs/2310.0811 18, 2023.
Chonghua Wang, Haodong Duan, Songyang Zhang, Dahua Lin, and Kai Chen. Ada-leval: Evaluating long- context Iims with anngth-ddaptable benchmarks, 2024a.
Xindi Wang, Mahsa Salmani, Parsa Omidi, Xiangyu Ren, Mehdi Rezagholizadeh, and Armaghan Eshaghi. Beyond the limits: A survey of techniques to extend the context length in large language models, 2024b. URL 32244
https: w/tpxvaaaivvvvvvv www./r///////ree.ee
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White. Doug Burger, and Chi Wang Autogen: Enabling next-gen llm applications via multi multi- conversation, 2023. http:/ttt///xiiiiiii wbwp//////////ss//////////// 08155
Yifan Xu, Xiao Liu, Xinghan Liu, Zhenyu Hou, Yueyan Li, Xiaohan Zhang, Zihan Wang, Aohan Zeng, Zhengxiao Du, Wenyi Zhao, Jie Tang, and Yuxiao Dong. Chatglm-math: Improving math moblem--olivnng in large language models with a self-critique pipeline. ArXiv, 2024.
Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Explor- ing large language models for communication games: An empirical study on werewolf. arXiv preprint arXiv:2309. 246608,00 2023.
Guang Yang. Yu Zhou, Xiang Chen, and Xiangyu Zhang, Codescore-r: An automated robustmess metric for assessing the functionaleorreetness of code synthesis, 2024a. htt.tptpp//wxi...rrrrr/..oe 06902
12

---

Published as a conference paper at ICLR 2025
John Yang, Carlos E. Jimenez, Alexander Wettig. Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. ArXiv, abs/2405.15793, 2024b.
Zibin Zheng, Kai- Kai-Chun Ning, Jiachi Chen. Yanlin Wang. Wenqing Chen, Lianghong Guo, Weicheng Wang. Towards an understanding of large language models software engincering tasks, ArXiv, abs/2308.1133 2023.
Wangchunshu Zhou, Yixin Ou, Shengwei Ding, Long Li, Jialong Wu, Tiannan Wang, Jiamin Chen, Shuai Wang. Xiaohua Xu, Ningyu Zhang, et al. Symbolic learning enables self-evolving agents. arXiv preprint arXiv:2406. 18532, 2024.
Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. Can large language models transform computational social science? Computational Linguistics, 50(1):237-291, 2024.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
BENCHMARK DETAILS
Table 3: Basic statistics for website and game do- mains, including the amount of samples, prompt length (mean/max), and number of test cases at both Basic and Advanced levels.
Game Category (%)
Website Category (%)
Initialization
Executability
Content Manage
Browsing
Log
Set Up
14.9
Communiy
4.9
Manage
42.9
Benchmark
Software
Test Case
Operation
Amtoun
Length
Basic
Advanced
Website
45
1011/1553
292
247
Display
Game
8
507/788
Figure 9: Statistics of Game and Website tasks.
7.1 FEATURES
Challenging and diverse software requirements. rSDE-Bench features long-context software requirements (averaging 507/1011 words for game and website tasks, respectively), unlike instruction- oriented benchmarks Chen et al. (202T); Austin et al. (202T); Jimenez et al. (2023) that rely on brief prompts. These detailed requirements better reflect real-world lengthy and complex software development challenges.
Requirement-aware precise and efficient evaluation. rSDE-Bench employs detailed software requirements and automated unit tests to precisely measure how well generated software meets its objectives. Generated codes are evaluated based on pass rates from running specific test cases, offering an accurate and efficient process. In contrast, instruction-oriented benchmarks rely on brief prompts, which lack constraints and make evaluation less reliable, often requiring labor-intensive or indirect evaluation.
7.2 CONSTRUCTION PROCESS
Step 1: Software requirement generation. Each task instance begins with the generation of clear, measurable software requirements. Given the inherent differences across various types of software, we adopt distinct approaches for their formulation. For game-related software, we focuses on common real-world games, capturing detailed task requirements such as GUT layout initialization, interaction methods, and game rules. To align more closely with actual game development practices, we also include game state logging part of the software requirements. Due to the complexity of logic in game software, these requirements are manually crafted by human. In contrast, for website-related software, we begin with a concise website name, and then leverage the large language model (qpt-44 1esp--4-miin in) to enrich the requirements according to predefined patterns. This approach ensures both efficiency and scalability in the creation of benchmarks for websites. By tailoring the process to the distinct characteristics of each software domain, we maintain precision in requirement formulation while addressing the unique challenges posed by each context.
Step 2: Requiremeen-baased test cases generation.
As illustrated in Fig and Fig unit tests offer precise evaluation of software completion. Each task instance includes black-box unit test cases that correspond directly to the software requirements, allowing for quantitative assessment of requirement fulfillment. To further assess the model's code generation capabilities, we categorize test cases into two levels of difficulty-basic and advanced, as outlined in Tab. 3] We also provide an overview of all websites and games in Tab. and Tab. respectivvl. As shown in Fig. g test cases for website and game software exhibit structural differences, reflecting the distinct nature each software type. They enable more targeted evaluation of code generation capabilities. Thus, similar to software requirements, the test cases are constructed differently based on the software type. For game-related tests, we manually create test cases, akin to the HvmanEval Chen et al. (2021) benchmark, which tracks state changes in response to specific inputs. In the game environment, we assess how game states evolve in response to GUI interactions. For website-related tests, large language model (ogpt-4o-mae based test cases aligned with the software requirements, followed by manual corrections to resolve any ambiguities. This structured approach ensures rigorous evaluation across diverse software domains.
generates Selenium-
14

---

Published as a conference paper at ICLR 2025
Table 4: Overview of Websites in rSDE-Bench.
Websites
CharitableGGivingPlafooo
DailyHealthTips
DailyJournalApp
EcoFriend
ElderCareResouress
EventPlanner
Finnsssqquppmenteenta
FitnessTracker
FreelancerMarkktplacce
GreenLivingCruide
RealthhonnsultaiionPlaatoio
PntivationalQuotssAA.e
MusieFesstivallirrecoory
Note NoteTTkkiggApp
NutriioolnfommatooiHtuu
OobineLbbaryyManneeeenSSSyta
Online Therapeutic
Thiineehrittttre
Peer/uutoringNeewookk
PersonalBlog
PersonalFinance
RecipeHub
Remote aaaooteeeteemshhpMareeppppp
Remote/ob Board
TravelDiary
VirtualBookuublishii
ViruaaWellnessseteeee
DigitalArtworKGalless
Sggitalttorteellnnglattoooo
Expense Tracker
FitnesSChallenges
BurdeniinFForeegineee
GourmetFooodSusscciinnn
MovieeReommeedattionSSyyte
MusicCollabone
Online UnlineCultuaalExchhag
OnlineCulturaFeettivlls
Online VintageMarket
ParentingAdvviceFFoo
PeCareCoommnni
PortfolioSite
SkillShare
Task Manager
VolunteerMatch
OnlineShoppinggCnntrr
Table 5: Overview of Games in rSDD-Bench.
Games
Balls
Tank
Racing
Ghostly
Mario
Bomberman
Sokoban
Brick
Basic and advanced requirements definition. The basics reflect the fundamental and more achiev- able requirements, such interaction, control, and logging. The advanced reflects more complex software functionalities, such game logical rules, and dynamic web content management. For the games, basic requirements involve straightforward user interactions that do not require complex logic, such as character movement interacting with simple GUI elements. Advanced requirements incorporate more intricate logic, such as managing game state transitions based on user actions or handling conditional game events. These cases focus on ensuring the correct execution of basic actions. In contrast, advanced cases incorporate more intricate logic, such as managing game state transitions based on user actions handling conditional game events. These cases challenge the model's ss ability to generate code that integrates dynamic decision-making and interaction within the game environment. For websites, basic cases focus ensuring that the necessary page ele- ments-such as input fields, buttons, and layouts-are present correctly. These cases assess the completeness of the webpage's structure. On the other hand, advanced cases evaluate more complex functionality, such as handling user authentication, managing dynamic content, or executing specific operations within a content management system. These cases require the model generate code that performs backend logic and manages user interactions at a deeper level.
ALGORITHM
In this section, we present the algorithm of EvoMAC in Alg, For more details, please refer to Section 3.
CASE STUDY
9.1
COMPLETE EvOMAC PROCESS
In this section, show complete process of EvoMAC RSD-BBnchh. Please refer Tab.
15

---

Published as a conference paper at ICLR 2025
Software description
Task: Davelop strple Sakoban gare. You rust design a GUT.
Requiroments:
board should divided into
Playens control the game using anrow keys keyboard.
3. As the game starts, a log file naned "gano. log' should be created to necord the gane's progress. The content the game. file should be appended with new entry after each player action. The content of the sane.log file should be cleared (if
Each log entry the start of each session.
fomat:
"tinestamp": timestarp, "EVENT_TYPE": INOEERRIGH "NOVE_LEFT" "HOVELIP"
"MOVE_DOWN"
"INWALIO_MOW" NOVE""
"player position": [x, y), "box_positions [DC, y1l, [x2, y2], ...b. "gane_status" "ONCOING" "COMPLETE"
victory
gone boxes pushed corresponding coordinate point.
5. initiall positioos of each element are required as follows:
player_position [3, 1]
box pox_posttions- [[3, 31, [4, 2]]
geal_positions
51,
box Mhose target position [5, 5]. [4, 2] is position the second box whose target position is [6, 3J.)
(I,, 3] initial pocitioo
wall positions [[e, 4], [2, 4], [2, (3, 4), 4]] (the
x-coordinate
secand nusber y-coordinate)
Evaluation functions
check, Excutablity coeck_log nove noveeright check_rove, left check_rove__r box
check_nove_wall check_seubox check_end
Figure 10: Test cases Game in ISDE-Bench
Software description
Requiresent Document for DallytoatthTips Meb Appliicttion
Objective
Bevelop web application named 'DatlylellthTips that provides users with datly health tips, allouing then to receive advico and inforsation about maintaining healthy lifestyle, using Python the development language. Noto that the website should start from the
sd 2. Language
The required developnent language for the DaslylissttTTips web application is Python.
Page Design
rdd Page 1: Login Page
**Page Title**; User Login
**Overview": page
log accounts.
**lementss*:
**Usernane Field**;
**ID**; usernane_ field"
*PPassoord Field**;
**Ip*>; password_fiell
"Lgin Button**;
*D*** login_ button"
Evaluation functions
test_Jogin_nen elenents test peat___ogiinnmeeefnntioooollll test_daily_tin pepplpttttppp_________ Siips__age_eeeeeene elements
test_dafly. past_daall__ts________ ttp__pago____ppppppppp jaage_functionll1111 test rest_ripe Ses___tp___ar_de archive_page__eeee elesents test rest_tips aips_archtiv__eeeeeeeee_e_ Bacttivee_ag___eee hatp:aaaag____ainoooee FunctSonality
Figure 11: Test cases of Website in rSDE-Bench.
9.2
UNIT TEST CASE
In this section, show more unit test written by coder RSD--Bench, please refer to Tab. and Tab. 16.
9.3 UPDATING PROCESS
In this section, we show additional examples of the updating process on RSD-Bench and HumanEval dataset. Please refer to Tab. 19 and Tab. 22 respectively. For RSD-Bench, due the code length, only show the texture updating crodess(codes are available at Sec, 9.I). We can see that the updating agent will adjust the job of each coder dynamically according the result of test team.
16

---

Published as a conference paper at ICLR 2025
Algorithm 1 Self-Evolving Paradigm
Require: X
Task input
Require: Ag (0)
Initialized MAC network: agent prompts and pipeline D Designed MAC network to generate target proxy Agent-based gradient function Agent-based update function Environment tool to generate loss 1: Define K as the number of self-evolving iterations, & as MACN generation process
Require: A
Require: G
Require: 4
Require:
2: Target Proxy
3: T = A.)
4: # Self-Evolving Procedure
5: for k = :,...;44 K
do
6: Forward Pass
7: G( (X, )
8: Loss Computation
9:
L (G T)
Use environment feedback as textual loss
Textual Backpropagation
11:
(
(k) G[L (M) )
Summarize textual gradient
12:
A
(k+1) U(A,)) VL
Update agent prompts and pipeline
13: end for
14: return AA ((R)
10 SOFTWARE PRESENTATION
In this section, we show some games and websites written by EvoMAC. Fig. 12 and Fig. 13 present the games and websites respectively. We see that: i) EvoMAC outputs games with well-written GUI and game rules. It can handle different kinds of GUI and game rule requirements from diverse games. ii) EvoMAC outputs websites with beautified, user-friee aeer-frindly web pages and correct transition logic. can handle the requirements of different websites.
17

---

Published as a conference paper at ICLR 2025
Table 6: A complete iteration process of EvoMAC on RSD-Bench
Notation X
Meaning
Example
Real example See Tab.
Textual description of the task to be completed.
A coding task such as: "Im- plement a code that sim- ulates keyboard input pro- cessing via Python."
Ag
MAC network representing the team responsible for generating the code.
The coding team consists of coding agents completing subtasks sequence.
See Tab.
G (XX,
Generated output produced by the coding team as a result of the feed-forward pass.
The
generated See Tab.
code:
....dff pro- cess _n_pnntkkeybbrrd__npp
A
MAC network representing the team responsible for generating the target proxy (unit tests).
The testing team generates See Tab. unit tests for the task.
T ((X,
Target proxy (unit tests) generated by the testing team based on the task de- scription.
Unit tests like: "...dff See Tab. test press_ees____ pllps_preesstte process input( 'Enter') == Processed Enter"
|]
input(): assert
< <, >E
Environmental feedback comparing the generated output G with the target proxy T using an objective environment (e.g.. compiler or test results).
The environment executes the generated code against the unit tests, providing feedback like: 'Failure: test_press_ing input'
The execution outcome of the unit test from the terminal. If the execution is successful, the outcome is "The software run success- fully without emror.. See Tab.
min < G,T>p >F
The optimization objective aiming minimize the dif- ference between the gen- erated output and the tar- get proxy using the environ- mental feedback.
Based on feedback, the sys- tem iteratively refines the coding team to generate code that better meets the task.
18

---

Published as a conference paper at ICLR 2025
Table 7: Textual description of the task to be completed.
Task: Design a Single-Player Tank Battle Game Requirements:
1. The interface should be divided into a 20x20 grid, though grid lines are not necessary. Fach tank occupies grid space, while obstacles may occupy multiple grid spaces. The background should be black, obstacles should be brown, enemy tanks should be silver, and the player's tank should be yellow. lank's movement using arrow keys on keyboard, allowing for movement one grid space at time. The 'enter' key is used to fire bullets.
The player can control
3. In the game, there are two enemies fixed at a certain position on the game interface, constantly firing bullets in four directions: up, down, left, and right. Two enemies and players cannot be initialized in the same row.
4. Both the player and the enemies have their own health points, which are initialized to 200. When hit by a bullet, the player's 's health decreases by 10 and the enemy's 's health decreases by 100. When health points drop to zero, the corresponding tank is destroyed. But the log still records information about the destroyed tank, with health points of Destroying an enemy tank earns the player 200 points. The game ends when the player's tank destroyed or all enemy tanks are destroyed, at which point the player's score will be displayed on the screen.
6. As the game start, The first log entry should capture the initial state the game. Each time new event occurs the player, a new log entry should be written in real-time. The logs should follow the format below. The EVENT TYPE can only be one of the following: "INIT", "MOVE LEFT". "MOVE RIGHT"" "MOVE UP", "MOVE DOWN". "FIRE" or "INJURED". The game state should capture the current state of the game. In this setup, the position coordinates defined with the top-left grid as 0], where the x-coordinate increases by one unit each grid space moved the right, and the y y-coordinate increases by one unit for each grid space moved downward. The player is initialized at position [0,0].
log file named 'omm should be created to record the game's 's progress.
"t "timest ammestappp: timestamp, "INII" "MOVE LEFT" LEFT" "MOVE_IOOWN" "FIRE" "INJURED",
RIGHT"
"MOVE UP"
"game_state";
"player";
"position"; [xp, ypl, "health"; shalth_playee "score":
enemies";
"position"; [xel, ycll, "health": health_enemyl
b,
"position"; [xe2, ye2), "health"; health
"obstacle_position [ [xol, yol),
(xon, yon) ]
19

---

check health status Published as a conference paper at ICLR 2025
Table 8: MAC network representing the team responsible for generating the code.
composition":
"Programmer 1": "Implement bullet firing logic that decreases health points for both the player and enemies when hit. This includes creating function handle bullet movement and collision detection, updating health points accordingly, and logging the events in the game, log file.",
"Frogranmer 2"; "Implement health management to check for health points dropping to zero and handle tank destruction. This includes creating function after each hit, updating the game state, and logging the destruction event in the game log file.", "Programner 3"; "Implement scoring logic to update the player's score when enemy tank is destroyed and display the final score when the game ends. This includes creating function to manage scoring and game-over conditions, and logging the final score in the game. log file."
"workflow":
"Programmer 1": [],
"Frogranner 2":
"Programner 1
"Programmer 3": "Programmer
20

---

Published as a conference paper at ICLR 2025
Table Generated output produced by the coding team result of the feed-forward pass. Due to the limit of page length, we only show partial of the codes.
This is the main file for the Single-Player Tank Battle Game. initializes the game, sets up the GUI, and manages the game loop, including player movement, enemy behavior, and logging events.
import pygame
import json import time # Constants
GRID_SI2R 20
CRII_SIZE 30
WINDON SIZE GRID SIZE * CELL SIZE
PLAYER COLOR = (255, 255, 0) Yellow
ENEMY COLOR = (192, 192, 192) $ Silver
OBSTACLE OBSTACLE_COLOR COLOR (139, 69, 19) i Brown
BACKGROUND_COL.oR (0, 0, 0) Black FPS
# Game state
player_position= [0, 0]
player_health 200
player_score enemies =
|"position": 15, 5), "health"; 2001,
("position": [10, 10], "health"; 200)
obstacles [[3, 3), (7, 7), (12, 12)] Log file
log file = open game. log', 'w')
initial state
"timestamp": time.time().
"EVENT_TYPE"; "INIT",
"game_state"; { "player":
"position": player_position
"health": player_health,
"score": player_score
enemiess; enemies,
postacllepositioni obstacles
log_file.wri (json, dumps (initial_state) 'An') Initialize Pygame
pygame.inith
screen pygame. bygame.display. weclllyy.et___nee mode (RINDOW__SSST SIZE, MINDON_SIZE) wwtpe/ccapllaaseeeeeeee
("Tank Battle Game")
clock eygame.time.clon. class Bullet: ""
This class represents bullet fired by the player.
def init init__sels (self, position, direction)
selt.position position self.direction direction def move (self):
if self.direction !UP!: sclf.position[1]
elif self.direction 'DONN'
self.position[1] += 1
elif sel = 'LEFT':
sclf.position(0] 1
elif sele.iirection 'RIGHT':
sels.position(0) += 121

---

Published as a conference paper at ICLR 2025
Table 10: MAC network representing the team responsible for generating the target proxy (unit tests).
composition":
"Frogrammer 1"; "Carefully test the logging mechanism according the task requirements. Ensure that legs happen inmediately after action is taken, recording the most recent state. Verify the logging order, ensuring basic operations are recorded first, followed by subsequent events. Check that the data format, keys, and values are accurate, paying attention to the nested data types and carefully checking each element.",
"Progranmer 2": "Test the logging mechanism for special triggeres conditions, such as when the player's health drops to zero or when an enemy tank is destroyed. Ensure that these events are logged correctly and that the game state reflects the changes acourately.",
"Progranmer 3": "Test the value initialization required by the task to ensure they are correctly achieved. Pay attention to the coordinates of the player and enemies, ensuring they are initialized in the correct positions and that ne two tanks occupy the same row.",
"Frogranmer 4"; "Test the function inputs and the global variables imported in each function. Ensure that the input values and global variables used the functions are valid and involved when the function is called.",
"Frogranmer 5": "Test that each event in the task is implemented correctly and that the logic triggered matches the conditions describea in the task. This includes movement, firing bullets, and health reduction for both the player and enemy tanks."
"workflow":
"Programmer 1": [],
"Programmer 2":
"Programmer 1
"Frogrammer 3";
"Programmer 1", "Programmer
},
"Frogramner 4":
"Progranner 1",
"Programner 2",
"Programmer
FFogrammer 5";
"Programmer 1"
"Programner 2"
"Programmer 3",
"Programner 4"
22

---

Published as a conference paper at ICLR 2025
Table 11: Target proxy (unit tests) generated by the testing team based the task description. Due to the limit of page length, we only show one test case written by one of the five test coders.
import unittest import json
import time
from main import player position, player_position, player phatth_ player_ score, enemies, log file, log_event, fire bulett, brack_health class TestTankBattleGame def setUp (self):
global player_position, playcr_health, player_secre, enemies playcr_position = [0, 0]
player_health 200
player_score enemies -
("position"; 15, 51, "health": 200), ["position"; 110, 101, "health"; 200)
log_file open ('game. 'w')
def tearDown (self);
log_file.close
def test test_move__es (self)
global player_position
player_posittion[0]
* Move to the right first
["o_oventt"" player_positioon00]
self seliaassstttsulllpaeeeooootoooo 10, 01)
def test_movee_right((e111]:
global player_pos__ronnooon
[og_event
player_positionn00
+= +
sel
11, 01)
def test test_move__ppseeff!:
global player_position
player_positionn[1] -
i Move down first
player_positionn[]] -
selt:aaserEEgnallpppeeeeoooooo.e [0, 0))
def test__mvee_down(ssll
global player_position
Log_eveatt"%%VV__o0
player_positionil]
self.asserru/upa
(plpa//uuaapllllla__a_aaall (player-poattinnn [0, 11)
def tett_fire_bolee__ht__eeeee/ (self):
global player_score
mnitial_health enemies 'hhallhh]
fire_bullet: ( (enemies [0] ["health"), initial health 100)
self
self.asserrU<qual (playrreee_allllllaa__ep_eeeee (player_score, 200)
def tess_fire_bulle__msss____
(self):
global player_positionn
player_position = [0, 0]
fire_bulle:()
self..aserr&Squal (enemies [0] ["health"), 200)
def test_plaayer_injuuyssslll (self))
global player_health
player_health - 10
Log_event ("INJURED")
check_healch()
self.assert//qual (player_health) 190)
def rest_enem__destruutiool (self) (self):
glcbal enemies
enemies ["health"] =
check (eeck_health()
self aclf.aasor1Equal (enemies [0] ("health"),
if name__. = m_inn''
23
unittest (maii))stmmimm))))))))))))

---

Published as a conference paper at ICLR 2025
Table The optimization objective aiming to minimize the difference between the generated output and the target proxy using the environmental feedback. According to the unit test results, Updating agent add more notes for the sub-task for Programmer and Programmer 3. To see complete updating process, please refer to Sec. 9.3
ccopossitoo""
"Programmer 1": "Implement bullet firing logic that decreases health points for both the player and enemies when hit. This includes creating function handle bullet mevement and collision detection, updating health points accordingly, and logging the events in the gane, log file.",
"Programmer 2": "Implement health management to check for health points dropping to zero and handle tank destruction. This includes creating function each hit, updating the game state, and logging the destruction event in the game. file. Additionally, ensure the game doop terminates properly when the player's tank is destrcyed."" ", "Programmer 3"; "Implement scoring logio update the player's score when an enemy tank destroyed and display the final score when the game ends. This includes creating a function to manage scoring and game-over conditions, and logging the final score in the game. log file. Ensure the final score is displayed correctly when the game ends."
check health status after
"workflow";
"Frogrammer 1": [],
"Frogrammer 2";
"Programmer ]"
p Programmer 3": "Programner 2"
Table 13: Unit test case: Game
Example
Game Ghostly
Requiremen((partia))
If the ghost controlled by the player eats superpellet (the big pellets), it gains the ability to eat other ghosts.
Subtask given by Test Organizer(partial)
"Programmer 5": "Test that each event in the task is implemented and that the logic triggered matches the conditions the task description."
Generated test case(partial)
please see Tab. 14
Evaluation test case(partial)
please see Tab. 15
24

---

Published as a conference paper at ICLR 2025
Table 14: The partial generated test case for Game Ghostly.
def test_eat_other__hhae (sel :
s wtw.suppppelet__cctiiiie True Activate superpel let
sain.move_ghost(MM
Move to [1, 0]
more move Liree_ghotttt.ee function call cmitted here
with o' 'r') as log_file:
logs log__ile.. readliiieeedd
last_log json (1ogs([11) self sclf.assertEquall (last_ "t [6, 51) anlf. (T0, 5], main.other_ghostt
"eat_other_ghost"
cotherr_host_posttiinns positicns)
Table 15: The partial evaluation test case for Game Ghostly.
def check_ Beack_eet___ee aac___a___thhtta other caacc__et_tthhreseeee with superpellet
Check if Character can eat others
try:
print (" eat othermmeeennn
process = self.start_gamel
time.sleep (0.1)
pg.press ('right! ptrsgghhttpsssssss.,, presssss-, interval-0)
time.sl (0me.s)eep(0.)))
more press omitted here
process.terminatell print ("terminated") Read the log logs self. read (logs) ent=0
for log in logs: if
cnt+=1
if cnt==2:
print ("Character can eat other with superpellettalaa)) return
print ("Character can eat other with superpelletinnnn" return
except:
print ("eat_other other error")
return
Table 16: Unit test case:
Website
Example 2
Website Deilh
Requiremen((arrial)
please see Fig.
Subtask given by Test Organizer(partial)
"Programmer 2": "Test whether the first page of the website is the login page and whether it
can log in correctly based on the example data provided in the Task."
Generated test case(partial)
please see Tab. 17
Evaluation test case (partial)
please see Tab.
18
25

---

Published as a conference paper at ICLR 2025
Table 17: The partial generated test case for Website Daily Health
Test whether the first page of the website is the Login page and whether it can Log in correctly based the example data provided in the Task.
import unittest
from selenium import webdriver from aeeeeiiu.webdrieer common by import
from seleenim.webbrriver.r www.iiveerrcooooc.c....m keys import Keys import time
class yettaiiiyiiillhhhipisioi (unittest. TestCase)
def setUp (
self.driver phooriverrChrome
self.driver.get ("http://occahhst::: def (est__ooin_paagesee))
driver self.driver Check is the current page is the login page assee:IIss: Login", driver.title)
print ("Login page title is correct.")
Locate username and password fields flerd driver. driver.find find_element (By.ID, "username_fiela"" password_field driver. find_element (By.ID, "password_field") login_button driver. element (By.ID, "login_butten"
Input example data
ussernam__Feeld.ennd keys ("jehn_doeel
passwerd__ieldd.ssn___ess ("sccurepassword")
Login_button.cliekk
Wait for the page to load
time.sleep(().
Check 1- redirested haily Health Tips page self.assert ("Daily Health Tips", trtle) print ("Successfully logged in and redirected to Daily Health Tips page.")
def tearDown
self.driver.quitt/
name "_main_"": unittest.man main
26

---

Published as a conference paper at ICLR 2025
Table 18: The partial evaluation test case for Website Tiis
class TestCase (unittest. TestCase)
def setUp (
"*"Set up the Selenium WebDriver before each test. selved = Ceborivvrr.Chromee
self.driver.
def tearDown
"*"lear down the Weboriver after each test.
self.driver.quit..
def login
self.drivers self.driver.deeete deaerddeeee__aae all_aall__ooooo ()l__cocies()
username = "john doe"
password "sccurepassword"
Ferforming login
solf.drivor.get ("http::/ /tt.o//oooo.o.oooooooot////ooooo :5000)
selff.driver.findeele.. (By.ID, ID, "username_field")
self.Griver.find_eles
(By. ID, "password_field")
seen_keeys felf..riverr.inndel......... (By. ID,
.click) ccickk()
===== Login Page Tests
def cest_login__age__leeeeee
(self):
Go back to the login page and check elements
helffassertTrus (elf ID, (uerramme_ffalldd__aa___aaaspss_ self tet..aaeerttunfeaa...ceeeeeeeee. (By.ID, 'Baasworr_fieelld.....aassssaaa oel.iaaaartreeell....eeeeee..ee.
IBy.ID, "login_butttrrr).....aasspleeee
def [est_llggiiaage_fnnnttoooooaes
$ Assume user login is verified by the presence
of daily tips page elements ausfaaassttIn"Usee Login", selé.driver. "john_doe"
password "securepasswora"
wefffdriverr.ind_elee.... (By.ID, 'username_field")
.send_keys (username))
self. felffddiveer.om findeeeeennneeemmen (By. ID, 'password_field")
send keys (password) self.drives
(By.Ih) 'login_button" cclick)
Check 1s we are redirected to the daily tips page self Isser:In ("Daily Health Tips", self.rriver.title) more test case omitted
Table 19: Updating process on RSD-Bench
RSD-Bench Game Tank
Example
Requiremenn((prttaa)
please see Tab.
Initial Code (ttraaoo 0)
please Tab.
8
Updating attertit 1)
please see Tab.
Updating agent(iteration 2)
please see Tab. 20
Updating aient(iteration 3)
please Tab. 2
27

---

Published as a conference paper at ICLR 2025
Table 20: Updating process on RSD-Bench Game Tank, iteration
"composition":
"Frogrammer 1": "Fix the health management Logic to ensure that when the player's health drops to zero, the destruction event is logged correctly, and the game loop terminates properly. This includes fying the "checkhheathh" function to accurately log destruction events for both the player and enemies.",
"Programmer 2": "Implement the gane-over conditions to ensure that the game ends when the player's s tank is destroyed or all enemy tanks are destroyed. This includes creating a function to manage the game-over state and logging the final score in the game, log file."
},
"workflow";
"programmer 1"; [],
"Programmer 2": "Programner
Table 21: Updating process on RSD-Bench Game Tank, iteration 3
"composition":
"Programmer 1"; "Implement constant firing behavicr for enemy tanks in four directions. This includes creating mechanism for enemies to fire bullets at the player and legging the correspending events.",
"Progranmer 2"; "Implement the display of the player's score at the end of the game when the game is over. This includes modi fying the game-over logic to ensure the final score is logged and displayed correctly.",
"Programmer 3"; "Fix the logging mechanism to ensure that the log file remains open during gameplay and logs events in real-time without closing prematurely. This includes reviewing the log event function and ensuring it is called correctly throughout the game lcop."
,,
"workflow":
"Programmer 1": [],
"Programmer 2":
"Programmer 1"
"F "Frogranmer 3":
"Programner 1 "Programmer 2
28

---

Published as a conference paper at ICLR 2025
Table 22: Updating process HumanEval
HumanEval
Example
Requirement
Write function that takes a message, and en- codes in such way that swaps case of all letters, replaces all vowels in the message with the letter that appears places ahead of that vowel in the english alphabet. Assume only letters.
Initial Code Organizer(iteration 0)
"Programmer I": "Identify and implement the logic to swap the case of all letters the mes- sage.", "Programmer 2": "Implement the logic to replace all vowels with the letter that appears places ahead in the English alphabet."
Generated test case(iteration 0)
please see Tab. 23
Main code(iteration 0)
please see Tab.
24
Updating agent(iteration
"Programmer T:
AssertionEEroor:
code "ede(")ess'))eee t= 'TGST' The actual output is:
'TgST'., "Programmer 2": AssertionError:
encode('Thi, This is a message") != 'tHKS KS C MGSSCGG'. The actual output is: 'tHKS kS MgSScGg'.
Final codertteatioon 1)
please see Tab. 25
Table 23: Updating process on HumanEval, Generated test case, iteration 0
Write a simple test case to verify the logic of the encode function. import unittest
from encodc_funct import encode
class TestEncodefunerionnnnniiii ( TestCase)
def [e sel selffaasserrrqppaleeeccccellll ('test'), 'TGST!)
def (est__erampl__caas_]][e111]]
sel aelf..aseertkgualleeee (encode ("This is message'),
"tHKS xs C MGSSCGG")
if name = main
tetitest.mainn.
29

---

Published as a conference paper at ICLR 2025
Table 24: Updating process on HumanEval, Main code, iteration
def encode (message):
vowels 'aciouAEIOU' encoded_message - for char in message: char. iaar. # Swap case
if char in vowels:
Replace vowel with the letter places ahead if char. shar.islower(::
new_char = chr (chrr - ('a') + 2) $ 26)
+ ord()a)))
else:
new rew_char: chr (char) ord ('A') 2) $ 26)
+ ord('A))
encoded_message append (new_char)
else:
Just swap for consonants pecooed_message.co
(char swapcase ())
else:
If it's not letter, we can just append it as is encoded message.... (opeendchhar
return " ".join (encoded message)
Table 25: Updating process on HumanEval, Final code, iteration
def encode (message):
vowels 'aeiouλEIOU' encoded_message for char in message: if char.isalpha!) $ Swap case
if char in vowels:
# Replace vowel with the letter places ahead and swap case
is char.islower():
new Sw__ chr (corr) ord('a') + 2) % 26)
ord()a))) (uppe)))upppr(() upper
else:
new_char chr (char) - ('d''A") + 2) $ 26)
+ ord (rower))).wwwrr() encoded messadeeeeageeeeee append (new_char)
else:
# Just swap case for consonants encoded (char. swapcase ())
clsc:
Preserve spaces
encoded meteeedmsssageeeaaa append (char)
return " ".join sencoded__eeseeeee
30

---

Published as a conference paper at ICLR 2025
Heb Ith: 100
(a) Balls
(b) Bomberman
Soore: 21900
bore: Health: 20p
(c) Mario
(d) Tank
Score: 0
Remaaniiig Lives of Bricks: 150
(e) Brick
(I) Ghostly
31

---

Published as a conference paper at ICLR 2025
Speed:
Distance: 231
(g) Racing
(h) Sokoban
Figure 12: Games generated by EvoMAC.
GardeningForBeginners
Guddning Tix foon Lopat
Login to CharitableGivingPlafffr
Cardening Tips
Username:
coutio your
Password:
Login
(a) ChariaableGivinnePattooo
(b) GardeningForBeeiinness
My Personal Blog
New Post
Logout
Blog Posts
My First Blog Post
Exploring Python
(c) PersonalBlog
32

---

Published as a conference paper at ICLR 2025
GourrneFFoodSubborittton
New Journal Entry
Tite:
Subsoription
fptiors
happy cay
Logout
Content:
happy
Your Subscriptions
Subscription Types:
Subsoription Frequencles:
Subecription Start Dalwx:
Select Bubocr piion Typec
Save Erity
a
Back Dashboard
Cobile Subicc Vilan
(d) GourneeFoodSubbeription
(c) Online/Terrapentibbommaningg
Festival List
Coachella Califoria 2023-04-14
Lollapalooza Chicago 2023-08-03
Add New Festival
Logout
(f) MusicFFestvaaDDircttory
Figure 13: Websites generated by EvoMAC.
33


Reviews:
[{"review_id": "NYx9Lgc5DT", "reviewer": "Reviewer_kPYh", "summary": "This paper proposes a multi-agent collaboration approach to address software development problems. EvoMAC obtains text-based environmental feedback by verifying the match between the MAC network's output and the target proxy, and it updates the network using a novel textual backpropagation technique, thereby achieving the final development outcome. Additionally, this paper introduces a software development benchmark called RSD-Bench, which provides more detailed and structured software requirements for documenting user needs compared to previous benchmarks. The final experimental results show that the proposed EvoMAC outperforms other single-agent and multi-agent methods on both RSD-Bench and HumanEval.", "questions": "1. Are there any issues with the citation format in the paper?\n2. Does the paper lack an appendix?\n3. In Table 2, there is no comparison of results with environment tools but without evolving. This should be added.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper proposes a multi-agent collaboration approach to address software development problems.\n2. This paper introduces a software development benchmark called RSD-Bench, which provides more detailed and structured software requirements for documenting user needs compared to previous benchmarks.\n3. Extensive experiments demonstrate that EvoMAC outperforms other single-agent and multi-agent methods on both RSD-Bench and HumanEval.", "weaknesses": "1. The benchmark proposed in this paper lacks data analysis and some basic statistical information, such as prompt length, the number of final generated files/functions, etc.\n2. The benchmark proposed in this paper is relatively easy, with the EvoMAC method already achieving around 90% accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kMzlT22g0q", "reviewer": "Reviewer_32VP", "summary": "The paper introduces a novel framework called EvoMAC, aimed at enhancing the capabilities of LLM-driven multi-agent collaboration (MAC) systems in software development. The authors argue that traditional MAC systems are heavily reliant on human-designed workflows, which limits their adaptability and performance in real-world scenarios. EvoMAC seeks to overcome these limitations by enabling self-evolution of agents and their connections during task execution.", "questions": "1. How sensitive is EvoMAC to the quality and specificity of feedback from unit tests? If the unit tests are incomplete or overly general, would EvoMAC still produce reliable code, or would it require stricter validation criteria?\n2. Can EvoMAC work effectively with models of different sizes, or does it rely on the power of high-capacity LLMs? Would it perform satisfactorily with smaller models that might be more efficient in constrained environments?\n3. Can the self-evolving mechanism be applied to other domains outside software development? If yes, how?\n4. Given EvoMAC’s iterative approach, how would it handle larger software projects with thousands of lines of code and extensive requirements? Are there specific design considerations for scaling it to more extensive projects?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This self-evolving paradigm allows MAC networks to adapt iteratively based on environmental feedback. The framework employs a mechanism similar to neural network backpropagation, where the output of the MAC network is verified against a target proxy, facilitating continuous learning and improvement.\n- The RSD-Bench provides a structured benchmark for software-level coding tasks, focusing on comprehensive requirements rather than isolated functions.\n- By incorporating unit tests and compilers as feedback mechanisms, EvoMAC reduces subjectivity and provides reliable feedback, which is critical for verifying the correctness of generated code. The objective environment-based feedback is an effective alternative to critique agents, which can introduce bias and hallucinations.", "weaknesses": "- The EvoMAC framework introduces significant complexity with its multi-agent setup, dynamic adjustments, and textual backpropagation mechanism. This complexity may limit the framework's accessibility and implementation ease for real-world adoption outside of specialized research contexts.\n- Although EvoMAC performs well with large models like GPT-4o-Mini, its performance with smaller or less capable models is unclear. This reliance may restrict its applicability, particularly in environments with limited computational resources.\n- RSD-Bench focuses on website and game software types, which may not comprehensively represent the diversity of real-world software development tasks. Expanding the evaluation to include other domains, such as enterprise applications or data processing software, would enhance the generalizability of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bFcW1S4xUX", "reviewer": "Reviewer_aNRL", "summary": "The paper presents EvoMAC, a self-evolving multi-agent collaboration (MAC) network designed to advance LLM-based multi-agent systems beyond function-level tasks to software-level coding tasks. EvoMAC employs a unique textual backpropagation mechanism to iteratively update agents and their connections in response to text-based environmental feedback, effectively enhancing task completion without human intervention. By formulating the evolving process to analogize neural network training, EvoMAC provides a clear structure for defining and extracting improvements. This approach underscores the significance of iterative refinement in the software generation process, enabling continuous improvement and adaptability in complex coding tasks. \n\nTo evaluate EvoMAC, the authors introduce RSD-Bench, a novel benchmark with complex and diverse software requirements that includes automated requirement verification. EvoMAC demonstrates superior performance on RSD-Bench and the HumanEval function-level benchmark, outperforming state-of-the-art methods and showcasing its robustness and adaptability across various evolving iterations and LLM configurations.", "questions": "Please refer to the questions in Weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Effectively demonstrates EvoMAC's evolution process as analogous to neural network training, establishing a reliable target proxy for evaluation and constructing a clear objective.\n\n-  The paper is well-structured and easy to follow, with thorough explanations of EvoMAC’s self-evolving process,the design of the RSD-Bench, and detailed descriptions of experimental procedures. Figures and benchmarks illustrate the methodology effectively, aiding comprehension.\n\n- By addressing limitations in traditional MAC systems and demonstrating EvoMAC’s efficacy on challenging software-level tasks, this work sets a promising precedent for adaptive agent frameworks in automated software development, making it valuable for both research and practical applications.\n\n- The RSD-Bench is more practical in software generation evalaution, as it aligns closely with real-world software development process. By incorporating unit tests at both the task and function levels, it establishes a rigorous and precise mechanism for evaluating software generation quality. Additionally, RSD-Bench demonstrates strong human alignment (0.9922), providing a reliable evaluation metric. This paper also conducts analysis the reasonality of RSD-Bench in comparison to existing benchmarks, demonstrates its necessity. \n\n- This paper provides thorough experiments and analyses that robustly demonstrate EvoMAC’s effectiveness and performance. EvoMAC’s strong performance on both the RSD-Bench and HumanEval benchmarks highlights its high quality and efficacy in handling complex coding tasks.", "weaknesses": "- Existing studies, such as [1-4] have also explored automation in LLM-based multi-agent collaboration. Please compare the differences between EvoMAC and these works. \n\n- EvoMAC's updating process includes removing agents that have completed their tasks. Can the entire agentic workflow be replayed once a task is finished, or is the removed agent permanently excluded from further iterations? \n\n- Given that EvoMAC includes multiple evolutionary iterations, direct comparisons with standard multi-agent frameworks may not be entirely fair. Could you also provide the number of LLM calls for tasks in RSD-Bench? This metric would offer a more clearer understanding of EvoMAC’s performance.\n\n-  EvoMAC primarily focuses on models like GPT-4, Claude 3.5, and Gemini, but it is unclear if the framework can adapt to less powerful models, such as GPT-3.5 or open-source options like DeepSeek. Presenting results across a broader range of LLMs would support EvoMAC’s claims of robustness and adaptability.\n\n- Can the authors provide additional examples of the unit tests designed within RSD-Bench?\n\n- The Testing Team’s performance significantly impacts the Coding Team's potential, particularly in the HumanEval benchmark. How is the Testing Team’s performance evaluated to ensure alignment with target performance objectives and to prevent divergence? Additionally, how is the Testing Team’s performance quantified within RSD-Bench? \n\n-  The paper does not specify the stopping criteria for EvoMAC’s iterative evolution process. Could the authors provide details on the stopping mechanism or criteria? \n\n- The paper lacks specific settings for the Coding Team in both the HumanEval and RSD-Bench benchmarks. Please provide these details to improve clarity on the experimental configuration and consistency across benchmarks.\n\n- Could the authors showcase additional examples of the textual gradient analysis and the updating process during the evolution for HumanEval and RSD-Bench?\n\n\n\n[1] Liu Z, Zhang Y, Li P, et al. Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization[J]. arXiv preprint arXiv:2310.02170, 2023.\n\n[2] Zhuge M, Wang W, Kirsch L, et al. GPTSwarm: Language Agents as Optimizable Graphs[C]//Forty-first International Conference on Machine Learning.\n\n[3] Hu S, Lu C, Clune J. Automated design of agentic systems[J]. arXiv preprint arXiv:2408.08435, 2024.\n\n[4] Qian C, Xie Z, Wang Y, et al. Scaling Large-Language-Model-based Multi-Agent Collaboration[J]. arXiv preprint arXiv:2406.07155, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jw6A5D6nqh", "reviewer": "Reviewer_sExs", "summary": "The authors provide a paper with twofold contribution: i) a new approach to developing multi-agent collaboration systems for software development and ii) a benchmark to compare their approach with existing approaches to solving the same problem.\nRegarding (i), their approach (called EvoMAC) intends to overcome the limitations of similar multi-agent collaboration systems development workflows, mainly on adaptability and generalization. EvoMAC was designed to mimic standard neural network development, meaning that the errors are \"backpropagated\" throughout the agents, creating a self-adaptive multi-agent network.\nRegarding (ii), the benchmark dataset RSD-bench was created based on the requirements of the software being developed, in contrast to the existing ones, which are usually based on the functionality of the generated code/software (i.e., unit tests).\nThe paper's results show that the EvoMAC approach outperforms other approaches when applying the RSD-Bench to adapted versions of standard benchmark datasets like HumanEval.", "questions": "- When explaining their approach (section 3.1), the authors did not mention problems regarding the context window of most LLMs. Since requirements can contain quite a large amount of textual data, is the approach capable of dealing with it without extra techniques (e.g., RAG)? If not, even though the authors did not highlight it as a problem, the context window limitations should be mentioned.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is very well written and clearly explained. The two proposed contributions (the EVoMAC approach and RSD-Bench dataset) are consistent with the objective expressed in the introduction.\nAltough the problem of how to (self-)organize multi-agents is not exactly a new problem, the rise of agentic approaches using LLMs brought new traction to this challenge, and the authors addressed a pain point on these approaches when dealing with complex software development. Indeed, most solutions rely only on function-level and ignore the requirements engineering perspective, which ultimately leads the developers to half-baked solutions. This is not the case with the proposed EvoMAC. It takes account of the particularities of user requirements when organizing the agents initially and also considers the evaluation of the generated code against initial requirements.\nTheir inspiration for neural network algorithms is broad. However, it is also a clever idea that sounds original and demonstrates a creative adaptation of backpropagation principles to multi-agent collaboration.\nThe authors also provide sound experimentation on how they implement their approach and when comparing with similar solutions.\nTogether with the approach, the authors provided a well-defined benchmark to overcome the limitations of the existing ones. A more detailed description of the RSD-bench could indeed compose a contribution per se.\nThe RSD bench is tailored to requirements engineering, which could address common limitations in agent-based collaboration research by bridging functionality-based assessments with requirement-based evaluations.", "weaknesses": "Most of the paper's weaknesses are minor problems that can improve its quality, even though I don't consider them mandatory.\n- I truly believe that the mathematical explanation of the problem (mainly between lines 169-184) is unnecessary, even though I see that this somehow facilitates some later explanations. Altough it provides some kind of generalization, the approach still relies on LLMs that are probabilistic by nature and then are not mathematically generalizable (i.e., the function $\\phi(\\cdot,\\cdot)$ is essentially a message sent to an LLM, so it does not behave exactly as a mathematical generic function as the authors may want to demonstrate). I advise the removal of the mathematical explanation. It can help sometimes but does not add value to the paper.\n- The paper lacks an actual example/use case in the opposite (or complementary, if the authors decide to keep it) of the mathematical explanation. The authors use this type of explanation in lines 270-271. This kind of example can be used as a running example throughout the paper.\n- Minor problems:\n\t- Typo on the Figure 3 caption. \"indrection\"\n\t- Figures 6 and 8 can be improved to ensure legibility and accessibility, maybe by adjusting font size and contrast.\n\t- Figure 1 is a bit confusing. I think it can be split into 3 different figures or be better explained in the paper. If kept as is, I suggest adding brief explanations of the arrows, such as \"add,\" \"revise,\" and \"remove.\"\n\t- Figure 5 can be rethought with a better color choice, especially considering the accessibility of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)