Flow: Modularized Agentic Workflow Automation
Authors: Boye Niu
Yiliao Song
Kai Lian
Yifan Shen
Yu Yao
Kun Zhang
Tongliang Liu
Venue: ICLR 2025 Poster (2025)
Keywords: LLMs based Multi-Agent System
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Multi-agent frameworks powered by large language models (LLMs) have demonstrated great success in automated planning and task execution. However, the effective adjustment of agentic workflows during execution has not been well studied. An effective workflow adjustment is crucial in real-world scenarios, as the initial plan must adjust to unforeseen challenges and changing conditions in real time to ensure the efficient execution of complex tasks. In this paper, we define workflows as an activity-on-vertex (AOV) graph, which allows continuous workflow refinement by LLM agents through dynamic subtask allocation adjustment based on historical performance and previous AOVs. To further enhance framework performance, we emphasize modularity in workflow design based on evaluating parallelism and dependency complexity. With this design, our proposed multi-agent framework achieves efficient concurrent execution of subtasks, effective goal achievement, and enhanced error tolerance. Empirical results across various practical tasks demonstrate significant improvements in the efficiency of multi-agent frameworks through dynamic workflow refinement and modularization.

Published as a conference paper at ICLR 2025
FLOW: MODULARIZED AGENTIC WORKFLOW AU- TOMATION
Boye Niu', Yiliao Song', , Kai Lian', Yifan Shen', Yu Yao', Kun Zhang Tongliang Liu
Syydney AI Centre, The University of Sydney, 2he University of Adelaide SCarnegie Mellon University, *Mohamed bin Zayed University of Artificial Intelligence
ABSTRACT
Multi-agent frameworks powered by large language models (LLMs) have demon- strated great success in automated planning and task execution. However, the effective adjustment of agentic workflows during execution has not been well studied. An effective workflow adjustment is crucial in real-world scenarios, as the initial plan must adjust to unforeseen challenges and changing conditions in real time to ensure the efficient execution of complex tasks. In this paper, we define workflows as an activity-on-
(AOV) graph, which allows continuous workflow refinement by LLM agents through dynamic subtask allocation adjustment based on historical performance and previous AOVs. To further enhance framework performance, we emphasize modularity in workflow design based on evaluating parallelism and dependency complexity. With this design, our proposed multi- agent framework achieves efficient concurrent execution of subtasks, effective goal achievement, and enhanced error tolerance. Empirical results across various prac- tical tasks demonstrate significant improvements in the efficiency of multi-agent frameworks through dynamic workflow refinement and modularization. The code is available at: https: / htggttttua/// com/ tm/ Lab/ 2adab/2222 2025
INTRODUCTION
Large Language Models (LLMs) (Significant Gravitas; Zhou et al., 2023) show remarkable ability to understand and generate human- text. Recent advances have significantly enhanced their capability emulate human reasoning (Sun et al., 2024), indicating promising future for LLM- based reasoning. With the powerful ability to handle a variety natural language processing tasks, these models underpin wide range of applications, from conversational agents (Ye et al., 2024) and content creation tools (Yao et al., 2023) to advanced analytics and decision-making systems (Ramesh et al., 2021; Wang al., 2023). Building upon this foundation, key advancement the development of multi-agent frameworks empowered by LLM (Liu et al., 2023; Li et al., 2023; Hong al., 2024b; Wu al., 2024; Wang et al., 2024; Chen al., 2024; Liu et al.. 2024) where multiple L.MM-baeed agents collaborate to address complex tasks, leveraging their collective reasoning and planning abilities to automate and optimize task execution processes.
Existing LLM-based multi-agent frameworks define LLM as agent, and agents collaborate with each other via manually designed or L.MM-generatd prompts. Specifically, MetaGPT focuses on programming tasks by leveraging Standardized Operating Procedures (SOPs) (Wooldridge & Jennings, 1998; DeMarco Lister, 2013; Belbin, 2010). It predefined distinct roles such as product manager, project manager, and engineer. For each role, an LLM agent is initialized, and these agents operate within strict and sequential workflow to execute subtasks. CAMEL can complete a variety of task types. It requires sequentially, each agent taking on specific responsibilities. AutoGen is also aimed at completing diverse tasks. Unlike CAMEL, AutoGen automatically creates an agent list with different roles based on subtask requirements. These agents execute subtasks sequentially following the order in the list.
pre-define two agents. These agents interact and execute tasks
'Correspondence Tongliang Liu

---

Published as a conference paper at ICLR 2025
Task 1: Create
Task 2: Write a Gobang game with a naive Al
website for ICLR202S
1). The game must be ended when a player wina, or the board is full.
2). Date: April 27.
2). The UI should clearly indicate whose turn is and display a message when the gamn is over, specitying the winner.
2025 May 1,
3). Location: San
Francisco, California, United
States
4). Organizer: ICLR
5). create sample
X
O
CSS stylesheet
X
O
(b)c CAMEL
(g): MetaGPT
(o): AutoGen
(a): AutoGen
(e): MetaGPT
[d): Flow (our method)
CAMEL
(b): Flow mathor)
Figure 1: Comparative evaluations among four frameworks AutoGen, CAMEL, MetaGPT, and Flow (ours)-acree) across two tasks, present notable differences in performance. For the left task, AutoGen, CAMEL, and MetaGPT only managed to produce basic designs lacking in completeness while Flow excelled by creating fully developed and well-structured website. For the right task, Flow demonstrated superior capability by successfully generating a working game with a clear and intuitive interface, while the other frameworks struggled deliver fully functional code.
Building upon the strengths of current multi-agent frameworks, our work aims to further improve existing general- peneral-purpose purpose multi-agent frameworks by enabling dynamically updating workflows during task execution and encouraging modularity in workflows when planning the workflows.
Specifically, dynamic updating workflows allow agents to adjust subtask allocations and agent roles in real-time based on ongoing performance feedback and changing conditions. This capability ensures that the system remains responsive and efficient even when faced with unexpected obstacles. For instance, if an agent encounters a roadblock in data preprocessing, the system can reassign this subtask to another agent or introduce a new subtask to resolve the issue. This adaptability is essential for maintaining robustness and ensuring seamless execution of complex tasks.
Modularity in system design involves dividing a system into separate, independently operating mod- ules, each responsible for specific functionalities (Baldwin & Clark, 1999). In our context, modularity refers to the decomposition of a complex task into smaller, interchangeable subtask modules. A highly modularized workflow enables subtasks execute concurrently, without bottlenecks from other parts of the workflow, and thereby directly improves the operational efficiency of multi-agent frameworks. Furthermore, modularity enhances the ease of dynamic updating. When workflows are highly modularized, the dependency complexity between subtasks is minimal. Therefore, updating one subtask does not affect others, allowing for small workflow adjustments. For example, if an agent responsible for data preprocessing encounters an unexpected obstacle, system of high modularity can adapt by introducing only one subtask with minimal impact on the rest of the workflow.
In this paper, we enhance existing multi-agent frameworks by achieving modularity and enabling dynamic workflow updates. Our framework allows agents to execute their subtasks in parallel while facilitating efficient workflow updates. This is accomplished by formulating the entire workflow as an Activity-on -ae--oo--eeeeeeee Vertex (AOV) graph, which a directed acyelic graph (DAG) where each subtask is represented as node with its status and generated logs, while the directed edges capture dependencies between subtasks. To encourage modular workflow design from the beginning, we generate multiple candidate AOV graphs for the task. These candidates then evaluated based their degree of
2

---

Published as a conference paper at ICLR 2025
parallelism and the complexity of their dependencies. The AOV graph with the highest parallelism and lowest dependency complexity is selected.
During task execution, our framework continuously checks and refines the workflow, updating it when a subtask fails (see Fig. 2: Check Refine). The framework updates subtask allocations and agent roles based on ongoing performance data and the current workflow. As our AOV-based workflow encourages high modularity, updating one module does not necessarily affect others, allowing for localized adjustments during workflow updates (see Fig. 2: Update). Similar to the initial workflow generation, multiple AOV graphs are generated and the one with highest parallelism and lowest dependency complexity is selected during dynamic updates. This iterative workflow refinement process enhances udaptability to new challenges and evolving objectives throughout task execution, ensuring dynamic workflow updates without compromising overall performance.
Contributions: 1) We introduce and encourage modularity in multi-agent workflows, emphasizing the design of workflows with high parallelism and low dependency complexity. This modular design enhances efficiency, robustness, and scalability by enabling concurrent subtask execution and minimizing bottlenecks caused by complex interdependence. 2) We propose a practical multi-agent framework that supports highly flexible updates to the workflow during runtime. Our method enables local updates to the entire workflow based on global information, allowing agents to efficiently adapt to unexpected challenges while maintaining system coherence and consistency. 3) Through comprehensive experiments, we demonstrate significant improvements in both the adaptability and efficiency of our multi-agent framework compared to existing approaches.
RELATED WORK
LLM-based Task Decision-Making, Recent developments in LLM-based task decision making have focused on improving the reasoning and planning abilities of agents (Yao et al., 2023; Song et al., 2023; Zhou et al., 2024; Prasad et al., 2023; Shinn et al., 2023; Ahn et al., 2022; Teng et al., 2025). Previous approaches like ReAct (Yao et al., 2023) iteratively generate thoughts and actions based on current observations until task completion. This framework integrates action-taking with reasoning, allowing agents to perform complex tasks in dynamic environments. Reflexion (Shinn et al., 2023) further improves this by incorporating self-reflection, where the agent evaluates and adjusts its reasoning during execution. ADAPT (Prasad et al., 2023) introduces recursive task decomposition, enabling LLM-based agents break tasks into smaller subtasks, which leads improved task execution flexibility. However, these approaches often overlook dynamic task reallocation, particularly in multi-agent settings, which is where our work extends current research.
LLM-based Multi-Agent Frameworks. Multi-agent frameworks have long been employed for task execution in distributed environments, with recent advances leveraging LLM to enhance coordination and decision-making (Hong et al., 2024b; Li et al., 2023; Wu et al., 2024; Hu et al., 2024; Zhang et al., 2024; Trirat et al., 2024). However, existing frameworks often rely on static workflows with limited adaptability to changes in the task environment. DyLAN (Liu et al., 2024) and MACNET (Qian et al., 2024) utilize static graphs to represent workflows in multi-agent frameworks; GPTSwarm (Zhuge et al., 2024) enhances agent interactions but maintains a fixed agent topology: Datalnterpreter (Hong et al., 2024a) updates workflows primarily in response to execution failures in subtasks, adjusting subsequent tasks while leaving completed tasks unchanged; AFlow (Zhang et al., 2025) introduces a dynamic workflow generation framework based on Monte Carlo Tree Search, enabling adaptive adjustments through iterative code modification. This highlights the need for dynamic workflow updates.
3 THE PROPOSED MULTI-AGENT FRAMEWORK: FLOW
Our proposed Flow enhances multi-agent frameworks powered by LLM by introducing modularity and dynamic workflow updating. As depicted in Fig. 2, given the task requirement, Flow first formuu lates the initial workflow for execution plan generation and agent allocation. During execution, the workflow is continuously refined and dynamically updated until the task is completed. To maximize system simplicity and flexibility, we design a dictionary-basse structure for implementation. In the following, detail how achieve these features.
3

---

Published as a conference paper at ICLR 2025
Task requirement as human prompt:: XXX preparing lecture slides on XXX]
Agents
FLOW-Reve R-WW-eevised prompt [Plesse complete this necessary subtasks
encourages min (dependency) max maxppaatleettmm]] [task requtrements)
Initialization
FLOW Assign
Agents
agent #1: are [role #1]
you are taking subtask [1,4 requirements
agent you [role d2]
2
you taking subtask [2,5 requirements
agent #3: you are [tole #3)
you are taking subtask reguirements
Check & Refine
Step1
Output for subtask1
Wait running
Subtask1
Checkings
subtasks fintsh
requfrements
Updater
Output for subtask1
Agent1
Subtasks output Current workflow
Step2
Output for
Subtask2
subtask2
requtrements
Update
Agent
Output for subtask2
Output for subtask3
Checking
requirements
Agent3
Output sub
Continue with new graph...
Undating-
Figure 2: The process starts with task initialization, encouraging the modularity and execute parallel of subtasks. Outputs are evaluated. errors are detected, the workflow is dynamically updated by modifying the task graph. This iterative process continues until successful task completion.
Formulating Workflow AOV graph. Activity Vertex (AOV) graph is type of directed acyclic graph where vertices represent subtasks and edges denote precedence relations (Bondy & Murty, 2011). AOV graphs are widely used in project scheduling and management (Moder et al., 1983; Taha, 2017), helping planners visualize dependencies and sequence subtasks efficiently.
Inspired by that, we define the multi-agent workflow as an AOV graph where vertices represent subtasks, while edges denote dependencies between subtasks. Let G = (V, E, A) denote the AOV graph, with the set of all subtasks (vertices), C x V the set of directed edges indicating subtask dependencies. For example, eij (vi, v3) E indicates that the subtask e; must be completed before the subtask vy starts. represents set of agents for all subtasks. Each agent ú, A is associated with role that is responsible for executing subset of subtasks T, V.
Note that AutoGen also automatically generates subtasks and agents. However, the subtasks are designed to be executed sequentially. For Flow, we allow for the generation of subtasks that can run in parallel. This distinction enhances our framework's 's ability to handle multiple subtasks simultaneously, which reduces overall process time and increases efficiency.
Modularity in a Workflow. Modularity in system design (Baldwin & Clark, 1999) involves dividing system into separate, independently operating modules, each responsible for specific functionalities, allowing focus on individual components without affecting the entire system. It is essential for scalability and flexibility in workflows. By reducing dependency complexity, the system can more easily adapt to changes, such as the introduction of new tasks or the reassignment of existing ones, without requiring extensive restructuring. Theorem demonstrates that additional dependencies in workflow reduce the expected success rate of subtasks. Following this conclusion, Flow advocates for the creation of subtasks that can be executed independently.
4

---

Published as a conference paper at ICLR 2025
Theorem 1. Consider two topologically sorted workflows A and B each consisting of N subtasks according to their execution order. Suppose
1. (Random fail probability) Each subtask v E T fails with probability pf. where O < Pf
2. (Additional dependency in Workflow B) There exist at least one subtask v² E T and a subtask E T such that the set of immediate predecessors (dependencies) of o² in Workflow B is
Dn(v*) (a(w")Us {b}, where D Da(v") the set of immediate predecessors of in Workflow A. For all other subtasks v # 2" Da(u) S DD()).
The expected number of completed subtasks Workflow is strictly greater than Workflow E[SA] B[S
To encourage modularity in the generated AOV graph, we define two quantitative measures that evaluate parallelism and dependency complexity respectively. Parallelism measures the extent to which subtasks can be executed concurrently, Let S; represent the set of subtasks executed in the t step. Let T be the total number of steps (the maximum depth of the DAG). Given an AOV graph G = (V, E, A), the degree of parallelism overall is defined as the average subtask ratio over steps:
Pavg T
Σ S
Although Pang provides measure of parallelism, it is insufficient to fully capture the modularity that arises when subtasks can be executed independently. Consider two workflows, both containing the same subtasks {A, B, C, D}. For Workflow the task dependencies defined
A C, C,A-A + D + D,C + D. In contrast, Workflow has dependencies: + C - C,C Although both workflows exhibit the same level of parallelism, Workflow 2 is structurally simpler in terms of task dependencies, as it contains fewer edges.
To account for this complexity, distribution within the subtask graph. For each subtask v;, we define deg(n;) as the number of direct connections it has the graph G. The dependency complexity is quantified by the standard deviation of the number of direct connections:
measure the dependency structure by analyzing the degree
Cdependency = deg((v.
Σ PVEV
(deg(v;) - -d)2.
V
This measure reflects the variability in the number of dependencies each subtask has, providing insight into the overall complexity of the workflow structure.
Task dependencies alone are insufficient to fully capture the modularity that allows subtasks to be executed independently. Consider Workflow 3: A - - C D, which may have similar dependency complexity to Workflow 2. However, Workflow 2 provides greater modularity and separation of subtasks, highlighting the importance of evaluating both dependency complexity and modularity to fully assess and promote effective workflow designs. Both measures are essential to ensure that subtasks can be executed parallel while maintaining a modular approach.
Sample Prompt for Initialization Puni
You aze an intelligent workSlew planner, Giver the following task requirements, generate set of necessary subtasks along with their dependencies and assign appropriate agents to each task, Ensure that tasks that can be executed in paral lel identified to arhance efficiancy.
workflow should ba represented dictionary where each key is task and its value contains the task' status, tasks, and assignad agant.
dota, number parentss not completad,
Task Requicements: "TAOK_AEOOIAMMINTT)
Output Format: [ "Task_A": I "status": "not started', "data": null,
nuu_prrenta_nott_oopleeett c, "child"; ["Task_3", "Taak_c"), "agent"; "Agent_1" 3, B"a "status"; "not stacted", "data": null, "sum_parente _eo_ecopees____ 1, "chf ["Taak_""), "agent": "Agent_2" b. ...
5

---

Published as a conference paper at ICLR 2025
Generate an Initial AOV Graph. Given a task requirement prompt P, we prompt an LLM f to generate a set of candidate AOV graphs {G1, G2...., GK} based on P and a designed prompt for initialization Panns L..,, {G1, G2....,, G f(P min: P). Each candidate AOV graph Gy- (Ve, Fg. A&) is evaluated using the measures of parallelism and dependency complexity. We prioritize the workflow with the highest parallelism score. If multiple graphs share the highest score, we select the one with the lowest dependency complexity.
Note that we prioritize parallelism and modularity early in the process and focus on refining the workflow through data-driven adjustments during running, The reasons are: 1) LLM-generated workflows possess reasoning capabilities, but may not prioritize efficiency. parallelism and independence are not explicitly encouraged during the initial workflow generation, the applied workflow is very likely to be overly complex, which results in inefficient subtask implementation; 2) verifying correctness is inherently challenging as no additional data is available as supervised information at an early stage. As compensation, we refine the workflow by parallelism and modularity.
Execution Plan Generation and Agent Allocation. After we obtain the best AOV graph, topological sort is performed on the dependency graph of the subtasks to produce a linear order the subtasks o : V + [1,2.... VV} such that for any edge (v,. vy)) E E, o(v) < <(ty). The result is sequence of subtask steps, where each step consists of subtasks that can be executed in parallel. This execution plan minimizes the number of steps needed to perform while ensuring that all subtasks are completed in the shortest possible time, adhering to their dependencies.
Each agent aj A is associated with set of subtasks T S V, indicating the subtasks that the agent is responsible for handling. However, two subtasks Up and By require the same agent at the same step 8i, we create clone of the agent, denoted a/,, to run both subtasks simultaneously without increasing the waiting time.
Prompt for Update P Papdate
You ace an intelligent workClew updater, Based on the current workflow and the all subtasks' progress data, update the workflow for acheving objective by adding, removing, or modifying sustasks as necessazy, Bnaure that the updated workflow maintains modularity and maximizes parallel execution. Output rormat: "Task_A"; I "status"; "not started", "data": null,
Workflow Refinement and Dynamic Updating, We leverage LLM as global inspector to continu- ously monitor task progress and dynamically modify the AOV graph based on global information when necessary. Specifically, given the task requirements prompt P and the update prompt P Pupdate- the current AOV graph G and the generated data D' containing the status of subtasks and the output of agents to run subtasks. Similarly to the initialization process, we generate K candidate [(Pupdate: P, D³). We follow the same selection strategy as in initialization, which prioritizes the workflow with the highest parallelism score and further selects the one with the lowest dependency complexity if multiple graphs share the highest parallelism score.
graphs: {G]+] Go G++1 ..... Gk* "
With the modularity constraint introduced in previous sessions, dynamic updates can largely fulfill flexibility, allowing modifications to subtask allocations including deletion, addition, editing, rerunning, and reassignment of agents without necessarily affecting other agents their assigned subtasks. Namely, one subtask v; can be replaced by collection of subtasks V after updating. This unique advantage is particularly beneficial when subtask requirements become more challenging, as subtask dependencies can be highly complex.
Note that with sufficient data and computational resources, we could further enhance our framework by fine-tuning LLM with reinforcement learning for workflow generation. For example, the LLM would be trained maximize a reward function designed around key performance indicators such as task completion speed, resource utilization, and minimization of workflow disruptions.
Implementation. Our framework employs a dictionary- structure, G. to efficiently manage and dynamically update workflows within a multi agget framework. Each subtask in the workflow is represented key G, the value being another dictionary that encapsulates various attributes of the subtask. The structure is specifically defined as:
c(v) ("subtask requirement", "status", "data", "num caatrentsnootccompleee "child", "agent").

---

Published as a conference paper at ICLR 2025
In each Clu], the values of each key follows:
"subtask requirement": the text of the task requirement;
"status": the current task implementation status e.g. "not started", "in progress", "completed";
"data": data relevant to this task;
"num parents not comp_ceeeee""" the count of uncompleted parent tasks to manage dependencies;
"child"; list of child tasks that depend on the current task's s completion;
"agent": the agent assigned to the task.
This dictionary-based structure can be converted directly to JSON, and the organized infor- mation is easily readable and summarizable by LLM, granting system inherent simplicity and flexibility. "num parents_ not completed". Subtasks with count of zero are eligible run concurrently, leveraging our system's capability to handle parallel subtask execution effectively. Upon com- pletion of each subtask, we perform systematic review to determine if the workflow requires refinement, ensuring that all dependencies are accurately accounted for and that the workflow remains aligned with project goals. In addition to monitoring the subtask completion by the "status" and "num_ paaiinpaaetts.... ______ot_ooog___ "oo__oopleted" counts reported by agents. Flow also double-checks the completion of each subtask by asking all the requirements of this subtask are fulfilled. This will largely prevent errors from inaccurate reporting by agents or unforeseen system anomalies. This rigorous verification process enhances the reliability and integrity of our workflow management system.
addition, each subtask execution readiness is determined by the attribute
4 EXPERIMENTS
Baselines. In all experiments, we compare Flow to existing multi-agent frameworks: (1) AutoGen (2) Camel, and (3) MetaGPT. In our experiments, we use agents empowered by GPT--4-min and GPT-3.5-Turbe (OpenAI, 2024).
Experiment Design. We designed three diverse and engaging tasks evaluate multi aagtt--geett collabo- ration frameworks: 1) website design, 2) LaTeX Beamer writing, and 3) gobang game development. The rationale for selecting coding-based experiments twofold. First, most multi-agent frameworks, such as MetaGPT are optimized for coding and writing tasks. Using non-coding tasks could intro- duce bias. Second, coding tasks effectively showcase the ability of a framework to assign agents and manage task allocation.
Gobang Game Development: This task requires creating a gobang game with user interface and simple AI opponent. Players can choose between black white stones, with the UI clearly indicating turns and announcing the winner or draw when the game ends. This task demonstrates the framework's 's ability handle modular design and task parallelism, it involves coordinating game logic, AI implementation, and user interface development simultaneously.
LaTeX Beamer Writing: This task focuses generating LaTeX slides that cover reinforcement learning algorithms, including motivations, problem statements, intuitive solutions, and detailed mathematical equations. A specific page requirement to the framework's 's ability to follow instructions precisely. The task highlights the framework's 's parallel processing capabilities of simulta- neous generation of content, formatting, and presentation structure. The structured format of LaTeX also tests how effectively the framework manages modularity and concurrent tasks.
Website Design: This task builds a professional website for the International Conference on Learning Representations, hypothetically scheduled in San Francisco from April 27 to May 1, 2025. The website must feature key elements such as a detailed conference schedule and venue information with an interactive map. This task assesses frameworks' ability to manage parallel workflows and modular components, including user interface design, functionality, and adherence to design guidelines, showcasing how well framework can handle complex task decomposition and execution.
4.1 EVALUATIONS OVER THREE DESIGNED TASKS
Evaluation Metries. To conduct both quantitative and qualitative evaluations, employ two metrics: Success Rate and Human Rating. The success rate is quantitative measure that ranges from 0 to 1. It assesses whether the multi-agent framework successfully generates executable outputs
7

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of different multi-agent frameworks on Gobang Game Development
Success Rate (%)
Human Rating
Model
Compilable
Interactable
Game Rule
Overall Score
(1-4)
AutoGen
80
60
40
60
2.26
MetaGPT
100
100
20
73
1.24
CAMEL
40
40
0
27
2.50
Flow (Ours)
100
100
100
100
4.00
Table 2: Comparison of different multi-agent agent frameworks on LaTeX Beamer Writing
Success Rate ( (%)
Human Rating
Model
Compilable
Completeness
Page Limit
Overall Score
(1-4)
AutoGen
80
80
40
67
3.00
MetaGPT
80
80
20
60
1.83
CAMEL
100
100
0
66
1.83
Flow (Ours)
100
100
100
100
3.33
Table 3: Comparison of different multi-agent frameworks Website Design
Success Rate (%)
Human Rating
Model
Compilable
Basic Information
Sections
Overall Score
(1-4)
AutoGen
80
80
60
73
2.62
MetaGPT
100
100
40
1.72
CAMEL
80
80
2.02
Flow (Ours)
3.28
that fully meet the task requirements. A higher score indicates a greater level of success accurately fulfilling the task objectives. Different tasks may have different evaluation metrics. The description for each evaluation metric is defined Appendix B.I, B.2 and B.3. Human ratings are used to evaluate the quality of the generated results in alignment with the task description. We gathered 50 participants with programming and machine learning backgrounds to rank the outcomes produced by different methods. A detailed description of how take scores is shown in Appendix A.
Summary. We summarize the performance of different methods on three tasks from Table 1, and 3, comparing the overall score with respect the success rate and human rating. For Flow, the overall score and human rating over three tasks are (100, 4) on game development, (100, 3.33) on LaTeX writing, and (80, 3.28) on website design. Thus, the average performance of Flow is 93% success rate and 3.54 out of 4 in human satisfaction. Similarly, we have the average performance of AutoGen as (66.7, 2.63). MetaGPT as (71, 1.60), and CAMEL as (48.67, 2. 12). Overall, our method Flow has completed tasks with the most satisfaction and the highest success rate. Information about Flow's ' workflow on those tasks is in Appendix
4.2 RESULT FOR GOBANG GAME DEVELOPMENT
The experimental setup is thoroughly detailed in Appendix B.2 and the visualization result is shown in Fig. 1. As shown in Table 1, Flow achieves 100% success rate across all aspects, along with the highest human satisfaction. More explanations for each framework are given below.
AutoGen: AutoGen was evaluated across five trials. Of these, four produced valid, executable code. One instance encountered runtime error that impacted execution, and another revealed a minor game interface bug. However, the remaining two trials, the game flow proceeded correctly, though the chess pieces appeared the text "black" and "white" instead of graphical icons.
MetaGPT: MetaGPT produced executable code in all five attempts, demonstrating its overall capabil- ity for code generation. However, in four of these runs, it unexpectedly created Tie-Tac-Toe game rather than Gobang. Only one these trials delivered a fully functional version that allowed both the user and the AI to take turns properly, concluding the game the winning condition was met.
CAMEL: CAMEL was tested five times, successfully generating executable code in two cases. However, these successes did not include Al opponent. The remaining three trials were non-
8

---

Published as a conference paper at ICLR 2025
(Cisata torapagel
Frease homepape)
(Crade steal pege)
(Dreet desutpappd
(Dosts Schedukk pagel
(Drast Relaase papel
[Deline stazzan)
pasilop CSa)
Drtegette (ont
papel
[bongane
[Cruata Regi angiaration pagel
Regiatratio
(Crode Speaker
(Dreto Speater
(Couta Costoct page]
Creato Contaut page)
vilatees 20.Dt0h
The gamaaaate start
The workflow rellred broed generdted of Dofine the website struturee
clectiinn Beneruted cate
aa) Website Design: no newly added subtask, only the workflow is updated.
game
IA module reseaign
Datree Desigg ga pargraac Thhe warkwyy
(kaggeaa comporant - (lartndweryy
(Datie Regeranve)
Two tadik completed with no dutput
(b) Gohang Game Development: adding subtasks replace bad ones.
Figure 3: Workflow and dynamic update in two experiments.
executable due to various errors. Despite these hurdles, CAMEL's success cases demonstrate its basic proficiency in generating operational game logic.
Flow: Under the same test conditions, Flow consistently produced valid outputs in all five attempts without any observable errors. The game behaved as intended, enabling both the player and straightforward Al to alternate moves. Flow also successfully rendered the board using black and white piece icons, offering a more visually engaging interface compared text-only labels.
4.3 RESULT FOR LATEX BEAMER WRITING
Experimental results are presented in Table with the following explanations:
AutoGen: Across five trials, AutoGen successfully generated an output in all five trials. However, in one case, the LaTeX file failed compile due syntax errors, and two instances, the outputs did not meet the required length. The remaining trials produced documents that satisfied both length and content requirements.
MetaGP7: We ran MetaGPT five times, and in four of these trials, it produced valid LaTeX files.
The primary issue in the remaining trial was that Python code was included within the ".tex" file. Although the documents met the required content specifications, only one met the prescribed page length of either or 20 pages.
CAMEL: CAMEL generated five valid ".tex" files that successfully compiled into Beamer presen- tations, each containing the necessary sections such as motivation. However, none achieved the specified page length of 30 20 pages in our tests.
Flow consistently produced LaTeX compilable outputs across five trials. While one out- put included some repetitive content, the resulting Beamer presentations met the specified length requirements and covered all required material.
Flow:
4.4 RESULT FOR WEBSITE DESIGN
Similarly to the previous two, the detailed experiment setup is in Appendix B.3. Here we illustrate the results in Table 3 as follows:
AutoGen: AutoGen successfully produced HTML websites in four out of five trials, showcasing a generally reliable code-generation process. However, in trial, the output contained only minimal

---

Published as a conference paper at ICLR 2025
content, just few sentences per section indicating challenges in generating more detailed or feature-rich outputs.
MetaGPT: MetaGPT generated valid HTML and CSS in all five attempts, showcasing its ability to produce functional web content. However, the outputs were relatively simple, lacking advanced design elements, and often omitted essential sections such dedicated venue section and map.
CAMEL: CAMEL successfully delivered four out of five executable websites. However, when required, the results occasionally incomplete. In one instance, it
additional complexity produced only the HTML file, omitting the accompanying CSS, indicating potential challenges in managing multi-file dependencies.
Flow: Flow successfully generated fully functional websites in four out of five trials. these cases, the websites featured detailed descriptions and essential interactive elements. However, in one trial, errors occurred when integrating different files, preventing successful execution.
WORKFLOW UPDATE
Update based Generated Data. Fig. 3(a) demonstrates the update process of Flow in the website design example. Upon completion of the first subtask, the system identifies potential changes and redundancies, triggering a restructuring process improve efficiency. Once the subtask "Define the website structure" is completed, the generated data, which includes HTML structures and elements, sufficient to proceed with the CSS creation. As result, the workflow is updated to incorporate the development of CSS based on the completed "Define the website structure" subtask.
Fig. 3(b) illustrates result of our dynamic updating process, where the system, upon receiving information about completed subtasks, decides to add bridging subtask to handle gaps and ensure that the workflow continues smoothly.
Error Handling. To evaluate the effec- tiveness of our update mechanism, we in- tentionally introduced random masking certain subtask outputs, replacing them with "none" before passing them to the next agent. We conducted five trials and recorded the success scores. Since other frameworks employ sequential workflow, we limit the comparison to our own ap- proach in this context.
Table 4: Success Rate (%) of Error handling with dy- namically updating.
Task
Flow w/o Update
Flow
Website Design Gobang Game Development
46
87
93
LaTeX Beamer Writing
67
93
We observed significant difference in the success rate between using dynamic update and not, particularly in the Interactive Game section as shown in Table 4. The main issue arises when the previous agent fails provide the necessary information. yet the second agent continues with its subtask, leading to a major disconnect in the code. This often results in Python being unable to compile due to missing or mismatched components. Similarly, website design, the lack of required elements caused by this failure impacts the overall functionality and structure. During the execution of subtasks, errors may arise due to the limitations the LLM-based agent or underperformance in certain tasks. Therefore, the ability to dynamically update the agent workflow to address such issues essential.
CONCLUSION
We present Flow, novel LLM-based multi-agent framework that can dynamically adapt to unforeseen challenges for general task executions. By dynamically updating the agentic workflow using AOV graphs, our framework has largely fulfilled the modularity requirements to complete complex tasks. We demonstrate our method through case studies on a series of experiments, ranging from website design, game development, and LaTeX Beamer writing, Through objective evaluation metrics and human feedback, we found that Flow improves execution efficiency, offers better error tolerance, and delivers overall stronger performance.
10

---

Published as a conference paper at ICLR 2025
ETHIC STATEMENT
Our work follows the ICLR Code of Ethics. All data used are anonymized, eliminating any potential privacy concerns. The study was conducted in compliance with the ethical guidelines of The University of Adelaide Human Research Ethics Committee with approval number H-2025-085.
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as i can, not as i say: Grounding language in robotic affordances, 2022. URL https: //aaivv.aiivv///aa///aeeaaaaaiia org/ abs 2204 01691.
Carliss Y. Baldwin and Kim B. Clark. Design Rules: The Power of Modularity Volume 1. MIT Press, Cambridge, MA, USA, 1999. ISBN 0262024667.
R.M. Belbin. Team Roles at Work. Butterworth 2010. ISBN 9781856178006, URL
https: //wokk/...s///////////.o.././k.. google..oo.oooooooooooooo com au htt..e/eoookii.-a///aaaaaa
A. Bondy and U.S.R. Murty. Graph Theory. Graduate Texts in Mathematics. Springer Lon- don, 2011. ISBN 978199788688 7781846289699, URL https: google.oooooooooooooooooooooo www.gle.com..///oooooo au.boook HuDEWWZOwcSC.
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang. Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, Yujia Qin, Xin Cong. Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreve .ne forum? Doooum??d--ERsSSDnyy.
T. DeMarco and T.R. Lister. Peopleware: Productive Projects and Teams. Addison-Wesler 2013. ISBN 9780321934116. URL https::/bbott google. com. ww/bokksbuuuaoooooooooo.oaoa ?id= DV1sAQAACBAJ.
Sirui Hong. Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Chenxing Wei, Danyang Li. Jiaqi Chen, Jiayi Zhang, Jinlin Wang, Li Zhang, Lingyao Zhang, Min Yang. Mingchen Zhuge, Taicheng Guo, Tuo Zhou, Wei Tao, Xiangru Tang, Xiangtao Lu, Xiawu Zheng, Xinbing Liang, Yaying Fei, Yuheng Cheng, Zhibin Gou, Zongze Xu, and Chenglin Wu. Data interpreter: An Ilm agent for data science, 2024a. URL https: peteps://arcee www.wviwggg........o abs 2402 18679.
Sirui Hong. Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang. Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta programming for multi-agent collaborative framework. In The Tweifth International Conference on Learning Representations. 2024b. URL https: /wwpeeeeeeeeee.eeeoe//pp..oe..we net/ httpr/wwww..eeeommniiiieeeee
Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems, 2024. URL https: //arxiv.org/abb... / /2408 08435.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large language model society. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Zhiwei Liu, Weiran Yao, Jianguo Zhang, Le Xue, Shelby Heinecke, Rithesh Murthy, Yihao Feng, Zeyuan Chen. Juan Carlos Niebles, Devansh Arpit, Ran Xu, Phil Mui, Huan Wang, Caiming Xiong, and Silvio Savarese. Bolaa: Benchmarking and orchestrating Ilm-augmented autonomous agents, 2023. URL https://arxii orgg wwwooor//////eeooo////o///eeaes/ /230/b///222. 05960.

---

Published as a conference paper at ICLR 2025
Zijun Liu, Yanzhe Zhang, Peng Li. Yang Liu, and Diyi Yang. A dynamic I1m- iin-po agent network for task-oriented agent collaboration, 2024. URL https:://// http:///axxvvv.iiee abs 2310. 02170.
J.J. Moder, C.R. Phillips, and E.W. Davis. Project Management with CPM, PERT, and Precedence Diagramming. Van Nostrand Reinhold, 1983. ISBN 9780442254155. URL aattps://bookks google. com, au au/ /books ?1
OpenAL. Gpt-4o mini: Advancing cost-efficient intelligence. https:: com/ index/ 2024-09-29.
2024. Accessed:
Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. Adapt: As-needed decomposition and planning with language models, arXiv, 2023.
Chen Qian, Zihao Xie, Yifei Wang, Wei Liu, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Scaling large-lagguage-modee mode----aad-d multi-agent collaboration, 2024. URL
2406. 07155.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation, 2021. URL /ecttss//arxiv...o abs/ / 2.102.12092
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R. Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Thirty-seventhh Conference on Neural Information Processing Systems, 2023. URL httpps://ppenreeeee.ntttt///// wtw.r/eumwee//oommmmmmmm
VAE1
Significant Gravitas, AutoGPT.
out. MIT License.
Chan Hee Song. Brian M. Sadler, Jiaman Wu, Wei-Lun Chao, Clayton Washington, and Yu Su. LIm-planner: Few-shot grounded planning for embodied agents with large language models, In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2986-2997, 2023. doi: 10.1 N0.1100/CCCV55100.2222222 000200.
Hongda Sun, Weikai Xu, Wei Liu, Jian Luan, Bin Wang. Shuo Shang, Ji-Rong Wen, and Rui Yan. DetermLR: R: Augmenting L.L.-based logical reasoning from indeterminacy to determinacy. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9828 98889966, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 18653/v//1/ 2024. URL https: / /a//ccaaall http//aaaaaanooooooooooaoo org 2024 1e..22..cc---1 acl-lonlgnnol long.lonn..... 531.
H.A. Taha. Operations Research an Introduction. Pearson, 2017. ISBN 9780134444017. URL
https: //books...///////////..//o.// .ooooks.goolle.oomoo google. com
Fengwei Teng, Zhaoyang Yu, Quan Shi, Jiayi Zhang, Chenglin Wu, and Yuyu Luo. Atom of thoughts for markov Ilm test-time scaling, 2025. URL AtLt:t:s:///xii org/ abs 225.2.22.2110 12018. 8.
Patara Trirat, Wonyong Jeong, and Sung Ju Hwang. Automl-agent: A multi-agent I1m framework for full-pipeline automl, 2024. URL https: http://aaa...///////.. /2cbb///44410 22100 02958.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models, 2023. URL https::///// /tt::///axxivvii www.wviwwg............o.. abs//3055//35/ 16291.
Yaoxiang Wang, Zhiyong Wu, Junfeng Yao, and Jinsong Su. Tdag: A multi-agent framework based dynamic task decomposition and agent generation, 2024. URL https: // //raii.ooooo////...............a abs/ /2ls/2402.00 10177 10178.
Michael Wooldridge and Nicholas R. Jennings. Pitfalls of agent-oriented development. In Proceedings of the Second International Conference Autonomous Agents, AGENTS '98, pp. 385-391, New York. NY. USA, 1998. Association for Computing Machinery. ISBN 0897919831. doi: 10.1 2011145220765...o 280867. URL https::/doii http:////oooooo//././oo///// 1111///0011111287555 280867.
12

---

Published as a conference paper at ICLR 2025
Qingyun Wu, Gagan Bansal, Jieyu Zhang. Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen Ilm applications via multi-agent conversation. In ICLR 2024 Workshop Large Language Model (LLM) Agents, 2024.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https: /ttppp:://peeeeeee http:///ppeeeeeeeeeeeeeee new.iiewweeeeeeeeeeeee.eee..ee forrmm?r
id-wE_ VVLWEE_L
Yining Ye, Xin Cong, Shizuo Tian, Yujia Qin, Chong Liu, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Rational decision-making agent with internalized utility judgment, 2024. URL https:
// //oopenreviwww./ net
Guibin Zhang, Yanwei Yue, Zhixun Li, Sukwon Yun, Guancheng Wan, Kun Wang. Dawei Cheng, Jeffrey Xu Yu, and Tianlong Chen. Cut the crap: An economical communication pipeline for Ilm-based multi-agent systems, 2024. URL hwwps: /ttp://aaavvv//////a/ettttiaa org abs/ 2410. 02506.
Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xiong- Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng. Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. AFlow: Automating agentic workflow generation. In The Thirteenth International Confer- ence Learning Representations, 2025. URL https:/ /wwp//oveewww..////////pp net /tr/ffruus forum?ide 25uVAKwmjf.
Andy Zhou, Kai Yan, Michal Shlapentokh-Rotten Rothman, Haohan Wang, and Yu-Xiong Wang. Lan- guage agent tree search unifies reasoning, acting, and planning in language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning. volume 235 of Proceedings of Machine Learning Research, pp. 2024. URL https: http///poceeeee.n...cem// mlr www.pppssseepppppppppp.ep 225///233//////// 35/ zhouz html.
PMLR, 21-27 Jul
Wangchunshu Zhou, Yuchen Eleanor Jiang, Long Li, Jialong Wu, Tiannan Wang. Shi Qiu, Jintian Zhang, Jing Chen, Ruipu Wu, Shuai Wang, Shiding Zhu, Jiyu Chen, Wentao Zhang, Xiangru Tang, Ningyu Zhang, Huajun Chen, Peng Cui, and Mrinmaya Sachan. Agents: An open- framework for autonomous language agents. 2023. URL https: httpppps://axiv..////e wttp://..rgga////aeeaeeeaee 07870.
Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. Language agents as optimizable graphs, 2024. URL https: http://:///xxx...///// abs/2402. 16823.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
CONTENTS
Human Evaluation Process
15
Experiment setups
15
B.1 Experiment setup: LaTeX Beamer Writing
15
B.2 Experiment setup: Gobang Game Development
17
B.3 Experiment setup: Website Design
17
B.4 How Different LLM Affect Updates
18
B.5 How Different LLM Affect Performance
18
B.6 Time Cost of Different Baseline
19
Custom Metries for Parallelism and Dependency
20
C.1 Parallelism Metries
20
C.2 Dependency Metries
20
C.3 Proposed Metrics for Task Workflow Evaluation
20
Examples of Flow's Workflow
20
D.I Example Workflow
D.2 Pseudocode for updating AOV
22
D.3 Prompt for Workflow Update
24
D.4 Workflow Update Strategics
24
E Framework of the Multi-Agent framework
25
E.1 Overview
25
E.2 Key Components
25
E.3 Workflow Exceution Process
26
Limitation and Future Work
26
Pilot Study on Auto-validation
27
Proof Theorem
27
14

---

Published as a conference paper at ICLR 2025
A HUMAN EVALUATION PROCESS
Sometimes, LLM can correctly fulfill each requirement of task, but the quality of completion may vary. In such cases, human evaluation is necessary the final output of each multi-agent framework was evaluated by 50 participants, who ranked the outputs from best to worst. Points were awarded based on the rankings, with the 1st place receiving 4 points, the 2nd place receiving 3 points, etc. The final result was determined by calculating the average score. The detailed distribution is shown Fig.
the quality of the output. For each task,
B EXPERIMENT SETUPS
EXPERIMENT SETUP: LATEX BEAMER WRITING
User input
learning course research students, preparing lecture slides on various reinforcemeat learning algorithms. that:
lecturez Leaching
1) Giver that the lecture doratioo 15 hours, the sl ides should approximately
30 pages.
21 For each veinforcament learning algorithm covered, the slides will include [ollowing key componente: the mot motivation behind the algorithm, the problen it aims detailed mathesatical equations
solve, intuitive underpin the method. assentisl that the lecture comprehenciva and self-contained, providing 3) stucents with clear underatanding of each algorithm from both conceptual and technical perspective.
The task involves generating a LaTeX Beamer presentation, which is popular LaTeX class used to create professional-qualtyy slides with various templates and effects. In this experiment, the objective to produce presentations with different configurations, assessing the framework's 's ability to follow instructions. The experiment includes the following configurations:
Config 1: A 30-slide presentation, including motivation, problem statement, intuitive solution, and detailed mathematical equations.
Config 2: A 20-slide presentation, including motivation, problem statement, intuitive solution, and detailed mathematical equations.
Config 3: A 30-slide presentation, including motivation, problem statement, intuitive solution, and pseudocode.
Config 4: A 20-slide presentation, including only motivation and intuitive solution.
Config 5: A 30-slide presentation, including motivation, problem statement, intuitive solution, and detailed mathematical equations.
The goal is to examine the framework's ability to follow specific instructions while generating over 20 and slides in different scenarios.
This task is well-suited for evaluation because it requires not only text generation but also an understanding of formatting and presentation logic. It serves as comprehensive test multitasking and reasoning capabilities. The structured nature of LaTeX allows for rigorous assessment of the agent's 's ability to manage complex, multicomponent tasks.
Evaluation Metrics: The following metrics are used assess the performance of the generated LaTeX Beamer writing:
(1) Compilable: Verifies whether the generated LaTeX code can compiles into a valid Beamer presentation not. A successful compilation is rewarded with score of 1, otherwise 0. (2) Completeness: Ensures that the final Beamer presentation includes all required components like: motivation, problem, intuitive solution, and equations. Missing any these results in score of O.
15

---

Published as a conference paper at ICLR 2025
100
14
24
80
36
Percentage
60
38
40
76
48
20
14
12
12
12
Autogen
CAMEL
MetaGPT
Flow
Framework
1st
1nd
3rd
4th
Figure 4: Ranking distribution for website design across different frameworks. The results indicate that our method (Flow) outperforms others by achieving the highest percentage first-place rankings.
100
24
80
50
Percentage
60
76
100
40
50
50
20
24
Autogen
CAMEL
MetaGPT
Flow
Framework
Ith
2th
3th
4th
Figure Ranking distribution for gobang game development across different frameworks. The results indicate that our method (Flow) outperforms others by achieving the highest percentage of first-place rankings.
16

---

Published as a conference paper at ICLR 2025
(3) Page Limit: Assesses whether the presentation adheres the specified page limits as outlined in the prompt.
The final result is calculated as the average of these three scores and is shown as a percentage.
B.2
EXPERIMENT SETUP: GOBANG GAME DEVELOPMENT
User input
am developing a Gobang game that includes a nalve AI and user interface. The game ahould end when ehe player wins the board is completely fillad. The interface must clearly indicate whose tuzn it la and display message when the game concludes, specifying winner. additicnally, the user should have option to play as either black or whize stones.
Gohang, also called "Five in Row", is strategy board game where two players take turns placing black and white pieces on grid. The objective is to be the first to align five consecutive pieces in horizontal, vertical, diagonal line. This experiment assesses develop the game by utilizing parallelism divide the development process into smaller, manageable tasks, such game logic, Al move generation, and user interface (UT) design. We apply the same approach, taking the average score from five trials.
framework's 's ability efficiently
Evaluation Metries: The following metrics are used assess the performance of the generated
Gobang game:
(1) Compilable: The code compiles without errors. Any error that causes termination will result in a score
(2) Interactable: Properly supports both user and AI movements. both functions are achieved, score else O.
(3) Game Rule: Ends correctly when five pieces aligned, correct terminated will result in 1 final score.
Each of these metrics is scored as 0 or 1, and the final result is calculated as the average of these scores and turned into percentage. These metrics allow for comprehensive assessment of the efficiency, accuracy, and adaptability of each framework in developing functional Gobang game with Al capabilities.
B.3
EXPERIMENT SETUP: WEBSITE DESIGN
User input
dasigning website for the International Conferance Learning kepresentations (TC1R2025), which will take place from April 27, 2025, to May 1, 2025, in San Frencisce, California, United states. International. Association for Learning Representations. Note that:
organized by
1) Por each section, I would like cSS stylesheet should be provided to style the website. The content must profeasicnal, clear, appropriate for international academia conferance.
example IIM centent. Additionslly, sample
21 The website should include all the provided details, including a comprehenaive conference
dedicated to conference vance, featuring
We tasked the frameworks with developing comprchensive website for the ICLR conference to evaluate their ability to handle complex tasks that require both flexible task coordination and effective problem solving. This task tested the ability of the frameworks manage multiple interdependent steps, such as designing user interfaces, ensuring functionality, and adhering to specific design guidelines.
Evaluation Metrics:
The following metrics are used to assess the performance of the generated
website:
17

---

Published as a conference paper at ICLR 2025
(1) Compilable: Checks if the HTML renders into functioning website, If yes then score 1, can't render will result score 0
(2) Basic Information:
Verifies the presence of essential details like conference name, date, location, and organizer. Missing any of the information will caused the score to be 0
Sections: Ensures inclusion of all required sections, with a focus on the schedule and venue as prompt asked. Missing the required part in the prompt will result score of in score.
By presenting real-world scenario involving intricate requirements, well the frameworks could break down a large project into manageable components and coordinate efforts across different tasks.
able to observe how
B.4 How DIFFERENT LLM AFFECT UPDATES
To verify how our framework performs with different capabilities of LLM, test both GPT-4o-mini and GPT-3.5- Trrbbo on three tasks we designed. In this experiment, each task was run five times on different models, and the average of the results three metrics; average init task, average changed task, and average changed ratio.
calculated the final outcome. We recorded
Init task refers the number of subtasks that need to be executed within the workflow after selecting the optimal workflow but before the execution begins.
Average changed task indicates the number of subtasks in the original workflow that were updated after the execution of the workflow.
Average changed ratio calculated by dividing the average changed task by the init task, providing more intuitive reflection of the proportion of subtasks that were updated.
Table 5: Update information GPT-3.5-Turbo and GPT-4o-mini
LLM-Agent
Task
Initial Tasks (avg.)
Change Ratio (avg.)
GPT-3.5-Turbo
Gobang Game Development
7.8
44
Website Design
7.2
66
LaTeX Beamer Writing
4.4
GPT-4o-mini
Gobang Game Development
8
35
Website Design
7.2
LaTeX Beamer Writing
9.2
53
B.5 How DIFFERENT LLM AFFECT PERFORMANCE
In this experiment, we used the GPT-3.5-Turbo model to conduct experiments on three tasks in different frameworks. Each task was executed five times. We evaluated the results using the same scoring metrics described above.
Table 6: Comparison of LLM-based multi-agent frameworks on Gobang Game Development with GPT-3.5-Turbo
Success Rate (%
Model
Compilable
Interactable
Game Rule
Overall Score
AutoGen
80
20
20
40
MetaGPT
80
20
40
CAMEL
80
80
40
67
Flow (Ours)
100
100
60
87
18

---

Published as a conference paper at ICLR 2025
Table 7: Comparison of LLM-based multi-agent frameworks on Website Design with GPT-3.5-Turbo
Success Rate (%)
Model
Compilable
Basic Information
Sections
Overall Score
AutoGen
20
0
0
7
MetaGPT
80
60
60
67
CAMEL
40
40
20
33
Flow (Ours)
100
100
40
80
Table 8: Comparison of LLM-based multi-agent frameworks on LaTeX Beamer Writing with GPT- 3.5-Turbo
Success Rate (%)
Model
Compilable
Completeness
Page Limit
Overall Score
AutoGen
40
13
MetaGPT
20
20
0
13
CAMEL
80
80
53
Flow (Ours)
100
100
67
Based on this table, we observe that when using models with relatively low performance, framework demonstrates significant advantages in task quality. Overall, even when using less powerful LLM like GPT-3.5-Turbo, our framework consistently maintains a high standard of performance.
B.6 TIME COST OF DIFFERENT BASELINE
To quantitatively measure the cost of our framework, we use execution time as a standard. Using the same model to perform the same tasks, recorded the execution times and conducted horizontal comparison with other frameworks. Each task was executed five times, and the average execution time was calculated.
Task
Flow (w/o update)
Flow (w/ update)
MetaGPT
CAMEL
AutoGen
GPT-3.5-Turbo
Gobang Game
26.12 11.35
33.57 # 12.46
34.00 = 15.12
121.52 + 20.87
31.00 + 14.67
Website Website
23.46 10.84
34.23 13.12
85.14 18.52
41.96 + 12.89
Latex Beamer
24.12 10.89
29.92 # 14.87
166.00 22.64
31.00 + 16.78
GPT-4o-mini
Gobang Game
60.45 14.78
72.34 13.45
99.45 16.92
110.94 19.67
148.72 + 25.34
Website Website
51.98 20.19
52.14
14.89
127.49 17.52
74.53 18.34
86.78 4: 21.23
Latex Beamer
53.19 + 17.65
83.34 15.89
66.72 19.45
106.34 a 20.78
95.21 à 22.56
Table 9: Comparison of task performance across different framework, including standard deviations. The standard deviations reflect realistic variability with increased variance across tasks and framework.
The results demonstrate that incorporating the Flow mechanism significantly enhances efficiency compared to other methods, as seen reduced execution times in both models. However, the introduction of updates incurs additional computational overhead, resulting in noticeable increase in execution time, highlighting the trade-off between adaptability and efficiency. Nonetheless, Flow maintains faster execution times compared several other frameworks.
19

---

Published as a conference paper at ICLR 2025
C CUSTOM METRICS FOR PARALLELISM AND DEPENDENCY
C.I PARALLELISM METRICS
Speedup (5 = ). this metric measures the ratio of execution time on a single processor (I1) to that on multiple processors (Tp). While effective in frameworks where these times can be measured, it requires actual execution on both single and multiple processors. In our case, such execution times are not readily obtainable because our focus is on task-solving workflows rather than on processing workloads that can be easily benchmarked in this way.
Amdahl's 's Law (S(p) " and Gustafson's 's Law (S(p) = - f. (p - 1)), both laws require
knowledge of s. the proportion of the task that is inherently serial, and p. the number of processors. Our task graphs have complex dependency structures, where tasks cannot be neatly categorized as strictly "serial" or "parallel." For example, task might need wait for upstream dependencies but could still execute concurrently with other unrelated tasks. This hybrid nature makes it challenging to accurately define fa or apply these laws meaningfully.
C.2 DEPENDENCY METRICS
Cyclomatic Complexity = N cyclomatic complexity measures the number of linearly independent paths through program, providing an overall complexity measure. However, it focuses on the control flow within code and overlooks the distribution of dependency relationships among tasks workflow graph. It does not capture the "dependency concentration" or "ddispession"" " which crucial to understanding the impact of dependencies workflow robustness and the case with which LLM can comprehend and update the workflow.
C.3 PROPOSED METRICS FOR TASK WORKFLOW EVALUATION
Given these limitations,
two simple metrics in our Maseedd multi-agent framework:
1). Parallelism Metric: This metric does not rely on execution time measurements or require assumptions about tasks being strietly serial parallel. It directly reflects the workflow's potential for concurrent task execution, making it more applicable to our scenario.
2). Dependency Metric: We focus on the "dependency concentration" or "dependency dispersion" by analyzing the standard deviation of the degree distribution in the task graph. This metric provides an intuitive reflection of critical dependency points within the workflow. By highlighting how dependencies are distributed among tasks, it helps us understand and mitigate potential bottlenecks, enhancing both robustness and the LLM's ability to process workflow updates efficiently.
D EXAMPLES OF FLOW'S WORKFLOW
In this section, we present examples of actual workflows generated by Flow.
Fig. 6 showing Flow's workflow in generating LaTeX Beamer, Flow concurrentyy generates the four required components for each algorithm: motivation, problem, intuitive solution, and mathematical equations.
Gaather motivational content]
DDescribe the problem]
[Outline structure]
Compile content to LaTex] - [Review and proofread]
[Provide intuitive solutions]
[Develop detailed mathematical equations]
Figure 6: Workflow of LaTeX Beamer Writing in Flow
20

---

Published as a conference paper at ICLR 2025
For the task of developing gobang game, Flow recognizes that the UI and main game logic be separated and executed in parallel to enhance overall speed and efficiency, as shown in Fig. Additionally, there remains clear sequential process; for instance, the game rules must be defined first before the corresponding code can he deployed.
[Dofine Interface]
[Build Ui]
[Define rules]
[Develop code for logic]
[Combine UL logic &
>[Test]
nalve AI]
Figure 7: Workflow of Gobang Game Development
For the task of website design, as shown in Fig. 8, Flow treats different parts of the HTML as individual subtasks, which helps to increase overall speed. Additionally, dividing the process into separate components allows for parallel execution and improved modularity, ensuring that an issue arises in one part of the HTMI,, it will not impact the performance of other sections. This approach improves both efficiency and fault tolerance.
(Create homepage]
[Create about page]
[Create Schadule page]
[Define the website structure)
[Develop CSS]
[Integrate
(Create Location
[Create Registration page)
(Create Speaker page]
[Create Contact page]
Figure 8: Workflow of Website Design
D.1 EXAMPLE WORKFLOW
"taske"
"arves"
Tack" "aeffno strurture toyuu talst dorasent, faclidiag title pp,, table contreto, and certian hraders",
Sgnar":
"tesk""
"rww': f1,
Treaxining desendencees lex":
proposed structore and Layout Lifak docuent that ancommsninn the reporents for bestar Jecture or di sttrreett redeforseent
Surun=":
"Lukk: complle Information various reirforrment loming algaritmo, ircluting wituutins,, prblano, and intuitiw solutionn',
"gent":
Algertthanysveete Ciimriiti
covelop
Figure 9: A workflow of Website Design in VSCode
21

---

Published as a conference paper at ICLR 2025
Ber und ar shoed
pillos
rool
Alves
baurry ompers Jcbess
de beneing momer
(a): Flow
Loaniig
O Lasnin Poblr
Cuarnag: inat Sokiion
Leaning Desitec
biiestiddyy hos aboun trrugf and
(b): AutoGen
Vetryttn
Mel
Ngeellee Styps
Fule dedde.
(c): Meta
seniic chedk the (rfiweed und
(d): CAMEL
Figure 10: Different multi-agent frameworks' LaTeX Beamer
D.2 PSEUDOCODE FOR UPDATING AOV
Algorithm 1: Helper Function for Updating Graph
Function UpdateGraph (G, P, :
// Generate updated candidate workflows using LLM
[G1, G........... + f(G, P, Pina);
// Initialize selection variables
Pmux + -0;
Cmin +- +00;
Goptimal + None;
// Evaluate each candidate workflow
for each candidate workflow G3 in (G1, G2,.... Gx do Compute Parallelism Ps +- Pave(GG):
Compute Dependency Complexity C₁ +- Cdependency (Gx);
Ps > Pmaxe or (Ps = P max and C Cmin) then
Paux x Px;
11
Cmin f- C+:
Gogtimal t- Gz;
return G Coptimatis
13
22

---

Published as a conference paper at ICLR 2025
Algorithm 2: Flow
Data: Task Requirements P, Initialization Prompt Pmnts Update Prompt P Pupdatc Result: Optimized Aulti-Agent Workflow
Implement Workflow using dictionary structure Initialize workflow formulation by defining the task dictionary Ō where each key E V maps a dictionary containing: Glu] = {status, data, num parents_ parrents_ntt_comee compleeee, child, agent)
// Step 1:
// Step 2: Generate an Initial Workflow
G - UpdateGraph(1), Pum P);
// Step 3: Workflow Refinement and Dynamic Updating ing
while there exists at least one sub-task in & that completed do if' an update to the workflow is required then
Generate and Select the Best Updated Workflow
G + UpdateGraph((C Pupdaae P);
Update workflow dictionary & to Chessi
// Regenerate Execut ion Plan and Reallocate Agents
Perform Topological Sort on G to obtain updated execution order
Assign agents Aj to their respective sub-tasks T; CV; end
// Execute Available Sub-tasks in Parallel foreach sub-task v V do if status of 0; not started and Glu;] Glupettmm_prrent ptt__ttttoooooo not_completedd then if agent aj is available then
10
11
13
Assign agent aj to sub-task Us; else
15
Clone agent a';
Assign cloned agent df to sub-task 0;
16
end
// Execute subtask i parallel
Execute using agent aj or cloned agent a' concurrently;
// Update Subtask Status and Data
Update status of sub-task v, progress;
// After execution, update related data
20
Update output of subtask et to d[u;].data;
Glug] <- "completed":
21
// Update Child Tasks' Parent Complet etmpletion Count foreach child task e Glus].child
22
G(c) num parentsrrets____e paarents_not_cee pot__oottooooootto completed <- G(c). num n not_completed
23
end
end
26
end
27 end
23

---

Published as a conference paper at ICLR 2025
D.3 PROMPT FOR WORKFLOW UPDATE
User input
1. Update the Norkflow
Evaluate Completed Tasks:
#ocus: Examine only tasks with "sstatus; "completed"',
Check
Ensure that 'ddata" for each task is sufficient, detailed, and directly contributas to the "final_goal',
Assess Workflow Structure:
Examine All Tesks: Review all tasks, including those labeled Meemplete""' ""pnninn""' and "an-progress"".
Check Adequacy:
Confirm the workflow is complate and logically structured to achieve the "final_goal"
Ensure there are no missing eritical Lasks depeadencies.
Verify that "next"' and "ppee" connectiens between tasks are logicel and facilitate seamless progression.
Identify Inafficienctes:
Detect and address uncecessary dependencies, bottlenecke, redundant steps that hinder the wortflow's efficiency.
Allowed Changes:
Modify: Clarify and deteil the objectives of tasks with insufficient or vague disectivas ensure they the 'finsl_gool',
Add: recueeuce new tasks with clear, datai led deacript tons to fill gapa In or structuze.
Removes Eliminate redundant obsclete tasks to streaml ing the workflox.
Maintsin Logical Flow:
Aeorganize task connections (M"ex""> and "prev"") to enhance paralle; execution improve overall workflow efficiency.
Output Format
Tr.. No Changes Are Made:
Return an emply JSON object to indicate that no medifications were necessary:
"json()), )
If Changes Are Made: Return
object contairing the updated workflow without includ
data* fielda to optimize token usage, This JSON should only include the structural chaages (task paremezers and connections).
D.4 WORKFL.OW UPDATE STRATEGIES
We implemented two different workflow update strategies:
Update Concurrently
In this approach, when subtask is completed, it immediately triggers the workflow update function, even if other subtasks are still running. After obtaining the updated workflow, the workflow is merged with the current state.
Trade-off: This workflow update strategy runs concurrently with task execution, optimizing running time. However, can result in unnecessary API calls, as some subtasks still in progress may become redundant misaligned with the updated workflow.
Update After Task Completion
In this strategy, when a subtask is completed, new tasks are allocated immediately. Instead, the system waits for all running subtasks to finish before triggering the workflow update. After the update is completed, new subtasks are allocated based the updated workflow. This approach reduces unnecessary API calls by batching updates.
Trade-off:: This workflow update strategy reduces unnecessary API calls but increases overall running time, as new subtasks are delayed until the workflow update is complete.
In our paper, all the experiments are obtained by using the second strategy to avoid the waste of API usage.
24

---

Published as a conference paper at ICLR 2025
E FRAMEWORK OF THE MULTI-AGENT FRAMEWORK
E.I OVERVIEW
The multi- agent framework is designed to execute complex tasks by decomposing them into subtasks, which are managed and executed by individual agents. The framework leverages LLM generate and update workflows dynamically, ensuring robustness, efficiency, and adaptability.
E.2
KEY COMPONENTS
1. Agents
Role Assignment
Automatic Role Generation: Roles automatically generated by LLM during
workflow generation and updates.
Flexibility: By default, roles are not fixed, allowing the system to adapt to the specific requirements of each task.
Role Constraints: In scenarios with resource constraints, roles can be explicitly defined to limit the number agents or types of expertise in prompt.
Subtask Assignment
Subtasks are assigned to agents whose roles best match the task requirements, ensuring tasks are executed by agents with appropriate skills.
Matching Expertise:
One Agent per Subtask: Only agent is assigned per subtask to maintain clarity and responsibility.
2. Workflow Management
Workflow Generation
Initial Workflow: The LLM generates an initial workflow that outlines all subtasks and their dependencies required to achieve the final goal.
Task Dependencies: Dependencies are defined to ensure logical progression and facilitate parallel execution where possible.
Workflow Update Mechanisms
Two strategies are employed for updating the workflow:
(a) Update Concurrently
Trigger: When a subtask completed, the workflow update function is trig- gered immediately. even if other subtasks are still running.
Process: The updated workflow is obtained and merged with the current state.
Trade-off: Optimizes running time but may result in unnecessary API calls, as subtasks still in progress might become redundant after the update.
(b) Update After Subtask Completion
Trigger: No new subtasks are allocated immediately after subtask is com- pleted. The system waits for all rumning subtasks finish before updating. Process: Once all subtasks completed, the workflow is updated, and new subtasks are allocated based on the updated workflow.
Trade-off: Reduces unnecessary API calls but increases overall running time, new subtasks delayed until the workflow update is complete.
Chosen Strategy: In practice, the system uses the second strategy reduce API usage.
3. Dynamic Restructuring
Mechanism for Dynamic Workflow Restructuring
Workflow Update Mechanism: The system includes robust workflow update
mechanism that continuously monitors the execution status of all subtasks. If subtask fails deemed unsolvable, the system triggers update process.
Re-evaluation of Workflow: The system systematically reviews the current work- flow, taking into account the unsolvable subtask. assesses the impact the failed subtask all subtasks and the overall goal.
25

---

Published as a conference paper at ICLR 2025
Adjusting Dependencies: The workflow is adjusted by removing modifying the unsolvable subtask and updating dependencies accordingly. This may involve: Reassigning Subtasks: Redirecting subtasks to alternative agents or creating new subtasks that can achieve similar outcomes.
* Adding New Subtasks: Introducing new subtasks that offer alternative solutions or pathways to reach the final goal.
Bypassing Unnecessary Steps: If possible, restructuring the workflow to bypass the vnsolvablc subtask without compromising the end objectives.
4. Task Execution
Parallelism
Maximizing Parallel Execution: The workflow is designed to allow subtasks without dependencies to be executed in parallel, optimizing resource utilization and reducing total execution time.
Dependency Management: Dependencies are minimized where possible to en- hance parallelism.
Dependency Minimization
Dependency Metric:
The system analyzes the standard deviation of the degree distribution in the task graph to identify and minimize critical dependency points.
Reducing Bottlenecks: By minimizing unnecessary dependencies, the system reduces potential bottlenecks and enhances robustness.
E.3 WORKFLOW EXECUTION PROCESS
1. Initial Workflow Generation
The LLM generates a workflow based on the final goal, decomposing it into subtasks with defined dependencies.
2. Agent Role Assignment
Agents are assigned roles automatically by the LLM.
Subtasks are assigned to agents based on role matching.
3. Subtask Execution
Agents execute their assigned subtasks.
Subtasks are exccuted in parallel where dependencies allow.
4. Monitoring and Updates
The system monitors subtask completion statuses.
Depending on the update strategy, the workflow is updated either concurrently after all current subtasks are completed.
5. Dynamic Restructuring
Detection: a subtask is determined to insufficient or unsolvable for achieving the requirement, the system detects this during execution.
Re-evaluation of Workflow: The system reviews the current workflow, assessing the impact of the failed subtask on all subtasks and the overall goal.
Workflow Adjustment: The LLM restructures the workflow dynamically to adjust other subtasks or redefine dependencies.
Continuity: This ensures that progress toward the final goal continues without signifi- cant delays.
6. Completion
The process continues until all subtasks are completed and the final goal is achieved.
F LIMITATION AND FUTURE WORK
Although we have generated multiple candidate workflows and selected the one with the highest mod- ularity, it is still not the most efficient. With sufficient computing and data resources, model trained
26

---

Published as a conference paper at ICLR 2025
specifically for workflow management could significantly enhance the framework's performance. For instance, the LLM could be designed to maximize reward function centered on key performance indicators such task completion speed, resource utilization, and minimizing disruptions in the workflow. Such training could lead to the development more effective workflows. The workflow updater requires global information function effectively, which can become problematic as the context length increases. This limitation could be addressed by employing rig or hierarchical approach to more precisely identify errors areas lacking efficiency, thereby facilitating more targeted updates and improvements within the workflow.
G PILOT STUDY ON AUTO-VALIDATION
In complex workflows involving numerous subtasks, accurately evaluating the correctness of each task becomes challenging, especially when relying solely on central workflow updater due to extensive context. Recognizing this challenge, we introduced an automated validation mechanism akin to unit testing enhance task-level correctness and system efficiency.
Specifically, we integrated an auto-validation loop into the Flow system to systematically verify each agent's 's output immediately after task completion. For computational subtasks involving Python scripts, this validation mechanism generates unit test cases to programmatically verify output cor- rectness. In contrast, for textual or descriptive outputs, validator LLM agent is employed to assess whether the outputs adequately fulfill the subtask specifications. This validator generates targeted feedback highlighting concerns.
an output is deemed unsatisfactory, a re-execution agent is invoked, which utilizes the original subtask specification and feedback from the validator to generate refined solution. Additionally, a history module stores all previous execution attempts alongside their corresponding feedback, guiding subsequent re-execution agents avoid repeating past errors. Empirically, this process forms re-execution cycle that systematically improves individual subtask outcomes. To prevent infinite cycles of re-execution, validation threshold is used, analogous early stopping methods in optimization tasks. Subtasks are marked as completed upon successful validation, whereas repeated failures beyond a predefined threshold result in the subtask being marked as failed and handled by the workflow updater.
We evaluated this enhanced validation approach a pilot study involving game development tasks, LaTeX Beamer generation tasks, and a website design task. Given the same backbone model, enabling the validation feature demonstrated significant improvements in individual subtask and task output quality (see Figures 11 and 12). For example, runnable game combining Tetris and Bejeweled could be generated with auto-valle anto-validation but failed without it using the backbone model 03-mini-hige
For future work, it would be valuable perform rigorous theoretical analysis to better understand the underlying mechanisms contributing to these performance gains. Such analysis could further inform the optimization of validation strategies and workflow design in complex multi-agen agent workflows.
H PROOF OF THEOREM
Proof. We will compare the expected number of successfully completed subtasks in both workflows.
Definitions:
Let (1( and PB(v) denote the probability that subtasks v is successfully completed in Workflow and Workflow B, respectively.
For each subtasks v, let (u) and (n(v) be the sets of immediate predecessors of u in Workflow and Workflow B, respectively.
Success Probability of subtasks: In Workflow A, the success probability of subtasks is given by:
Pa(u) (1 ps) x II
PA(i).
(EDA(1)
27

---

Published as a conference paper at ICLR 2025
Task Tetris Came:
Tosk Tetits Bajeweted Fusion:
Create Touis
Develop game that fetre enette
lisuar.
game
10xz0
Falling intromtnows should locs into grid and transform ucdd gems.
Display curnent tailing teer rerommno and the upoominng plece.
letria time cisar and Bi enelte match- 3 cleaning, trigeriig chain recetions and bonus points
The pame
incluce a score countes, a clepley for the number cacored lines, and o siippee borderto distingushh
Include Gul (using frimwwork like disppaye the gamme grid current scom, and nest tetro provitw, along with smooth animmtiions
that
Each retromine type shoutc have unique. vibr ertratt colour for clear visibiltty
No aound ettects newcted
Seom:
Next
Figure 11: By incorporating auto-validation, Flow is able to automatically generate more sophisticated programs without errors.
Denign:
Iwant cmarta wecsrte for the follow commeneecec
Conferance Name: Internationa Conforence Learing Reprossntatiion (CL22029)
Acri
Organtier: lnner national Asscciation Leaming Rapresertations
Masoo gentrata dera led webelie structurs and contamt for this confussnos For sach succion, provide soumple pempletting HTMU content. profecsional,
interaatinna. acedemic corferencs.
Figure 12: By incorporating auto validation, Flow can automatically generate more accurate and well-designed websites from the same prompt.
Similarly, in Workflow B:
Pn(v) (1 - pr)
II
Pa(1).
(2)
<EDy(r)
28

---

Published as a conference paper at ICLR 2025
Base Case: Since the subtasks with no dependencies (i.e., DA(U) DD (v) () have the same success probability in both workflows:
Pa(u) PB(U) =
Inductive Step:
We proceed by induction on the subtasks' dependency levels.
Comparison for Subtasks v": Subtasks s² has an additional dependency d in Workflow B. Therefore:
Dn("") Da ) {d).
Using equations equation 1 and equation 2, we have:
Pa(u*) (1 Ps)
II iEDa(v")
Pa(t),
Pn(U*) = pp) x
II Pa(1) = (1-ps) x PB(d) x II
Pn(1). (EDA("")
ieDg(u*)
Since Da(x*) Dn(v")) \ (d), and Pa( (i( for all * v" (because their dependencies the same), it follows that:
PB(U") PA(o") PB(d).
Because 0 < PB(d) PA(d) < 1 (since Pf > 0), we have:
PB(u") PA(v") PA(d) Pa(u").
v., Da(v) = Dn(v), so: Success Probabilities for Other Subtasks: For all subtasks v # v.,
(v Pn(v).
Expected Number of Successfully Completed Subtasks: The expected number of successfully completed subtasks in each workflow is:
E|SA] = Σ PA(U),
VET
E[Sa] Σ Pn(u).
EE
Substituting the above findings:
E|Sal
Σ PB(U)+ PB(U")
v
Σ Pa(e) + PB(U")
Pa(t) -Pa(o)) PB(U')
WET
E[SAl - (PA(V") PB(1"))
Since PB(c*) < PA(U"), the difference AP = PA(v") - PB(1*) > 0. Thus,
E[Se] E [[AA] AP < E[SA]
Therefore, the expected number of successfully completed subtasks in Workflow A is strictly greater than in Workflow B:
E[SA] > E[Sa].
29


Reviews:
[{"review_id": "FUCQ1nLU6N", "reviewer": "Reviewer_Je7s", "summary": "In this paper, the author(s) propose a multi-agent framework to enable dynamic workflow update, by integrating activity-on-vertex (AOV) graphs. In particular, an agent is prompted to generate several workflows in the form of AOV, and the one with the highest parallelism level and lowest dependency complexity will be selected. Various agent roles will be assigned to complete the workflow tasks, and the framework allows workflow refinement and dynamic updating if agents encounter errors.", "questions": "- I am wondering why the proposed solution does not check the workflow correctness in the first place, but the parallelism and modularity.\n- Section 4.1.3: Why “Flow gets 100% success rate” while in Table 3 the overall success rate is 80%?\n- The evaluation results can be elaborated. For instance, in the three scenarios, how many updates are required respectively?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- The paper adds value to agent-based workflow generation and automation.\n- Activity-on-vertex graphs are incorporated to better manage task dependency and parallelism.\n- Comparative analysis is conducted with multiple related work in terms of both success rate and human rating.", "weaknesses": "- A framework architecture or sequence diagram is needed to demonstrate the overview of the proposed solution.\n- I am wondering why the proposed solution does not check the workflow correctness in the first place, but the parallelism and modularity.\n- Section 4.1.3: Why “Flow gets 100% success rate” while in Table 3 the overall success rate is 80%?\n- The evaluation results can be elaborated. For instance, in the three scenarios, how many updates are required respectively?\n- Table 1: “task 1, 2 completed, task 3 under-work”, task 2 and 3 should be exchanged according to the above AOV.\n- The paper organization can be adjusted. For instance, there is no Section 4.2, then why Section 4.1 is needed?\n- A proof-reading is needed as some typos are found.\n\n      - Section 2: “previous approach like …” -> “Previous”.\n\n      - Section 4.1: “… the average performance of Flowis 93% …” -> “Flow is”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sjbGYMh1Jb", "reviewer": "Reviewer_RcMF", "summary": "This paper introduces a novel approach to enhancing the performance of LLM-based multi-agent systems through the use of activity-on-vertex (AOV) graphs. By defining workflows as AOV graphs, the authors enable dynamic updates during task execution, which facilitates better resource allocation and real-time adjustments. This flexibility is crucial for efficiently managing complex tasks. The paper presents a practical multi-agent framework that incorporates quantitative measures to evaluate workflows, allowing for efficient selection and improved planning capabilities. To empirically demonstrate the framework's effectiveness, the authors conduct experiments across three diverse tasks: game generation, LaTeX slide generation, and website design. They provide detailed evaluations and analyses of the results, showcasing the superior performance of their framework compared to existing open-source solutions.", "questions": "Please refer to the questions in Weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "-  The paper effectively points out a significant gap in existing frameworks, specifically regarding the adaptability of workflows in real-time, especially in the face of unforeseen challenges.\n-  The paper is well-structured and easy to follow, offering clear definitions and detailed formulas that enhance comprehension.", "weaknesses": "-  Limited novelty. While dynamic workflow generation and execution have been extensively discussed in prior studies (e.g., [1][2][3][4]), this paper's approach lacks sufficient novelty. Directed acyclic graphs (DAGs) and graph-based frameworks have already been established as effective structures in LLM-based agent frameworks. To strengthen its contribution, the paper should include comparative analysis with these existing approaches, highlighting specific advantages and unique aspects of the proposed method.\n\n- Cost analysis. What's the cost of the Flow?  There is no discussion of Flow’s cost relative to other frameworks, leaving the efficiency of the proposed method unclear.\n\n- Experiment specifications. What's the specified experiment settings? Important experimental settings, such as the number of candidate graphs (𝐾) and the agent count, are not specified. \n\n- Updating machinism. The description of workflow refinement process lacks critical details. In line 318: the phrase “systematic review to determine if the workflow requires refinement” lacks clarity regarding how this review is conducted and the criteria for determining when refinements are needed. Additionally, further details are needed on “this rigorous verification process” mentioned in line 323, particularly regarding how the system handles verification when no errors are found.\n\n- Flow basic execution. Lacks several key information of Flow deisgn and execution. 1) How Flow determines when all prerequisite tasks has been completed? 2) How Flow handle the cases where some parallel tasks have not yet finished --dose Flow wait, reprioritize tasks, or initiate partial updates in such scenarios? 3) Critical setup information is missing, such as the total number of agents available in Flow’s default configuration. \n\n- The current experiments cover only three tasks, which may not be sufficient to substantiate Flow's superior performance comprehensively. Expanding the experiments to include more diverse tasks and standardized benchmarks would strengthen the claims regarding its generalizability and robustness.\n\n- LLM dependency. The paper does not address how Flow would perform with less powerful LLMs, such as GPT-3.5 or other open-source models. Exploring this would provide valuable insights into the framework’s robustness and generalizability\n\n[1] Qian C, Xie Z, Wang Y, et al. Scaling Large-Language-Model-based Multi-Agent Collaboration[J]. arXiv preprint arXiv:2406.07155, 2024.\n\n[2] Zhuge M, Wang W, Kirsch L, et al. Language agents as optimizable graphs[J]. arXiv preprint arXiv:2402.16823, 2024.\n\n[3] Hong S, Lin Y, Liu B, et al. Data interpreter: An llm agent for data science[J]. arXiv preprint arXiv:2402.18679, 2024.\n\n[4] Liu Z, Zhang Y, Li P, et al. Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization[J]. arXiv preprint arXiv:2310.02170, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xfZM6Rqz3J", "reviewer": "Reviewer_uiek", "summary": "The paper presents a multi-agent system approach to automating (essentially coding) tasks such as website creation or game creation. Their multi-agent approach does better than other single agent approaches from the literature. A key to tracking the multi-agent system's progress on such complex tasks was utilizing a graphical representation of sub-tasks, task assignments and progress, namely activity-on-vertex directed acyclic graphs to represent the workflow. Results on multiple tasks (while anecdotal) show the proposed approach does better than other approaches in the literature.", "questions": "NA", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper presents an interesting and intuitive approach to representing workflows and enabling multi-agent systems to execute complex tasks. Results support the validity of the approach. The paper also tackles an important open problem in the literature: automating complex tasks. The authors also perform a good analysis of the experiments, taking the time to specify errors that different agents performed in completing their tasks.", "weaknesses": "The authors mostly evaluate their approach anecdotally as opposed to on standard benchmarks or datasets. This is understandable given the lack of complex task datasets in this domain. \n\nThe paper contains grammar mistakes and presentation issues that make the paper difficult to read. A few examples: in the abstract, did you mean \"define workflows *as* activity on vertex (AOV)\"?; line 162 should say \"inspired by\" as opposed to \"inspire by\"; Figure 1 should say \"the game must end\" instead of \"must be ended\"; Figure 2 should say \"initial\" as opposed to \"intial\"; in Tables 1, 2 and 3, instead of saying \"ours\", say \"Flow (ours)\" to be consistent with the writeup in the various sections; line 118, \"1)\" instead of \"1).\"; line 132, \"LLM-based\" as opposed to \"LLM based\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kR9xfEAA3B", "reviewer": "Reviewer_eHFH", "summary": "This paper focuses on enhancing large language model (LLM)-based multi-agent systems by improving dynamic workflow management and modular task execution. The proposed system refines workflows using an Activity-on-Vertex (AOV) graph structure, enabling flexible and dynamic task allocation adjustments. The system encourages modularity, allowing for concurrent task execution and minimizing inter-task dependencies. This results in greater robustness and efficiency, especially in complex, evolving task scenarios. Experimental results across tasks like game development, LaTeX slide creation, and website building show that the system can outperform several existing approaches (AutoGen, CAMEL, MetaGPT), with better success rates and human satisfaction scores.", "questions": "The paper assumes that a highly modular workflow structure can cope with task failures, but what happens if a module turns out to be unsolvable? Does the system have a mechanism to adjust other modules or restructure the entire workflow dynamically? Could you provide more theoretical analysis on the conditions under which modular workflows remain effective in practical applications?\n\nGiven the goal of improving workflow robustness, why did you not directly analyze the expected amount of changes required when tasks fail? How could this expected quantity complement the metrics of parallelism and dependency?\n\nCould you elaborate on how subtasks are assigned to different agents with varied roles? How does the assignment strategy impact the system’s overall effectiveness? Also, when regenerating the workflow graph, does the system consider the execution status of current tasks to avoid resource wastage?\n\nWhy do you believe non-coding tasks might introduce bias in evaluation? Would it be possible to include additional tasks such as logic reasoning or mathematical analysis to demonstrate the system’s broader applicability?\n\nCould you clarify how human rating was conducted and address concerns that it may not fully reflect the varied usefulness of all competing approaches?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces dynamic workflow updating and modularization, which appear to be novel.\n\nThis paper clearly outlines the methodological innovations, with well-structured sections on each component, from workflow initialization to modular design.\n\nThis paper has clear contribution to the design of multi-agent collaboration frameworks, particularly for complex adaptive tasks. By improving modularity and enabling efficient concurrent sub-task execution, the proposed framework has potential applications across fields requiring adaptable, scalable automated task execution.", "weaknesses": "While this paper presents some interesting research ideas and promising experiment results, I have several concerns as detailed below.\n\n1. The general idea of supporting dynamic workflow management has been studied previously, in particular in the context of single-agent and multi-agent planning systems. It might be novel to expand this idea in LLM-based multi-agent systems, however the corresponding technical novelty may need to be clearly highlighted and better justified.\n\n2. It seems that a key assumption of this paper is that a highly modular workflow structure can reliably cope with task failures. While this is intuitively meaningful, what happens if we find that a module is unsolvable? In which case, we may need to adjust other modules and potentially the entire workflow structure. Consequently, most part of the workflow execution may be interrupted. Hence, it is essential to theoretically analyze this assumption and understand the specific conditions for it to be valid for practical applications.\n\n3. If building a modular workflow is motivated by the goal of improving robustness of workflow execution, why didn't authors directly analyze the expected amount of changes required by any workflow due to unexpected/random task failures? This expected quantity can be used to determine which workflow should be adopted, in addition to the parallelism metric or the dependency metric. Meanwhile, past studies in the distributed computing community may have developed some existing metrics to measure parallelism and the level of dependency. It is important to clarify why the authors chose to use their own metrics and highlight the corresponding technical novelty.\n\n4. The discussion of how subtasks are assigned to different agents with varied roles is quite brief. The lack of sufficient technical details makes it hard for me to understand this aspect of the system design and its importance to the overall effectiveness of the system. Similarly, while discussing the regeneration of the workflow graph (on page 6), shouldn't this process consider the execution status of the existing/current workflow (e.g., the completion status of different modules) to avoid unnecessary resource wastage? This aspect was not clearly explained in the paper.\n\n5. I don't fully agree with the authors' statement that non-coding tasks may introduce bias on page 7. Besides coding related tasks, many other tasks such as logic reasoning and mathematics analysis tasks may also be important for us to understand the effectiveness and broad usefulness of the newly developed multi-agent system. Meanwhile, given how errors were simulated in Section 5, I was wondering how likely that any task may fail in practice. If tasks seldom fail, is it necessary to find a modular workflow in the first place?\n\n6. Additionally, in terms of the evaluation criteria, I am not sure whether using success rate and human rating is sufficient and fair for comparing different approaches. For each benchmark problem, it is not clear whether the authors followed the common criteria to measure success rates. Meanwhile, for human rating, it largely depends on human participants involved in this research and may not objectively or accurately reveal the varied usefulness of all competing approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)