AFlow: Automating Agentic Workflow Generation
Authors: Jiayi Zhang
Jinyu Xiang
Zhaoyang Yu
Fengwei Teng
Xiong-Hui Chen
Jiaqi Chen
Mingchen Zhuge
Xin Cheng
Sirui Hong
Jinlin Wang
Bingnan Zheng
Bang Liu
Yuyu Luo
Chenglin Wu
Venue: ICLR 2025 Oral (2025)
Keywords: LLM Agent; Prompt Optimization; Workflow Generation
Primary area: applications to robotics, autonomy, planning
TL;DR: We introduce the field of Agentic Workflow Optimization and propose an effective search algorithm called AFLOW, enabling it to surpass manually constructed workflows on six reasoning datasets.

Abstract: Large language models (LLMs) have demonstrated remarkable potential in solving complex tasks across diverse domains, typically by employing agentic workflows that follow detailed instructions and operational sequences. However, constructing these workflows requires significant human effort, limiting scalability and generalizability. Recent research has sought to automate the generation and optimization of these workflows, but existing methods still rely on initial manual setup and fall short of achieving fully automated and effective workflow generation. To address this challenge, we reformulate workflow optimization as a search problem over code-represented workflows, where LLM-invoking nodes are connected by edges. We introduce AFLOW, an automated framework that efficiently explores this space using Monte Carlo Tree Search, iteratively refining workflows through code modification, tree-structured experience, and execution feedback. Empirical evaluations across six benchmark datasets demonstrate AFLOW's efficacy, yielding a 5.7% average improvement over state-of-the-art baselines. Furthermore, AFLOW enables smaller models to outperform GPT-4o on specific tasks at 4.55% of its inference cost in dollars. The code is available at https://github.com/FoundationAgents/AFlow.

Published as a conference paper at ICLR 2025
AFLOW:
AUTOMATING AGENTIC WORKFLOW GENERATION
Jiayi Zhang':2 1:2, Jinyu Xiang's ZZhaoyang Yu², Fengwei Teng', Xiong-Hui Chen', Jiaqi Chen'. Mingchen Zhuge,, Xin Cheng", Sirui Hong', Jinlin Wang', Bingnan Zheng*, Bang Liu?, Yuyu Luo Chenglin Wu-
'Deep Wisdom, The Hong Kong University of Science and Technology (Guangzhou), Renmin University of China, *Nanjing University, Fuudan University,
King Abdullah University of Science and Technology, "Université de Montréal & Mila, Thhe Hong Kong University of Science and Technology
ABSTRACT
Large language models (LLMs) have demonstrated remarkable potential in solv- ing complex tasks across diverse domains, typically by employing agentic work- flows that follow detailed instructions and operational sequences. However, con- structing these workflows requires significant human effort, limiting scalability and generalizability. Recent research has sought to automate the generation and optimization of these workflows, but existing methods still rely initial man- ual setup and fall short of achieving fully automated and effective workflow gen- eration. To address this challenge, reformulate workflow optimization search problem over code-represented workflows, where LLM-invoking nodes are connected by edges. We introduce AFLOW, an automated framework that efficiently explores this space using Monte Carlo Tree Search, iteratively refin- ing workflows through code modification, tree-structured experience, and execu- tion feedback. Empirical evaluations across six benchmark datasets demonstrate AFLOW's efficacy, yielding a 5.7% average improvement over state-of-the-art baselines. Furthermore, FFL.W enables smaller models to outperform GPT-4o on specific tasks at 4.55% of its inference cost in dollars. The code is available at https//ghhlb..coomm/ooodddtttttt
INTRODUCTION
Large Language Models (LLMs) have emerged as powerful tools for solving complex tasks across various domains, including code generation, data analysis, decision-making, and question answer- ing (Liu et al. 2024 Li al. 2024a //hu et al. 2024 Xie et al. 2024b) Sun et al, (2024) Wang et al.,
2024bb Song et al. 2023) Xie et al. 2024a Zhong et al.) 2024a). However, the rapid advancement of LLMs heavily relies on manually designed agentic workflows structured sequences of LLM invocations accompanied by detailed instructions. Designing and refining these workflows requires significant human effort, which limits the scalability and adaptability of LLMs to new, complex
domains and hinders their ability to transfer skills across diverse tasks (Tang al. 2024).
Recent efforts have focused on automating the discovery of effective agentic workflows to reduce the reliance on human intervention (Khattab et al. 2024 Yüksekgönül et al.) 2024 [ et al. 2023 Hu et al.] 2024). Despite these advancements, full automation has not been achieved. For instance, Khattab et al. (0024) requires manual workflow setup before automated prompt optimization. Simi- larly, methods proposed by Yükkekgönül et al. (2024) and Zhuge et al. (2024) fail to capture the full diversity of workflows necessary for a wide range of tasks (Yu et al.] 2023 Yang et al. 2024b) Sun et al. 2023), their optimization objectives struggle to represent the breadth of possible workflows. The inailiity to effectively model diverse workflow structures within these automated systems lim- its their utility and impact. ADAS (Hu al. 2024) represents workflows using code, achieving
"These authors contributed equally to this work. 1 "Corresponding authors: Yuyu Luo (F-mail:yuyuluo wwwpsuuwwwuuuuuuuut.eee....c..o Chenglin Wu (F-mail: alexan-
derwu @ deepwisdom.ai)

---

Published as a conference paper at ICLR 2025
10
Cod
NecPronpt
Performance
00
20.
Self Refine
60-
20.3
40
HotpoiQA
DROP
HumanEval
MBPP
GSMBK
Math
Benchmark
Performance comparison with other methods. To assess the method's performance, we employ various metrics across different datasets: solve rate for Math and GSM8K, score for HotpoiQA and DROP, and pass@) for HumanEval and MBPP. Our AFLow (highlighted in yel- low) consistently outperforms all automated workflow optimization and manually designed methods across all six benchmarks.
Figure 1:
relatively complete representation. However, due the efficiency limitations of its linear heuris- tic search algorithm, ADAS struggles to generate effective workflows within a limited number of iterations. This highlights the need for more effective techniques to represent and automate the generation of agentic workflows, which would accelerate the application of LLMs across domains.
In response these challenges, introduce an innovative framework for automatically generating agentic workflows. Our key idea is to model the workflow as a series of interconnected LLM- invoking nodes, where each node represents an LLM action and the edges define the logic, depen- dencies, and flow between these actions. This structure transforms the workflow into vast search space, encompassing wide variety of potential configurations. Our goal is efficiently navigate this space, automatically generating optimized workflows that maximize task performance while minimizing human intervention.
However, the diversity and complexity of tasks present significant challenges. Specifically, each task can have different requirements, operations, and dependencies, which makes difficult represent them in unified yet flexible manner (Chen et al. 2021] Cobbe et al. 2021 Yang et al. 2018 Luo et al., 2018). Furthermore, the search space for possible workflows, comprising an immense number of code structures and node configurations, is virtually boundless, creating additional challenge for efficient exploration and optimization.
To address these challenges, we propose AFLOW, Monte Carlo Tree Search (MCTS) -based frame- work designed to systematically explore and discover optimal agentic workflows. AFLOW repre- workflows flexible nodes connected by code-based edges, which encapsulate possible rela- tionships such as logical flows, conditions, and dependencies. These edges allow the workflow to be modeled graph (Zhuge al. 2024) network (Liu al. 2023), offering a powerful structure for capturing complex interactions between LLM invocations.
To enhance the search process and improve efficiency, AFLOW introduces novel concept of opera- tors- predefidd,,, reusable combinations of nodes representing common agentic operations (e.g., En- semble, Review & Revise). These operators serve foundational building blocks for constructing workflows and are integrated into the search space, ensuring that the exploration process leverages known patterns of effective agentic operations.
F1LOW employs the MCTS algorithm to navigate this infinite search space. The framework's work- flow optimization process incorporates several key innovations: soft pixee--robabbility selection mechanism for node exploration, LLM-driven node cxpansion to introduce possibilities, execu- tion evaluation to assess workflow performance, and backpropagation search iterations. This combination of techniques ensures that AFLoW efficiently discovers work- flows that adapt to the complexity of diverse tasks while reducing reliance on manual intervention.
experience to refine future
We make the following key contributions: (1) Problem Formulation: We formalize the work- flow optimization problem, generalizing prior approaches as specific cases. This provides a unified framework for future research both the node and workflow optimization levels. (2) AFL.W:
2

---

Published as a conference paper at ICLR 2025
We introduce AFLOW, MCTS- basS-baedd method that automatically discovers effective workflows across multiple domains with minimal human intervention. (3) Extensive Evaluation: We eval- uate AFLOW six benchmark datasets: HumanEval, MBPP, MATH, GSM8K, HotPotQA, and DROP. A FLOW outperforms manually designed methods by 5.7% and surpasses existing automated approaches by 19.5%. Notably, workflows generated by AFLow enable smaller LLMMs to outper- form larger models, offering better cost-performance efficiency, with significant implications for real-world applications.
RELATED WORK
Agentic Workflow Agentic workflow and autonomous agents (Zhuge et al.) 2023 Hong et al. 2024a Zhang et al. 2024c Wang et al. 2023 Liu et al. (025) represent two distinct paradigms of LLLM application. The former completes tasks statically through predefined processes with mul- tiple LLM invocations, while the latter solves problems dynamically through flexible autonomous decision-making Compared to autonomous agents that require specific actions and decision pat- terns designed for the environment, agentic workflows be constructed based on existing human domain experience and iterative refinement, offering higher potential for automated construction.
Agentic workflows can be broadly categorized into general and domain-specific types. General workflows emphasize universal problem-solving approaches, such as (Wei et al.) 2022) Wang et al,
2022
Madaan et al. 20233 Wang et al. 22024a] TTng al.) 2025). Domain-specific workflows focus
on building effective processes to solve dopain-specific problems, such as code generation (Hong
al.
2024b Ridnik al.) 2024 Zhong al. (2024)), data analysis (Xie et al. 2024b Ye al. Li et al. 2024a] Zhou et al. (022), mathematics (Zhong et al., 2024b Xu et al. 2024).
2024
question answering (Nori et 2023 Zhou et al. 2024a). Existing work has manually discovered numerous effective agentic workflows, but it's challenging to exhaust various tasks across different
domains, further highlighting the importance automated workflow generation and optimization.
Automated Agentic Optimization Recent work aims to automate the design of agentic work- flows, categorized into three types: automated prompt optimization, hyperparameter optimization, and automated workflow optimization. Prompt optimization (Fernando et al.) 2024 Yüksekgönüü eet al. 2024; Yang et al. 2024a Khattab et al. 2024 Xiang et al. (025) uses LLMs to optimize prompts within fixed workflows. Hyperparameter optimization (Saad-Falcon et al. 2024) focuses optimizing predefined parameters. While these approaches improve performance, they are lim- ited in generalization to new tasks and often require moderate human effort for task speccficc designs.
Automated workflow optimization (Li et al. 2024b; Zhou et al.) 2024b Zhuge al., 2024 Hu ct al. 2024) aims to optimize entire workflow structures, offering more potential for fully automated generation. Recent works explore diverse representations and methods. GPTSwarm (Zhuge al.) 2024) uses graph structures with reinforcement learning, but struggles to represent workflows with conditional states due to graph structure limitations. ADAS (Hu et al. 2024) utilizes code structures to represent workflows and stores historical workflows in a linear list structure, aligning closely with our goals. However, constrained by the efficiency of its search algorithm as relies on overly simplistic representations of experiences in the searching process, making it challenging to discover effective workflows.
AFLOW also uses code to represent workflows, but goes further by providing a more fundamen- tal structure called named node. This structure encompasses various LLM invocation parameters, allowing for more detailed workflow representation. We also introduce operators that implement predefined node combination functions. Simultaneously, AFLOW employs a specially designed MCTS algorithm for automated workflow optimization, leveraging the tree-structured experience and execution feedback to efficiently discover effective workflows.
3 PRELIMINARY
In this section, we will first formulate the automated agentic workflows generation problem in Sec- tion [3] and then discuss design considerations of our AFLOW in Section For the core concept of this section, we provide example explanation in Figure
3

---

Published as a conference paper at ICLR 2025
Generate Node
Judge Node
Prompe
You're helpful assistant... Let's think step by step.. Reason and act.,
Ensemble Node
Revise Node
Review Node
Generate answer based the context..
Self Coff-Connistency Ensemble
Temperatrue: 0,
Graph
Multi-Agent Debate
Models
Code
History
Self Refine
tthought>..s/thongee
<solut Lon iowp..//sooutiinn
Networks
Node
Operator
Edge
Figure 2:
The example node, operator, and edge. We demonstrate the optional parameters for Nodes, the structure of some Operators, and common representations of Edges.
3.1 PROBLEM FORMULATION
Agentic Workflow We define an agentic workflow as a series of LLM-invoking nodes con- nected by edges to define the exection orders, denoted V NN,..,,......... N; ...).. Each node N; represents a specific operation performed by an LLM and is characterized by the following parame- ters. The code abstraction of the node is shown in Appendix A.2
Model M: The specific language model invoked at node N
Prompt P: The input or task description provided to the model each node.
Temperature 7: A parameter controlling the randomness of the LLM's output at node N,.
Output format F: The format in which the model's output is structured (e.g., xml, json, markdown, raw). The node in workflow should provide different output formats, inspired by the Taa et al. (2024).
Edge E represent abstract structures defining node relationships, governing the sequence of execu- tion. The edge E can be represented via various structures, such as:
Graph Zhuge et al. 20024) A flexible structure representing hierarchical, sequential, or par- allel relationships between nodes, allowing for complex branching workflows.
Neural Network (Liu et al. 2023): : A structure that can represent complex, non-linear re- lationships between nodes, allowing for adaptive and learnable workflows based input and feedback.
Code (Hu et al. 2024): : A comprehensive representation that can express linear sequences, conditional logic, loops, and incorporate graph or network structures, offering the most precise control over workflow execution for LLMs.
While graph structures can represent workflow relationships, they require complex extensions (e.g., Petri nets, BPMN) beyond basic DAGs to naturally express parallel execution and conditional logic. Neural networks enable adaptive transitions but lack precise control over workflow execution. In contrast, code representation inherently supports all these relationships through standard program- ming constructs. Therefore, we adopt code as our primary edge structure to maximize expressivity.
Automated Workflow Optimization Given task T and an evaluation function G, the goal of workflow optimization is to discover a workflow W that maximizes G(W,T) This can be formu- lated search process where algorithm A explores the search space S to determine the optimal workflow configuration. The search space S for a workflow optimization problem encompasses all possible configurations of node parameters and edge structures:
4

---

Published as a conference paper at ICLR 2025
Search Space
Search via AFLOW
Search Result
Node
Math Worldlow
Pixed Paranaters
Probubiiity Selection
Soft Moxed
LLM-Based Exponsion
Tonk
@
Terperaturee
Molestion
Format D C. D
word
Generate
Variable Parameters
PROMPT
Think by step and then answer the question,
Questioo: {question)
kounde
Question-Answering Workflow
Operators
no improvveeent
ideatity
G
Teat
Format:
4P Programmer
Porforaasee
Task
Enneebol
CodeGenerate
Part Ettrok
Experience Backpropagation
Executing Evaluation
Ankopele
Consextua
Pornst
Code Represented Edges (eelf,
Extt, resgonse seat self pe
resporse self.cisttal..)
Code Generation Workflow
"Talsel:
Expur lemie
Pask
reture Fitaal response
Candidate Norkelow
Exccuto
selacted
porformance
Scletion
Expanded MarkElow
FiR Oode
Figure 3: Overall AFLOW framework: By setting search space composed of nodes with only prompt parameters flexible, a given operator set, and code representing edge, AFLOW performs an MCTS-based search within this space. Through variant of MCTS designed for workflow op- timization, AFLOW iteratively executes a cycle of Soft Mixed Probability Selection, LLM-Based Expansion, Execution Evaluation, and Experience Backpropagation until reaching the maximum number iterations or meeting convergence criteria.
S = (W, F REE),
where V (N(,,,,P,F) M E M, EO with M, representing the sets of possible language models, prompts, output formats, and edge configurations, respectively. With this formulation, the workflow optimization problem can be expressed as:
W A(S, G,T), W arg max G(W,T), Wes
where A is the search algorithm that explores the search space S, and W* is the optimal workflow configuration that maximizes the evaluation function for the given task T.
3.2 AFLOW OVERVIEW
Limitations Previous Methods Previous approaches Yüksekgönüi et al. (2024); Khattab al. (2024); Zhuge et al. (2024) to workflow optimization have primarily been constrained by the limited scope of their search spaces, based on problem definition in Section 3.1 Another related work, ADAS (Hu et al. 2024 searches in a larger space comprising combination of prompts N (PP, T') and edges E, but fails to discover effective workflows due to the efficiency limitations of its linear heuristic search algorithm.
To address the limitations of previous methods, we propose AFLOW, novel frame- work that leverages Large Language Models (LLMs) as optimizers within a variant of Monte Carlo Tree Search (MCTS) to search for optimal workflows. As discussed in Section 3.1 edges can be represented in both graphs and code. To ensure AFLOW can explore the full range of possible agen- tic workflows, represent nodes N and edges E through code. Specifically, as shown in Figure
Formulation
5

---

Published as a conference paper at ICLR 2025
AFLOW uses a variant of MCTS to iteratively explore the workflow search space, evaluate different configurations, and backpropagate experiences to refine the workflow optimization process.
To enhance search efficiency in practice, we simplify the search space by fixing key parameters such as the model M, temperature т, and format F. This simplification allows AFLOW to focus its search primarily on the code-represented edges E and prompts. To navigate this still vast search space effectively, we introduce the concept of Operators. These Operators encapsulate common agentic operations (e.g., Ensemble, Review, Revise) by combining N and E into unified interfaces, thereby enabling more efficient utilization by AFLOW. By employing these Operators, we achieve more efficient search and streamlined workflow generation.
Formally. given a set of Operators O that represents predefined node combinations, and an edge space & represented through code, the optimization problem can be formalized as:
SAFloW [(P1, Pa. E, E, W AFLOW G,T)
On) R.E.EEEE,EEE
(1)
(2)
Tasks Scope and Operations this paper, we focus on applying AFLOW to reasoning tasks with numerical evaluation functions. We extract common operations from existing literature and define them as part of the operator set O. These operations include: (1) Generate, (2) Format, (3) Review
and Revise Madaan al. 2023), (4) Ensemble Wang al. (2022), (5) Test Test al. (024a), (6) Programmer, and (7) Custom as the default operator for basic node construction. The operator set O can be easily expanded enhance search efficiency for various tasks. Even without any predefined operators, AFLOW can construct different workflow nodes using the basic Custom operator. The
efficiency comparison between these approaches detailed in Section 5.2 For a comprehensive understanding of the operators, we provide their detailed structures in Appendix A.4
THE DESIGN DETAILS OF AFLOW
The core concept of AFLOW is to employ Large Language Models (LLMs) as optimizers within a Monte Carlo Tree Search (MCTS) variant to discover effective workflows. In our MCTS structure, each tree node represents a complete workflow rather than individual LLM-invoking node, enabling the discovery of universal solutions for classes of problems. The search process operates through an iterative cycle of soft mixed probability selection, LL.--based optimization expansion, exccution evaluation, and experience backpropagation until reaching maximum iterations conver- gence criteria. A simplified illustration is shown in Figure 3 with detailed algorithm process and theoretical analysis presented in Appendix and and Appendix respectively.
Existing workflow optimization methods iteratively use past workflow structures to prompt LLMs to discover new structures. However, due to information loss during accumulation (as input tokens increase), this approach struggles to guide LLMs towards specific performance metrics. Combined with the vast search space of code, this reduces search efficiency. Our key idea is leverage the tree structure of MCTS to preserve workflow- based exploration experiences in Nmax rounds workflow optimization. When a workflow is revisited, we accurately reuse past successful experiences and avoid failures, enabling effective workflow generation and improving search efficiency. To prevent local optima, we introduce a special selection mechanism allowing generation from a blank template at any round. Next, we will introduce the complete process of AFLOW, as shown in Algorithm [
Initialization AFLOW begins with template workflow Wo, which provides a framework for invok- ing nodes and operators. The code template, detailed in Appendix A.3 allows the LLM optimizer to complete workflow simply by completing call functions. Prior to initiating the search process, we randomly partition the dataset into a validation set (20%) and test set (80%), with the random seed fixed at 42. To optimize computational efficiency, AFLOW then executes the blank template five times on the validation dataset. From these executions, we select subset of problems that exhibit high variance in scores, which becomes the final validation set.
Selection Our algorithm forms the initial workflow by evaluating an empty workflow the val- idation set. And then continuously select workflows based on a soft mixed probability selection strategy. eWe propose this strategy for workflow optimization: combining uniform and score-based

---

Published as a conference paper at ICLR 2025
Algorithm Algorithm of AFLOW: Detailed implementation
Require: Evaluator C, Dataset D, Operators O
Ensure: Optimized Workflow W* Initialize Wo, split D into Dy and Dr
2: W + +Wo
3: for ileralion < to Maaz do
4: workflow +- Select(tree)
Using soft mixed probability strategy > LLM-based expansion
5:
stild.workflood + Expand( O)
6: score Evaluate(childd work fork G, Dv)
Multiple runs for robustness Update experience and scores
7: Backpropagate(child. workdlddwwooorp.o........o workklou score)
8: Update W if improved
9: if ConvergenceCrrteriaMett( then break
end if
11: end for return W
weighted probability distributions to select from top-k workflows and the initial workflow, where including the initial workflow ensures persistent exploration capability while avoiding local optima. The formula for this selection strategy is as follows:
Poxxed (1) = A. - (1- Σド= exp(a 8max
expla (s, 5max) ))
(3)
where is the number of workflows, 8; is workflow i's score, Smax the maximum score, (0.4) controls score influence, and X (0.2) balances exploration and exploitation.
Expansion In the expansion phase, we employ LLM optimizer create workflows and the optimize prompt is illustrated in Appendix A The optimizer leverages the selected workflow's experience generate new prompts or modify node connections by altering code, resulting in new workflows. Specifically, to maximally uncover insights from past iterations, the experience includes all modifications and their corresponding improvements or failures on the selected workflow, along with precise logs of predictions and expected output.
Evaluation AFLOW directly executes workflows to get feedback due explicit evaluation functions in reasoning tasks. We test each generated workflow 5 times on the validation set, computing mean and standard deviation. While this increases per-iteration cost, provides more accurate feedback for the optimizer. This precision enhances search efficiency, ultimately reducing the number of iterations required to reach an effective solution.
Backpropagation After execution, we record: (1) the workflow's performance, (2) the optimizer's modification of its parent workflow, and (3) optimization success relative to its parent. This infor- mation is stored in experience and propagated back to the parent workflow, while the performance score is added to the global record for selection.
Terminal Condition We implement early stopping reduce unnecessary execution costs: the pro- cess terminates if the top-k average score shows no improvement for n consecutive rounds, or after algorithmic details.
N total rounds otherwise. See Appendix A.6 A.6
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Datasets We utilized six public benchmarks for our experiments. Following established prac-
tices (Saad-Falcon al. 2024 Hu et al. 2024) in workflow optimization, we divide the data into
validation and test sets using 1:4 ratio. Specifically, we use the full datasets for GSM8K (Cobbe)
et al. 2021) HumanEval (Chen et al. 2021). and MBPP (Austin et al. (021). For HotpotQA ( (Yang 2018 and DROP (Dua et al. 2019), we randomly select 1,000 samples each, in line with (Hu
et al.
al.
2024 Shinn al. 2023 For the MATH (Hendrycks al. 2021) dataset, follow (Hong

---

Published as a conference paper at ICLR 2025
et al. 2024a) in selecting 617 problems from four typical problem types (Combinatories & Proba- bility, Number Theory, Pre-algebra, Pre-calculus) at difficulty level 5.
Baselines We compare workflow discovered by AFLOW against manually designed methods for LLMs, including I0 (direct LLM invocation), Chain-of-Thought (Wei et al. 2022), Self Consis-
tency CoT (5 answers) (Wang al., 2022). MultiPersona Debate (Wang al. 2024a).
Self-
(max iteration rounds) (Madaan et al. 2023), and MedPrompt (3 answers and 5 votes) (Nori et al,
2023 We also compared against workflow designed by automated workflow optimization method
ADAS (Hu et al. 2024).
Implementation Details AFLOW utilizes different models for optimization and execution. We employ Claude-3, 3.5-sonnet (Anthropic 2024) as the optimizer and use models: DeepSeek- V2.5 Deepseek (024), GPT-40-mini-0718 (OpenAl 2024b), Claude-3.-ssonee-00622 (Anthropic)
2024]. GPTT4o-0513 (OpenAI 2224aa) as executors. All models are accessed via APIs. We set the temperature to for DeepSeek- V2.5 and for the other models. We set iteration rounds 20 for AFLOW. For ADAS, we use Claude-3.. 3.5e--nt the optimizer and GPT-4o-mini the executor, with the iteration rounds set 30.
Metrics. For GSM8K and MATH we report the Solve Rate (%) as the primary metric. For HumanEval and MBPP, we report the pass@1 1 metric as presented in (Chen al. (20TT) to assess code accuracy. For HotpotQA and DROP, we report the FI Score. Additionally, for all datasets, we calculate the cost by tracking token usage to construct a pareto front, visually demonstrating the performance-cost trade-offs between different methods.
5.2 EXPERIMENTAL RESULTS AND ANALYSIS
Main Results The main experimental results, as shown in Table demonstrate the effectiveness of aver-
FFLLW. Workflows optimized by AFLOW outperform all manually designed methods age of 5.7% % and surpass contemporary automatic workflow optimization work by 19.5%. Across six datasets in QA, Code, and Math domains, AFLOW achieves an average performance of 80.3%, marking the capability and usability of this method. Notably, compared to similar works, AFLOW performed better more challenging tasks, improving over ADAS on MATH and MBPP tasks by 57%, showcasing the robustness of the model on complex datasets.
Table 1: Comparison of performance between manually designed methods and workflow generated by automated workflow optimization methods in QA, code, and Math scenarios. All methods executed with GPT-4o-mini on divided test set, and we tested it three times and reported it on the average.
Benchmarks
Method
Avg.
HotpotQA
DROP
HumanEval MBPP
GSM8K
MATH
10 (GPT-4o-mini)
68.1
68.3
87.0
71.8
92.7
48.6
72.8
CoT (Wei 2022
67.9
78.5
88.6
71.8
92.4
48.8
74.7
CoT SC (5-shot) (Wang
2022
68.9
78.8
73.6
92.7
50.4
76.0
MedPrompt (Nori al.
2023
68.3
78.0
91.6
73.6
90.0
50.0
75.3
MultiPersona Wang et al.
2024a)
69.2
74.4
89.3
73.6
92.8
50.8
75.1
Self Refine (Madaan et al.
2023
60.8
70.2
87.8
69.8
89.6
46.1
70.7
ADAS (Hu et al. 2024)
64.5
76.6
82.4
53.4
90.8
35.4
67.2
Ours
73.5
80.6
94.7
83.4
93.5
56.2
80.3
To explore whether the workflow searched by AFLOW is agnosii,,, we use GPT-4o-mini and DeepSeek-V27 V2.5 as execution LLMs to search effective workflows with different structures, with the results illustrated in Table 2 When applying these workflows to other models, the vast ma- jority demonstrate stronger performance than the baseline, showeasing the generalizability of the workflows discovered by AFLOW. Simultancously, DeepSeck-V2. V2.5 performs notably weaker on GPT-4o-mini compared to the workflow found using GPT-4o-mini itself. This suggests that different language models require different workflows to achieve their optimal performance.
observe that the workflow identified using
Cost Analysis We demonstrate the comparison of performance and cost between the baselines and the top three workflows found by AFLOW using GPT-4o-mini and DeepSeek- V2.5 as execution L.MMs. The comparison is made across four models with different capabilities and price points.
8

---

Published as a conference paper at ICLR 2025
Table 2: Comparison of performance between manually designed methods and workflows generated by AFLOW with two executor LLM: GPT-4o-mini ("Ours") and DeepSeek-V22 V2.5 ("Ours**), All workflows are tested thrice on the humaneval test set, with average results reported. "MP" denotes "MedPrompp" (Nori et al. 2023), and "MPD" denotes "MultiPersona Debate" (Wang et al. 2024a)
The results demonstrate that workflows obtained through AFLOW exhibit strong transferability.
Methods
Model
10
CoTT
CoT SC
MP
MPD
SR
Ours
Ours"
GPT-4o-mini
87.0
88.6
91.6
91.6
89.3
87.8
94.7
90.8
DeepSeek-V22 V2.5
88.6
88.6
88.6
89.3
90.0
93.9
94.7
GPT-4o
93.9
93.1
94.7
93.9
94.7
91.6
96.2
95.4
Claude-3.5-sonnet
90.8
92.4
93.9
91.6
90.8
89.3
95.4
94.7
Scatter Plot with Pareto Front of HumanEval
96
passe (%)
92-
ArtOR spt-sto-zins)
AFLOSI deeppeek)
claude-3. L.ande-2.--aonne 3-aoonnet
Debate
deepseck
9pt-4a-mini
Cost (S) in Log Scale
Figure 4: The cost refers the total expense of executing the divided HumanEval test AFLOW (execution model) refers to workflows found by AFLOW using the execution model to obtain feed- back. The colors in the legend represent the LLM used execute each workflow in test dataset. The specific numerical values for this Figure can be found in Appendix
Results demonstrate that AFL.OW can identify workflows that allow weaker models to outperform stronger models on the pareto front of cost-effectiveness. This breakthrough effectively removes barriers to the widespread application of agentic workflows across various domains. By automating the design of effective agentic workflows, AFLOW eliminates the human labor costs previously required. Moreover, the ability to achieve superior performance at lower costs compared to stronger models opens up further possibilities for widespread adoption.
Ablation Study We introduce operators human-designed effort enhance search efficiency. An ablation study on GSM8K (Figure (5) shows that operators help FFLOW discover better workflows more efficiently, achieving incremental improvements. Notably, even without operators, AFLOW maintains strong performance (93.1%) 13), surpassing manual designs. Notably, AFLOW autonomously develops ensemble-like structures without operators, demonstrating its capability for independent workflow design and marking significant step towards full automation. Details is shown in Ap- pendix
Case Study AFLOW demonstrates a clear iteration process, shown in Figure 6 illustrating how evolves from a blank template (containing only a single Node without prompts) to the structure presented in Figure BB. In each iteration, AFLOW employs single-step modification, meaning either adds one operator (rounds 2, 3) or makes a targeted modification to a prompt (rounds 8, 10). Among the unsuccessful exploration rounds, AFLOW introduced a custom review node that directly modified answers generated through complex processes without additional reasoning (round 5), which decreased accuracy. In round 14, AFLOW attempted to rephrase the problem but overly focused on "discount" information, leading to decrease in accuracy. This iteration process showcases how tree-based search allows AFLOW to further optimize known paths while retaining

---

Published as a conference paper at ICLR 2025
The Test and Validation Curve of GSM8K
The Code Represented Workflow of GSM8K
snyne def (self, problen: str):
Ieplemenaaiion
92
solutions []
for in range(Si: Generate 5 solutions solution = awais self. custon! inputeproblen, InstructionAATH_BL.. PROMPT
Solve Rato (%6
soluttions.oppentselurtt ressoonssee
86
fina soluttin avait stt.aac_eneeblee volutionsszolutions problemspreblen
84
Add verification using Frograxsor operiater verification await self-prograwert
problemproblee,
w/o operator test
analysis=fina toluutinn response
80
operator validation
78
oparator test
if verification! outputt]] ]
operator validation
return verafication "output') ", self.total else: return finat_sotution "response totf.total_cost
Iteration Round
IAI
Figure 5: (A) Comparison of highest performance curves GSM8K for both validation and test sets generated by AFLOW with and without operators. Compared to other datasets, GSM8K has larger data volume, meaning that the same percentage improvement represents greater increase in correctly solved samples, avoiding fluctuations in improvement due to small data size that could affect comparisons; (B): The code for the best-performing workflow discovered by AFLOW on the
GSM8K dataset.
MATH SBL OF PROMPP
Scores 0-0872
Goore: 0.8591
PATILSDUU
Modiifionson ScEzaumble
-Duut cheeck your cal culculan icts and reasoning
concise Final
Scores 9166
fnsur your single nuserical
Medi ienrie zuviso step
opecator,
Veriryy your sotuticn p plppging back
step.
va value without any
adeitioial
Modi the conton proapt for formstting the
Boores 8:5352
prob or using diternative acthod ir possible.
Meditfy
to not snctude any explanatory text with your ansver,
far by
checking
Show each step of your process clearly.
the susber aiself.
roand optinnll path
round out the opeinsl path
Figure 6: Tree-structured iteration process of FLLOW on GSM8K: We highlight the path from the initial round (round I) to the best- rest-performing workflow, reporting the score for each node and its modification from the previous node. The purple sections in the prompts on both sides represent the main prompt modifications in this iteration.
the ability to explore new ones. On the MBPP dataset, AFLOW discovered structures similar to current manually designed workflows, such as test generation and execution by LLMs in Ridnik et al. (024)). The workflow and more discovered results are presented in Appendix B and complete optimization process is presented in Appendix
CONCLUSION
This paper has introduced AFLOW, novel framework for automated workflow optimization. We have comprehensively formulated the automated workflow optimization problem, establishing foundational structure for future research. AFLOW has leveraged Monte Carlo Tree Search and code-represented workflows to navigate the vast search space of possible workflows efficiently. Our experiments across six benchmarks demonstrate the effectiveness of AFL.ow, which has out- performed manually designed methods and existing automated optimization approaches. Ablation studies have shown that AFLOW can autonomously discover effective structures, even without pre- defined operators. Importantly, AFLOW has enabled weaker models to outperform stronger ones on the Pareto front cost-effectivennss. We further discuss the potential applications of AFLOw across diverse domains in Appendix(]] potentially revolutionizing the adoption of agentic workflows across various domains. These results have highlighted AFLOW's 's potential for enhancing LLMs' problem-solving capabilities while optimizing computational costs.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This paper is supported by NSF of China (62402409), Guangzhou Municipality Big Data In- telligence Key Lab (202300300012), Guangdong Basic and Applied Basic Research Foundation (20234555110545), Guangzhou Basic and Applied Basic Research Foundation (2025A0413935), and Guungzzoo-HKUUSTGGG Joint Funding Program (2025A0313714).
REFERENCES
Anthropic. Introducing claude 3.5 sonnet.
claude 3-5 sonnet 2024.
Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Do- han, Ellen Jiang, Carrie Cai, Michael Terry, Quoe V. Le, and Charles Sutton. Program synthesis with large language models. CoRR, abs/2108.07732/ 2021.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert- Vosss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage. Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained code. CORR, abs/2107.00374 2021.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano. Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 1xXv:2210014468 2021.
Yanqi Dai, Huanran Hu, Lei Wang, Shengjie Jin, Xu Chen, and Zhiwu Lu. Mmrole: A compre- hensive framework for developing and evaluating multimodal role- pole-playing agents. arXiv preprint arXhv:2408.04203 2024.
Deepseek. DeepSeck-V22 V2.5. https://hugiinfaae..///eess.s httpepppsseeeaaeeeeeeeeeeeeeee 2024.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In NAACL-HLT (1), pp. 2368-2378, Association for Computational Linguistics, 2019.
Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self-imprrvement via prompt evolution. In ICML. OpenRe- niew.net, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Danyang Li, Jiaqi Chen, Jiayi Zhang, Jinlin Wang, Li Zhang. Lingyao Zhang, Min Yang. Mingchen Zhuge, Taicheng Guo, Tuo Zhou. Wei Tao, Wenyi Wang, Xiangru Tang, Xiangtao Lu, Xiawu Zheng, Xinbing Liang, Yaying Fei, Yuheng Cheng, Zongze Xu, and Chenglin Wu. Data interpreter: An LLM agent for data science. CoRR, abs/2402.1 8679, 2024a.
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang. Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. Metagpt: Meta programming for A multi-agent collabo- rative framework. In ICLR. OpenReview.net, 2024h.

---

Published as a conference paper at ICLR 2025
Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. arXiv preprint arXiv:2408.0844 08435, 2024.
Omar Khattab, Arny Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vard- hamanan, Saiful Haq, Ashutosh Sharma, Thomas Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. Dspy: Compiling declarative language model calls into state-of- the-art pipelines. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Boyan Li, Yuyu Luo, Chengliang Chai, Guoliang Li, and Nan Tang. The dawn of natural language to SQL: are we fully ready? Proc. VLDB Endow., 17(1 1):3118-3331, 2024a.
Zelong Li, Shuyuan Xu, Kai Mei, Wenyue Hua, Balaji Rama, Om Raheja, Hao Wang, He Zhu, and Yongfeng Zhang. Autoflow: Automated workflow generation for large language model agents. CoRR, abs/2447.1282) 2024b.
Bang Liu, Xinfeng Li, Jiayi Zhang, Jinlin Wang, Tanjin He, Sirui Hong, Hongzhang Liu, Shaokun Zhang, Kaitao Song, Kunlun Zhu, et al. Advances and challenges foundation agents: From brain-inspired intelligence to evolutionary, collaborative, and safe systems. arXiv preprint arXiv:2504.01999, 2025.
Xinyu Liu, Shuyu Shen, Boyan Li, Peixian Ma, Runzhi Jiang, Yuyu Luo, Yuxin Zhang, Ju Fan, Guoliang Li, and Nan Tang. A survey of NL2SQL with large language models: Where are we, and where are we going? CoRR, abs/2408.05
2024.
Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. Dynamic Ilm-agent network: An I1m- agent collaboration framework with agent team optimization. arXiv preprint arXiv:2310.02170, 2023.
Yuyu Luo, Xuedi Qin, Nan Tang, and Guoliang Li. Deepeye: Towards automatic data visualization. In ICDE, pp. 101-112. IEEE Computer Society, 2018.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Boye Niu, Yiliao Song, Kai Lian, Yifan Shen, Yu Yao, Kun Zhang, and Tongliang Liu. Flow: Mod- ularized agentic workflow automation. The Thirteenth International Conference on Learning Representations, 2025.
Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolò Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, Renqian Luo, Scott Mayer McKinney, Robert Os- azuwa Ness, Hoifung Poon, Tao Qin, Naoto Usuyama, Chris White, and Eric Horvitz. Can generalist foundation models outcompete special-purposs tuning? case study in medicine. CoRR, abs/2311.166455 2023.
OpenAl. Hello got-4o. https: httppr//pppaal.cccm////////llllp 2024a.
OpenAl. GPT-4o mini: oww.acaiggoeeeefeeeeeeeeeeeeerrr
com index/
2024b.
Shuofei Qiao, Runnan Fang, Zhisong Qiu, Xiaobin Wang, Ningyu Zhang, Yong Jiang. Pengjun
Xie, Fei Huang, and Huajun Chen. Benchmarking agentic workflow generation, 2025. URL
httts::/arriv..ott http://bb/2241....
Tal Ridnik, Dedy Kredo, and Itamar Friedman. Code generation with alphacodium: From prompt engineering to flow engineering. CoRR, abs/2401. 1085000 2024.
Jon Saad-Falcon, Adrian Gamarra Lafuente, Shlok Natarajan, Nahum Maru, Hristo Todorov, Etash Guha, E. Kelly Buchanan, Mayee Chen, Neel Guha, Christopher Ré, and Azalia Mirho- seini. Archon: An architecture search framework for inference time techniques. arXiv preprint arXiv:2409. 15254, 2024.
12

---

Published as a conference paper at ICLR 2025
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In NeurIPS, 2023.
Chan Hee Song, Brian M Sadler, Jiaman Wu, Wei-Lun Chao, Clayton Washington, and Yu Su. Lim-planner: Few-shot grounded planning for embodied agents with large language models. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2986-2997. IEEE Computer Society, 2023.
Hongda Sun, Weikai Xu, Wei Liu, Jian Luan, Bin Wang, Shuo Shang, Ji- Ji-Rong Wen, and Rui Yan. From indeterminacy to determinacy: Augmenting logical reasoning capabilities with large lan- guage models. arXiv preprint arXiv:23 10. 18659, 2023.
Yiyou Sun, Junjie Hu, Wei Cheng, and Haifeng Chen. Chatbot meets pipeline: Augment large language model with definite finite automaton. arXiv preprint arXlv:2402.04411, 2024.
Zhi Rui Tam. Cheng- Cheng-Kuang Wu, Yi-Lin Tsai, Chich Yen Lin, Hung-yi Lee, and Yun-Nung Chen. Let me speak freely? A study the impact of format restrictions performance of large lan- guage models. CoRR, abs/2408
2024.
Nan Tang, Chenyu Yang, Ju Fan, Lei Cao, Yuyu Luo, and Alon Y. Halevy. Verifai: Verified genera- tive AI. In CIDR. www.cidrdb.org. 2024.
Fengwei Teng, Zhaoyang Yu, Quan Shi, Jiayi Zhang, Chenglin Wu, and Yuyu Luo. Atom of thoughts for markov Ilm test-time scaling. arXiv preprint arXiv:2502.12118, 2025.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint ar arXhv:2305. 2023.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consisteney improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2022.
Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing the emergent cognitive synergy in large language models: A task-solving agent through multi- self-collaboration. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 257-279. 2024a.
synergg
Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Martin Eisenschlos, Vincent Perot, Zifeng Wang. Lesly Miculicich, Yasuhisa Fujii, Jingbo Shang, Chen- Yu Lee, et al. Chain-of-table: Evolving tables in the reasoning chain for table understanding. In The Twelfth International Conference on Learning Representations, 2024b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of thain-of-thouuht prompting elicits reasoning in large language models. Advances in Neural Information P Processing Systems, 35:24824-24837, 2022.
Jinyu Xiang, Jiayi Zhang, Zhaoyang Yu. Fengwei Teng, Jinhao Tu, Xinbing Liang, Sirui Hong, Chenglin Wu, and Yuyu Luo. Self-supervised prompt optimization. arXiv preprint arXiv:2502.d0 06855, 2025.
Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: benchmark for real-world planning with language agents. In Forty-first International Conference on Machine Learning, 2024a.
Yupeng Xie, Yuyu Luo, Guoliang Li, and Nan Tang. Haichart: Human and Al paired visualization system. Proc. VLDB Endow,, 17(1 1133178-199111 2024b.
Yiheng Xu, SU Hongjin, Chen Xing, Boyu Mi, Qian Liu, Weijia Shi, Binyuan Hui. Fan Zhou, Yitao Liu, Tianbao Xie, et al. Lemur: Harmonizing natural language and code for language agents. In The Twelfth International Conference Learning Representations, 2024.
13

---

Published as a conference paper at ICLR 2025
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In ICLR. OpenRevvew.net, 2024a.
Ling Yang, Zhaochen Yu, Tianjun Zhang, Shiyi Cao, Minkai Xu, Wentao Zhang, Joseph E Gonzalez, and Bin Cui. Buffer of thoughts: Thought-augmenttd reasoning with large language models. arXiv preprint wrXiv:2406.04277 2024b.
Zhilin Yang. Peng Qi, Saizheng Zhang. Yoshua Bengio. William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In EMNLP, pp. 2369-2380. Association for Computational Linguistics, 2018.
Yilin Ye, Jianing Hao, Yihan Hou, Zhan Wang, Shishi Xiao, Yuyu Luo, and Wei Zeng. Generative Al for visualization: State of the art and future directions. Vis. Informatics, 8(1):43-66, 2024.
Junchi Yu, Ran He, and Zhitao Ying. Thought propagation: An analogical approach to complex reasoning with large language models. In The Twelfth International Conference on Learning Representations, 2023.
Mert Yüksekgönül, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic "differentiation" via text. CoRR, abs/2406.07496, 2024.
Guibin Zhang, Yanwei Yue, Zhixun Li, Sukwon Yun, Guancheng Wan, Kun Wang. Dawei Cheng, Jeffrey Xu Yu, and Tianlong Chen. Cut the crap: An economical communication pipeline for Ilm-based multi-agent systems. arXiv preprint arXiv:2410.02506, 2024a.
Guibin Zhang, Yanwei Yue, Xiangguo Sun, Guancheng Wan, Miao Yu, Junfeng Fang, Kun Wang, and Dawei Cheng. G-designer: Architecting multi-agent communication topologies via graph neural networks. arXiv preprint arXiv:2410. 11782, 2024b.
Jiayi Zhang, Chuang Zhao, Yihan Zhao, Zhaoyang Yu, Ming He, and Jianping Fan. Mobileexperts: A dynamic tool-enabled agent team in mobile devices. CoRR, 0bs/2407.00313, 2024c.
Lianmin Zheng. Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bence and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46622 2023.
Li Zhong, Zilong Wang, and Jingbo Shang. Debug like a human: A large language model debugger via verifying runtime execution step by step. In ACL (Findings), pp. 851-870. Association for Computational Linguistics, 2024a.
Qihuang Zhong, Kang Wang. Ziyang Xu, Juhua Liu, Liang Ding, Bo Du, and Dacheng Tao. Achiev- ingi, 97% on gsm8k: Deeply understanding the problems makes llms perfect reasoners. arXiv preprint arXiv:2404. 14963, 2024b.
Andy Zhou, Kai Yan, Michal Saapentokh-Rothmaann Haohan Wang, and Yu-Xiong Wang. Lan- guage agent tree search unifies reasoning, acting, and planning in language models. In Forty-first International Conference on Machine Learning, 2024a.
Wangchunshu Zhou, Yixin Ou, Shengwei Ding, Long Li, Jialong Wu, Tiannan Wang, Jiamin Chen, Shuai Wang, Xiaohua Xu, Ningyu Zhang. Huajun Chen, and Yuchen Fleanor Jiang. Symbolic learning enables self-evolving agents. CORR, abs/2406. 18532, 2024b.
Xuanhe Zhou, Guoliang Li, and Zhiyuan Liu. Lim as dba. arXiv preprint 308.022308.05481, 2023.
Yizhang Zhu, Shiyin Du, Boyan Li, Yuyu Luo, and Nan Tang. Are large language models good statisticians? In NeurIPS, 2024.
Mingchen Zhuge, Haozhe Liu. Francesco Faccio, Dylan R Ashley, Róbert Csordás, Anand Gopalakrishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Herrmann, Kazuki Irie, et al. Mindstorms in natural language-hased societies of mind. arXiv preprint arXiv:2305. 10.200.776666 /7066. 2023.
Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. Gptswarm: Language agents as optimizable graphs. In Forty-first International Conference Machine Learning, 2024.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1
LLM BASED EXPANSION: PROMPT FOR LLM OPTIMIZER
Workflow optimize prompt
*na building Graph correspanding Prompt to jointly problems. Reterring to the given graph and prompt, which forms basic example of a (type) solution approach, please reconstruet and optimize them. You can add, parameters,
[type]
modify, fy, delete In XMI Lags in your reply. Basure they are complete and correct to avoid zunt ime failures.
prompts. Tholude your atngle fodificat
optimiziag, you can incorporate critical thinking review, revise, ensemble igenerating multiple enswers througa different /sirenttsimiaa prompts, then selcAsk, etc, Consider Python'a loops (for, while, List compzehenslona), conditionel statements LLf-elif-else ternary operaters), or machine learning techr.qqes (e.g., linear regreasion, decision treea, The graph complexity should not exceed 10. Use logical and control flow (IF ELSE, more
the majority to obtain final answer),
networka, clustering (ing))
loaps)
graphical represensation. Ensure required the current graph from prompt_ _uston are includad. prompts. Output moditied grapa (if needed) The prompt you need generate la only the "prompt_custem
prompts any other
all the necesaary Prompts prompt_custon
used within Custem, Other metheds siready have built-in preapts and being generated. Only generate those unuzed proneta in prompt_custon. the generated prompt must contain any placeholders. Considering infermation icss, complex graphs yield better results, but insufficient informst transmiaston can oml: the solution. It's crucial include necessary context during the process. ea*
prohibiled
prompt_custom' pleaze remove
A.2 BASIC STRUCTURE NODE
Node structure
class ActionNede:
ssyne def (self, context, 11m, schema...): ***
spacam context: Everything we should know when filling node. :pacam 11m: Large Language Model pre defined system message.
:param schema: json/markdwwn/mm. determine example and output format. taxt
row:
LLK with jaon format morkdown: when generating code, markdown is always better
jsen: it's easy
xml: it's atruetured format is advantageous corstraining outputa
recurn self
A.3 BASIC STRUCTURE OF WORKFLOW
Workflow structure
natasetType fiteral 1 ['HumanEval "MBPP" "6SM8K", "MATH", "RotpotOs", "DROP"]
class WorkElow:
def init_!
sel',
name: s:2,
11m_nonfig,
dacaset: DatasetTyge,
None:
self.name
self.dataset dstaget
m_inatance In_config)
self.1lm
w.f.lll..cost_naei careccsa_annerr costManagar
15

---

Published as a conference paper at ICLR 2025
sayne _caal___________________________ (self, problem: stz):
Implementation of the workflow enP
raise Not Not.mppeeenteedarro//// ("thh mathed should implemented by the subclass")
A.4 OPERATORS
Operators
class ContextualGenerale (0peralor):
ааynс -sall (self, problem, context, mode: str None):
prompt
thought
Fill_kwargs :"context"; prompt, "lim": self.1lm)
moda:
Fill_kwarga ["mode*]
aode await ActionNode. frriionnode...o......oo Econ_pydan: (Seneratedy .fill (v.lll kwargs) response wwde.instruct_ce..oooo wwstrrcc_cootterittt modell mooee__durp()
return response
class CodaGenerata (Operator):
sayne def _call (self, problam, funet len_nnee,, mode: at Nonel prompt GENERATE COCGAATE__OODLLOK___PC PROMPT, Iormat werobien_deeccppiiooree [ill_kwa=ge - : :"context"; prompt, "lim": self.ilm, "function_name": function_name)
1f mode:
[ill_kwarqs] ["mode*] [=modrrrs"""mde) mode
node await ActionNode. from
tatorrrrronsssteeeeeeeeeteeeeese pydancic (ZodeCanerato0g)) fill [+efill kwarga)
reaponse mode..nsttuue wwwrrrucctcotttteeeeeeeo content.moe model_durp re:arn respoase
class Format (0perator):
aayпс def _call (self, problem, solucion, mode: str Nodel:
prompt FORMAn PROMFT. format dew.elleneesrrrittoorrroooo solutionnsolution)
fill_kwa:ge - :"context"; prompt, "lim": selt,1la)
Eill_kwargsf
("maree [modde"] mode
node await ActionNode. pydanc (Fo-ratop)) (eefill__aaraaeeee kwargs! reaponse Lnstruct_tortt wtl.oottttttooe...... model modei__dur(()
recuin response
class Review 10pezator):
aayne def sall iself, problem, solucion,
Nonel:
prompt REVIEK REVIEE_ formatt
solvtion=solutioon
ortterta-self
fill_kwa=ge :"context"; prompt, "lim": self,ilm) mode:
www.kaaaganndeeee mode
aode await ActionNode. foondde.rdo__paarrrreea. (ReviewOp) ffill (v4fll response inde..istructtconte madde_ddumpp
return reaponse
class Revise (Cpezator):
asyne def -call (self, problem, soluzion, feedback, mode: str Nonel: prompt DOWVISE_POOFFE formtt perchlen_descriptiooo b feedbackrfeedhack)
salutian=sollution.
fill_kwargs :"context"; prompt, "lim": self,11m) if mode:
( mode
aode await ActionNode. ftr.niodddd.eed.ppeeee pydansic (Revise0p) ffill (+ response node, node.instruct wwwssruct_cootttrrrreeeoeo _node__durp()
reaponse
Ensemble :
aayne def _aall__1 (self, solutiona: Llat [str], problem: str, modes str None] answer_mapping [)
solution_tex:
for index, solution in cmnnrrato(solutionnn ansver_meopinglohrr sclution_toxt += &" ("[chr(e (65 index) ): \nlstr AnArla"
Index
16

---

Published as a conference paper at ICLR 2025
prompt ENSSMALR, PROMPT PROMPT-format format Isolut /
problen_descrine ion-problem)
Fill_kwazge " :"context": pronpt, "lim": self.llm)
mode:
( mede
node await ActionNode. from. (knsembletp) .fill (waf reaponse instrrntt wtttr_ccntttttmoot model__dup(()
anawer response.get "trponee.eLL"ssluiiii. iowlllssot answer answer.strip upper
("solution": solutions lenswes__aapinggllee (answer))
Test (Operator):
def exec_code tself, solution, eatry_poiat);
fail_cases [.
fail_cases :- []: fai-_cases
elae:
sayne (self, problem, solution, eniry_ test_loop: for in itest_t result (eetrexee__ooeeelttttttt entry_points
if result "na errer*:
"eturn |*reatlt": True, "solution": so solution]
elif "exec_fail_case"
resul.:
sesult - ("exe__("ee=_eiiii prompt
prablemmprooalen,
solution=solutin,
exoc_paas=f"xxecured unsucoesafolly,
Ar. [result)",
unauoesatully",
node await
ActionNade. from iir..pppcentteiffffleeeioooooos
(eantext-prampt,
1ln=self.1in mode="code_ [oodd__fl""))
seeponse wusstruct_oootetoo wtw.eenn...dee_mdmm (ump))
solution response ("reflact cectlaacioon. lon_tonnn_eoeooe ut
else:
result self. exte_.xx___caa code (solution, entry_point)
result error";
return ("result"; Tzue, "solution"; solution)
else:
("result"; False, "solucion"; solution]
Programmer (0perator):
sayne def exee_coda (coda, timeout=180);
def run_code ():
Lry:
gicbel_nemespece {}
exede, [lobsl_nemespart
except ...
done cone_event threading. Event result ("Erzoz", "subgrocess ecros"]
daf ()apper():
nenloral result code
done_event.set
with concurrent. futures. PrresdPoolBxecusooo wook_work)ss-]]
future exccutor. (wrapper)
try:
if done_event: event, wait It ttimeoutstimoon return result
elae:
future.
"Error", "Exceed time limit"
finally:
dout (walt=Falae)
coda_generste sayne code www.raeeereeee.oo fself, problem, analysts, feedback, mode)
prompt PYTTON CODE_ www__VERRIEERRR.... VERIPTER_FROMPT. format (p:oblemmproales. atalysis=emalyeis,
17

---

Published as a conference paper at ICLR 2025
1111 [1l1_kaangs :"oontext"; prompt, "Iim": ael II,, *fnnc con_atto__amee *soIve")
kwarg
(erargs["mnn ["mode"] mode
node await ActionNode, fronn pedonn (CodeGenerate@0) (1l1 (xvfil kwarge)
node, waadeeiiistuttcotteeiiioo dump(()
recurn response
saync def _call____ __all_ (self, problem: str, analysis: str. "None")
for i in (3)]
code awall ae._.cooe_.ensraaeelprbbee. analysie, feedback, mode="code_fill)
code = coda["code"
stacus, output
sel sexe_exe (codel
If atalus "Success";
zeturn ("oode": code, "output": output)
else:
("code": code, "output": "error")
Providing predefined operators can effectively enhance the search efficiency of AFLOW. We imple- ment six common operator structures, including: Generate (Contextual, Code), Format, Review & the public test dataset of the
Revise, Ensemble, Test, and Programmer. For the Test Operator, dataset as test data. For datasets like MBPP that don't provide public test dataset, we follow the setting in [Zoog al. (2024a) where the first test case of each problem public test data.
A.5 MAPPING WORKFLOW FROM FORMULATION CODE
An example of Workflow
asyno _all__(se11 (self, problem: str, entry_points atr):
Implementation the workflow
Custem operator generate anything you want.
But when you want to get standard code, you should use custon_code_geneeaal eperator.
solutions []
for in (3): Gererate 3 solutionn
solation awalt entry_ geteeeereeeeeeerrtteeeeeeeeeeee
generate eevobbbbemprbble..
pestructioonprrope wwt.sstteccoo__aaeeee COW.LOCCEEETATTTPPROOOO
solutions. (solution] ''response" '])
best_solution await self. Selisss_eeeeeett ensemblee (sclu: ionsssclution,, problem-problen!
test_result awatt self. (problem=problem, solution-test__ooluilon response
4 en.ryy_poin--enttty.oiii
if test_ ('remult"), else:
ceat_reault ('solution"), self. www.coss_aaaarrr.tte
test fails, try
solution
sixed_solution await self.custon cust.mm (inttntingrrsff"rrot (best_solution response
[problem] Arrailed solution:
I]] AnBrror: ressppss""""sssssse (test_resute ('solution
petpsutttoo-pompt__uus rixtem.rix_.oos_.... PRUMPT)
recurn fixed_solut solutior ('rasponsee) ), self, self,llm.co comt satt__nnggr.tttl____
In this example.
self.custom the interface for building nodes, through which the Optimizer can generate/mod-
ify its prompts.
test and self.sc ensemble are interfaces for using Operators (In this example, this workflow only use operators).
Edge in AFLOW represented through code, controlling the flow of all input/output vari- ables between Nodes and Operators to form complete workflow. Given this definition, the traditional concept of 'node having outgoing edges' does not apply to this formulation.
A.6 MCTS ALGORITHM OF AFLOW.
18

---

Published as a conference paper at ICLR 2025
Algorithm Detailed Explanation of the AFLOW Algorithm
Require: Initial Workflow Wo. Evaluator G, Dataset D, Number of rounds N, Operators O, Top
k, Early stopping rounds n2
Ensure: Optimal Workflow W
1: Initialize results < 0, experiences t- 0, N <- 20, k: +- 3, n f- 5
2: Dv, Dr RandomSplit( D. 0.2, 0.8) >> Split dataset: 20% for validation, 80% for training
3: scores + E G, Dv)
4: Dv SelectHiigh SadectlighhaaaaanccnnssaecsC///
scores, threshold)
Select instances
5: for round t- 1 to N do
6: round then
7: parent Wo
8: else
9:
parent Selec(Parent(resuutts)
10: end
11:
context + LowddConexxpparent.ccpoppp exppp/eereetttcceerreneeeeess Wround modiffiaiiii Optiiiieeezmicmtterttt O) for i f to 5 do
12:
13:
14:
score, cost + reesulss.appendrroundd score, cost)
Executor( W rounds E, Dv)
(orouto
16:
end for
augScore Calullec-werggSSonrrrrrteeeeedde experience CreneFxperrience(ppreent modi fication, avgSeore)
18:
axpriieneesppeenceepeeeene
20:
if augScore >> best then
21:
W* + Wround best.Score + augScore
22:
23: end
24: if The Top & Workflows remains unchanged in rounds then
Early stopping
return W 25: end if
26: end for
return W
28:
procedure SELECTPARENT(resutt) rested. SortDescendin(reesult,,,,, key=lambda r: r.scores)
key-eaambbtskkkkmmbblb t_p_krrssults t sorted k]
30:
31:
scores <- [resull.scores for resull in top resullss
32:
probabililiess Caleuuaaeiiceeeroaaaiiiiiseeeeee
33:
return Sump@fFroonCaggggraallpobbblllls
34: end procedure
35: procedure CALCUUAATMMXEDDORRRBBBBELEESECCC
36:
length(scores), < 0.4, a: 0.2, Smaz f- max(scores)
37:
Wi + cxp(a (s; 8max)) for [1,n]
38:
Pacore wi/ 2/1 Wj for [L,n]
Puniform &- 1/n for E [1, n]
40:
Pmized f- X Puni form + (1 X) Pscore
return Pinixed
42:
end procedure
43: procedure OPTIMIZER(COntezt, Operators)
44:
LLM Optimizer, generate new workflow and modification.
45:
return newW orkflow,
46:
end procedure
47:
procedure EXECUTOR(W, evaluator, datasel)
48:
L.LM Executor, execute workflow dataset and return score and cost return score, cost
49:
end procedure
19

---

Published as a conference paper at ICLR 2025
B
CASE STUDY
B.1 CASE STUDY OF AFLOW
Alpha Codium like workflow for MBPP
CODE TEE_GGENERATE.CC PROKET
Python function solve given problem. function zame matches the one specitied in the problem. Include necessary imports. Use clear variable names and add comments for slarity.
Problem:
[problen]
Function signature:
(entry_point:
Generate the complete function below:
FIX_CODE PROXPT PROXPP
provided solution failed pass tests. please analyze error and the code. Ensure the function name and signature remain unchanged. If recassory, add modify imports, correct logical excors, and inprove the implementation.
em: [inpu-]
Provide ccrrected
GENERATE __RMPPT ***
Given the problem and potential solution, generate additional test cases to thoreughly evaluate function. Include edge cases typical scenarios. Pornat the test cases as assert statements that can be directly added to sython test function.
Problem: [1nput]
Generate additions: cases as statemeats:
def calll (self, problem: str, entry_point: atr):
solutions =
(3): Generate solutions
solution - await se.ff.cstto_cood generate (problemmpzoblen)
entry_
instructioo-ppoop..com COW.:/CCCCCCOLLEEEAAAEEEEEEE
solutiins. (solution] (rrapponse" )
best best_solure ton_ssoltionn await self.sc_ sellfss__eneee ens.mnneeel.. (solus wonssocllttonnn probl probleem-robbln
Generate additional Lest
addaaioool_cest awaft self.custoom (Input=f ems (probl em) InSolution:
(best_solution, (tsppoonnnessseee
Combine original problem and additionel tests
mniensed_problee. (prooeelmm] \nknaddizional test 4 inn
''aapooss")) ]
test_result awalt self.test (problemmerhanceee problem,
solutionn-ees_eollu..e collution''resposee respponse' satry_pointreattry.ooitt
if tess_resule(rresils
recurn Lest_result ('solution'),
else:
test fails, try
solution
soed_solution awalt wfl..austom solff..stomminnoo=fff (Pppos [problem ArFal led solutions
wtt.:oolttiooi"eeppscee I]] AnErzoz: [test_result wtw.reesslltsssatee. solution']]",
insssuction-prompe wttppppp___sttee..__...o.. BB___CCO___CCPPP__S_C
rezurn fixed ("espprr""s"" self, wwm..ost_mnnn..oo.mmm..m.....sss paga__mnnger.tttal.ccm
FLOO demonstrates its ability to reduce human effort by evolving from an empty workflow to a solution highly similar manually designed workflows like Ridnik et al. (2024) in the code gener- ation scenario. This showcases AFLOw's capability to generate efficient workflows comparable to expert designs with minimal human intervention.
20

---

Published as a conference paper at ICLR 2025
The optimal workflow generated for MATH
SEPINN_ASSERR ANSWER_ PROKFT ***
Siven the mathematical proo en and the output fror the code execution, please provide Follow these guideliaes:
formatied
detailed
1. Begin with a clear statement of the problem.
Explain the approach and any formulas or concepts used.
Shew step-by-step calculatiens, using LaTeX notation for mathemetical expressions.
4. incerpret the code output and incorporaze it into your explanation.
Provide final
arclosed in \boxed]] LaTeX notation.
5. Enaure all mathematical notation 18 in LaTeX format.
Your response should be compreieneive, nathenatically rigorous, easy follow.
GENERATE SOEERATESSOLUTONLL COAUTTOO_FROMEE
Please sclve the given mathematical problem step by step. Follew these guidelines:
1. state the problcm ciearly.
Oueline
approach and any relevart formulas concepts.
3. Provide detailed calculations, using LaTeX notation for nathematical expressions.
Explain each step of your ressoning.
5. Present the final ansver enclosed in \boxed() 1arex notation. notation Your solution should be thorough, mathematically sound, and easy understand.
Fnaure mathemat'
format.
DETAILED DOTATtSD_GOLUTIONLEEE FROMPT .u.
Provide comprehensive, step by step soluzica to the given mathematical problem. Your response should include: 1. A clear restatement of the problem.
An explanation of the mathematioal concepts and thecrems involved.
detailed, loginal progressior of steps leading to the solation.
4. Clear expl anat ona for each step, ircluding the reasoning behind it. matheratical expressions 5. Visual aids oz diagrams if applicable (described in text].
equations IaleX format.
answez clearly marked and enclosed in \boxed!) LaTex notation.
8. A brief explanation of the fignifi of the result, if relevant.
Ensuze your solution is rigorous, easy to follow, and educational for someone learning
the concept.
asyae _call_aaeee_____________________ (self, problem:
Implementation the graph
Use Programmer to generate and execuse Pyshon code code_solution awalt self.programme (probl emaprob Use Castom to refine and Tormat the answer refined seiinie_sslution await custom (input=pzoblen g"AnCode output: Dat ANN_AANNR_RPPPPN
S poode_solution"" "outtntottuutuu
Generate detaf step-by-step solution vaing Cuaton
swalt self.cuatom input-problem, + pestruesionnpcomp__cco wtw.ee.ttttt...aa....e.. BAALLL___SLUTIOO_PPAAA SCIUTION__RRMPT)
detailed_solute
Generate multiple solutions using Custom solutions -
refined_eolusion "sespense"), sorailed_ooluttonn respansee]
Ior in (2):
solution swait custom (saputtpreblen,
inseouction-propp. www.mpp_pusten.cieeee www.ttt.GGEEAATSSSOLL....
(eoiution "rrsioonnss
scinsemble to select
solution
final_aolution await self.sc_engembe tatllutiooooooolltttoo problereprooee emmproblam) am) return solat_solutioni poolutoonnrrepponee "responee)) 11m. cost_manager..otaa
This optimal workflow generated for the MATH task showcases the model's ability to generate com- plex, task-specific solutions from task-agnostic initial settings. It combines programmatic solutions with various reasoning strategies, culminating in an ensemble selection process, and spontaneously formats the answer into the required form. This adaptation demonstrates the model's flexibility in tailoring workflows to different problem domains, while maintaining sophisticated moolem--solvigg structures.
21

---

Published as a conference paper at ICLR 2025
The optimal workflow generated for MBPP
CODE CEEEEAATE_ PROKFT ***
Generate a Python function to solve the gf problem, em. Ensure the runct fon name mae problem. Include necessazy importa. Use clear variable 4 names and add comments for clarity.
the specified
Problem:
(prcblem]
Function signaturer
jentry_point?
Generate complete below: ***
FTX_COPR_ PROMPT .**
The provided solution failed to pass the Lests. Flease analyze the error and fix the code.
funetion. name signature unchanged. necessacy, add modify importe, corract logical errors, and improve the implementation.
Problem: [Inpur]
Provide the corrected function below: *n*
asyne def __ fself, problem: str, entry_point: atr):
Tmppl ementatioon of the workflow
Custom operalor generate anything you
But when you want to get standard code, you should use custom_code_generatae operator.
solutions [
for range Generate solutions
solution = await ae. I.cuaton_code poode_generate perrobeeerprbeem,
instruction wttp:c/toorppoopccsttttooo
solations.appent (soiction( (ressoons"" '1)
best_solution await self. s.f_.sc_nemms (solu: lons=cclutorre.. probl pmmpoblem
test_result awatt self. (problem=prableee em, volutioon-best_colur..oo 'response + sn..yy_poin-eentty_ooiii
if test_result ('resull'" return seat_reault ("soluttinn 1 hmm..oor_rannggrrtora.
else:
15 the test fails, try to fix the solution
fixed_solution await self.custen (ingu==f"Froblee [problem) AnFailed solution: (best_ (o'luttiooo"respzeee AnErrer: [Lest_result (nssult['solitte solution']]",
LLI__COD__PRCOPPPPPP recurn siledd_olluioott ("rooppccee""" lif.fllm..oottmmaaaa totaager.tota_ccss
PROMPT)
The optimal workflow generated for the MBPP task simply combines operators with an ingenious FIX-CODE PROMPT, achieving the optimal workflow the iteration the fourteenth round. Al- though this workflow is simple, its score is extremely high and stable, demonstrating AFLOWWs potential to find the optimal cost-performance balance.
The optimal workflow generated for HotpotQA
FORMAT DORWAT_ANSWER. ANSWER_PROKET PROKFT
Given question
answer, the ansver concise, accurate, and directly addressing the question. snsure the ansver is clear, brief statement without addit: onal expl inatioon "easoning, the answer name, profession, short phrase, previde that information without Torming compleze sentence.
For axample:
person's name, just provide name. If the answer in profession, state only the profesaion. the answer short phrase, give only that phrase.
the
22

---

Published as a conference paper at ICLR 2025
no not Include any prefixes like "The answer 18" or "The profession 1a", Just provide the anawer itself.
asyne def _aal__ (self, problem: strl:
impemmmttaiion of the workflow
solutions
for in (3):
initial_:eeponse await aes....swee..eeeeraaa /ttpueraae///pttpproooio (input-problen)
thought_processs initisl_responsal ('thought"
initial_answe anttial_response "answe=")
solstions.aapenn ttattlal_anseer
ensemble_reaie
self.ac_ensemble itw.eoooooooiiesoosooooios best_answes ensemble_resulk ('response
await
Inpat=f (problem) \nBest anawar: [best_answerr]", instruct wttp:ocooooncmmpppueeeee PORKAT_ANSNER_
return
) self, 1i.. wtw.:/mmoootmmnggr.ttoo.. taaaager.totolccor
The optimal workflow generated for the HotpotQA task demonstrates the effectiveness of execution feedback. Apart from logical reasoning. another factor affecting QA problem scores is effective formatting. AFLOW can effectively identify the correct format and automatically perform formatting through learning from execution feedback, showcasing the efficacy of this design.
An ensemble structure that emerged in the GSM8K ablation experiment
SOLVE SONV__APPROCCII_EEE
Solve given problem step by step using a standard algebraic approach. solving, excract the final numerical answer and format it as follows:
Final Answer: (Insert the numerical value here]
only numerical 4 units or additions: text.
provided aftez "Pinal Anawers", without any
Problem:
Solve given probles sLep step using a viaual diagrammatic approach, 1C + applicable. If not applicable, use an alternstive method different from the solving, extract final answer
standard algebrais approach.
format it fol lows:
Final Answer: [Insert the numerical value here]
only numerical units or additions". text.
provided aftez "Pinal Answer:", without any
Problem:
SOLVE SOLVE_ APEROACH3 PROMPT
given problen step by step using estimetion approximation techniques, then refine the ansker for accuracy. After solving, extract the firal numerical answer and format 't follows:
Final Answer: [Insert the numerical value here]
Ensure thas only the numerical value is providad after "Final Answer:", ", without any units additions. text.
Problem:
COMPARE_ BELDDESEEEECCC PROMFT N*P
23

---

Published as a conference paper at ICLR 2025
Compare the three solut ions provided for the given math problem. Ana Analyze each solut fon [oz corceclness, compleleness, consistency with problem statement. Selecs
4 the most accurate and zeliable solution, or if all solutions ageee, confirm their consis.ency,
1f the selutions differ, explain the diffarences and justify your selection of the most If all solutions agrae, this conal stency.
Provide final answer following forma::
Final Answer:
[Insert the numerical value here]
Ensure that only the numerical volue is providad ofter "Final Ansxer:", without any unita additiona" text.
Problem:
asyne def _aal__ (self, problem:
Implementation worktlow
4 inatruc:ton=pp:omtt. (tt.:ccustommmuutppprreeeee
solution2 awalt se self.ovatom ( (Input-problem,
solution3 = awalt ael f.osstomoomommmommooooooooo..o.o (Input
instruet wwwp::eoooooppmpeeeeee
3OINE_APPRCACUI
(solutionn (M ('repposse' sumbined_solutionn " inSolutton
E"Solution 1: (salutionl 'zesponse") AnSolution 2:
final_ao in await self self www.ff..tooommmmmmommoo (inputaproble "\n" +
inat www.:cooooooompooeeeeee owm.oostttttooooooooooooooomoo.t COMPARE, AND SELECT_FROMPT)
return finsl_solute respanse") self, catn.cost_mmaaa wwwt_mamoggorrr.oooooooooooooo.e aaanoger.totalccar
In the ablation study, where predefined operators were deliberately removed, AFLOW surpris- ingly developed this simplified yet effective workflow. Most notably, it independently evolved an ensemble-like operator, mirroring key aspect of the optimal workflow. This emergence of multi- solution generation and selection process, despite reduced guidance, highlights AFLOW's inherent tendency towards robust problem-solving strategies. The spontaneous development of this ensem- ble approach in a constrained environment underscores AFLOW's ' ability to identify and implement effective techniques, even when operating with limited resources instructions. This unexpected convergence between the ablated and optimal workflows further demonstrates AFLOW's s capacity for developing sophisticated, human-lite problem-solving paradigms across different experimental conditions.
B.2 CASE STUDY OF ADAS
Iterative Knowledge-Enhancee Refinement workflow HotpotQA
asyne forwardiself, taskinfo!:
import asyncia
Step Initial ressoning by diverse expert agents inieisl_ins::oction 'Flease think step by step and solve the task based on your expertise."
expert_agen:s= =
LaMagen: 'answer'), 'Expert Agent', rolesrole, t1
for role in ('Reading Specialist', 'Logio Specialist', 'Generalist"]
sayne def con_expert (agert):
recurn await agent [taskinfo],,
initial await asyncio.gather [run_expect (agent) for agent in expert_agentsi]
combined infon (taskInfo) (info for result in initial_rencte for info in reault] Plattening instial_resulta
24

---

Published as a conference paper at ICLR 2025
Step 2: Tceratiive rement with excernal knowledge Artegration iex_itecatonns
for iteration in zange (max itsa_ittetaiions
Retrieve external knowledge
'Retzieve relevant information from knowledge base that can assist in refining the solution."
LiMAgantRase retrriveed__ee.. info),, 'Knowledga
+ Retrieval Agent")
resrieved_eesuits
retwleeje_eetrivval.... (combined_in.fes) rearieved retrieved_resulltsi0
$ Verify external knowledge
verification ooattooninstruation 'Verlfy relevancy accuracy the retrieved
4 information:
verification_sgen. LiskgentBase (I ['verifiid__inoo]] 'Verification Agenz")
verifiad = axzit verification ([taskinfo, vetrieved_infol, vers verified_info verifiee_reemits(00
Refinement using verified knowledge zef
'Review and refine the insights provided by other agents using verified external knoxledge."
derinement_agente preerined_thinkingg "retiine_anawer"" 'Retinement Agent', 11MAgentRase ([' solesrale, temperature=0.5) for vole in ('Reading Spectaliat', 'togle Spectalist', (Gereraliat")
combined [verified[
asyne def run run_ref serrrffinmeet (agent):
return awalt agent (oombined wnwd_iioos____ee www_innnossshheeee___eeo_ee www.h_eeeeireeatiooo.... sefinement_ rsttrcciion
setinement__esuile await asyncis.çathes refunnent (agent) agent
refi seefinement_agentee
iinbsenodiiffos..xxonlllll result]) Plattening rerinement_resulta.
rasult rafinement_reaite
Step Final synthes's agert integrates all refined traighta detal_deeisionnnttttetee answer."
- 'Synthesize all refined insights and provide final
final_ secelldeecisi__neeeee = LMMAgettsaee (I'shikkin 'ansver'), 'rinal Decision Agenz',
tempersture=0.33
dinal_shinkin final_answer await
agent (combined_infos,
return [insl_answer]
When designing workflows, ADAS incorporates all workflows from the search history into the prompt, distinguishing them only by their generation order and scores. However, the complex infor- mation embedded in the intricate structure of workflows, coupled with the accumulation of search iterations, the vast amount of information, and the continuously accumulating irrelevant information, poses significant challenges for LLM reasoning. ADAS stores experience from previous searches at directly storing all complete workflows. This approach causes the LLM designing workflows in ADAS to behave more like explorer of infinite possibilities within & rather than a designer seeking the optimal workflow.
the coarsest granularity -iircct
As shown in the code in Appendix B.2 the optimal workflow discovered by ADAS assigns diverse roles and multiple steps for refinement and summarization. However, for multi-hop reasoning tasks, the correct approach is to continuously reduce the problem scale to single-hop reasoning. Contrary to this, ADAS's optimal workflow actually increases the problem scale, ultimately attempting to the LLM's summarization ability to synthesize information, rather than gradually reducing the number hops based on the characteristics of multi-hop reasoning scenarios.
C OPTIMIZATION PROCESS OF AFLOW
Taking AFLOW's search process on the Math dataset as an example, we demonstrate how AFLOW iteratively improves workflows based on tree-structured experience and execution feedback.
25

---

Published as a conference paper at ICLR 2025
C.I
EXPERIENCE.
Processed Experience (formatted as tree sturcture)
"score": 0. 0.4873949579831933,
"modificatioz":
Programmer operator generate execute Python code for mathematical calculations, and use the Custem operator to and the final. answer.",
"score": 0.524367747891197
"failuze":
*9"
imodification *Add Scknsemble operator to generate multipla so uons and select the best one. help improve the accurazy of the final answer.",
"score": 0,433613453781112
1r
"2":
"score": 1.52436974788911597, "success":
"modification" ScEnsemble opezalor improve the reliability of the final answer by genersting multiple solutions and selecting one.", "score": 0.527731C924369947
"failure";
"6";
"modificatioz": *Add a Scknsemble operator to improve the relisbility answer by genarating multiple solutions salecting the most consistent one.", "score"; 0.5722689075530252
"modificatioz": ScEnsemble opezator improve the reliability of the final answer by genersting multiple solutions and selecting most conaistant one.",
"score"; 0.52436974789911977
Fr
"score": 0.5277310924366748,
"sucoesa":
"14": "modification": "Modify the Cusson operator to generate more detailed step-by-step solution, add
operator review and refine the firal answer. This will improve the clarity, accuracy, process.",
completeness "score": 0.53109243697478999
"modification" *add a new Custom opezator to generate a deteiled atep-by-step solution, modify the ScEnsemble operator to compare and select the best solution from multiple approaches.") ", "score": 0.5512605042016807
"modificstion": "Add Custom opezator generate detailed step-by-step solution, and modify the Scžnsemble operator to comрaгe and seleat tha best solution from miltiple approaches.",
"score": 0.537815126050401
"failure": '10"
"rodificatior", "Add a new Custom operator to generate a step-by-stap sttution, modify ty the SoEnsemble operator compare and sectct the best solution from multiple approaches.",
26

---

Published as a conference paper at ICLR 2025
"score"; 0.30420166800 6806122684
113"
ino":
"Modiry the cus.on: operator to generate more detailed step-by- solution, and add Custom operator to refine and format the final answer, This will improve the clarity and accuracy of solution process.",
"score"; 0.319327731092436
"modification" "Add a new Custom opezatoz to generate multiple solutions using different approsches, then Scänseable to select the best solution. This will increase the diversity of solutions potartially improve acouracy.", "score"; 0.0
Fr
*9":
"score": 0.5378151266554263,
"sucoesa": (),
"failuze":
"modification": *Add new Custom operator to generate detailed step-by-steg solution with explanations, and incorporate it into the ensemble prosess. This will previde comprehensive approach to solving mach problem em.", 1.5159662865546219
12":
"nodi cation*: *Add new Custom operator to generate multipla solution approaches, then use SeBnsemble to select the best solution. This will increase the diversity of solutions and potentially improve accuracy.", "score": 0.0
16
"modi cat *Add new Custom operator to generate multiple solution approsches, then use SeEnsemble to select the best ution. will increase verers ty of solutions and potentially improve accuracy.",
"score": 0.5210084033813455
b
"14":
1.5310022439777889,
"success": (),
"failure":
*15*:
"modi catfor"; a new Custom operator generate multiple soiutions, then use SoBaseable to select the best This modification sims to improve the sccuracy and consistency of the Cinal answez.",
"score": 0.5243637478991596
18"
"nodification" *Add a new Custom operator generate a more detailed step step solution, modify the ScEnsemble operator to compare and select the best solutien from multiple generated solutions." "score": 0.5176470586235293
F.
"5":
"score": 0.5512605042016607,,
"aucoess": (),
"failure":
*17":
"modificatior" *Add new Custom operator to generate multiple solutions using different approsches, and modify the ScBasemble operator to select the best solution frem larger pool of cancidates.",
0.0
19*;
27

---

Published as a conference paper at ICLR 2025
"nodif cation": *Add a new Custom operator to generate a simpli simplified solution, which solution to provide more comprehensive answer. This simplified solution will added to the 1iat of solutions for the ScBasemble operator to consider.".
fied aloagside existing detailed
"score"; 0.545577815126503
More optimization trajectories will be made available in an open-source repository upon publication.
Less Effective Optimization Steps:
Round 1 + Round 8 (Score decreased from 0.4873 to 0.4336); The key change was the removal of the Programmer operator, relying solely on Custom ScEnsemble, which lost the compu- tational precision provided by programmatic solutions, demonstrating that removing concrete computational capabilities significantly hurts performance.
Round - Round 16 (Score decreased from 0.5378 to 0.5210): The key change involved simplifying the solution generation process without maintaining the review step, which resulted in the loss of the quality control aspect of solution refinement, showing that solution quality checks are important for maintaining performance.
Successful Optimization Steps:
Round 1 + Round (Score improved from 0.4874 to 0.5244): The addition of the Programmer operator to generate executable Python code and use of the Custom operator to refine re- sults introduced concrete computational capabilities alongside human-like reasoning, creating a more robust solution approach and providing a foundation for both numerical accuracy and explanation quality.
Round Round (Score improved from 0.5244 0.5277): The introduction of the ScEnsemble operator to select from multiple solutions added solution diversity and reliabil- ity through ensemble selection, creating more robust system by considering multiple solution approaches.
Round 3 Round (Score improved from 0.52773 to 0.5513): The key change was the ad- dition of detailed step-by-step solution generation, which enhanced solution clarity and com- prehensiveness, ultimately improving the pedagogical value of solutions while maintaining accuracy.
The tree-structured optimization process helped guide LLM workflow improvements in sev- eral ways:
The tree structure allowed exploration of multiple optimization directions simultaneously, enabling the development of successful paths while pruning less successful branches, which facilitated the efficient discovery of effective combinations of operators.
Path Discovery:
Incremental Improvement: Each node in the tree represents specific workflow configura- tion, and the success/failure feedback at each step helped identify which modifications were beneficial, with the scoring system providing quantitative guidance for optimization decisions.
Pattern Recognition:
The tree structure made it easier identify patterns in successful versus unsuccessful modifications, revealing common elements in high-scoring branches. such as the combination of Programmer Custom ScEnsemble, which informed future optimization decisions.
Error Recovery: When modification led to decreased performance, the tree structure facil- itated easy backtracking, allowing exploration of alternative optimization paths from previous successful states and preventing the process from getting stuck in local optima.
28

---

Published as a conference paper at ICLR 2025
C.2 EXECUTION FEEDBACK.
In the process of optimizing the overall response generation, we implemented a concise and task- agnostic prompt: "Below the logs of some results with the aforementioned Graph that per- formed well but encountered errors, which can be used as references for optimization: {log}". This approach enabled execution feedback assist LLM in workflow optimization. The following ex- amples demonstrate how adjustments to the prompts enhanced the quality, consistency, and clarity of the answers, with particular emphasis on answer formatting as key illustration.
Optimization Steps:
adds "Provide final answer, enclosed in \boxed() LaTeX notation", resulting the ability to identify patterns in the scoring feedback without knowing the specific rules of the scoring function.
Round 1 + Round 2: SEPINELANSWER..ROEE
Round + Round 8: While REFINELANSWER PROMPT adds "Provide clear, concise final answer" to shift focus towards presenting answers more concisely, Round 8 lacks the strict LaTeX formatting constraints present in Round 2. This leads to Round 8 scoring lower than the baseline Round However, the emphasis high-scoring Round 19, indicating that this remains a valid optimization direction.
concise answers also appears in the
introduces two new requirements: "If there multiple possible answers, list all of them separated by commas within the \boxed{]" )r and "Simplify expressions where possible without losing accuracy". While the former aims standardize the format for multiple solutions, including this general instruc- tion may interfere with the solution process. Statistical analysis shows that Round 13 contains 55 instances of comma-separated answers, notably higher than better-performing rounds such as Round (43 instances) and Round (42 instances).
Round + Round 13: REFINELANSWER PROMPT PROMPP
29

---

Published as a conference paper at ICLR 2025
D PARETO FRONT: DETAILED COST-PERFORMANCE DATA
Detailed cost-performance data for HumanEval. Executing AFL.ww (GPT-40-mini) with deepseek achieves parity with GPT-4o 10 at 4.55% of the cost. Executing AFL.Ow (deepseek) with deepseek
and AFPoo(GPT-tt-minn) with GPT-4o-mini outperform GPT-4o IO at 5.92% and 8.05% of the cost, respectively.
Model
Method
Score (%)
Cost (S)
gpt-4o-mini
I0
0,8702
0.0223
gpt-4o-mini
CoT
0.8860
0.0277
gpt-4o-mini
CoT SC
0.9160
0.1794
gpt-4o-mini
MedPrompt
0.9160
0.2200
gpt-4o-mini
LLM Debate
0.8930
0.2278
gpt-4o-mini
Self Refine
0.8780
0.1232
gpt-4o-mini
AFLOW (gpt-4o-mini)
0.9470
0.0513
gpt-4o-mini
AFLOW (deepseek)
0.9084
0.0669
deepseek
10
0,8860
0.0127
deepseek
CoT
0.8930
0.0180
deepseek
CoT SC
0.8860
0.1168
deepseek
MedPrompt
0.8860
0.1433
deepseek
LLM Debate
0.8930
0.1484
deepseek
Self Refine
0.9000
0.0802
deepseek
FFL. (gpt-4o-mini)
0.9390
0.0291
deepseek
AFLOw (deepseek)
0.9466
0.0377
gpt-4o
I0
0,9389
0,6371
gpt-4o
CoT
0.9310
0.7772
gpt-4o
CoT SC
0.9470
5,0345
gpt-4o
MedPrompt
0.9390
6.1756
gpt-4o
LLM Debate
0,9470
6.3952
gpt-4o
Self Refine
0.9160
3,4589
gpt-4o
AFLOW (gpt-4o-mini)
0.9620
1.0111
gpt-4o
FFLOW (deepseek)
0.9542
1.6600
claude-3.5-sonnet
I0
0,9084
0.6987
CoT
0.9240
0.6412
Mlaude-3.5-sonne
CoT SC
0.9390
4.1534
Slaude-3.5-sonee
MedPrompt
0.9160
5.0949
claude-3.5-somee
LLM Debate
0,9080
5.2761
-laude33.5-somnee
Self Refine
0.8930
2,8536
Saaue-33.5-oome
AFLOW (gpt-4o-mini)
0.9540
1.1612
claude-3.55 3.5nonett
AFLOW (deepseek)
0.9466
1.3252
E DISCUSSION ON WORKFLOW, AGENTIC WORKFLOW, MULTIAGENT SYSTEMS
In the field of LLM applications, concepts such as Workflow, Agentic Workflow, and MultiAgent Systems are frequently referenced, each with distinct emphases. The notion of Workflow in (Qiao) et al. 2025) primarily focuses on the structured decomposition of real-world tasks, such gen- erating execution sequences for specific tasks in digital environments. (Qiao et al. 2025) work represents significant advancement in workflow benchmarking through its comprehensive multi- faceted approach spanning function calls, problem-solving, embodied planning, and open- gpennggoodded planning scenarios.
30

---

Published as a conference paper at ICLR 2025
Despite terminological differences, the multi-agent systems described in Zhang al. (2024a b); Zhuge et al. (0224) are functionally equivalent to agentic workflows. Both consist of series of LLM-invoking nodes that lack autonomous capability and execute in predetermined logical quence. Though the terminology varies, these approaches essentially describe identical technical architectures.
Meanwhile, Niu et al. (2025) explores methods for dynamically adjusting workflows during task execution, enhancing the system's ability address complex problems by modifying workflows in real-time based on execution feedback.
AFLOW proposed in this paper focuses the automated generation and optimization of workflows, aiming to reduce dependence on manually designed initial workflows while improving the scala- bility and generalizability of agentic workflows. This approach complements existing research and collectively advances the field of LLM applications.
F DISCUSSION ON OPEN-ENDED TASKS
F.1 ADAPTING AFLOW FOR OPEN-ENDED TASKS
While we emphasized AFLOW's powerful capabilities solving reasoning tasks with numerical feedback in the main body, there many tasks in the real world without numerical feedback and non-reasoning requirements (Dai et al. (2024). These scenarios lack numerical feedback because for open-ended tasks, performance is typically evaluated by humans who propose the tasks, making difficult to evaluate using fixed and quantitative criteria. By introducing LLM as a judge (Zheng al. 2023), this challenge can be addressed to some extent, as the LLM-as-judee evaluation prompt can effectively incorporate human preferences to adaptively evaluate open-ended tasks. In this sec- tion, we discuss how leverage LLM judge while maintaining AFLOW's core architecture to search for agentic workflows for these types of tasks.
To extend AFLOW's capabilities to open-ended tasks, we modified the original workflow optimize prompt by removing reasoning-specd instructions. The new prompt is as follows:
Workflow optimize prompt for open-ended tasks
You constructing a graph cocresponding prempts jeintly (type) problams. Referring to the provided graph and prompts, which form basic example Itypa] solution approach, please reconstruct promota. Include
opcimisg them. You single
may add, modlcy, delete nodes, pazameters, modification enclosed in XML tags in your reply. Easure they are complete and correct aveid runtime failures. logical end control flow (such IF-ELSE end lcops) to schieve more advanced code irppresentation.aaee current graph are
sll prompts not include any need to generate are limited to those usad in
required by additional prompts. The prompts "prorpt_euste wwstem.tttmmo..o xxx, Other methoda al have It-in prompts prohibited from being generaled, Generate only the prompts needed the graph and renove any unused prempts from pprompt_custoo The generated proupts must not contain any pizeeholders. Considering information loss, cemplex graphs may yield better results, but insufficient information transmission might omit the solution. nacessary context
"prompt_custee
are
Ensure
throughout the process.
For tasks without numerical feedback, we propose evaluation prompt based on LLM Judge. We utilize GPT-4o as our evaluation model, implementing the following prompt for scoring open- ended tasks:
Evaluation prompt
2VAL_ PROMPT You are an expert evalustor tasked with scoring responses to open-ended questions. You will provided with:
1. The originsl quest on/prompt golden (seference)
(if available;
31

---

Published as a conference paper at ICLR 2025
3. candidace reaponse
lowling dimenatons, each scored from - 1 5. Khen nо reference answer is provided, your expert Judgment to assess the expected quality level for the given task type:
please evaluate candicate resperse
1. Contert Relevance (1-5):
Perfectly addresses
aspects
prompt
4: Addresses most key aspects with minor amiasions
3: Addresses main points but misses some Important elements
Only partially relevant to the prompt
1: Largely irrelevant or off-topic
2. Content Quality (1-5):
Exceptional depth, insight, originality
4: strong maalysts/oreatioii with good supporting ing detailla
3: Adequate development with some supporting elementa
Superficial sreatment with minimal development
1: Poor quality with major floxs in reasoning/ exxenniingxxeetiion
3. Coherence and Structure (1-51:
Excellent orgenization with seamless
4: Clear structure with ainor tranaition lasues
3: Generally organized but some awkward transitions
Poesly organized with frequent disconnects
1: Chaotic or illogiesl structure
4. Reference Compacleoa (1 5):
Matches exceeds expested quality this type
Slightly below ideal but strong performance
3: Moderately below ideal but acceptable
significently below expected quality
1: Far below anceptable quality atandards
Please provide:
each dimensicn. (1-5)
2. Brief jussification for each
sentences)
3. Total score (sum of the four dimensiens, maximum 20 peints)
4. Summary feedback (2-3 sentences)
Tormal your response as: Content Relevence: [score] points Justification: [brisš explenation]
Content Quality: [score] points
diatification: [briaf explanation]
Coherence: [score] points Jnstification: [brieé explanation]
Reference Comparison: [score] points Justification: [briaš explenation]
Sunmacy Feedback:
seatences highlighting key strengths areas improvement]]
<ecore> [sum dimenaionn ons) (pare number) </score>
This evaluation framework replaces the original evaluation function in AFLOW's executing evalu- ation stage, enabling assessment of open-ended tasks while maintaining consistent evaluation stan- dards.
F.2 CASE STUDY
We demonstrate the effectiveness of our adapted AFLOW through two open-ended task scenarios: long-form novel generation and academic idea generation. Both cases lack numerical feedback and utilize adapted methodology. To evaluate AFLOW's performance on these tasks, we first hired three human annotators at S10/hour to score and rank the results generated during AFLOw's iteration compared results generated directly by LLM with those generated by AFLOW to observe the differences. Due to space limitations, we provide the complete results of the two task generations in the supplementary materials.
process. Additionally,
32

---

dateiled character profiles for Published as a conference paper at ICLR 2025
F.2.1 LONG-FORM NOVEL GENERATION
In this case study, employed Claude-3.5-somee the execution and optimization model and GPT-4o as the evaluation model due to its strong language capabilities. The system was tested with single question without reference answers:
Novel data
"question" "Create novel-length narrative, exactly 20, 20, words long (please ensure this spaci fic length is metl, exploring world where time's 'a flow varies for each peraon based on their deepest regrets, The story should meticulously examine how emotionsl burdens affect temporal perception, with careful attention maintaining the required length. socus on developing several characters whose distinctly differant experiances time's passage.", requitem"tt" a long nevel wita emphaala on aubstant length, logical chapters,
unique regreta
interconnection Manswer"; "No reference answer"
language style.",
The optimized workflow obtained after eight iterations of AFLOW was:
Novel generation workflow
OUILINE ..*
Create a detailed outline for a novel based the given requi rements. include:
1. 1ist main charaeters
their deepest regrets how affacta
4 perception of time
2. chapter by chapter breakdown of the plot, ensuring logical interconnection.
3. key themes and motifs be explored throughous the nevel
4. rough word count estimate for each chapter to aim for the required total length
Provida this outline in acructured format.
CHARACTER PROPILE
PROPILE_ PROMPT
sased the given requirements and estline,
< each main charscter. For each character, inal include:
Name, age,
physical description
2. Rackground and personal history
3. Their deepest regret and how affecta their perception of time
4. Fersonalicy traits, metivations,
goals
5. kelazionships with other characters
6. Character throughout the novel
Provide profiles structured
FROMTE n**
write single chapter of the novel based on the given requirements, previded outline, character profiles. Fellow these guidelines:
1. Adhere to the chapte: structure from the outline
Maintein logical intercoanectione
previous
chapters
3. Use retined language and vivid descriptions
4. Develop cheracters' exes based en their zegrets and time perception
5. Incorporaze the thenes and motifs
6. Aim for opproximataly 10,000 words per chapter
Write the complete chapter without stopping or summacizing, Do not include any neta-ccmeentary or explenations outside chapter text itself.
asyne def _call_ problem: strl:
Trolementation of workflow
self.custom/ (Elf.
instructioasprroe _esttte_sooooooeeo [NE__NEEP_]]]______
Character_profilee axait self.cust
#"\aOutline:
joutline ['response']]]
ruat cnuproopt_oosee musstom.CHARACTRR CHARACTER PROFT1R, PROFT1R_PROMWT PROMPT)
33

---

Published as a conference paper at ICLR 2025
chapter_coune = 10 Adjust as needed
in range (chapter_pouunt]] countt
chaple: await self,custont
input-prcblen + f"Mnoutline: [outline( sespponse } AnCharacter
Profiles: [ciaractor_profilee]] r'repoonse \nChapter Number: + [1+1]",
novel P'AnnnChapter: Nitncchheee reaponse
']
return novel, self. wwtt.oootmmmannn maot__.aaage..otal....
For comparison, below is the baseline response generated using only the Claude-3.5 -oonnnett,, without any AFLOW optimization:
LLM-Generated Novel
apologize, but I's. not able Lo generate novel length narralive of exactly 20, 000 this fornat. Writing full extrenely lengthy and complex task that goes well beyond the scope of what I can produce aingle response.
specific length would
Additionally, maintaining an exsct word count while crefting a cohesive narrative with for human
developed charsctars and themes would be extremely challenging, + authors. Novals typically go through many drafta and ravisions to reach their final
Rowever, can offer some suggestions for approaching this interesting concept:
1. Start by outlining the key characters and their deepest regrets. Consider how each regret night impact thair perception time.
Devalop
plot structure temporal expariences intersect and affect each other.
you explore
different
3. Consider the werldbuilding sspects how does society function when everyone experiences time differently? what are the implicatiens for relationships, work, st&?
segin writing scenas chapters to explare ideas, without worrying about exact word count initially.
5. you write, track of word and adjust pacing/detail needed to work < towards your serget length.
6. Flan for multiple drafts and revisions to refine the story and adjust length.
If you'd like, could help brainstorm points within this cencept. But for full novel- -ength werk, especislly one with such precise length requirements, you'll likely need to undertake the writing process yourself over on extended period of time.
apecific Ideas for characters or plot
Here the simplified response generated using the AFLow-optimized workflow:
Hlow-Generrated Novel
Chapter 1: The Ticking of Regret
Zame Sinclsir's fingers hovered over the keyboard, the cursor blinking accusingly on rhythmic ticking louder with each passing second, constant reminder of the impending deadline locming over her. She glanced at the time display on her cemputer screen poasible that only three minutes had passed since she last
document
clock
2:37 PM.
chacked?
34

---

Published as a conference paper at ICLR 2025
Chapter 2: The Scientific Discovery
Dr. Bvelyn laappr stood the podiutt, her silver hair gleaming under the harsh auditorium lights. Bez piereing blue eyes scanned the crowd scientista, doctors, end journalists who had gathered for her greuncbreoking presentation. She took deep breath, fealing the weight of decades of research pressing down on her shoulders.
Chapter 10: New Understanding of Time
The grand auditoriur of the National Science Center buzzed with anticipation. Journaliats, acientists, and curious
the public filled every seat, thair excited whi whispers creating a low hum that reverberated through the space, At the center of all, Dr, Evelyn Narpez atood behind the podium, ser silver hair breath, her fingers lightly
gleaming under the bright lights. She tock tracing the adges of her notes.
Dr. faappe cleared her throat, and the room fell ailent. "Ladles and gent guesta," began, stand before to preaent the calmination of years of research into what once thought was immutable: the nature of time itself."
colleagues,
steady. "Today,
And in + that nosent, as they moved forward Logether, tine flewed perfectly, neither Leo nor slow. simply was, they were fully, completely present within
brightly overnead,
reminder
present
it. The future beckoned, full of promise possibility, and they aLepped Lowards it with open hearts and minds, ready for whatever adventures time night bring.
The optimized workflow (obtained after eight iterations) produced 27,000-word novel, demon- strating significant improvements both quality and efficiency compared to baseline responses. The cost and performance metrics across iterations show that AFLOW can achieve high-quality long-form content generation with minimal resource expenditure.
Round
Avg LLM Score
Avg Human Score
LLM Score Rank
Human Score Rank
Cost (S)
12
8
8
0.005
2
16
10.3
6
7
0.291
18
2
2
0.323
17
13
3
4
0.360
5
16
12.3
6
5
0.370
6
17
12.3
3
5
0.365
17
3
2
0.336
20
19.3
1
0.863
Table A1: Novel generation Workflow Performance Comparison between LLM and Human Scores Across Different Rounds, with Rankings and Total Costs per Iteration
F.2.2 ACADEMIC IDEA GENERATION
For this case study, we specifically chose GPT-4o-mini as the execution model and Claude-3.5- sonnet as the optimization model and GPT-4o as the evaluation model to better demonstrate AFLOW's optimization capabilities. While the system was tested with 10 different questions, we present one representative example here:
Idea data
"quest or ven the current research landscape in Envi ronmental Anthropo logy, propose
through. logical analysis*,
35

---

Published as a conference paper at ICLR 2025
"requirement"; "Thzough rigozcus anslysis, propose single research adareasad significant chellenges demonstratis feasibility with current techrology. NOTE: only concrete idea should provi focus on deve oping single, well reasoned proposal rathes than multiple options.", "answer"; "No reference answer"
The optimized workflow obtained after six iterations of AFLOW was:
Idea generation workflow
GENERATE_ IDEA
Given
landscape specified propose and fessible zesearch idea through logical analysis. Focus on developing a single, well-reasoned proposal
-
addresses signifieent challenges demonstrates feasibility with current technology.
Your reepense should be concise, providing the title and brief deseription of t the research idea in no more than sentences.
.*.
ERIORITIZE
Analyza the given research ideas and prioritize the most promising one based on its 4 potential impact and feasibility. Consider the following criteria:
1. Novelty and originality
Potential impact on field
3. yeasibility with current technology
4. Alignment
current research trends
Provide a brief explanation 12 sentences) for your selection, highlighting its strengths in relation the shove criteria.
ELASCRTEE IDEA
aborate the prioritized research Idea, Provide comprehenstvo ana laaa inoluding:
1. Research objective
2. Methodelogy
Expected outcomes
4. Fotantial chal lengas
Insure your response is well structured, logically sound, and demonstrates the b feasibility of the proposed research with current technology.
EVALUATE RESBARCH n*.
Rvaluate the elaborated research proposal. Consider the foll loxingg aspects:
1. Novelty and originslity
2. Feasibility with current technology
Potantial Irpact the field
4. Clarity and coherence of the proposal
Provide concise evaluation hignlighting strengths and areas for improvement.
REF INE PROPOSAL
Saaed elaborated and evaluation, refine
proposal, Address + any weaknesses ideatified in the evaluation and enhance the proposal 's strengiha. refined proposal:
1. Clearly scates the research objective
Outlines feasible methodelogy
3. Describea expacted outcomes and their significance
Addresses potential. challenges nitigation strategies
refined propesal well-struetured suitable submission.
asyno daf call 1 (self, problem em: str):
36

---

Published as a conference paper at ICLR 2025
Implementation
Ideas
for in r3)g 4 Generale 3 ideas
idea awsit self.cust (input-problemm, instructionnprrmmt..c cttipppt_sssmmmteeemm wtt.em...CEEeRD0000
ideas.append dea response" ']]
best_idea awatt
(solutiongeidens) problemmprableem)
prioritized ides await self, custaniingu- (ideas) \nBest idea: b (best_ ideal response 1)",, resppooe
pepooratd__deee = await se self.cuatoo (irput-probeen I"NPrroorit idea: (prioritized peepppppsse response instruct Lom-propt wetprrproottttttttte BLABORATE_
evaluation awall self.custon (ipptttelaboeeeeeeee idea "response"]
instructioorrremsssssssoom BOALLAATERRREAARR..
final uniution awaft custtoo nepttellll aoputellabbn aborsted deaa reapense"))
[" ("MaBvaluation: (evaluat ion! "rrespnsse]]
instruetioorppomss_ccae REPINE PROPOSAL)
seturn finel_ solutioon "rrptioonrrssseeee" self, self.1lm.ce wtw....m...ettttm motgeee wew._mmmnger..ttaa..com Lotel_cost
Due space constraints, provide the original LLM and AFL.o-geenertted ideas in the sup- plementary materials. The results below demonstrate substantial improvements in idea generation quality and specificity compared baseline responses, with consistent performance gains achieved after just six iterations.
Round
Avg LLM Score
Avg Human Score
LLM Score Rank
Human Score Rank
Cost (S)
18.4
15.6
6
7
0.004
2
18.1
15.7
7
6
0.220
19.2
17.3
2
4
0.234
17.6
9.7
8
8
0.223
5
19.2
17.7
2
2
0.234
6
19.7
19.1
1
1
0.235
19.0
17.6
3
0.248
8
18.8
16.5
5
5
0.253
Table A2: Idea generation Workflow Performance Comparison between LLM and Human Scores Across Different Rounds, with Rankings and Total Costs per Iteration
G DISCUSSION ON THEORETICAL PROPERTIES OF AFL.W
G.1 SEARCII SPACE COMPLETENESS
AFLow's search space completeness relies on two key properties:
Code-represented edge structure express all valid node relationships
LLM expansion generates valid workflow modifications with non-zero probability
These properties ensure AFLOW can traverse from any initial workflow to any point in the search space, avoiding local optima.
G.2 CONVERGENCE PROPERTIES
FFLOw achieves optimal performance within finite iterations under three conditions:
Bounded evaluation function G(W, T)
Valid workflows maintained by code-represenned edge structure
37

---

Published as a conference paper at ICLR 2025
Non-zero probability of LLM generating improvements
While the convergence sequence may not be strictly monotonic, MCTS properties and soft mixed probability selection balance exploration and exploitation achieve convergence.
G.3 SEARCH EFFICIENCY
AFLOW enhances search efficiency through three key mechanisms. First, Operators increase the probability of generating improvements in each iteration by providing predefined node combina- tions that encode successful patterns. Second, Tree-Structured Experience enables efficient reuse of successful modifications while avoiding repeated failures through systematic path tracking. Finally, Execution Feedback provides direct performance measurements that guide the optimization process, helping AFLOw identify and prioritize promising directions in the search space.
38


Reviews:
[{"review_id": "3jDaPy0Fv3", "reviewer": "Reviewer_ZL3t", "summary": "The paper introduces Aflow, a new automatic workflow optimization framework based on MCTS and code-represented workflows. It models the workflow as a sequence of LLM-invoking nodes, where nodes represent LLM action and edges represent logic, dependencies and flows between the actions. The experiment results on 6 benchmarks show preliminary effectiveness over SOTA, and that AFlow can enable smaller LLMs to outperform larger models, offering better cost-performance efficiency, with significant implications for real-world applications.", "questions": "The authors chose 6 agentic workflow benchmarks for the experiments. Are there more rationale and explanation behind how those benchmarks are chosen to best represent the agent workflow optimization capability?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Overall, this paper is clear, well-motivated and provides a new framework for automatic workflow optimization, which has significant potential impact on agent design and workflow optimization for the broader machine learning community. It proposes a novel, original approach to model the workflow as a sequence of LLM-invoking nodes in a graph structure, with prompts, operators, and code-represented edges in the search space. By leveraging MCTS, the paper reaches SOTA performance on major workflow benchmarks and shows the potential of enabling smaller, cheaper models reaching similar performances as large models. The documentations of the experiment setup, code representation, case studies and results are clear and technically sound, and this paper can provide great inspiration for other researchers in the domain of agent workflow optmization.", "weaknesses": "The paper could benefit from discussions with regards to the following points:\n1. To reduce the search space, the paper focuses on custom prompts, operators and code-represented edges by fixing parameters such as model choice, temperature and output format - which is a sound choice. Could there be more discussion on the potential effect of these parameters on model performance?\n2. The authors mention some of the parameters used in MCTS in the appendix (e.g. $\\lambda = 0.4$ used to balance exploration vs. exploitation), but not in the main paper. It would be helpful to include key parameter values and brief discussions about the choices. \n3. Similarly, a quick discussion around why models are chosen for specific parts (executor vs. optimizer) would be helpful for context as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "svtodtkdwj", "reviewer": "Reviewer_ciCS", "summary": "This paper introduces AFLOW, a framework that automates the generation and optimization of agentic workflows for Large Language Models (LLMs) by reformulating workflow optimization as a Monte Carlo Tree Search (MCTS) problem. The workflow structure consists of LLM-invoking nodes connected by edges, represented in code. The system leverages tree-structured experience and execution feedback to refine workflows iteratively. AFLOW is evaluated across six benchmark datasets, demonstrating performance improvements over manual and automated baseline approaches and enabling smaller models to achieve competitive results at lower cost. This novel approach aims to reduce human intervention in workflow design, providing a scalable and efficient framework for workflow optimization in structured tasks.", "questions": "Scope and Generalizability:\nHow do you envision AFLOW handling tasks that do not have well-defined success metrics, such as creative writing or exploratory research? Are there specific adaptations you would recommend for such tasks?\nThe paper primarily discusses AFLOW’s applicability to benchmark tasks with structured goals. Could you provide examples or suggestions for how AFLOW might be applied to open-ended, real-world applications beyond these benchmarks?\nCould you clarify how AFLOW maintains context and coherence in longer workflows or workflows requiring memory across stages? Is there a mechanism in place to support long-term contextual awareness?\n\nPrompt Adaptation and Handling for Broader Tasks:\nGiven that AFLOW’s prompt templates seem tailored to structured problem-solving tasks, what modifications would be necessary to adapt these prompts to open-ended or creative tasks?\nAre there mechanisms for adapting or evolving prompts dynamically based on task progress? How does AFLOW approach prompt optimization in situations where task objectives may evolve or remain undefined?\nCould you provide examples of prompt templates used in AFLOW, and explain how execution feedback specifically informs prompt revisions, if at all?\n\nImplementation Details for Reproducibility:\nCould you provide a clearer example of how workflows evolve iteratively during AFLOW’s optimization process? Specifically, how does feedback inform changes in node structure, prompt design, or operator selection?\nHow does AFLOW ensure consistency and reliability across different task types, especially when tasks involve diverse workflows or require different prompts and operators?\n\nComparison with Other Approaches:\nHow does AFLOW’s MCTS-based approach compare to DSPy’s instruction optimization or the Tree of Thoughts approach in handling complex, multi-stage tasks? Are there specific advantages or limitations relative to these frameworks?\nTree of Thoughts is briefly mentioned as related work but not fully explored. Could you elaborate on how AFLOW’s workflow optimization extends or diverges from the principles underlying Tree of Thoughts?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Novel Approach: AFLOW’s integration of MCTS with code-represented workflows introduces a new direction in automating LLM workflows. This reduces the reliance on manual design and allows efficient workflow discovery and optimization.\n\nComprehensive Problem Formulation: The paper formalizes workflow optimization with a general mathematical framework, effectively unifying prior approaches and broadening the potential for future applications.\n\nDetailed Critique of Prior Work: The authors present an insightful analysis of existing methods, identifying the limitations of prior frameworks like ADAS in handling information accumulation and search efficiency. This sets a strong foundation for AFLOW’s proposed contributions.\n\nEmpirical Validation: AFLOW is rigorously evaluated across diverse benchmark tasks, with quantitative comparisons to multiple baselines. Ablation studies further illustrate the impact of different operators, and cost analysis demonstrates AFLOW’s efficiency.", "weaknesses": "Limited Scope and Generalizability: The paper primarily demonstrates AFLOW on benchmark tasks with clear success metrics, which raises questions about its applicability to more open-ended tasks, such as document generation or creative exploration. There is limited discussion on how the standardized prompts used in AFLOW would generalize to tasks without clear success criteria. The current prompts seem tailored to test-taking scenarios and may lack the flexibility required for tasks that demand creative or exploratory outputs. A clearer strategy for adapting or evolving these prompts to support open-ended workflows would strengthen the paper’s position on generalizability.\n\nImplementation Details for Reproducibility: While the automated workflow design is a strength, the paper lacks sufficient details on prompt handling, tool calling, and how workflows evolve with execution feedback. Specific examples of workflow changes during optimization and consistency maintenance across components would improve reproducibility.\n\nLimited Comparative Analysis: Although the paper provides a robust critique of ADAS, the treatment of DSPy is brief, and Tree of Thoughts is only briefly mentioned despite its relevance. A more detailed comparison with these approaches would clarify AFLOW’s unique contributions and limitations.\n\nTheoretical Analysis: The paper lacks a theoretical analysis on the convergence of the MCTS optimization process, completeness of the search space, and performance bounds, which are important for understanding the robustness and scalability of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dFD1Wu9Pvb", "reviewer": "Reviewer_hYzq", "summary": "This paper proposes a novel approach to agentic workflow that leverages MCTS to search over sequences of actions that are jointly able to achieve automation of various tasks.", "questions": "- Page 3: “We define an agentic workflow W as a sequence of LLM-invoking nodes”. Does this definition of a sequence really make sense? It appears that it can also be some sort of graph structure that allows for loops, decision points (branching), and parallel relations. Terminology-wise, wouldn’t that graph be the workflow, and a sequence of LLM-invocations would then be a particular instantiation or execution of that workflow? This relates also to Page 4: “the goal of workflow optimization is to discover a workflow W …”, given the definition of W as merely a sequence, is this what is intended here?\n- Page 4: “The edges E can represent various structures, such as: Graph: A flexible structure representing hierarchical, sequential, or parallel relationships between nodes, allowing for complex branching workflows”. The graph that is shown in Figure 2 is merely a DAG. How can a DAG represent both sequential and parallel relationships in one graph? Typically richer graph languages are required to model rich types of branching workflows, such as workflow nets [1] or BPMN. Is the graph structure that is intended here a DAG and thereby unable to represent parallel relationships, or richer than a DAG and thereby able to represent parallel relationships and other complex workflow patterns [3]? Note that efficient search spaces for have been proposed for some of those workflow representations (e.g., [4]).\n- Page 4, about Code: “offering the most precise control over workflow execution”. Why would this be the most precise? Various variants of Petri nets and other graphical representations that are commonly used in the literature to define workflows are turing complete.\n- Page 6: Equation 2: “W^* = AFlow(S_{AFlow}, G, T)“. It is unclear what method AFlow here refers to. My thought initially was that it refers to Algorithm 1 “Algorithm of AFlow”. But that algorithm is defined to take 5 arguments (defined in the algorithms require), while equation 2 passes three arguments. Without clarity on this, the procedure is not well specified.\n- Page 6: “AFlow can perform searches with an empty Operator Set”. If the set of operators is empty, then what is the set of mutations to the workflow that are considered? I didn’t find a definition of this.\n- Page 7: “Our algorithm forms the initial node by evaluating an empty workflow on the validation set, which is distinct from the root node in MCTS”. I did not understand this. Why is the initial node of the workflow not the root node in MCTS?\n- Page 7 about expansion: not clear if the LLM generates new workflow only (i.e., new code), or whether it also generates/modifies the contents of the nodes (e.g., the prompt/model/output format).\n\n[1] van der Aalst, W. M.P. (1997). Verification of workflow nets. In: International Conference on Application and Theory of Petri nets (pp. 407-426). Springer.\n\n[2] Dijkman, R. M., Dumas, M., & Ouyang, C. (2008). Semantics and analysis of business process models in BPMN. Information and Software technology, 50(12), 1281-1294.\n\n[3] van der Aalst, W. M.P., Ter Hofstede, A. H., Kiepuszewski, B., & Barros, A. P. (2003). Workflow patterns. Distributed and parallel databases, 14, 5-51.\n\n[4] Esparza, J.: Synthesis rules for Petri nets, and how they lead to new results. In: International Conference on Concurrency Theory. pp. 182–198. Springer (1990)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "The methodology is interesting, and the experimental setup is convincing in showing that AFLow enables smaller models to achieve superior performance to larger models. This lifts the cost/accuracy Pareto front. Given these results I am appreciative of this work.", "weaknesses": "While the methodology and results are nice, I do however believe that the presentation of the paper requires improvement. The description of the AFLow methodology lacks precision and can at times even be called handwavy (details below), which makes the paper hard to read. In case authors are able to address those issues I may be willing to increase my score. Some concrete examples (more examples in the questions):\n- There is a tree structure involved in the MCTS search process, but there is also a graph and nodes involved in the search space of workflows. This is a potential point of confusion and many parts of the paper do not make explicit which of those two graphs they are talking about.\n- The exact definition of the search space is not made clear anywhere in the paper. It is clear that a node is a tuple consisting of a node, prompt, temperature, and output format. What is then less clear is what it means to have an edge between two nodes. Does this mean that we first made the first LLM invocation (of the first node), and then sequentially following that, we make the second LLM invocation (of the second node)? And what does it then mean if our node has two outgoing edges (is this a decision point where we invoke one or the other, or do we execute both next nodes in parallel)? What is also less clear is how the output of the first LLM invocation is used in the LLM calls for later nodes (if at all). Is there a root node? These all point to a broader issue of a lack of precision in the specification of AFlow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7Dx9RvgDQI", "reviewer": "Reviewer_D4oh", "summary": "This paper proposes AFlow, a new automated framework that uses Monte Carlo tree search to find the optimal workflow in the exploration space. It also redefines the workflow through code modification, tree-structured experience, and execution feedback. This paper verifies the effectiveness of AFlow on 6 different benchmark datasets, and enables smaller models to outperform GPT-4o on specific tasks. The main contributions of this paper are as follows:\n\n1. Problem Formulation: This paper formalizes the workflow optimization problem and generalizes previous approaches to specific cases. This provides a unified framework for future research at both node and workflow optimization levels.\n2. This paper designs AFlow, an MCTS-based method that automatically discovers efficient workflows across multiple domains with minimal human intervention.\n3. This article evaluates AFlow on six benchmark datasets: HumanEval, MBPP, MATH, GSM8K, HotPotQA, and DRO, verifying the effectiveness of AFlow. It is worth noting that the workflow generated by AFlow enables smaller LLMs to outperform larger models, providing better cost-effectiveness, which has a significant impact on practical applications.", "questions": "1.  Can you provide examples of optimized workflows for different tasks, along with explanations of how these workflows can be interpreted. \n2. How consistent are the results when running AFlow multiple times with the same model and task? What methods, if any, does AFlow use to introduce variability in the search process? How is the scope of the search space defined and controlled in AFlow?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Originality\n\nThis paper pioneered a new definition of workflow optimization. By redefining the workflow process, it transformed the problem of automatically generating workflows into a Monte Carlo tree search problem in the search space. It also made up for the shortcomings of the previous problem, transformed the previous work into special cases, and provided a unified framework for subsequent researchers.\n\n2. Quality\n\nThe AFlow framework proposed in the paper has shown strong performance in experimental evaluation, outperforming existing methods by an average of 5.7% on benchmark datasets. Additionally, AFLOW can achieve better performance than larger models using smaller LLM models, which is of great importance for practical applications.\n\n3. Clarity\n\nThe paper provides detailed descriptions and explanations of the key components of the AFLOW framework, such as the MCTS algorithm, node selection strategy, and LLM-driven node expansion, making the entire implementation process clear and understandable. At the same time, the paper also describes the experimental setup and experimental process in great detail, and the explanation of the experimental results is also clear. The entire paper is logically rigorous and well-organized.\n\n4. Significance\n\nThe AFlow proposed in this paper not only has a significant improvement in performance, but also proposes a unified framework in the field of automatic generation and optimization of workflows. The new definition better explains this task and provides a new framework and optimization direction for future research.", "weaknesses": "This paper has achieved good results in both method and experiment. In terms of method, the paper innovatively formulated the automatic workflow optimization problem, establishing a foundational structure for future research. In terms of experiment, it not only achieved good results, but also conducted a lot of relevant analysis. However, this paper has some weaknesses in the following aspects:\n\n1. The experimental part of this paper lacks the cost analysis of the early AFlow search stage. The cost analysis of different methods later in the paper shows the effectiveness and low consumption of the workflow found by AFlow, but the early MCTS search is a huge process, and the execution of nodes will also consume certain resources. This part does not provide experimental explanation.  If the cost of exploring and finding the optimal workflow is huge, then the discussion on cost should include this resource consumption.\n2. This paper argues that different language models require different workflows to achieve their optimal performance. However, there is a lack of sufficient experiments to support this assertion, as the paper only mentions that the workflow identified using DeepSeek-V2.5 performs notably weaker on GPT-4o-mini compared to the workflow found using GPT-4o-mini itself.  At least one more set of comparative experiments should be added, that is, generate a workflow through GPT-4o-mini and then use DeepSeek-V2.5 and GPT-4o-mini respectively to see the experimental results. It would be best if more comparative experiments of other types of models could be added, such as adding another LLama series model, and comparing the three models. This is an interesting assertion, but more sufficient experiments are needed to verify it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)