Benchmarking Agentic Workflow Generation
Authors: Shuofei Qiao
Runnan Fang
Zhisong Qiu
Xiaobin Wang
Ningyu Zhang
Yong Jiang
Pengjun Xie
Fei Huang
Huajun Chen
Venue: ICLR 2025 Poster (2025)
Keywords: workflow generation
graph structured planning
large language model
agent
Primary area: datasets and benchmarks
TL;DR: A unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures.

Abstract: Large Language Models (LLMs), with their exceptional ability to handle a wide range of tasks, have driven significant advancements in tackling reasoning and planning tasks, wherein decomposing complex problems into executable workflows is a crucial step in this process. Existing workflow evaluation frameworks either focus solely on holistic performance or suffer from limitations such as restricted scenario coverage, simplistic workflow structures, and lax evaluation standards. To this end, we introduce WorfBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures. Additionally, we present WorfEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms to accurately quantify the LLM agent's workflow generation capabilities. Through comprehensive evaluations across different types of LLMs, we discover distinct gaps between the sequence planning capabilities and graph planning capabilities of LLM agents, with even GPT-4 exhibiting a gap of around 15%. We also train two open-source models and evaluate their generalization abilities on held-out tasks. Furthermore, we observe that the generated workflows can enhance downstream tasks, enabling them to achieve superior performance with less time during inference. Code and dataset are available at https://github.com/zjunlp/WorfBench.

Published as a conference paper at ICLR 2025
BENCHMARKING AGENTIC WORKFLOW GENERATION
Shuofei Qiao Rumnan Fang Zhisong Qiu Xiaobin Wang
Ningyu Zhang Yong Jiang S Pengjun Xie Fei Huang Huajun Chen Vi
Zhejiang University AAlibaba Group
Zhejiang Key Laboratory of Big Data Intelligent Computing
shuofei, zhangningyu) 02 (tw.ggyyjjejeeeeee edu edu. on
ABSTRACT
Large Language Models (LLMs), with their exceptional ability to handle a wide range of tasks, have driven significant advancements in tackling reasoning and planning tasks, wherein decomposing complex problems into executable workflows is crucial step in this process. Existing workflow evaluation frameworks either focus solely on holistic performance or suffer from limitations such as restricted scenario coverage, simplistic workflow structures, and lax evaluation standards. To this end, we introduce WORFBENCH, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures, Additionally, we present WORFEVAL, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms to accurately quantify the LLM agent's 's workflow generation capabilities. Through comprehensive evaluations across different types LLMs, we discover distinct gaps between the sequence planning capabilities and graph planning capabilities of LLM agents, with even GPT-4 exhibiting a gap of around 15%. We also train two open-source models and evaluate their generalization abilities held-out tasks. Furthermore, observe that the gener- ated workflows can enhance downstream tasks, enabling them to achieve superior performance with less time during inference!.
"If you can't describe what you are doing as a process, you don't 't know what you' re doing." "
W. Edwards Deming
INTRODUCTION
The remarkable advances in Large Language Models (LLMs) (Dubey al., 2024; Yang al., 2024a; OpenAl, 2024) are gradually recovering the considerable potential of LLM-driven agents towards tackling complex real-world problems, such as function calls (Qin et al., 2024; Tang et al., 2023; Qu et al., 2024; Liu et al., 2024a), embodied planning (Song et al., 2023; Zeng et al., 2024a; Xiang al., 2023; Song al., 2024; Qiao al., 2024a), code generation (Hong et al., 2024; Qian al., 2023; Zhang et al.. 2024c), etc., wherein decomposing complex problems into executable- exccutable-raanulariy subtasks is a crucial capability that L.IM agents must possess to achieve practically deployable.
A set of subtasks with execution dependencies is typi- cally referred to as workflow. Workflows can serve intermediate state for solving complex tasks,
Check of adie 1et Bovir mooeds..
J:Chock of the wnol adviingull coes disccaall
aiding agents in bridging the gap between tasks and specific executable actions (Wang al., 2024). More- over, an explicit workflow can enhance the agent's ss debuggability and interpretability, facilitating human- machine interaction. Many previous studies have examined the potential of LLMs for automatic work- flow generation (Zeng et al., 2023; Ye et al., 2023; Xue al., 2024: Li et al., 2024c). Recent researchers in the realm of LIM agents emphasize workflows as a form of prior knowledge or experience, guiding agent planning to avoid the occurrence of hallucinations (Zhu al., 2024; Xiao et al., 2024; Wang et al., 2024). As for benchmarks in this field, most works are confined to the end-to-end planning
-Chuck the emell mannaerngin to chopoaable
recorda the amall untrongiet cor
Suquerttal Enecuton trafficiet
Pandial Excutio Effiient
The Apullcationnnooooooooo Workfoor
Chatn-o--Thoghtt Augieriation
Kenwbadge
Shartes Planeiin Stope
Figure 1: Workflow and its application.
Equal Contribution. 1 Corresponding Author.
'Code and dataset are available httpp:://gtthbb.cmm//// hww.e//

---

Published as a conference paper at ICLR 2025
abiliity of LLM agents (Qin et al., 2024; Patil et al., 2023; Yang et al., 2023b; Wu et al., 2024; Liu et al., 2024c), while only a few studies attempt to evaluate the workflow generation (problem decomposition) capability from a finer granularity perspective (Shen et al., 2023; Chen et al., 2024b; Ye et al., 2024; Zheng et al., 2024a; Valmeekam et al., 2023). However, the current evaluation benchmarks unavoidably suffer from the following issues: 1) Limited scope of scenarios. They only focus on function calling or reasoning tasks. 2) Sole emphasis on linear relationships between subtasks (Figure left). Real-world scenarios often involve more complex graph structures, including parallelism. 3) Evaluations heavily rely on GPT-3.5/4, yet LLM themselves exhibit hallucinations and ambiguity. These limitations make the evaluation of workflow generation lack some systematicity, Therefore, to address the above issues, we introduce WORF BENCH, unified workflow generation benchmark with the following features:
Multi-faceted scenarios. We cover four complex scenarios for LLM agents, including problem- solving, function calling, embodied planning, and open-grounder planning. The dataset comprises 18k training samples, 2166 test samples, and 723 held-out tasks to evaluate generalization.
Complex workflow structures. We model the workflows as Directed Acyclic Graphs based on dependencies between subtasks, enabling a more precise representation of complicated serial or parallel structures in the real world (Figure right).
Strict quality control and data filtering. We introduce an intermediary structure called node chain between the original task and the workflow graph and employ the Topological Sorting algorithm for rigorous validation of the graph structure followed by human evaluation.
Accurate quantitative evaluation. We further propose WORFEVAL. to evaluate the workflow generation ability of LLM agents, which applies developed subsequence and subgraph matching algorithms for accurate quantitative assessment.
We evaluate the performance of most mainstream LLMs with various model scales WORFBENCH. We observe that, compared to linear structures, the models' ability to predict graph-structured workflows falls far short of real-world requirements, with even GPT-4 (OpenAI, 2023) only achieving performance of 52.47%. Furthermore, we train two open-sourced models on the training set and evaluate their generalization abilities on held-out tasks. Finally, we analyze how workflows can enhance end-to-end model performance as CoT (Wei et al., 2022) augmentation and prior knowledge, reducing end-to-end inference time by paralleling and shortening planning steps (Figure bottom).
To sum up, we summarize our main contributions follows:
We propose WORFBENCH, unified workflow generation benchmark with multi- factief scenarios and complex workflow structures. We conduct strict data filtering and human evaluation to ensure the quality of WORFBENCH.
We introduce WORFEVAL using effective subsequence and subgraph matching algorithms to evaluate the workflow generation ability of LLM agents from both chain and graph structures.
We conduct comprehensive evaluation on various closed-sourced and open-sourced models with different scales. We further exploit the generated workflows to facilitate downstream tasks and achieve superior and efficient performance.
WORFBENCH
2.1
TASK FORMULATION
Given specific task and candidate action list, our goal is enable the language agents generate a graph-structured workflow, where the nodes in the workflow satisfy the minimum executable granularity. Our action list here can include function APIs, tools, embodied actions, or mixture the above to simultaneously adapt various scenarios. Formally, given the task description action list A and language agents Ms, the workflow generation can be modeled as:
g(V.E) +- A),
where G is a Directed Acyelic Graph (DAG) with nodes V {U1, U2...., .., } being subtasks and edges& & {(v, 03),, I + j & representing the execution relationships between nodes
2

---

Published as a conference paper at ICLR 2025
Sector Benchmark Construction
Sector Quality Control
Task Description
How pack
bag full fan for long
plane riee" (Open (Open-drounde drounded)
" clean soap har tollet" (Embootied) "who is the present food minister the state where Kankumbi located?" (Problem Solsing)
Rutrieve
precipitation forecast swxt?
Action List
days Swattte? (Cunction
Socted
GPT-4
Workdfow Groph
CPP
Node Cherl
Sector 3: Werlf Eval
Sector A Belailed Case Worf Bench
Gold Graph
Imput Task:
Check the emails adminesite.comm com and managerdvite.comm are
Teprstopiaal Sort
disposable and get their sx records.
The action list you can select from:
[name: check for disposable smalls, descripion: parameters: -.)
(name: walddan smal descriptin: ... parameten:
(name: get the mo records descippion:..... parametent -.]
Medimamn Oenenun
(name: malize descripion: parameters ..)
Predicted Graph
Output hode:
adminguits.com dispssable.
Ratoogarce Mitckin
mansgemuutt.com dispssable.
Predicted Node Chain
NX records email managengntte.com Edge
(sTAatl) (STAR),)) (2)00
Figure 2: The overview framework of our WORF Sector is the benchmark construction where we first synthesize the node chain and then the workflow graph ($2.2). Sector is our data filtering process ($2.3). Sector 3 describes the algorithms in WORFEVAL to evaluate the predicted workflow of LLM agents ($2.4). Sector 4 is detailed data point of our WORFBENCH. Note that each node in this figure is uniquely identified by its color. Numbers the nodes represent their indexes in the gold chain. Nodes matched with gold chain or graph are circled by O in Sector 3.
(vy must be executed after vz). Note that all nodes V need to be executed according to their dependencies before task q is considered completed. It is evidently a bit challenging to directly instruct the language agents generate graph structure. To accommodate the generation habits of language models, we introduce the node chain C(V) before graph ((V,E):
G(V.E)+ + C(V) A).
Here node chain C is one of the Topological Sequences of graph G, where the order of nodes in the node chain ensures the relative order of nodes in the graph. Thus, the workflow graph generation can be reformulated by first creating node chain and then establishing edges for the generated nodes.
2.2 BENCHMARK CONSTRUCTION
We mainly collect various tasks q and the corresponding action lists A from existing well-known datasets. To facilitate a better understanding of the benchmark construction, we provide a detailed exposition of each dataset utilized in our paper Appendix A. To ensure the quality of the data, we also adhere to the strategy of first constructing the node chain and then building the workflow graph during benchmark construction. Depending on the varied ways of acquiring workflow nodes in different scenarios, we categorize node chain C construction into the three following types:
Function Call Tasks. Our function call data is collected from ToolBench (Qin et al., 2024) and ToolAlpaca (Tang et al., 2023), both of which involve combining different functions to accomplish multi-step user tasks. As the two datasets only have golden function calls for each task, make the synthesized node chain more reasonable, following the thought noought-action--e observation REACT (Yao al., 2023) format, we first utilize GPT-4 to reverse-engineer the thought based on the given function call (action) and then execute the function call to obtain the observation. Next, we carefully design the few-shot prompt for GPT-4 to generate the node (subtask) for each step based on its thought- oestion-bbeevaation loop. By iterating through all the function call steps, we obtain the node chain for the specific task. In addition, we also include Seal-Tools (Wu et al., 2024) as the held-out tasks, where the construction of the node chain follows the same process as described above.
Embodied Tasks. We collect the REACT format gold trajectories of ALFWorld (Shridhar et al., 2021) and WebShop (Yao et al., 2022) from ETO (Song et al., 2024) and OS (Liu et al., 2024b) from AgentInstruct (Zeng et al., 2024a). Unlike function call tasks, where each function call corresponds to node, embodied scenarios evolve dynamically based on the environment. It is hard to decompose granularity solely based on the initial environment. Therefore,
tasks into a seee-nodd-e----ceee we only decompose tasks into fixed granularity based the task and initial environmental information available. However, the advantage of embodied tasks is that the workflows of the same kind of tasks are similar. So for each kind of task, we seriously analyze and manually design few-shot
3

---

Published as a conference paper at ICLR 2025
examples, enabling GPT-4 to synthesize node chains directly based the gold trajectories. Similarly, we also include InterCodeSQL (Yang et al., 2023a) as held-out tasks for embodied scenarios.
x Problem-Solving and Open- Grounded Tasks. We also introduce problem-solving tasks like math, commonsense, and multimodal reasoning tasks from LUMOS (Yin et al., 2024), as well as a challenging general open-grounded dataset, WikiHow (Koupaee & Wang, 2018). Since the LUMOS-O O version of LUMOS and the WikiHow have already contained gold planning chains, we directly process them into the unified format we require. Furthermore, since WikiHow does not provide candidate action lists, we use the specific task as the query to retrieve the most similar actions in the public action library as distractors and then mix the distractors with the gold actions create the action list. The aim of retrieving similar actions is to increase the difficulty of the task.
After obtaining the node chain C(V), we further use GPT-4 to generate edges for the node chain. Since each node is a piece of text describing subtask, it is formally difficult to add edges to the text directly. Thus, we sequentially assign numbers to the nodes in the node chain and use these numbers instead of the node text when generating edges (e.g. (f.j) instead of (e,, 22)). All the generated edges finally form G(V,E), E). For ease of evaluation, we add the START and END nodes to represent the beginning and end of workflow. A detailed data point is illustrated in Figure 2 Sector 4. All the detailed prompts we use during the benchmark construction process can be found in Appendix A.7.
2.3 QUALITY CONTROL
Although we use GPT-4 to synthesize workflows, rationality still cannot be guaranteed. So we conduct restricted quality controls and data filtering for both node chain and workflow graph generation. Quality Control for Node Chain. Due to the low variability of the node chains in embodied tasks, our primary focus lies in filtering the node chain of function call tasks. Two critical factors are whether the order of nodes logical and whether each node accurately decomposes the task. The former has been guaranteed when constructing the node chain based on the sequence of gold function calls. For the latter, we use each synthesized node as a query to retrieve the function list. If one node within the node chain retrieves function that does not align with the gold, we will discard this task. We filter out 15.36% data through the quality control for the node chain.
Quality Control for Workflow Graph. The key priority is to ensure that the order of nodes in the workflow graph is consistent with the order of nodes in the node chain. Therefore, we perform a Topological Sort on the graph generated by GPT-4. Since the topological sort of a graph is not unique, ensure uniqueness and facilitate matching with the node chain, during the topological sorting process, when there are multiple nodes with an in-degree of in the graph, we sequentially remove them in ascending order based on their node number defined in node chain. We discard the data points with topological sorting results not aligned with node chains. We filter out 29.77% data through the quality control for the workflow graph.
To ensure complexity, we filter out data points with only 1 node or 1 edge. Furthermore, to maintain data balance across different scenarios, we randomly sample datasets with excessive data volume, subsequently dividing them into training and testing sets. Then we manually check the test set for a fair and effective evaluation (The detailed human verification process can be found in Appendix A.2). The final data statistics are outlined in Appendix A.3.
2.4 WORFEVAL
To ensure accuracy, apart from utilizing GPT-4 or simple semantic similarity matching, we quantita- tively evaluate both the node chain and workflow graph using restrict algorithms. Assuming the nodes and edges of the gold workflow are denoted as V> = [o] vi, ..., u/py) and E9 = [e% eq.., el while the agent predicted nodes and edges are represented as y" = [of ug,... "/vyy and g = Jef, eg ..., c'eyl- Firstly, we calculate the similarity matrix S between 1/9 and vp:
o (v) v')) = 0,
a (vg of) 2 a ( <
S.
Here, 1 1 |7 represents the cosine similarity function and in this paper, we encode the semantics of nodes using Sentence-BERY (Reimers & Gurevych, 2019). / serves as
https: / /huggi face. ww/.eonneeenne t rans formers/ al 111-mpet-bbol -aaeett-aa---ee v2. This model is also used as the retriever in benchmark construction process.
2a1 1 base-
4

---

Published as a conference paper at ICLR 2025
threshold, and we consider two nodes to be semantically matched only when their similarity is greater than or equal to 3. Therefore, S can be viewed as a bipartite graph, where one part consists gold nodes and the other part consists of predicted nodes. Since predicted node may match multiple gold nodes and a gold node may be matched by multiple predicted nodes, we utilize a max-weighted bipartite matching algorithm (Hoperoft & Karp, 1973) to find the best matches. Ultimately, the matched gold nodes and predicted nodes form two new node V C 19 and y C Vp, where the nodes in VP and the nodes in 19 correspond one-to-one. After obtaining the aforementioned variables, we proceed to evaluate both the node chain and workflow graph predicted by the agent.
Node Chain. Supposing the agent's 's predicted node chain is C(V"), based on the gold workflow graph G(V". E.). ) we can obtain all its possible topological sequences
C(V"),, }.
Assuming C(VP') ) is permutation of Vr such that it forms a subsequence of C(VP), and considering the one-to-one mapping between ye' and Vo we can derive sequence C(>ss) that aligns in order with C(VP'). ). Next, following Chen et al. (2024b), based the indexes of V in C(V*);, siis n, we can calculate a Longest Increasing Subsequence (LIS)* of C(Vs') for each C(V9);
4 = LIS(C(V")), ), C(V))))
Then, / = max (11), |, [/2]...,, (el....,,/// (al)) is obtained to represent the length of the longest valid subsequence for the agent's predicted node chain C(VP). Finally, the score for the node chain is denoted as:
2pchain Cchain
I [Pp] Tchain
flchain
Pchain
Tehain
PPehain
|
(5)
where Pchain and rchain are the precision and recall of the generated node chain respectively.
Workflow Graph. Based on Vr', we can derive the subgraph G(V , where gr' satisfies Ee' C and V(ui, U2) E EP', Ui, Uj Yp', By once again leveraging the one-to-one correspondence between yo and V we can utilize the Maximum Common Induced Subgraph (MCIS)² matching algorithm to find the maximum match between G(V Ey') and G(V9,
Gmeis (Vmeis Emess) = MCIS (G((V (g')), " (.9(V,
Assuming the number of nodes in this maximum common induced subgraph is h = [mmiss the score for the egent-generated workflow graph is denoted as:
fl grghahh 2g graphh -Pgrpph
T "graph
(7)
Pgraph + T'graph
where Pgraph and r 'ggaph are the precision and recall of the generated workflow graph respectively.
EXPERIMENTS
3.1 EXPERIMENTAL SETTINGS
To comprehensively evaluate the workflow generation capabilities of existing LLM agents, we validate total number of 18 models on the test set of WORFBENCH, including: 1) Four representative closed-sourced LMs: 01 (ol (ol-preview) (OpenAI, 2024), GPT-4 (OpenAL, 2023), GPT-3.5 (opp--..5tturbo-0000 2022), and Claude-3.5 (claude-3. 5tw.nnet--ssnnne-1---- art open-sourced LMs ranging from 7B to 72B: Llama series models and their variants, Llama- 3. (Meta-1.lama-3, (8, ) B-Inst (Dubey al., 2024), Llama-2-13B (1.) 10llmm-2-11 1a0--221--chtttttee (Touvron et al., 2023), Vicuna- 13B (vi al., 2023), and WizardLM-(133700 (RizardLM- .2 and 3iizardLM-70B-- 0) (Xu et al., 2024); Qwen series models, Qwen-2-(7,72jjB (Qwen2 - 72]B Instruct)
(OpenAI, (Anthropic, 2024). 2) Fifteen state-of-the. thee
(Zheng
'When the number of nodes and edges is considerable, matching all possible topological orders of a graph becomes a high time complexity issue. Therefore, we limit the output to maximum of 20 topological orders. For a specific analysis of why it is 20, please refer to Appendix A.9.
*nttps: / /en, wik i ped a org/ wi k http://ntnnnntnnnneeeeeeeeeeeeee innreasnnsssse www.nneeasinngssueene subsequencce Phttps://a //en. wik a wi Mww.maximmmmma wwtpaiiimmmmmmmmmeeeeeeme wnw.ooeddnndnndddineeo nndued_subgrrahh
5

---

Published as a conference paper at ICLR 2025
Table I: Main Results. We evaluate all the models with identical carefully designed instructions and two-shot examples. We categorize the models based on whether the models are open-source and their scales. The best results for each category are marked in bold, and the second-bes results are marked with underline.
Model
M Function Call
7
Problem Solving
Embudied
Open-Grounded
Averuge
Agreh
Agaah
flyshh
Aavin
Agrzh
faasiin
grap
Soureed
Claude-3.5
66.44
55.06
67.28
$5.30
71.74
56.71
61.33
42.88
666.20
$2.53
GP1:3.5
73.36
60.32
69.86
$4,50
64.57
49.17
47.67
28.
63.86
48.02
GPT-4
74.87
62.11
20.94
56.17
56.30
36.36
67.32
52.47
Ol-preview
70.68
57.11
72.76
59.25
59.90
54.19
53.47
35.97
66.70
51.63
Open Sourced
GLM-4-99
59.27
53.17
53.85
33.80
Phi-3-small
$7.66
40.71
55.76
39.75
54,77
37.52
44.65
22. 06
53.21
35.16
1lama-3.1-8B
63.30
43.62
64.49
46.79
56.23
36.40
44.-88
25.48
57.15
38.08
Mistral-718
67.30
51.67
40.97
58.53
Qwcn-2-7B
70.79
$5.50
68.6 65
52.13
62.83
46.25
39.29
20.89
60.39
13.69
M-2.511
68.43
$2.99
72.92
$7.80
65:77
48.09
40.84
21.27
61.99
45.03
2-13B
53.32
34.33
53.74
38.69
44.27
30.55
37.17
23.14
47.12
31 6.
WizardI.M-133
55.78
69.41 12
49.71
37.23
53.46
36.41
Vicuna-13B
53.75
37.66
64.55
50.25
$7.99
42.61
38.93
23.11
53.81
38.41
-meen-1.5-400
65.73
46.86
58.80
43.89
60.55
44.14
41.73
21.44
56.70
39.08
Phi-2-modium
67.71
47.26
71.15
54.85
65.11
49.99
42.73
23.77
61.68
43.97
Wizandl M-708
63.47
45, 4/
63.92
47,93
59.15
42.87
45.27
26.89
57.95
40.79
Mixtral-8x 7B
66.13
48.83
71.89
57.58
72.08
54.94
42.96
23.21
63,26
46.14
Llama-3. 3-0B
64.41
52.72
70.37
57.05
69.98
55.52
53.64
33.06
64.60
49.59
Qwen-2-72B
71.67
52.31
20.63
58.13
73.24
58.49
53.43
32.89
67.24
50.46
(Yang et al., 2024a) and Qwen-1.5-14B (Qwen1. ries models, Mistral-7B 8x7B Bea-rral--xx--Inssuuut------- (small, medium] (Phi-3- (smal1, medimaammmediiiiim 122 -18k-instrrtttt (Abdin al., 2024); other models including GL-M--9BB (glm-4-9b-chat) (Zeng et al., 2024b) and Internl.M-2.5-7B (internlm2. 5-7b-chtt)
(Bai et al., 2023); Mistral se- vo.tt (Jiang et al., 2023) and Mixtral- (Jiang et al., 2024); Phi series models, Phi-3-
(Cai et al., 2024).
We evaluate all the models using the LlamaFactory (Zheng et al., 2024b) framework with two-shot prompting. For models above 70B parameters, we utilize vLLM (Kwon et al., 2023) to accelerate inference. The hyperparameters used during decoding are all set to default values except for the temperature, which is 0.5. For all the models, the semantically matching threshold |3 is set to 0.6.
3.2
MAIN RESULTS
Table I displays our detailed experimental results. With the help of this table, we aim to analyze the following four research questions.
Q1: Which is more challenging for LLM agents, linear planning or graph planning? For each kind of task, we show both the node chain score f chaan and workflow graph score /1 graph. representing the linear planning and graph planning abilities of L.M agents respectively. It can be observed that the workflow graph scores of all models are significantly lower than the node chain scores, with the largest disparity found in GLM-4-9B, reaching an average difference of 20.05% Even the model with the smallest difference, -Jama-3.1-708 exhibits a difference of 15.01% %. We also explore what will happen if we provide the gold node chains the agents and task it solely with predicting the edges of the workflow graph (in Appendix A.4 Table 5). Despite the noticeable improvement after being provided with node chains, the model's performance in graph planning is still unsatisfactory, aligning with our analysis in Q4 Figure 4. So it appears that graph planning is more challenging than linear planning, and overall, the graph planning capability of models is positively correlated with their linear planning capability. However, there also several exceptions to this trend. For instance, in the reasoning tasks, Qwen-2-72B has a lower flehain compared to Mixtral-8 8 7B, but it leads in fl graphh On the other hand, in function call tasks, although Qwen-2-72B has a more satisfying /lhhain its fgaraph is lower than that of Llama-3.1-7055
Q2: How is Scaling Law manifested in workflow generation? Model size, training data scale, and training time are three crucial indicators of the Scaling Law (Kaplan et al., 2020). For open-source models, we categorize them into three groups based on model size: around 7B, 13B, and 70B. By observing the performance of models within the same series, we can discern the power of model size for workflow generation. For instance, in the Qwen-2 series, the 72B model outperforms the 7B model by 6.77% in flgraph: in the Llama-3.1 series, the 70B model surpasses the 8B model by 11.51%. An unconventional phenomenon is that some 7B models outperform the majority of 13B models. One reason for this is that most of the selected 7B models were released within the past six months, and trained on a greater amount of high-quality data, which is crucial for workflow

---

Published as a conference paper at ICLR 2025
70
.coc gragh
Table 2: Generalization Results of fine-tuned (FT)
& g.graah
50-
models on held-out tasks compared to baselines.
00-
Hehl-in Tocks
Hehl-ont Tooks
Model
Average
TiSral-Tiuls
TlaterCuleSQLL
Itais
Fansph
Famils
Jasis
GPT-3.5
63.86
48,02
95,91
76.63
65.30
53.07
GPT-4
67,32
52.47
96.58
50.25
66.35
54.36
10-
Qwecn2-788
60.30
43,69
92.68
14.75
54.20
30.72
Interal M-2.5-7B
61.99
45.03
93.07
14.43
53.36
42.20
Phi-3 mil-3-mmeeimm
61.68
43.97
94.11
79.45
58.45
46.62
Node & Edge Num
10
21 32
Llauss3 1-708
Qwen-2-12B
64.60
67.24
49.59
50.46
94.40
94.47
73.90
63.49
63.96
52.47
Figure 3: Performance Distribution of GPT-4. The distribution of L'Lehain for the number of nodes and the distribution of flaraph for the number of edges.
Qwen-2278---27
79.36
06.40
62.62
62.57
78.98
05.83
63.78
generation. In contrast, the 13B models were mostly released last year, and their training data and techniques may have become outdated, Another possible reason is our speculation that models around 13B may have reached point where the trade-off between training costs and model effectiveness has become challenging to navigate. This could also explain why there have been few models of this scale released in the past six months in the open-source community.
Q3: How far are existing LLM agents from being real workflow planning experts? Even though the node chains and workflow graphs are synthesized by GPT-4 in our benchmark, after we remove the gold trajectories and let it generate directly, its absolute performance in / l'lehain and I /'graph averages only 67.32% and 52.47%, respectively. In the challenging open-grounded tasks, the top performer, Claude-3.5, only achieves 61.33% and 42.88%, which is far from the level of practical and deployable workflow planner. The current most powerful reasoning model 01 (OpenAI, 2024) only performs relatively well problem-solving (reasoning) tasks, falling short other tasks that require more environmental knowledge. In addition, we analyze the performance of GPT-4 across different numbers of nodes and edges in workflow, as shown in Figure 3. With the increase of nodes and edges, both the f flchain and f graphh performance of GPT-4 tend to decline, with occasional brief spikes likely caused by uneven sample distribution. Therefore, for complex planning tasks with more planning steps, the performance of GPT-4 is unsatisfying no matter for linear planning or graph planning, let alone other models. This is clearly inadequate for many complex real-world scenarios, which is why many agent architectures are currently only at the theoretical level.
Furthermore, we attempt to train the 7B models Qwen-2-7B and LnternL.M-2.5-77B (both models show excellent performance in Table 1) on the training set We evaluate the trained models' capabilities on both held-in and held-out tasks. Results are presented in Table 2. While the trained models have shown the best performance on Seal-Tools (surpassing GPT-4 by 3.5%% in flgraph), their advantages are not as pronounced as on held-in tasks (where they surpass GPT-4 by 10%+ and 15%+ in f flchain and figraph, respectively). Moreover, the workflows of Seal-Tools are relatively simple (about 2~3 nodes per task on average), with even untrained 7B models achieving around 74%. When it comes to the more complex InterCodeSQL. tasks, the trained models fall slightly behind, only outperforming models around 7B and 13B. To sum up, although the trained models have made significant breakthroughs on the held-in tasks, their performance does not exhibit remarkable generalization when extended to held-out tasks, especially on embodied tasks. This implies that the structured workflow planning capability cannot be learned solely through fitting a large amount of data. So whether through prompting or training for generalization, LLM agents still have a long distance to reach real workflow planning experts.
Q4: What shall we do to enhance the workflow generation capability of LLM agents? First, we answer this question by analyzing samples where GPT-4 scores less than 0.5 on fl flgraph- Through meticulous manual checks and categorization, we identify four kinds of typical errors: 1) Granularity. The decomposition of subtasks does not meet the minimum executable granularity. 2) Explicitness. The summary of subtasks is overly vague. 3) Graph. The subtask is correct, but the graph structure is incorrect. 4) Format. The output does not adhere to the specified text format. Figure 4 displays the probabilities of different types of errors. We also show some cases of 01 in Appendix A.6. Among them, case (a) represents the error of Graph, while case (b) represents the error of Granularity. We summarize that most of these errors can likely be
granulaari ity
explicitness
graph
format
others
Figure 4: Error Statistics.
'The detailed training setups can be found in Appendix A.5
7

---

Published as a conference paper at ICLR 2025
attributed to the agent's lack of environmental knowledge. Granularity pertains to deficiency in prior knowledge of the environment. For instance, in case (b), O1 lacks the knowledge that the fridge can be used for cooling, assuming that cool potato can only be achieved inside fridge rather than somewhere else. Explicitness reflects a lack of understanding of the environmental task, resulting in insufficiently specific subtasks. Graph issues arise from a lack of comprehension regarding the dependencies of environmental actions, leading to errors in the relationships between subtasks. Some prompt optimization algorithms (Yüksekgönül et al., 2024; Zhou et al., 2024) or multi-agent architectures (Zhuge et al., 2024; Zhou et al., 2023; Hong et al., 2024) may lead to improvements to some extent. However, achieve a higher level of intelligence, the key may lie in truly integrating world knowledge (Yu et al., 2024; Qiao et al., 2024a; Guetta et al., 2024) or world model (Dawid LeCun, 2023; Hu & Shu, 2023; Wong et al., 2023) into agents to advance their understanding of the real world (Sumers et al., 2024; Durante et al., 2024a; Yang et al., 2024d).
4 THE ROLE OF WORKFLOW FOR AGENT PLANNING
In this section, we delve into how structured workflows can aid downstream tasks such as function invocation or embodied planning in achieving more precise and expedited results. this section, we default to utilizing the trained Qwen-2-7B (mentioned in $3.2 Q3) as the workflow generator.
4.1 ENHANCE END-TO-END PERFORMANCE
Table 3: End-to-end Performance augmented by work- flow as prior knowledge.
Workflow as Structured Prior Knowledge. Given that a workflow encompasses detailed execution process for a task, an evident use case is to employ as prior knowledge to directly guide agent planning. This is particularly ad- vantageous in embodied scenarios where L.M agents often lack prior knowledge of the real en- vironment and rely on brainless trial-and-error (Qiao et al., 2024a). Therefore, put the generated workflow along with the task and design instructions for the LLM agent to plan based on the guidance of the workflow. We choose GPT-4, Llama-3.1 1-8B, and Qwen-2-72B as the L.M agents and report the results in Table 3, illustrating that models with varying capabilities can benefit when enriched with structured workflow knowledge. For ALFWorld with greater diversity in environmental changes and more complex tasks, workflow knowledge yields greater advantages. Furthermore, we observe that these workflows are generated by a 7B model, providing guidance even to the significantly more powerful 72B model. This leads us to contemplate the weak woak--uide-strroe paradigm, wherein small model possessing specific environmental knowledge supervises the planning of larger, more general model.
World
Model
WebShop
seen
unseen
GPT-4
27.14
28.36
55.62
GPT-4-W
40.71 113.57
47.01 118.65
56.49 10.7
LJama-3, 1-88
1.49
5,00
$1.03
Llama-3,118BB
8.21 16.72
12.14
17.14
52.28
11.25
Qwen-2-728
53.57
$6.72
directly in-
Qwan-2-12++W
56.43 12.86
62.29
15.57
60.55
11.60
Workflow as CoT Augmentation. Chain-of-Thought (CoT)) (Wei et al., 2022) has been widely acknowledged for enhancing the reasoning abilities of LLMs and plays a crucial role in OpenAl's latest reasoning model, ol (Ope- nAI, 2024). However, a tricky issue lies in its long-context nature, which may mislead LLM agents in making erro- neous decisions, especially when there are multiple plan- ning steps involved. Based our workflow construction process where each node corresponds to a function call, we can leverage this characteristic induce agents to en- gage in more focused planning. Specifically, we prompt Qwen-2-7B to generate a CoT each step based on the corresponding node and then use the node as query to retrieve the most similar API from the API list as the func- tion for that step. Ultimately, we allow the model to decide how to invoke the function based on the CoT and the selected function. In this process, the workflow plays a role similar to augmenting CoT, assisting the agent in thinking at each step, serving the query for retrieval to provide the agent with more relevant APIs, thereby alleviating the agent's 's burden and enabling it to focus more on how invoke tools effectively. By comparing the accuracy of function calls with lama
Cat.
47.50
13
Inst.
12 Inst.
Oven 2-120
bana
Figure 5: Relative Function Call Accuracy of workllow-augmenee Qwen-2-7B (Qwen- 2-7B+W) on StableToolBeence (Guo et al., 2024b) compared with various basclines.
8

---

Published as a conference paper at ICLR 2025
(Qin al., 2024) and two one-shot baselines (Qwen-2-72B and GPT-4) StableToolBench 7, find that the above procedure is effective (shown in Figure 5). Unlike kind of external knowledge for reference, the workflow here actively participates in the planning process, leading to improved accuracy in function invocation.
4.2 REDUCE END-TO-ENT TED- INFERENCE-TING TIME
Parallel Planning Steps. In graph-structured workflow, nodes without dependencies can be executed in parallel. This can significantly reduce the time required to complete tasks compared to linear step-by-step execution. Continuing our analysis on StableToolbeech, for specific task, we calculate the time taken by ToomL to complete each node when execut- ing step by step (including generating thought, generating function calls, executing functions, and returning results). We then mark the nodes in the workflow graph based on their completion times. So our objective can be transferred to identify the longest path between the S START and END nodes, also known as the Critical Path of the graph. Fi- nally, we compare the average time taken to complete all tasks with the linear TooaL as shown in Figure 6. It can be observed that with graph-structured parallelization, there is significant reduction in the average time to com- plete tasks, with reductions ranging from approximately one-fifth to one-third across different test sets. The parallelization feature of graph structures allows for substantial savings in inference time real-world applications. Moreover, the execution of a node does not necessarily depend on all previous nodes, which to some extent alleviates the issue of long contexts in multi-step complex tasks, thereby enhancing the quality of task completion.
ToolLiama
ToolLlama+W
50
39.31
39
76
30
time
20-
26.36
26.05
10-
0
Cat.
Inst.
13 Inst
Figure 6: Average Task Execution Time
linear ToolL.ama and parallel IooaL
Shorten Planning Steps. In addition to the hori- zontal reduction of inference time brought by parallel subtask execution, can vertically decrease the planning steps of the LLM agent. This finding emerges during our experiments on workflow as structured knowledge. When the LLM agent lacks prior knowledge of the environment, it often accumulates knowledge through random trial- and-error the environment, which may introduce irrelevant noise and lead to drop in long-text disaster. Introducing knowledge makes the agent's s actions more purposeful, reducing the steps of blind trial-and-error. In Table 4, we quantitatively analyze the average planning steps required for the model to complete tasks with or without workflow knowledge, which corroborates our discoverics.
Table 4: Average Planning Steps.
Model
ALFWorld
WehShop
also observe that workflows
seen
unseen
GPT-4
17.19
17.43
5,80
GPT-4+W
15.64 11.55
15.85 11.58
5.72 10.08
Llama- 3.1-8B
19.81
19.43
7.08
IJama-3. 3.1-8-3.1-8::21
19.09
18.38
6.77
Qwen-2-728
14.67
Qwcn-2-72B++W
14,05
10.34
13.94
10.73
3.73
1O.15
RELATED WORK
5.1 LARGE LANGUAGE AGENTS
The rise of Large Language Models (LLMs) has established them at the forefront of the quest for Artificial General Intelligence (AGI), offering substantial support for the advancement L.LM- centered AI agents (Wang et al., 2023; Xi et al., 2023; Guo et al., 2024a; Yang et al., 2024b; Durante et al., 2024b; Li et al., 2024a; Zhang et al., 2024a; Yang et al., 2024c). Numerous efforts have been dedicated to employing language agents for tool utilization (Qin et al., 2024; Tang et al., 2023; Yuan et al., 2024; Ye et al., 2024; Qiao et al., 2024b; Chen et al., 2024a; Qu et al., 2024), embodied planning (Huang et al., 2022: Yao et al., 2023; Song et al., 2023; Xiang et al., 2023; Palo et al., 2023), software engineering (Hong et al., 2024; Qian et al., 2023; 2024), etc. However, these agent
As defined in ToolBench (Qin et al., 2024), 12 and 13 stand for intra categoy and intra-collection multi-too instructions. respectively, based on the tools belonging to the same RapidAPI (https: // /app dapi. con/ hub) category collection. Inst. represents unseen instructions for the same set of tools in the training data. Cat. denotes unseen tools that belong to unseen category of tools in the training data.

---

Published as a conference paper at ICLR 2025
methods frameworks tend to concentrate primarily the end-to-end performance of the task at hand, overlooking the assessment of the inherent reasoning and planning capabilities that are actually the bases for achieving stable and reliable agent performance.
5.2 WORKFLOW AND LANGUAGE AGENT PLANNING
Outside the context of LLMs, there have been various mature works in the field of concurrency theory that theoretically study the formal models for concurrent execution, including Petri nets (Peterson, 1977; Reisig, 1985; Murata, 1989) and process calculi (Milner, 1989; Hoare, 1985). Building upon these concepts and methods, much previous literature concentrates on workflow modeling and mining within the realm of business process management and process mining (van der Aalst et al., 2003a; van der Aalst, 1997; Dijkman et al., 2008; van der Aalst et al., 2003b; van der Aalst, 2016). Recently, more focus has been placed on integrating workflows with LLM agents to automate the generation of workflows or to enhance the capabilities of LLMs in handling complex problems (Wang et al., 2024; Zhang et al., 2024b; Li et al., 2024c; Xiao et al., 2024: Zeng et al., 2023). On the one hand, workflows can serve as an intermediate state for solving complex tasks, aiding agents in bridging the gap between tasks and specific executable actions (Li et al., 2024c; Zhang et al., 2024b). Explicit workflows can enhance the interpretabilityy of the LLM agent, facilitating human involvement in interaction). On the other hand,
debugging and ensuring the agent's 's security (human-mache workflows can serve structured prior knowledge, assisting agents in handling knowledge-intensive tasks to avoid planning hallucinations (Zeng et al., 2023; Ye et al., 2023; Zhu et al., 2024; Qiao et al., 2024a; Xiao et al., 2024; Wang et al., 2024). Note that the DAG-based workflow introduced in paper is equivalent in expressivity to Marked Graphs (Commoner et al., 1971), one of the many restricted subclasses of Petri nets.
5.3 AGENTIC WORKFLOW GENERATION AND EVALUATION
The most straightforward approach is human-designed workflows that constrain the planning process of language agents in the form of natural language prompts or state machines to prevent hallucinations (Hong al., 2024: Li et al., 2024b; Guan et al., 2024; Zhu et al., 2024; Wang et al., 2024). However, manual design is cinecconguiing and lacks flexibility. As a result, some studies (Zhou et al., 2023; Ye et al., 2023; Zeng et al., 2023; Li et al., 2024c; Xue et al., 2024; Zhang al., 2024b) try to enable language agents to automatically generate workflows. For example, Zhang et al. (2024b) use MCTS (Monte Carlo Tree Search) to guide LLM in exploring the workflow space, which effectively advances the quality of the generated workflows. So it is crucial to effectively evaluate the quality of workflows generated by language agents. Previous studies have explored automating workflow generation evaluation in tool learning or reasoning scenarios (Chen et al., 2024b; Ye et al., 2024; Shen et al., 2023; Valmeekam et al., 2023), using fine-grained metrics to assess each stage of tool utilization. In particular, problem decomposition ability is evaluated through semantic similarity matching or GPT-4 scoring. However, these works suffer from the following limitations: firstly, most of them focus only on function-calling or reasoning scenarios, neglecting embodied interactive scenarios with the real environment; secondly, their workflows are all single linear structures, making it difficult to represent more complex tasks; lastly, they mostly rely GPT-4 or human evaluation. Moreover, Xiao et al. (2024) also explore workflow- -uidedd planning, but it is noticed that they mainly examine the workflow compliance capability of language agents. Lal et al. (2024) examine the understanding of node dependencies in workflows by LLM in a question-answering format. The above two can be seen as the downstream procedures of our work.
CONCLUSION
In this work, we introduce WORFBENCH, a unified agentic workflow generation benchmark with miscellaneous scenarios and intricate graph-structured workflows. To precisely assess the workflow generation capability of LLM agents, we further present WORFEVAL, which utilizes quantitative algorithms to evaluate both the linear and graph workflows. Through comprehensive experiments across various kinds of LLMs, we investigate large performance gaps between the traditional linear- structured and complex graph-structured workflow generation. We also train open-sourced models and evaluate their generalization abilities on held-out tasks. Finally, we explore the role of workflows for downstream planning tasks from model performance and inference time efficiency.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We have submitted all the test datasets (comprising both held-in and held-out tasks) of our benchmark, along with the test code, to the supplementary materials. The detailed benchmark construction and quality control processes can be found in Section 2.2 and 2.3. Additional data source information, human verification processes, and benchmark statistics are provided in Appendix A.I, A.2 and A.3. Specific test configurations: the code framework used, the model versions used, and the inference hyperparameters are mentioned in Section 3.1. All the training settings involved in our paper are detailed in Appendix A.5.
ETHICS STATEMENT
This study is carried out in strict accordance with ethical guidelines and best practices in research. All the data utilized are sourced from publicly available datasets, and no proprictary or confidential data are used. Throughout the paper, every mention or use of these data sources has been properly and accurately cited. We strongly urge all users adhere to the highest ethical standards when using our dataset, ensuring faimess,, transparency, and responsibility in their research. Any usage of the dataset that may lead harm or pose a detriment to society is strictly forbidden.
ACKNOWLEDGMENTS
We would like express our great gratitude the anonymous reviewers for their kind comments. This work was supported by the National Natural Science Foundation of China (No. 62206246, No. NSFCU23B2055, No. NSFCU19B2027), the Fundamental Research Funds for the Central Universities 1226-2033-011888 Yongjiang Talent Introduction Programme (2021A-156-G), CIPSC- SMP-Zhipu Large Model DissilDiscyylinary Fund, Ningbo Science and Technology Special Projects under Grant No. 20232212, Information Technology Center and State Key Lab of CAD&CG, Zhejiang University. We gratefully acknowledge the support of Zhejiang University Education Foundation Qizhen Scholar Foundation.
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat S. Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck. Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Amit Garg, Abhishek Goswami, Suriya Gunasekar, and al. Phi-3 technical report: A highly capable language model locally on your phone. CoRR, abs/2404.14219, 2024. doi: 10.48550/ARXIV.240044 14219.
URL http::://oo...oo//100555555
14219.
Anthropic.
Claude 3.5 sonnet, 2024. claude- 3-5 sonnne..
https:://ww.anthheeee
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, and et al. Qwen technical report. CoRR, abs/2309. 16609, 2023. doi: 4488550/RXIV..399..U.. 2309. 16609.
URL https: /tt. or1/100
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, and et al. Internlm2 technical report. CoRR, http///44...199,, 2024. doi: 10. 48550/ARXIV.240..2225
URL https: /oi. orgg100 11.11..8555555 48550/ arxiv 2403 .1133.1.22 17297.
Mingyang Chen, Haoze Sun, Tianpeng Li, Fan Yang, Hao Liang, Keer Lu, Bin Cui, Wentao Zhang, Zenan Zhou, and Weipeng Chen. Facilitating multi-turn function calling for llms via compositional instruction tuning. CoRR, abs/2410. 12952, 2024a. doi: 10.48550/ARXII22410.0 129522..... URL hetps://dof httpss////ooooo//.////e 48050..8555aara 2410 12952. 222...1291...........

---

Published as a conference paper at ICLR 2025
Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, and Feng Zhao. T-eval: Evaluating the tool utilizztion capability of large language models step by step. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACI. 2024, Bangkok, Thailand, August 11-16, 2024, pp. 9510-9529, Association for Computational Linguistics, 2024b. URL https: //aclanthol a/aclanthology org/ 2024. acl-aaalaaeaeeeeee -oog..n.................... 515.
Frederic G. Commoner, Anatol W. Holt, Shimon Even, and Amir Pnueli. Marked directed graphs. Comput. Syst. Sci., 5(5): 1-5533,,,,, 1971. doi: 10.1016/50022- //doi. org. 10. 1016 //0002/00222 0000 (71) 80013-2. 2.
0100000222200007780001111 URL https:
Anna Dawid and Yann LeCun. Introduction to latent variable energy-based models: A path towards autonomous machine intelligence. CoRR, abs/2306. 02577,,,.00002..... 2023. doi: 10.48550/ARXIV.2306. 02572. URL https: / doii wtt.::........4///.............. 02572.
48550 http55//aXxi..v
Remco M. Dijkman, Marlon Dumas, and Chun Ouyang. Semanties and analysis of business process models in BPMN. Inf. Softw. Technol., 50(12)::281-1194 2008, doi: I0.016/..NFSOF22008.00 006. URL https: /tt.//oiiii/// org/ 10 1 infsoiifff... 2008. .2.022..0000 60.6
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshorn Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, and et al. The llama 3 herd of models. CoRR, 11x/2407.21783. 2024. doi: N0.48550/ARXIV224...211 48550/arXis 2407. 21783.
URL https: //do:///oo.o/// ott//00..111...
Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong, Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, Katsushi Ikeuchi, Hoi Vo, Li Fei-Fei, and Jianfeng Gao. Agent AI: surveying the horizons of multimodal interaction. CoRR, abs/2401 03566, 2024a. doi: 10.48550/ARXIV.24000 400.0366.. 03568.
wtt.::i..rr11....55000. URL 0 2401
Zane Durante, Qiuyuan Huang. Naoki Wake, Ran Gong, Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, Katsushi Ikeuchi, Hoi Vo, Li Fei-Fei, and Jianfeng Gao. Agent Al: surveying the horizons of multimodal interaction. CoRR, /wt.22401.03668.. 2024b. doi: 10.48550/ARXIV.400 2.0.000........56...... URL htttps://dot http:////ooo..g/../../oo/ 48550/555///0 03568.
2401.
Jian Guan, Wei Wu, Zujie Wen, Peng Xu, Hongning Wang, and Minlie Huang. AMOR: A recipe for building adaptable modular knowledge agents through process feedback. CORR, abs/2402.01166, 2024. doi: 10.48550AARXIV224022000 24C2.
URL https://////.. :ttss:///oiiii org/o0rg.//.. 48550/arXix
Nitzan Bitton Guetta, Aviv Slobodkin, Aviya Maimon, Eliya Habba, Royi Rassin, Yonatan Bitton,
Idan Szpektor, Amir Globerson, and Yuval Elovici. Visual riddles: a commonsense and world knowledge challenge for large vision and language models. CoRR, abs/2407. 2024. doi: 10. 48550/ARXIV.2407.11M URL https: //w.///oo//////////.oo wtt.ddi..oooooooooooooo www.ooogg/o.o.oo.oooooooo 081100..855000 48550/ arxiv. 2407 112.0...94477 19474.
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. CoRR, abs/2402.01680, 2024a. doi: 10.48550/ARXTV22422.0666 URL https: //doii.or/////// wttp://.....51///1 48550/ wwt.xiii..42....2 01680.
Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large language models. In Lun-Wer Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACI. 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 11143-11156. Association for Computational Linguistics, 2024b. URL. https: httsss//aclaathhloggo.. org/2024. f findings acl 664.
C. A. R. Hoare. Communicating Sequential Processes. Prentice-Hall, 1985. ISBN 0-13-153271-5.
12

---

Published as a conference paper at ICLR 2025
Sirui Hong. Mingehen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran. Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. Metagpt: Meta programming for A multi- agent collaborative framework. In The Twelfth International Conference on Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL .et/
https://openreve/
John E. Hopcroft and Richard M. Karp. An n 5/2 algorithm for maximum matchings in bipartite graphs. SIAM J. Comput., 2(4))225-231, 1973. doi: 10.1137/0202019 URL https: /doi.
org/10.1137 02020011.
Zhiting Hu and Tianmin Shu. Language models, agent models, and world models: The LAW for machine reasoning and planning. CoRR, abs/2312.05230, 2023, doi: 10448550/RRXIV23322.0222 URL wwt.t:tps///o
/ooii org/ / 10. 48550/ arxi
Wenlong Huang, Picter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero- shot planners: Extracting actionable knowledge for embodied agents. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 9118-9147. PMLR, 2022. URL https: / wwtppppooosgssooeeeeppppeepp.sse mlr press / 62/ huang22a. htmll
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Ane Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William EI Sayed. Mistral 7b. CoRR, abs/2330.00685, 2023. doi: 10. 48550/ARRIVV23100066222 URL https: doi. /t.///...... 48550/ www.wiivv...... 06225.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mixtral of experts. CoRR, abs/2401 004088, 2024. doi: 10,48550ARXXI.2401.00 https: http///ii.oggo////////oo 48550/ arxiv 2401. 04088.
URL
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, 0as/22001.08361 2020. URL https: / aarxiaiivv.o.. wrg/orggg//.......o. /awaab/0000 2001. 08361.
Mahnaz Koupaee and William Yang Wang. Wikihow: large scale text summarization dataset. CoRR, abs/ 2018. URL http: http//aaxiv..v http:/aatwar.raaa///////aaaaaaaa 1810.1111 09305.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Jason Flinn, Margo I. Seltzer, Peter Druschel, Antoine Kaufmann, and Jonathan Mace (eds.), Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23 26, 2023, pp. 611-626. ACM, 2023. doi: 10.1145/ 166000663613165 URL. www.t:ttt/////.t. /oi. org. 0 1 145 360006 3613165.
Yash Kumar Lal, Vanya Cohen, Nathanael Chambers, Niranjan Balasubramanian. and Raymond J. Mooney. Cat-bench: Benchmarking language model understanding of causal and temporal dependencies in plans. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 16-16, 2024, pp. 19336-19354. Association for Computational Linguistics, 2024. URL https:: http:////caanthooo.o.oo org/2024 camlp-mmain 1077.
Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, Rui Kong, Yile Wang, Hanfei Geng, Jian Luan, Xuefeng Jin, Zilong Ye, Guanjing Xiong, Fan Zhang, Xiang Li, Mengwei Xu, Zhijun Li, Peng Li, Yang Liu, Ya-Qin Zhang, and Yunxin Liu. Personal LLM agents: Insights and survey about the capability, efficiency and security. CoRR, abs/2401 2024a. doi: (0.48550/ARXIV.2401 005459. URL https:/ //do:.///....... .org/10...... 485508850/// rxiv 2401 05459.
13

---

Published as a conference paper at ICLR 2025
Zelong Li, Wenyue Hua, Hao Wang, He Zhu, and Yongfeng Zhang. Formal-llm: Integrating formal language and natural language for controllable llm-based agents. CoRR, abs/2402.00798, 2024b. doi: 1044550/ARXV..442..0777 00798.
URL https: :
Zelong Li. Shuyuan Xu, Kai Mei, Wenyue Hua, Balaji Rama, Om Raheja, Hao Wang, He Zhu, and Yongfeng Zhang. Autoflow: Automated workflow generation for large language model agents. CoRR, abs/2407, 12821, 2024c. doi: 10.4850/ARXIV.247...222 12821. URL https: 48550/ att.s///arii... 12821.
Weiwen Liu, Xu Huang, Xingshan Zeng, Xinlong Hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, Zezhong Wang, Yuxian Wang, Wu Ning. Yutai Hou, Bin Wang, Chuhan Wu, Xinzhi Wang, Yong Liu, Yasheng Wang, Duyu Tang, Dandan Tu, Lifeng Shang, Xin Jiang, Ruiming Tang. Defu Lian, Qun Liu, and Enhong Chen. Toolace: Winning the points of Ilm function calling. 2024a. URL https: / wawpiv.vvvvv.///aa//.a.a/.aaatv. org/ abs 2409. 00920.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng. Zhengxiao Du Chenhui Zhang, Sheng Shen, Tianjun Zhang. Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong. and Jie Tang. Agentbench: Evaluating lims as agents. In The Tweifth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024b. URL https: /tlp:://ppeee /ttp:////eeeeeeeeeeeeeeeeeeeeeee vew. net/ forum? d==AdUBOaCTT.
Xiao Liu, Tianjie Zhang, Yu Gu, lat Long long. Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, Jiadai Sun, Xinyue Yang, Yu Yang, Zehan Qi, Shuntian Yao, Xueqiao Sun, Siyi Cheng, Qinkai Zheng, Hao Yu, Hanchen Zhang, Wenyi Hong, Ming Ding, Lihang Pan, Xiaotao Gu, Aohan Zeng, Zhengxiao Du, Chan Hee Song, Yu Su, Yuxiao Dong, and Jie Tang. Visualagentbench: Towards large multimodal models as visual foundation agents. CoRR, 2024c. doi: 1044550/ARXXV.24000000 063277. URL htttps://doi. http//oi..ooo....////oo///
485 48550/arXiv. www.aaiwwii........... 14088.06321
Robin Milner. Communication and concurrency. PHI Series in computer science. Prentice Hall, 1989. ISBN 978-0-13-1150077
Tadao Murata. Petri nets: Properties, analysis and applications. Proc. IEEE, 77(4)5441-580, 1989. doi: 10.1109/5.24143. URL https: wtt.://......19/.22499....
OpenAI. Chatgpt: Optimizing language models for dialogue, 2022. https:/ com/ blog/cha chattpt/..
OpenAl. GPT-4 technical report. CoRR, abs/2303.08874, 2023. doi: 10.4850WaaXi..22330077744 URL
2023, orgoiooogg1000 10111...555500
OpenAI. Introducing openai o1-preview, 2024.
Norman Di Palo, Arunkumar Byravan, Leonard Hasenclever, Markus Wulfmeier, Nicolas Heess, and Martin A. Riedmiller. Towards A unified agent with foundation models. CoRR, abs/2307.0966 2023. doi: 10.4855/ARXIV.220..0000 2307.08 09668.
URL https://doi. oro/ /tt.g//....8500000
Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis. CoRR, abs/2305. 15334, 2023. doi: 10.4855/ARR113.2355.555. URL httph::t///dbe //to:.///oo... org/ 1tt.111...55500 2ww.008855000ra1r... http:/aawww.........5 1533.4
James L. Peterson. Petri nets. ACM Comput. Surv. 93)):23--52,, 1977. doi: 10.1145/35669883557022
URL wwtphhtss///.i.ooooo /10 10.
356702.
Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. CoRR, abs/2307 abs/2307.07924, 2023. doi: 10. 48550//ARXIVV2307777222 URL https: //ooi org/10 48550/855050 awvivv 07924.797.....
14

---

Published as a conference paper at ICLR 2025
Chen Qian, Jiahao Li, Yufan Dang, Wei Liu, Yifei Wang. Zihao Xie, Weize Chen, Cheng Yang, Yingli Zhang, Zhiyuan Liu, and Maosong Sun. Iterative experience refinement of software- developing agents. CoRR, fass/2405.04219, 2024. doi: 10.4850/ARRXI222000.000.. URL
https: //ooi
wwo./r//...... 4.544/55////0 /0/50///rxii/ 2405 04219.
Shuofei Qiao, Runnan Fang, Ningyu Zhang, Yuqi Zhu, Xiang Chen, Shumin Deng, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Agent planning with world knowledge model. CoRR, abs/2405. 14205, 2024a. doi: 10.48550/ARXIV.2405.5 14205. URL https://doi org//0. 48550/arXiv 2405 14205.
Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Eleanor Jiang, Chengfei Lv, and Huajun Chen. Autoact: Automatic agent learning from scratch for QA via self-planning. In Lun-Wer Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 3003-3021. Association for Computational Linguisties, 2024b. URL https::/p//...c...t.
www.yooooy.....oooooooooooo..o.. ogy org/ 2224 acc- long. g. 1 65.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan. Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang. Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL net forumr forum?id=ddngg200.
Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang. Dawei Yin, Jun Xu, and Ji-Rong Wen. Tool learning with large language models: A survey, CoRR, abs/2405.17935, 2024. doi: 10.48550/ARXII.2465..777 17935.
URL https: /tw.///oi.oo//// ort....gg/..... 1..0..8.0
2405.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System op- timizations enable training deep learning models with over 100 billion parameters. In Ra- jesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash (eds.), KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pp. 3505-3506. ACM, 2020. doi: 10.1145/394486.44007700 https:
URL
http://...11//39444141 3406703.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese nett-ns. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 3980-3990 Association for Computational Linguistics, 2019. doi: 10.18653/V1/D199 14100-199011 URL https: htt. ort/ /00 11118/66vv/// wt//119-141111
Wolfgang Reisig. Petri Nets: An Introduction, volume 4 of EATCS Monographs on Theoretical Computer Science. Springer, 1985. ISBN 3-540-13723-8, doi: 10. 10.1007/998833442269990 9. URL
https://doi.org:
1007 /978 3 642 69968-9.
Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yueting Zhuang. Taskbench: Benchmarking large language models for task automation. CoRR, 0bs/2211..18666 2023. doi: 10.48550/ARXI..231..1888 10. 48550/ arxiv 2311. 18760.
URL https: / org/
Mohit Shridhar, Xingdi Yuan, Maee-Alexandre Côté. Yonatan Bisk, Adam Trischler, and Matthew J. Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https:/ http:://pppeeeeeeeeeee eew net/ /f forum? idd= OIox0YcCaTn.
Chan Hee Song, Brian M. Sadler, Jiaman WU, Wei-Lun Chao, Clayton Washington, and Yu Su. Lim-planner: Few-shot grounded planning for embodied agents with large language models. In IEEE/CVF International Conference Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 2986-2997. IEEE, 2023. doi: 10. 10011090CC1100002222..0000 //doi org/ 1109/ wCC http///...111//1/11110111111 2023 22230002800000
URL https:
15

---

Published as a conference paper at ICLR 2025
Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and er- ror: Exploration- based trajectory optimization of LLM agents. In Lun-Wet Ku, Andre Mar- tins, and Vivek Srikumar (eds.), Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACI. 2024, Bangkok, Thailand, Au- gust 11-16, 2024, pp. 7584-7600. Association for Computational Linguistics, 2024. URL https: http:s//accanthllggeooooooo
a01-222...looonn 409.
Theodore R. Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. Cognitive architectures for language agents. Trans. Mach. Learn. Res., 2024, 2024. URL https:
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. CoRR, abs/2306.05301, 2023. doi: 10448550/ARXI.23066005 2306. 05301.
URL attps://doi... org/ 48555/001.880001111 48550/arXiv
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton- Moya Chen, and et al. Llama 2: Open foundation and fine-tuned chat models. CoRR, 0abs/2307.092880 2023. doi: 10.45550ARRXV.2337000222 URL https: //ooi. org/ 10. 4.5855/55xxi 2307 .02288.
Karthik Valmeekam, Matthew Marquez, Alberto Olmo Hernandez, Sarath Sreedharan, and Subbarao Kambhampati. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change. Alice Oh, Tristan Nau- mann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informa- tion Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 Serape__fieest 2tales/paaer/22233
16. 2023, 2023. URL http://ppaprs..ppp hash/7a Benchmare rks. html.
Karthik Valmeekam, Kaya Stechly, and Subbarao Kambhampati. Lims still can't plan; can Irms? preliminary evaluation of openai's on planbench, 2024. URL httpt:ttt////cc //arxii org/ abs/ 2409. 13373.
Wil M. P. der Aalst. Verification of workflow nets. In Pierre Azéma and Gianfranco Balbo (eds.), Application and Theory of Petri Nets 1997, 18th International Conference, ICATPN '97, Toulouse, France, June 23-27, 1997, Proceedings, volume 1248 of Lecture Notes Computer Science, pp. 407-426. Springer, 1997. doi: 10.100773-554--6313999 1888999898.8......
URL https://doi. oroo/ooorgg000
63139 9_48.
Wil M. van der Aalst. Process Mining Data Science in Action, Second Edition. Springer, 2016. ISBN 978-3-662-498500
doi: 10. 00.1007/978-3-6624490 1007/978- 3- 3-2----66666666 49841-8811-1
URL https::/doi.
Wil M. P. van der Aalst, Arthur H. M. ter Hofstede, Bartek Kiepuszewski, and Alistair P. Barros. Work- flow patterns. Distributed Parallel Databases, 14(1):5-51, 2003a. doi: 10.1123/A:10222837772222 URL https:://// / doi. org/ 10. 1023 / / 102288372720S
Wil M. P. van der Aalst, Boudewijn F. van Dongen, Joachim Herbst, Laura Maruster, Guido Schimm, and A. J. M. M. Weijters. Workflow mining: survey of issues and approaches. Data Knowl. Eng., 47(2):237-26)) 267, 2003b. doi: 10. 10.1016//0169-01 0000060000000000066666 1. URL https::/ddi.org// 101 2016/50110 69 023x (03) 00066-1.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen. A survey on large language model based autonomous agents. CoRR, abs/2308. 1432, 2023. doi: 10.48550/ARXIV. 2308. 1432. URL https::tt//.......t. //dp:::::/dd.... orgg /10. 18550/arxis 2308 11432.
Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory, 2024.
URL https: /. .rg. wtggabs/////e.oo. /249 07429.
16

---

Published as a conference paper at ICLR 2025
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian lchter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou, Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo. S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processngg Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http: /tattp://papess..ee nips, oc Abstract -bstraaa Conference html.
wwtpppppppeppceeeeeeeeee fillrsfiillee__eee htpppaapp////////// / 2022/ /hshh/
Lionel Wong, Gabriel Grand, Alexander K. Lew, Noah D. Goodman, Vikash K. Mansinghka, Jacob Andreas, and Joshua B. Tenenbaum. From word models world models: Translating from natural language to the probabilistic language of thought. CoRR, abs/2306. 02233366.126777 12672, 2023, doi: 10.48550/ ARXIV.2306. 12672. URL https: //w.i//oi///////////////... org/ /00 48500/ www.///a.ivvv... 2306. 12672.
Mengsong Wu, Tong Zhu, Han Han, Chuanyuan Tan, Xiang Zhang, and Wenliang Chen. Seal-tools: Self-instruct tool learning dataset for agent tuning and detailed benchmark. CoRR, abs/2405.08355, 2024. doi: 10.48550/ARXIV220000 2405. 08355.
URL https:/ /doi. httg/100.885//arxii
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Jun- zhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, Wensen Cheng, Qi Zhang. Wenjuan Qin, Yongyan Zheng. Xipeng Qiu, Xuanjing Huan, and Tao Gui. The rise and potential of large language model based 2023. doi: 10.48550/ARXIV23000000
agents: A survey. CoRR, abs/2309.0788 https:/ /ooi. wrw.o0.............o... 1..00.88550 48550/ arxiv. 2tt.v.23..0088800 07864.
URL
Jiannan Xiang, Tianhua Tao, Yi Gu. Tianmin Shu, Zirui Wang. Zichao Yang, and Zhiting Hu. Language models meet world models: Embodied experiences enhance language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: / wwwp/papppppppaaap/eeeeeeeeeeeee nips oc cc/paper_files httpse/fiielepeppeeeeaeee pappr/202333 htt///2222hhha// hash// httml.
Ruixuan Xiao, Wentao Ma, Ke Wang, Yuchuan Wu, Junbo Zhao, Haobo Wang, Fei Huang, and Yongbin Li. Flowbench: Revisiting and benchmarking workflow-guideed planning for llmebased agents. CoRR, abs/2406. ab4/2406..444844 14884, 2024. doi: 1144850WAXXIV.2446.148888 URL https: /doi.
240.. 200 14884.
Can Xu, Qingfeng Sun, Kai Zheng. Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. Wizardlm: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representaatins,, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: /twp//eeieee/.eeep////p/.. net/ Cerum?id-Cixh3 3NDgH.
Xiangyuan Xue, Zeyu Lu, Di Huang, Wanli Ouyang, and Lei Bai. Genagent: Build collaborative ai systems with automated workflow generation case studies on comfyui, 2024. URL https: //aaxiivorrg//eeeaaaaoaaaaooaaae htcp//bb/22// 2409. 01392.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang. Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma. Jianxin Yang, Jin Xu, Jingren Zhou, and et al. Qwen2 technical report. CoRR, abs/2407.10671 2024a. doi: 10.48550/ARXIV. 1407.10671. URL https: //tp:::::/d..oooo org/ 48550
John Yang. Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Stan- dardizing and benchmarking interactive coding with execution feedback. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023a. URL. http:/ hile:///papeee httpp/aappaaaaaaaa//ee//ee/aaees ni is.. ec cwtppapreeaaiiieeeeeee firffffillsseeeee paper/
17

---

Published as a conference paper at ICLR 2025
2023/ h11p:////11111110000033333311111 and Lid__Beccharks html.
Ke Yang, Jiateng Liu, John Wu, Chaoqi Yang, Yi R. Fung, Sha Li, Zixuan Huang, Xu Cao, Xingyao Wang, Yiquan Wang, Heng and Chengxiang Zhai. If LLM the wizard, then code the wand: A survey on how code empowers large language models to serve as intelligent agents. CoRR, doi: 1044550/ARXXV.2401.0002 URL htttpp:///oi.org//// 48550/arXiv 1401.00811
abs/2401.005 10012, 2024b.
Rui Yang. Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model to use tools via self-instruction. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Ad- vances in Neural Information Processing Systems 36: Annual Conference on Neural Infor- mation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023b. URL http: wwtppaappppee/aa///ee n ps. cac/paper_fiilie files/illeeeeeeeee haallespppprrr httppppppr///// 624 (11.77336666444222222ccccdddd..0 Attraaa---aaaaatttoootoreeee
html.
Zonghan Yang, Peng Li, Ming Yan, Ji Zhang, Fei Huang. and Yang Liu. React meets actre: When language agents enjoy training data autonomy. CoRR, abs/2403. 14589, 2024c. doi: 10.48550/ ARXIV.2403. 14589. URL https::tt..............tt /doii org/10. 48550/ 2403. 14589.
Zonghan Yang, An Liu, Zijun Liu, Kaiming Liu, Fangzhou Xiong, Yile Wang, Zeyuan Yang, Qingyuan Hu, Xinrui Chen, Zhenhe Zhang, Fuwen Luo, Zhicheng Guo, Peng Li, and Yang Liu. Position: Towards unified alignment between agents, humans, and environment. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024d. URL https: /ttp:p:///ppeeeee hwtppppeeeeeeeeeeeeeeeeeeeeeeeee net/ forum? diddiiddDDnaaaaa DcFFL.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scal- able real-world web interaction with grounded language agents. In Sanmi Koyejo. Mo- hamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Process- ing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http: http://aappps..e/ee nips cc/ sacpperrfillsseeeeeee wwwpe//spppppp/////e2.ee //hash/ Abst -ateaaaaetbo--aattatt-----ooooo Con.raacct-oooeeeeeeeeeeeeeee
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https:ttt..... opwnreview.........m........ net teee/fforue wwwrumiiddwwwwwlya
Junjie Ye, Guanyu Li, Songyang Gao, Caishuang Huang, Yilong Wu, Sixian Li, Xiaoran Fan, Shihan Dou, Qi Zhang, Tao Gui, and Xuanjing Huang. Tooleyes: Fine-grained evaluation for tool learning capabilities of large language models in real-world scenarios. CoRR, abs/2401.00744 2024. doi: (0.45550/ARXI..2000000 00741.
URL https: /ttp:::///o...oo www.oogggo........ 48w58855500110 2401. http:///ooooorooo./oooo
Yining Ye, Xin Cong, Shizuo Tian, Jiannan Cao, Hao Wang, Yujia Qin, Yaxi Lu, Heyang Yu, Huadong Wang, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Proagent: From robotic process automation to agentic process automation. CoRR, abs/2311. 2023, doi: 1048550/ARXIV.211.0000 URL https:://dl. org/10 w.t.0.85550/// wwt.//5////...... http://awww.....1....0.
Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Raghavi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. Agent lumos: Unified and modular training for open-source language agents. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). ACL 2024, Bungkok, Thailand, August 11-66, 2024, pp. 12380-12 12400. Association for Computational Linguisties, 2024. URI. https:tppp........p.....t : / antho ogy- org/ 2024 aacl- long. 670.
Jifan Yu, Xiaozhi Wang. Shangqing Tu, Shulin Cao, Daniel Zhang-Li, Xin Lv, Hao Peng, Zijun
Yao, Xiaohan Zhang, Hamming Li, Chunyang Li, Zheyuan Zhang, Yushi Bai, Yantao Liu, Amy Xin, Kaifeng Yun, Linlu Gong, Nianyi Lin, Jianhui Chen, Zhili Wu. Yunjia Qi, Weikai Li, Yong Guan, Kaisheng Zeng, Ji Qi, Hailong Jin, Jinxin Liu, Yu Gu, Yuan Yao, Ning Ding, Lei
18

---

Published as a conference paper at ICLR 2025
Hou, Zhiyuan Liu, Bin Xu, Jie Tang, and Juanzi Li. Kola: Carefully benchmarking world knowledge of large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: http:///peeevvvee ew net/ forum? d=RANN23
Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Yongliang Shen, Kan Ren, Dongsheng Li, and Deqing Yang. EASYTOOL: enhancing lim-based agents with concise tool instruction. CoRR, abs/2401.06201, 2024. doi: 10.48550/AKXV.240000 48550/arXiv 2401 06201.
URL
Mert Yüksekgönül, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and
James Zou. Textgrad: Automatic "differentiiaion" via text. CoRR, abs/2406.07496, 2024. doi: 10. 48500ARRX1.24460074966 URL https: http//dtiorg//////////e wwg/10..............o.o ..0..4455 48550/ hat.://axivv... wiw.:iiv...00.... 2406. 07496.
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agent- tuning: Enabling generalized agent abilities for 1lms. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 3053-3077. Association for Computational Linguisties, 2024a. URL https: //a/ccaannna aoclannthology org/ 2024. f Findings-aae acl .11..
Aohan Zeng, Bin Xu, Bowen Wang. Chenhui Zhang. Da Yin, Diego Rojas, Guanyu Feng. Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang. Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong. Mingdao Liu, Minlie Huang, and et al. Chatglm: A family of large language models from GLM-130B to GLM-4 all tools. CoRR, hts/2406.11795. 2024b. doi: 10.48550/ARXI.2246.11 12793. URL https: 2406 12793.
//doi. orggrrgg..... 0...48550///2
Zhen Zeng, William Watson, Nicole Cho, Saba Rahimi, Shayleen Reynolds, Tucker Balch, and Manuela Veloso. Flowmind: Automatic workflow generation with llms. In 4th ACM International Conference on Al in Finance, ICAIF 2023, Brookly, NY, USA, November 27-29, 2023, pp. 73-81. ACM, 2023. doi: 10.1145/3604237336669999 URL https:://oi... 3604237. 3626908.
1145/
Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, Zhiwei Liu, Yihao Feng. Tulika Awalgaonkar, Rithesh Murthy, Eric Hu, Zeyuan Chen, Ran Xu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Silvio Savarese, and Caiming Xiong. xlam: A family of large action models to empower ai agent systems, 2024a. URL https: / arxiv. org/ has///////// / 2009 03215.
Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. Aflow: Automating agentic workflow generation, 2024b. URL https: / /rxivv.rgg/////// abs/2410, 10762.
Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 13643 13658. Association for Computational Linguisties, 2024c. URL https: ac http///caaaanttocoooooooooo
org/ 2024 acl- long. 737.
Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng-Tze Cheng, Quoc V. Le, Ed H. Chi, and Denny Zhou. NATURAL PLAN: bench- marking lmms on natural language planning. CoRR, abs/2406 0452000 2024a. doi: 10.48550/ARXIV. 2406.04520. URL https: /doi. woo/oogg///0 /0/.00..000 48550/ att./arxivvv/ 2406. 04520.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bhnch and chatbot arena. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, IA, USA, December
19

---

Published as a conference paper at ICLR 2025
10 16, 2023, 2023. URL http://ppp/paaae........caaaeaea htt.://2laaaasssaae2222222222222 Ben.charks .tmll.
Yaowei Zheng, Richong Zhang, Junhao Zhang. Yanhan Ye, Zheyan Luo, and Yongqiang Ma. Lla- mafactory: Unified efficient finettunng of 100+ language models. CoRR, abx/2403.13372, 2024b. doi: 10.4850/ARRXV.240...353 13372.
URL https: /ttp///iiiiia/ otgp htttg//....0///a....vvooo
Wangchunshu Zhou, Yuchen Eleanor Jiang, Long Li, Jialong Wu, Tiannan Wang. Shi Qiu, Jintian Zhang, Jing Chen, Ruipu Wu, Shuai Wang, Shiding Zhu, Jiyu Chen, Wentao Zhang, Ningyu Zhang, Huajun Chen, Peng Cui, and Mrinmaya Sachan. Agents: An open-source framework for autonomous language agents. CoRR, abs/2309.078 2023. doi: 1044850/ARXX.22300007777 URL https: /tt///ddo doi. org/10. 48550/ arwiv..iiivvv////vvv 22.9 07870..
Wangchunshu Zhou, Yixin Ou, Shengwei Ding. Long Li, Jialong Wu, Tiannan Wang, Jiamin Chen, Shuai Wang, Xiaohua Xu, Ningyu Zhang. Huajun Chen, and Yuchen Eleanor Jiang. Symbolic learning enables self-evolving agents. CoRR, abs/2406. 18532, 2024, doi: 10.4855/ARXIV.24066 18532. URL https:tppp.........p. www.ddo..o...oooooooeoooooooeeee /1lg///..00 48550/550//a arxiv 2406. 18532.
Yuqi Zhu, Shuofei Qiao, Yixin Ou, Shumin Deng, Ningyu Zhang, Shiwei Lyu, Yue Shen, Lei Liang, Jinjie Gu, and Huajun Chen. Knowagent: Knowledge-augmented planning for Ilm-based agents. CoRR, abs/2403.03101 2024. doi: 10.4850/ARXX..200.55000 URL https: //oii/oi..// org/ 10 40044/55//222 artivv.iiivvvao///vva 2403. 03101.
Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. Gptswarm: Language agents as optimizable graphs. In Forty-first International Conference Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL
net 2ttp://wwwwweeeeeutmmmmmmaa
Max Zuo, Francisco Piedrahita Velez, Xiaochen Li, Michael L. Littman, and Stephen H. Bach. Planetarium: A rigorous benchmark for translating text structured planning languages. CoRR, abs/2407.03322 2024. doi: 10.48550/ARXIV.2407..353 URL https: /ttp///oi////////////o/ /or//.11..1/.....////o 48550/arXiv 2332-0331..
A APPENDIX
A.1
SOURCE DATA INFORMATION
In order facilitate better understanding of our paper, here we provide detailed exposition of the dataset utilized in our paper.
Held-in Tasks
ToolBench (Qin et al., 2024). Toolbench is a function call dataset generated by selecting several APIs from an API library and synthesizing instructions using GPT-3.5. Due to the potentially disparate nature of the extracted APIs, many instructions within Toolbench may pose challenges in logical comprehension. To address this issue, we manually establish templates and filter them based on the complexity of function invocations. Additionally, there is another version of ToolBench called StableToolBent (Guo et al., 2024b) that utilizes GPT-4 and caching mechanisms to achieve for end-to-end task evaluation.
stable API calls, which
ToolAlpaca (Tang et al., 2023). The construction method of ToolAlpaca is akin to that of ToolBench. The quality of instructions relatively higher, yet most instructions can be executed within 1-3 function calls. We filter out instructions with only single function call.
ALFWorld (Shridhar et al., 2021). ALFWorld is a household dataset requiring the agent to navigate through the room and manipulate objects. It includes uumaa--ndntated gold trajectories, which we directly utilize to construct node chains and workflow graphs.
WebShop (Yao et al., 2022). WebShop is an online shopping dataset in a website environment. We the gold trajectories collected through GPT-4 in Song et al. (2024) construct our benchmark.
20

---

Published as a conference paper at ICLR 2025
OS (Liu al., 2024b). OS is an interaction dataset based operating systems, where agents are required to complete operations through shell commands. We gather gold trajectories from AgentInstruct (Zeng et al., 2024a) to aid in the construction of our benchmark.
LUMOS (Yin et al., 2024). LUMOS is dataset that models agent planning-related datasets using a unified format. Within LUMOS, the OnePass planning data (LUMOS-O -oo aligns well with our workflow construction. We select the math, commonsense, and multimodal reasoning components as the foundation for building our reasoning tasks' workflow.
WikiHow (Koupaee Wang, 2018). WikiHow contains open-world planning tasks and complex linear process data. We collect and filter data based on task topics and directly construct graph workflows based on the process data.
Held-out Tasks
Seal-Tools (Wu et al., 2024). Seal-Took. is a function call dataset obtained entirely through self-instruction using ChatGPT.
InterCodeSQL (Yang et al., 2023a). InterCodeSQL is embodied dataset that generates SQL instructions based on user intent and interacts with a graph datahase.
A.2 HUMAN VERIFICATION
We divide the test data into five parts and invite five NLP volunteers to evaluate the quality of the node chains and workflow graphs based on the following principles:
1. Granularity. The decomposition the node chain should meet the smallest executable gramularity that be derived from the task description and action list. This means that nodes should not combine subtasks (indicating granularity too large) or introduce information that cannot be obtained from existing information (model self-association, indicating granularity
small).
2. Logic. The workflow graph should follow logical sequencing that satisfies the execution relation- ships between nodes.
Task Quality. The tasks themselves should not exhibit any obvious quality issues.
We discard data that does not adhere to the above criteria and obtain the final test set.
A.3 BENCHMARK STATISTICS
Figure 7 illustrates the statistics of our benchmark. Our training set comprises 18,679 instances, with data evenly distributed across four types: function call, groblem-solving. embodied, and open- grounded. The test set consists of 2,146 instances, with 33.69% dedicated to held-out tasks to evaluate the generalization capability of the trained model. Figure 8 the distribution of our benchmark based on the number of nodes in the workflow. The majority of the data is in the range of 2 to 10 steps, with smaller portion falling within the 10 to 20 steps range. The average number nodes across all data points is 4.17.
A.4 ABLATION STUDY
Here we investigate what changes in the performance of the agents would occur if we provide the gold node chains to the agents and task solely with predicting the dependencies between nodes (i.e., the edges of the workflow graph). The evaluation is still in a two-shot manner. The experimental results are shown in Table We can observe a clear improvement in performance after providing the gold node chain. Giving the gold node chain can alleviate issues related to granularity and explicitness (as defined Figure 4). However, predicting the graph relationships between nodes remains a challenging task, which contributes to an overall performance that is still subpar.
A.5 TRAINING SETUPS
We also apply the LlamaFactory (Zheng et al., 2024b) framework to train the models. We fine-tune Qwen-2-7B and Internl with full parameters using DeepSpeed (Rasley et al., 2020). We
21

---

Published as a conference paper at ICLR 2025
Train
Test
Arpugo 4.27
ADV
Past Bare
JLRN
2 TT
12NN
Total 18679
Total 2146
Function Call
Embodied
Held-out
Problem Solving
Open
Stops
Figure 7: Statistics of Our Benchmark. We also include held-out tasks in the test set.
Figure 8: Workflow Steps Distribution on the whole benchmark.
Table 5: Ablation Study. We select some representative models examine what will happen if we provide the gold node chains the agents and task solely with predicting the edges of the workflow graph. w/o represents traditional f'l graph without gold node chains and w/ stands for generating edges with the aid of gold node chains.
Model
Fumetion Call
Problem-Solving
Embodiid
Open-Grounded
Average
w/o
w/o
Claude 3.5
55.06
72.75
55,50
89.27
56.71
78.80
42,88
62.06
52,53
75.72
GPT-4
52.47
74.63
Qwen-2.7B
55,50
79.71
$2.13
74,81
46.25
50.54
20.89
42.93
43.69
62.00
InternI.M
67.63
57.80
45.03
62.20
Owen: 1,5-14B
46.86
65.51
43.89
77.29
44.14
57.68
21.44
39.10
39.08
59.89
Phi-3-medium
47.26
65.80
$4.85
80.98
49.99
52.21
23.77
33.33
43.97
58.08
Llama-33 3.1. 7.70
52.72
69.62
57.05
86.91
55.52
72.70
33.06
42,70
49.59
67.98
Qwen-2-7218
52.31
70.81
58.13
79.99
58.49
72.46
32.89
53.59
50.46
69.21
include two-shot prompt in the input to enhance the model's generalization during both training and testing. Both models utilize identical hyperparameters. The detailed hyperparameter settings are outlined in Table 6. All the experiments are conducted on 3 NVIDIA 80GB A100 GPUs.
Table 6: Detailed training hyperparameters used in our paper.
Name
Value
cutoff len
4,096
epochs batch size batch size device gradient accumulation steps
3
12
2
2
learning rate scheduler type warmup ratio b
le-5
cosine
0.1
true
A.6 CASE STUDY OF 01
Below are two cases we test on OpenAl's current most powerful reasoning model, 01 (OpenAI, 2024). In the first case, OI successfully identifies various subtasks but makes an error in predicting the dependency between node 1 and node 2/3 when generating the graph, failing to recognize their parallel relationship. In the second case, based on environmental priors, the agent needs to first locate the potato and then cool it using the fridge. Due to lacking this specific knowledge, 01 assumes the potato already inside the fridge, thus missing the subtask of finding the potato. These observations confirm what has been mentioned in Valmeekam al. (2024) that Large Reasoning Models (LRM) still can't plan. We analyze that the complexity of planning lies in the interaction with the environment, which requires substantial amount of environmental commonsense knowledge.
22

---

Published as a conference paper at ICLR 2025
Tasks like math reasoning or code generation only involve step-by-step deduction from the problem to the solution. Therefore, to enhance the planning ability LLM agents, integrating world knowledge (Yu et al., 2024; Qiao et al., 2024a; Guetta et al., 2024) world models (Dawid & LeCun, 2023; Hu Shu, 2023; Wong et al., 2023) into the agent system remains crucial. This also why a correct workflow serve prior knowledge to enhance agent planning.
Case Study of Ol (a)
Task: Please analyze the access_logs.tn file using the machine learning algorithm to detect any potential malicious activity. Afterwards, retrieve the network policy for the library with ID "wscDOqu63Giq regarding internet access. Finally, get the digital PR metrics for the 'Beauty Revolution' campaign that took place from '2022-01-01' to '2022-12-31' platforms such as Twitter, Facebook, and Instagram.
The tool list you can select from: [tool_]ish
Gold Nodes and Edges
Node:
1: Analyze access access_logs.tx for potential malicious activity using machine learning.
2: Retrieve network policy for library "wscDOqa63Giq regarding internet access.
3: Get digital PR metrics for 'Beauty Revolution' campaign from '2022-01-01 to '2022-12-
31' on Twitter, Facebook, and Instagram.
Edge:
(START, 1) (START, 2) (START, 3) (L, END) (2, END) (3, END)
Predicted Nodes and Edges
Node:
a. Analyze the sccess_logs..e file for potential malicious activity using machine learning algorithm.
b. Retrieve the network policy for the library with ID 'wscDOqa63Giq' regarding internet
access.
Get the digital PR metrics for the 'Beauty Revolution' campaign from '2022-01-01" to "2022-12-31" on Twitter, Facebook, and Instagram.
Edge:
(START, (1, (2, END) (3, END)
Case Study of OI (b)
Task: You are in the middle of room. Looking quickly around you, you see cabinet 20, cabinet 19, cabinet 18, cabinet 17, a cabinet 16, cabinet 15, cabinet 14, a cabinet 13, cabinet 12, a cabinet 11, a cabinet 10, a cabinet 9, a cabinet 8, a cabinet 7, cabinet 6, cabinet 5, cabinet 4, cabinet 3, cabinet 2, cabinet I, coffeemachine 1, countertop a countertop 1, a diningtable 2, a diningtable a drawer 6, a drawer 5, a drawer 4, drawer drawer 2, drawer 1, fridge garbagecan 1, microwave 1, sinkbasin stoveburner
4, stoveburner 3, a stoveburner 2, a stoveburner 1, and a toaster Your task is to: put cool potato in garbagecan.
The action list you can select from: go to recep
2. task obj from recep
put obj in/on recep
4. open recep close recep
6. toggle obj recep
clean obj with recep
8. heat obj with recep
cool obj with recep
where obj and recep correspond objects and receptacles.
Gold Nodes and Edges
23

---

Published as a conference paper at ICLR 2025
Node:
1: go to where the potato is located
2: take potato from where is located
3: to fridge
cool potato with fridge
5: go garbagecan
6: put potato in/on garbagecan.
Edge:
(START, (). (2. (3, 4) (4. (5, 6) (6, END)
Predicted Nodes and Edges
Node:
Go fridge
Take cool potato from fridge 1
to garbagecan
Put potato in garbagecan 1
Edge:
(START, 1) (L. (2, (3, 4) (4, END)
A.7
PROMPT FOR BENCHMARK CONSTRUCTION
Prompt for Node Chain Generation
Function Call Tasks
Here is multi-hop query and segment of its solution trajectory in conversation format. Please summarize the content of the last step based on the "Action" results, and generate node task. Node tasks should focus the API related the current action, and there should be no duplication between node tasks, make sure the task clear, concise accurate, and specific.
Please note that the trace may contain some API call errors. Please ignore the errors and focus the last Thought and Action step. Node task focuses more on planning rather than the specific API call or the results after execution so that specific API names should not appear. Here are two examples for you:
Examples
Now it's your turn.
Query: Query
Trajectory: Trajectory Segment
Embodied Tasks
I will provide you with analysis of successful trajectory of a task that interacts with the environment. Please identify the key factors that contribute to success. Based on this analysis, you need generate workflow help increase the success rate of future endeavors. The available actions are Action List
Your output should be sequence of subtasks that you would take complete the task and in the format of: Workflow: A -> B - C Here are two examples for you:
Examples
Now it's your turn.
Query: Query
Trajectory: Trajectory
Prompt for Workflow Graph Construction
You are a planner who is good at task planning. Next, will give you a task and some node of the subtasks. Some subtasks may not be dependent on each other, while others may have
24

---

Published as a conference paper at ICLR 2025
dependencies. Please convert these nodes of subtasks into topology diagram based on the task relevance in the workflow. The Graph should start with the START node, and end with the END node.
Here are two examples for you:
Examples
Now it's your turn.
Task: Task
Nodes: Subtask Nodes
A.8 LIMITATIONS
This paper still has certain limitations that must be acknowledged: a) While we have enforced strict quality control on the node chain and workflow graph, is inevitable that some queries themselves may have quality issues. The synthesis of complex queries remains an unresolved issue, which we leave for future work. b) All our data is collected from existing general domain datasets, inevitably missing scenarios that not covered. For example, some tasks require heterogeneous actions (e.g. needing both function calls and embodied actions) to be completed. c) In addition to natural language, workflows can also be represented in code form (e.g. PDDL) (Zeng al., 2023; Li et al., 2024c; Zuo et al., 2024), which we plan to incorporate in the future. d) Our workflow currently follows a one-pass generation paradigm. In the future, plan introduce iterative paradigm where the workflow can be iteratively generated and evolve based on environmental feedback. e) Workflows in this paper assume that all nodes need be traversed complete the whole task. We do not cover some scenarios where choices, loops, or more complex structures (van der Aalst et al., 2003a) are in the graph.
A.9 ANALYSIS OF THE TOPOLOGICCAL. ORDERS NUMBER NODE CHAIN EVALUATION
Due to the high time complexity of matching all the topological sorts of graph, especially when there are many parallel nodes in the graph, we opt for sampling approach to save time on matching. We analyze the distribution of the number of topological sorts for all workflow graphs in the test set, as shown in Table 7. It can be observed that the majority of workflows have number of topological sorts within 10, and data exceeding 20 quite sparse. Therefore, striking balance between computational efficiency and evaluation accuracy, we chose 20 as the sampling quantity.
Table 7: The distribution of the number of topological sorts for all workflow graphs in the test set.
Topo Num
10
20
< 50
S
100
Percentage 86.39%
92.82%
96.01% %
98.22%
98.32%
25


Reviews:
[{"review_id": "RiXHp8YVfZ", "reviewer": "Reviewer_jY5E", "summary": "This paper introduces WORFBENCH, a unified benchmark designed to evaluate the workflow generation capabilities of large language models (LLMs) across diverse scenarios and complex graph-based workflow structures. It also presents WORFEVAL, a evaluation protocol using subsequence and subgraph matching algorithms for accurate assessment of workflow generation is proposed. Author claimed following key contributions: 1)new features (Multi-faceted Scenarios and Complex Workflow Structures, Strict Quality Control); 2)WorFEval Evaluation (utilizes advanced matching algorithms to assess LLM performance on both linear and graph-based workflows quantitatively); 3)Comprehensive Experiments", "questions": "Q1)Is the training code included in the supplemental material? If yes, where to find and run it\nQ2)The paper provides some number such as ”1k training samples, 2146 test samples..” → how significant of these numbers towards the task and state of the art?\nQ3)How are the gold nodes and edges checked for correctness?\n\n-------------------\nSome questions were answered by authors, so, I raised the score to 6", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "S1)The problem highlighted by the paper is valid and emerging\nS2)The dataset has some interesting features\nS3)The experiments seem to be extensive", "weaknesses": "W1)the evaluation scores f1_chain and f1_graph in Section 2.4 were introduced as the measures for all evaluations in Section. But they are given without solid foundation why they are formulated and the right measures for the workflow chain/graph. \nW2) Quality control protocol is very subjective and manual, it’s difficult to judge the quality of data of the benchmark\nW3) Many technical details are not very clear (see questions)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DMEcmXHPhw", "reviewer": "Reviewer_eFu6", "summary": "This paper proposed a new benchmark to evaluate the abilities of LLMs in generating agentic workflows.\nThe insight of this paper lies in that the agentic workflows can be considered as a directed acyclic graph and therefore the generation of the workflow can be formulated as the generation of the nodes and edges in the graph.\nThe evaluation is done by measuring the similarity of the model generated workflow graph to the ground graph.\nThe experimental results across multiple models demonstrate that current models cannot handle the planning on the workflow graph level very well, and the most advanced models are still not performing on a good level on this benchmark.", "questions": "1. What if the model is given the ground truth nodes and they only need to predict the edges between the nodes? Would this improve the performance? \n2. Does the benchmark contains tasks that can be solved by using multiple different workflow graphs? Can the current evaluation metrics account for it?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of evaluating agentic workflow generation is interesting and novel.\n2. The paper puts in the work to evaluate a wide range of models.\n3. The evaluated scenarios cover a wide range of agentic use cases.", "weaknesses": "1. The matching between the model generated nodes and the ground truth nodes are done by using sentence bert. This may introduce errors in the evaluation step if the matching is not correct. \n2. The evaluation metrics mostly focus on the similarity to a ground truth graph, but not on how the generated workflows can complete the task correctly. There might be more than one graph than can complete the given task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V9Yn9EDfUi", "reviewer": "Reviewer_Dn7v", "summary": "The paper presents WORFBENCH, a benchmark designed to evaluate LLM-generated workflows in complex reasoning and planning tasks. It introduces subsequence and subgraph matching algorithms for assessing workflows across diverse and intricate planning scenarios. Experiments highlight a notable performance gap between linear and graph-based workflows, with even GPT-4 showing a 15% deficiency in graph planning. It also reveals that well-structured workflows can improve downstream task performance and inference efficiency.", "questions": "This paper introduces WORFBENCH, a benchmark designed to evaluate LLM-generated workflows across diverse scenarios and complex graph structures, utilizing subsequence and subgraph matching algorithms for precise assessment. The evaluation is comprehensive, offering valuable insights into how structured agent workflows can enhance LLM planning capabilities. This work makes a significant contribution to the agent community. However, regarding insights of the graph-based agent workflows, I have several questions. \n- In the evaluation, the authors consider different actions as nodes across scenarios such as function-calling and embodied tasks. Have you considered more complex scenarios with heterogeneous action nodes, where some nodes represent function calls while others represent embodied actions? An evaluation or discussion on this point could be interesting. \n- More information on the training process to embed structured workflows as knowledge in LLMs would be helpful. For example, is there any insight of which training method can be more beneficial to training the LLMs with well-structured graph workflows, such as instruction-tuning or reinforcement learning through self-exploration?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The writing and presentation is good.\n- The motivation of standardizing the evaluation of agent workflow is impressive.\n- The evaluation is extensive and the insights from the evaluation are helpful.", "weaknesses": "N/A", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1FQHWg8zFB", "reviewer": "Reviewer_EjZY", "summary": "This paper proposes extending the evaluation of workflows in agentic workflow contexts from purely sequential workflows to a slightly richer graph structure that include parallelism, represented as DAGs. \n\nI think this is an important step, but also just a first step into much needed generalization from agentic workflows to real-work scenarios that often involve much more complex graph structures. The area of analysis workflow analysis and optimization of workflows has existed as a rich and mature area of study in the research fields of business process management and process mining [1,2], and decades of work has been done in those fields to study various representations of workflows. One difference is that those communities originally focused on representations of workflows that represent business processes and/or human execution of work, while the focus here is on workflow representations of LLM execution steps. I believe that there is no meaningful difference between the two from a workflow perspective.\n\nI applaud this paper for taking a first step in the direction of bringing those fields closer together, as there seems to be much to gain.\n\n**References**:\n\n[1] van der Aalst, W. M. P., Van Dongen, B. F., Herbst, J., Maruster, L., Schimm, G., & Weijters, A. J. (2003). Workflow mining: A survey of issues and approaches. Data & knowledge engineering, 47(2), 237-267.\n\n[2] van der Aalst, W. M. P. (2016). Data science in action (pp. 3-23). Springer Berlin Heidelberg.", "questions": "- In WorkFEval, nodes are represented using sequence-BERT, where it is checked if the similarity of the predicted and the ground truth node *at the same index* is above some similarity threshold. I wonder how this works within a parallel block. Take for example the Graph Workflow of Figure 1 as example: nodes 1 and 2 are in parallel and if the ground truth workflow and we also have a predicted workflow that has two parallel nodes at the start, then how do we know which predicted node to similarity-match to which node from the ground truth graph?\n- There is over a decade of work in workflow graph similarity metrics in the business process management literature. See e.g. [8] for a survey. Have authors considered using any of those existing methods out-of-the-box for comparing the ground truth workflow graph to the predicted workflow graph?\n- Comparing the node chain to the gold workflow graph: it seems that this problem can just be reduced to the reachability problem in Petri nets. Since the chosen DAG representation is just a marked-graph, there is a well-known result from Petri net theory that reachability in marked graphs is solvable in polynomial time [9]. The current implementation decision of generating all possible topological sequences seems exponential in time (in the case where all nodes are parallel). This may perhaps be OK with the size of DAGs that we practically encounter today. I wonder if authors envision that in the future we could be dealing with DAGs that are large enough that this may become a limitation? (I believe that the alignments algorithm can provide a more efficient comparison of node chain to workflow graph and also provide more principled metrics for this problem (see [10, 11]).\n\n**References:**\n\n[8] Schoknecht, A., Thaler, T., Fettke, P., Oberweis, A., & Laue, R. (2017). Similarity of business process models — a state-of-the-art analysis. ACM Computing Surveys (CSUR), 50(4), 1-33.\n\n[9] Esparza, J., & Nielsen, M. (1994). Decidability issues for Petri nets. Petri nets newsletter, 94, 5-23.\n\n[10] van der Aalst, W. M. P. , Adriansyah, A., & van Dongen, B. (2012). Replaying history on process models for conformance checking and performance analysis. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 2(2), 182-192.\n\n[11] van Dongen, B. F. (2018). Efficiently computing alignments: using the extended marking equation. In Business Process Management: 16th International Conference, BPM 2018, Sydney, NSW, Australia, September 9–14, 2018, Proceedings 16 (pp. 197-214). Springer International Publishing.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper distinguishes gaps between sequence planning capabilities and graph planning capabilities of LLM agents that were previously unknown. This is an important finding. \n\nMore importantly, I see great value in the steps that this paper has taken (even if limited) to extend (benchmarking of) agentic workflows from purely linear workflows to slightly richer workflows that also cover parallel execution. This is a small step towards integrating decades of progress of workflow analysis from areas outside of the LLM and machine learning communities into agentic workflows.", "weaknesses": "- The paper chooses to formalize workflows as DAGs. This representation is able to capture both sequential execution as well as parallel execution, but it still is unable to capture many other relevant structures that play a role in real-life scenarios. A DAG is for example unable to represent a decision point (choice), and unable to represent loops (repeated execution). I refer to [3] for foundational work on workflow patterns that contains a rich collection of workflow patterns, of which only patterns 1, 2, and 3 are captured in the current representation.\n\n- Modern representations of workflows in the business process management community include so-called workflow-nets [4] (a subclass of Petri nets [5]), and BPMN [6]. Note that the DAG formalism proposed in this paper is also equivalent to a subclass of Petri nets, namely the class of so-called “marked graphs” [7]. While it seems natural to study prior literature in workflow analysis outside of the LLM context when extending (evaluation of) agentic workflow to rich workflow patterns, this seems to be lacking, which is a missed opportunity. I encourage authors to at least incorporate some of these works into their related work section.\n\n- Minor: Figure 2 is small and hard to read.\n\n**References**:\n\n[3] van der Aalst, W. M.P., Ter Hofstede, A. H., Kiepuszewski, B., & Barros, A. P. (2003). Workflow patterns. Distributed and parallel databases, 14, 5-51.\n\n[4] van der Aalst, W. M.P. (1997). Verification of workflow nets. In: International Conference on Application and Theory of Petri nets (pp. 407-426). Springer.\n\n[5] Peterson, J. L. (1977). Petri nets. ACM Computing Surveys (CSUR), 9(3), 223-252.\n\n[6] Dijkman, R. M., Dumas, M., & Ouyang, C. (2008). Semantics and analysis of business process models in BPMN. Information and Software technology, 50(12), 1281-1294.\n\n[7] Commoner, F., Holt, A. W., Even, S., & Pnueli, A. (1971). Marked directed graphs. Journal of Computer and System Sciences, 5(5), 511-523.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hLjLpQpIAH", "reviewer": "Reviewer_nenD", "summary": "This paper introduces WORFBENCH, a workflow generation benchmark designed to assess LLMs' ability to handle multi-faceted scenarios and complex graph-structured workflows. The experiments reveal significant gaps in LLM performance, especially in graph-based workflow prediction, which offer some interesting insights.", "questions": "- 1. It seems that GPTSwarm [3] and TextGrad [4] offer an effective solution for DAG-based task solving, improving workflows in graph-based settings. It would also be helpful to discuss related work, and suggest supplementing at least 3 related papers with earlier studies on workflow generation or graph-based agentic systems like [3], [4].\n\n[3] Zhuge, Mingchen, et al. \"GPTSwarm: Language Agents as Optimizable Graphs.\" Forty-first International Conference on Machine Learning.\n\n[4] Yuksekgonul, Mert, et al. \"TextGrad: Automatic\" Differentiation\" via Text.\" arXiv preprint arXiv:2406.07496 (2024).\n\n- 2. The difficulty of this task is not very challenging. I suggest adding (or splitting) a hard set of tasks that are visibly more difficult and showing the results. \n\n- 3. To be honest, Q1 in the paper did not provide much insight. It seems to this point that is more like common sense and doesn’t require much space to verify.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- 1. The scope of this paper makes sense, and evaluating the workflow is an important factor for agentic task solving.\n\n- 2. Figure 2 clearly presents the main idea of the paper.\n\n- 3. This benchmark includes some comprehensive topics of different agentic tasks.", "weaknesses": "- 1. A concern is that similar benchmarks have already been proposed but are not compared and mentioned in this paper, which may affect its overall soundness. For example, PlanBench [1], published two years earlier, covers similar scopes, and CAT-Bench [2], released four months before the ICLR submission, also overlaps in focus, etc.\n\n[1] Valmeekam, Karthik, et al. \"Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[2] Lal, Yash Kumar, et al. \"CaT-BENCH: Benchmarking Language Model Understanding of Causal and Temporal Dependencies in Plans.\" arXiv preprint arXiv:2406.15823 (2024).\n\n- 2. It is unsure about the specific usage of the dataset. Since it is widely recognized that workflows are difficult to benchmark and planning can be dynamic, it is hard to determine if one decision path is truly superior to others as tasks become more complex.\n\n- 3. There is a lack of discussion on whether the datasets remain reliable as the number of planning steps increases. Does the generated workflow still prove useful in such cases? This might introduce significant hallucinations, potentially harming task-solving effectiveness. \n\n- 4. The results in Table 3 are not convincing. If adding workflows (+W) improves performance so significantly, wouldn't it be logical to have these models first generate the workflow and then run the experiments (for many previous projects or papers)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)