WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models
Authors: Shengda Fan
Xin Cong
Yuepeng Fu
Zhong Zhang
Shuyan Zhang
Yuanwei Liu
Yesai Wu
Yankai Lin
Zhiyuan Liu
Maosong Sun
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Process Automation
Workflow
Tool Learning
Primary area: generative models

Abstract: Recent advancements in large language models (LLMs) have driven a revolutionary paradigm shift in process automation from Robotic Process Automation to Agentic Process Automation by automating the workflow orchestration procedure based on LLMs. However, existing LLMs (even the advanced OpenAI GPT-4o) are confined to achieving satisfactory capability in workflow orchestration. To address this limitation, we present WorkflowLLM, a data-centric framework elaborately designed to enhance the capability of LLMs in workflow orchestration. It first constructs a large-scale fine-tuning dataset WorkflowBench with 106, 763 samples, covering 1, 503 APIs from 83 applications across 28 categories. Specifically, the construction process can be divided into three phases: (1) Data Collection: we collect real-world workflow data from Apple Shortcuts and RoutineHub, transcribing them into Python-style code. We further equip them with generated hierarchical thought via GPT-4o-mini. (2) Query Expansion: we prompt GPT-4o-mini to generate more task queries to enrich the diversity and complexity of workflows. (3) Workflow Generation: we leverage an annotator model trained on collected data to generate workflows for synthesized queries. Finally, we merge the synthetic samples that pass quality confirmation with the collected samples to obtain the WorkflowBench. Based on WorkflowBench, we fine-tune Llama-3.1-8B to obtain WorkflowLlama. Our experiments show that WorkflowLlama demonstrates a strong capacity to orchestrate complex workflows, while also achieving notable generalization performance on previously unseen APIs. Additionally, WorkflowBench exhibits robust zero-shot generalization capabilities on an out-of-distribution task planning dataset, T-Eval. Our data and code are available at https://github.com/OpenBMB/WorkflowLLM.

Published as a conference paper at ICLR 2025
WORKFLOWLLM: ENHANCING WORKFLOW ORCHES- TRATION CAPABILITY OF LARGE LANGUAGE MODELS
Shengda Fan' Xin Cong Yuepeng Fu", Zhong Z Shuyan Zhang", Yuanwei Liu", Yesai Wu?, Yankai Lin I Zhiyuan Liu?, Maosong Sun?
*Remmin University of China TTsinghua University "The University Manchester
"Wuhan University fanshengda, yankai lin@ruc wewneeruc.ed....oo cn, congxin1995 & wtt.emmmmaaaaaiaaeeeee.mme www..ssinguuaeeeoo edu,
yanka
ABSTRACT
Recent advancements in large language models (LLMs) have driven revolution- ary paradigm shift in process automation from Robotic Process Automation to Agentic Process Automation by automating the workflow orchestration procedure based on LLMs. However, existing LLMs (even the advanced OpenAl GPT-4o) are confined achieving satisfactory capability in workflow orchestration. To address this limitation, we present WorkflowLL.M. a data-centric framework clab- orately designed to enhance the capability of L.LMs in workflow orchestration. It first constructs large-scale fine-tuning dataset WorkflowBench with 106, 763 samples, covering 1, 503 APIs from 83 applications across 28 categories. Specif- ically, the construction process can be divided into three phases: (1) Data Collec- tion: we collect real-world workflow data from Apple Shortcuts and RoutineHub, transcribing them into Python-sty code. We further equip them with generated hierarchical thought via GPT-4o-mini. (2) Query Expansion: we prompt GPT-4o- mini to generate more task queries to enrich the diversity and complexity of work- flows. (3) Workflow Generation: we leverage an annotator model trained on col- lected data to generate workflows for synthesized queries. Finally, we merge the synthetic samples that pass quality confirmation with the collected samples to ob- tain the WorkflowBench Based on WorkflowBench,
fine-tune Llama-3.1 1-8B to obtain WorkflowLlama Our experiments show that WorkflowLlamaa demon- strates strong capacity orchestrate complex workflows, while also achiev- ing notable generalization performance on previously unseen APIs. Additionally, WorkflowBBenc exhibits robust zero-shot generalization capabilities on an out- of-distribution task plamning dataset, T-Eval. Our data and code are available at attps://gitta: www. wwt./pee.w///////cctcct. BWw.B///RM/OORROOOLOO
INTRODUCTION
Process Automation (PA) (Cichocki et al. 1997). a long-standing pursuit of the human race, aims to automate repetitive tasks to minimize human labor and improve efficiency. Robotic Process Au- tomation (RPA), the current predominant PA technique, abstracts the repetitive task into workflow (i.e., a program that can execute automatically) by orchestrating various actions (e.g., functions or APIs) [Ivanči et al. 2019 Hofmann et al., 2020 Wewerka & Reichert 2020 Agostinelli et al.)
2020 Ferreira et al. 2020). While RPA successfully reduces the human labor via automated work-
flow execution, the process of orchestrating workflows still requires substantial manual effort. Re- cently, large language models (LLMMs) (OpenAI, [222] 2023) Touvron et al.) 2023a bj Dubey et al,)
2024) have achieved remarkable performance beyond natural language processing (Ahn al. 2022
Cheng et al. 22033 Qian et al. (2024). The emergence of LLMs has unveiled a paradigm shift trend.
moving from Robotic Process Automation Agentic Process Automation (APA) (Ye et al., 2023
Zeng et al. 2023 Huang et al. 2024; Wornow et al. 2024 Li et al. (2024) which automates the
workflow orchestration process by utilizing LLMs to build the workflow.
Indicates equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2025
Workflow@ench Construction
ChatoPY
jhbi
Lotp
Drncc
Graine
red shoedis
Figure 1: Overview of WorkflowLLM. It first constructs WorkflowBench through three-phase pipeline and fine-tunes WorkflowLlama which can generate workflows based on the user's query (appointment management in this case).
The paradigm shift is hindered by LLMs' limited ability to orchestrate complex workflows, lead- ing to two main limitations in current APA methods: (1) Limited Action Scale: LLMs, even ad- vanced models like GPT-4, can only orchestrate workflows with an average of 6.1 actions, falling short of real-world complexity requirements (Ye al.) (023). In contrast, Apple Shortcuts typically involves 70.4 actions.
(2) Simple Logical Structure: Most current methods focus on sequential actions (Yao et al., 2022 Qin et al., 2024, Chen et al. 2024), while real-world workflows require more complex structures, including nested branches/loops, as seen in Apple Shortcuts' average of 2.6 nested logical structures. Thus, there is an urgent need to unlock the workflow orchestration capability of LLMs to expedite the paradigm shift in process automation.
To address these challenges, we propose LorkflowLLLM, a data-centric framework including dataset construction, model training, and evaluation to enhance LLMs' workflow orchestration ca- pabilities (shown in Figure D. Specifically, we first construct Berch, the first dataset explicitly designed for workflow generation. WorkflowBench consists of 106, 763 instances, en- compassing 1, 503 APIS across 83 applications, structured through three primary phases:
Data Collection: We select shortcuts from RoutineHub as high-quality data sources because they represent a robust RPA application with numerous expert-developed workflows available. We curate 14, 771 buman-annotated,, high-quality shortcuts spanning 28 diverse categories, alongside associated metadata including titles, functionality descriptions, and API documentations. As the raw shortcuts are not suitable for I.LMs to process, and considering that Python allows convenient parameter passing and control logic (Wang et al.) 2024b), we transcribe the shortcuts into Python code. Subsequently, we prompt GPT-4o-mini to generate comments, task plans, and task queries at varying levels of granularity- from fine-grained to coarse-gras coarse-grained -eeigggaaiiiidiee to enrich the data with detailed thought processes and enhance the learning efficacy of LLMs (Wei et al., 2023).
Query Expansion: To enrich the diversity and complexity of workflows, we utilize GPT-40- mini to generate additional task queries. Specifically, we first sample applications with diverse functionalities and select their APIs, along with built-in APIs, to prompt GPT-4o-mini generate task queries that leverage these sampled APIs to accomplish specific tasks. To further ensure workflow complexity, we also sample real-world workflow examples as demonstrations to guide GPT-4o-mini in generating similar workflows.
Workflow Generation: Since existing LLMs, including GPT-4o, struggle with workflow gener- ation, we first train workflow annotator model on real-world shortcuts. We then use the trained annotator to generate workflows for expanded task queries. To maintain dataset quality, we per- form quality confirmation step. Specificallyy, GPT-4o-mini refines the workflows to fix minor bugs, followed by rule-based filtering to remove workflows with logical errors.
To evaluate the capability of LLMs in workflow orchestration, we employ two metrics: the reference-code-bbaed metric CodeBLEU and the model-based metric Pass Rate. Experimental results demonstrate that WorkflowLlama consistently and significantly outperforms all baselines, including GPT-4o even with the in-context learning technique, across both metrics under unseen in- structions and unseen APIs settings. Furthermore, WorkflowBende demonstrates strong generaliza- tion capabilities in out-of -fitrrribititiiionn (OOD) scenarios, particularly on the T-Eval benchmark (Chen et al. 2024), where it achieves an F1 plan score of 77.5%.
2

---

Published as a conference paper at ICLR 2025
Collection
Query Expansion
Werffen Domplls
Shortzuts
Filee
Transcription
Thought Comeration
Querry Exparssion
Afl
Gasnci
Collaction
Supervised Fine-tuning
Workflow Generation
Asnotstor Training
fp
Quality Corferection
Workfos
Workfow
Woriffoetooee
Generted Worifonn
Figure 2:
Illustration of our WorkflowLLM which contains three phases to construct Workflow- Bench, followed by the supervised fine-tuning phase to derive WorkflowLlama.
2 RELATED WORK
RPA has gained considerable attention for automating repetitive tasks in var- ious productivity scenarios (Ivančić et al.) 2019 Hofmann et al., 2020 Wewerka & Reichert [020)] Agostinelli et al. 2020: Ferreira et al. 2020). RPA predominantly relies handcrafted workflows (e.g.. programming recording human behavior), making them highly suitable for automating well- structured, routine processes (Herm et al.) 2020). However, such approaches require substantial efforts and in-depth domain expertise, resulting in high setup costs and limited adaptability. Re- cent advancements in LLMs have spurred interest in integrating these models into RPA enhance flexibility and reduce dependency on manual workflow creation. Ye et al. (2023) introduced the concept of APA, which utilizes LLMs to autonomously orchestrate workflows based on human in- structions. Subsequently, several studies have sought to apply APA in various domains, including travel planning (Xie et al., (2024), smartphone applications (Huang et al. 2024), enterprise automa- tion (Wornow al.] 2024)) financial question answering (Zeng et al. (023), and data analysis (Li cet al. (024). Despite relying on advanced LLMs (c.g., GPT-4), these approaches have often ex- hibited suboptimal performance, highlighting challenges faced by existing LLMs in workflow chestration. While Li et al. (2024) made an effort to fine-tune Mixtral-8x 7B (Jiang et al.) 2024), it could only orchestrate sequential workflows with an average of 15.6 actions, remaining insuffi- cient for real-world requirements. This work addresses a critical gap by proposing Workflow LLM framework enhance the workflow orchestration capabilities of LLMs meet real-world demands.
Process Automation
Tool Learning Workflow orchestration driven by LLMs frequently depends on external tools to extend their operational capabilities. Recent studies show that LLMs can acquire and utilize tools by learning from documentation, enabling them to tackle complex tasks beyond their native abili- ties (Wu et al. 2023] [Schick et al., 2024 Qin et al. 2023b 2024). This integration allows L.MSs to access real time knowledge and perform specialized operations, particularly for intricate pro- cesses (Yang al. 2023 Nakano al.) 2021 Qin al. 2023a Wang et al. 2024c Gao et al. 2023). To further enhance this capability, several efforts have introduced datasets to fine-tune LLMs for tool interaction (Zhuang et al., 2024 Qin al. 2024 WWang al., 2024a). However, they have a limited action scope and lack complex logic control, focusing primarily on dynamically se- lecting APIs and filling parameters. Compared to tool learning scenarios, orchestrating workflows demands more sophisticated planning and reasoning that current LLMs have yet to fully realize. In response to these limitations, we present WorkflowLLMN to significantly improve LLMs' capabili- ties in workflow orchestration. Besides, Shhn et al. (2024) also used Apple's Shorteuts but aimed to assess L.LMs' tool utilization ability. In contrast, we emphasize a different scenario, workflow orchestration and aim to enhance the workflow orchestration ability rather than evaluation alone.
LORKFLOWLLM
As Figure 2 shows, WorkflowLLM introduces a data-centric framework to enhance the capability of LLMss in workflow orchestration by constructing high-quality supervised fine-tuning dataset
3

---

Published as a conference paper at ICLR 2025
WorkflowBench. In this section, outline the dataset construction process, which is carried out in three distinct phases: Data Collection, Query Expansion, and Workflow Generation.
3.1 DATA COLLECTION
We first introduce Apple Shortcuts and RoutineHub, and describe how we crawl and filter to get high-quality data. We then convert the shortcuts into Python-style workflow code. Finally, we prompt GPT-4o-mini to generate hierarchical thoughts for each shortcut.
Apple Shortcuts and RoutineHuu Apple Shortcuts, as a representative application of RPA, is developed by Apple Inc. This tool facilitates the automation of a series actions, enabling users to efficiently perform a diverse range of tasks. The actions within Shortcuts are APIs provided by both built-in Apple applications, such as Safari, and third-party applications like OpenAI. Each application may provide multiple actions. For instance, OpenA/ provides APIs that facilitate voice conversations and text interactions with ChatGPT. Through simple drag-and-drop interface, users can construct complex workflows, such as navigating to the nearest coffee shop or downloading watermark freee images from TikTok.
HoutincHut is prominent community for sharing shorteuts, with a collection of thousands of shortcuts across both iOS and macOS platforms. All shortcuts on RoutineHub are categorized into 28 workflow categories (c.g.. Business, Health & Fitness, Productivity, etc). RoutineHub records the metadata of each shorteut (e.g., title, description, iCloud URL), providing valuable information.
Crawling and Filtering For each shorteut, we crawl the title, peveloper-provideer description, and iCloud URL linked Apple. As RoutineHub does provide the source code for these shortcuts, we further crawl it from their iCloud URLs. Besides, we merge shortcuts collected by Shortcuts- Bench (Shen et al. 2024), sourced from platforms like ShareShortcut and MacStories"] further expand the scale of our dataset. However, the source code of these shortcuts lacks detailed infor- mation about the involved actions, such as API metadata. Inspired by ShortcutsBench (Shen al.) 2024]. we extract action information from macOS's ' built-in definition files and third-pann party appli- cation interface definition files. For each API, we record its name, description, parameter names, parameter types, default values, return value types, and return value name, which provides valuable resource for L.LMs to efficiently interpret and utilize these APIs, even in zero-shot scenarios.
To ensure compatibility between the crawled shortcuts and the action interfaces, we implement stringent filtering mechanism to verify that all API calls are executed correctly. During this process, we identify that some shortcuts contain non-interpretable binary sequences as API parameters, po- tentially disrupting the training process of language models. To maintain data quality, we remove these samples from the dataset. As result, we curate final set of 14, 771 high-quality shortcuts, ensuring the reliability of the dataset for subsequent data expansion and model training.
Shorteuts Transcription The original shortcut source codes are written in property lists for- mat (Hummert & Humphries 2022), which sequentially encodes logical constructs like branches and loops. This encoding is notably different from the types of data commonly used in the pre- training of LLMS.. To address this gap, convert the shortcuts into abstract syntax trees (ASTs), apply pre-order traversal to transform them into Python code, with further algorithmic details pro- vided Appendix A Furthermore, the original shortcuts use hexadecimal strings variable names, leading to reduced semantic clarity. To improve interpretability, we use GPT-4o-mini to automat- ically reassign these variables with more contextually meaningful names, thereby enhancing the overall readability and utility of the code for further language model training. A typical comparison between property lists and Python code can be found in Appendix
Thought Generation
To provide informative guidance for LLMs in orchestrating workflows, we design three-level thought hierarchy from fine-grained to coarse-grained: (1) Low-level comments
We selected 114-4--mni-22224-07- due to its balanced performance and cost efficiency. /https//toutiinhhbb.oo
hhtpp://shareshooteusscomm whtns::/www..mmsstrrrssseeeeeeee
4

---

Published as a conference paper at ICLR 2025
APT Back
Moekieer Cate
(iphoord caps plans, cortento
AFI Derk
Wiltichhud Jettir so sente altee day)
9
nuch prigredd
Enghe
Perfere dippeee Cettee Call
Contrrct dee formiter Naneer ehiy
Withe putt ote Yeer's
oomant oighor cs
the
key stonis jage sauni seit
laffeaint Micoomen Brach Viir
Mitppccios 4ith Ocfestia 'ite Taet Vilan
Workdaww Cade
Figure 3: Illustration of data field composition in WorkflowBench comprising Task Query, API documentations, Task Plan, and Workflow code with Comments.
intended to clarify the purpose of each action within the workflow. (2) Median-level plans repre- sent an abstraction over a sequence of actions, outlining the collective goal of these steps. (3) High- level queries reflect the user's requirements, specifying the intended outcome without prescribing specific methods to achieve it. These three levels of thought are generated through bottom-up ap- proach. Specifically, given the transcribed workflow w, let the set of actions within the workflow be denoted as A, where each action a; E A corresponds to a Python function call. The documentation dj 'D associated with each action includes key information such as the function name, input and output parameters, and a description of its functionality. For each action d, generate a corre- sponding comment C₁ by prompting GPT-4o-mini. Subsequently, given the action set A {a,} and comments C } of workflow w. we prompt GPT-4o-mini to generate the corresponding task plan P. We combine the task plan P. the comments C, and the action set A of the workflow w to generate the high-level task query Q. This bottom-up approach, resembling the summarization task, may contribute to content reliability and potentially reduce the risk of hallucination.
(2, where the workflow w consists of the task query Q, action documentation DD for all involved actions, the task plan P, and all actions represented as annotated Python code A. A detailed example can be found in Appendix F
Finally, Figure 3 shows, each workflow w is represented
3.2 QUERY EXPANSION
After performing comprehensive statistical analysis the collected data, we find that the data exhibits significant complexity, with an average of 70.4 actions and 12 branches, surpassing the complexity of existing workffow-related benchmarks. However, the diversity of the data relatively low. Specifically, 40.3% of the workflows fall under the iiciiiiies category, and over 99% of the APIs used are Apple's 's built-in APIs (i.e., those classified is s_wwrrkf www.wwwwaaooooooooooooo........w ons APP).
Therefore, we intend to expand the dataset by focusing on two key aspects: (1) Diversity: making up for the lack of diversity real data and covering broad range of APIs and workflow categories to enhance the model's 's utility and robustness; (2) Complexity: matching the action scale and logical complexity of the real-world data to ensure that they can effectively represent real-world problems and orchestrate nodes accordingly. To this end, we sample APIs from diverse applications and multiple workflows with representative logical structures (e.g., whether they contain branches or loops) to synthesize additional queries.
To ensure that the number APIs the synthesized dataset aligns with real-workee usage, we sample APIs based on real-world distributions. Approximately [n/2] are drawn from Apple's 's built-in API set (c.g., openurl or sendemail), with the remainder from third-party applications (e.g., OpenAl). The total number of built-in and external APIs is thus
To ensure that the sampled APIs can interact coherently, we do not sample directly from the entire API set. Instead, first randomly select 1-5 applications and then choose all APIs from these selected applications. This method ensures that the selected APIs are functionally compatible and capable of representing real-world workflows.
5

---

Published as a conference paper at ICLR 2025
01.00
38
224
(a) Discributions the number of factions of collected data
(b) Distribttoons of weekflow caleguries of collected data
(6) Distribations applications of collected data
115
11:04
50
24
(d) Dietributions of the number of a'aciiuss of final data
(c) Distribucions of workflow cntegories of final data
(1) Distribatios of applications of final dats
Figure 4: Comparison of the distributions across workflow categories, APPs, and action counts in the collected data and the final dataset. The upper section shows the original data collected from Apple Shortcuts and RoutineHub, while the lower section presents the expanded dataset distributions.
The prompt used for GPT-4o-mini to synthesize queries consists of four components: (1) general prompt to describe the task query generation task, (2) documentations for the sampled APIs, (3) in-context examples from the collected data for reference, and (4) the workflow category which the query belongs to. By controlling the workflow category and in-context examples, we can ensure the diversity and complexity of the generated data. As seen from Figure g the synthesized query has a more balanced category distribution and uses more third-party APIs. Although most of the APIs used are still built-in APIs, this is reasonable considering that they carry necessary operations.
3.3 WORKFLOW GENERATION
To annotate the corresponding workflows of the synthesized queries effectively, we train an an- notator model based the collected shortcuts data to support more diverse applications and categories, while ensuring consistency with the real-world data as much as possible.
Table 1:
Detailed statistics of WorkflowBench Seed. refers to the collected data from Shortcuts. Train. and Test. refers the training set and the test set of WorkflowBern respectively.
Statistics
Seed,
Train.
Test.
Num. of Instances
14,771
105,573
1,190
Annotator Training First, we construct the supervised fine-tuning (SFT) dataset based on the collected human-labeled shorteuts. Specif- ically, each workflow data point comprises query Q, the corresponding action documenta- tion D, the task plan P. and the workflow rep- resented as annotated Python code Acommented- shown in Figure 3 we take the query Q, the corresponding ac- tion documentation D as the input to guide the model to generate a task plan P. followed by the step-by-step generation of the current thought (i.e., the comment e;) and the corresponding action a, which includes the action name and its associated parameters. We use the trained annotator to generate workflows A'' from synthesized queries.
Num. APPs
71
31
Num. of APIs
584
1,503
324
Num. Categories
28
28
Avg. Action
70.4
78.5
41.7
Avg. IF
12.0
7.4
7.9
Avg. LOOP
0.7
0.5
0.5
Avg. Nested Depth
2.6
2.7
2.1
During the SFT process,
Quality Confirmation Due to the limited accuracy of the annotator model, the generated work- flows may contain emrors to extent. For example, manually identify issues in A' (e.g., extrancous branches not relevant to the query and incorrect function call formats). To enhance the overall quality, prompt GPT-4o-mini with in-context samples refine both A'' commentedd and Pl

---

Published as a conference paper at ICLR 2025
Then, use rule-based filtering to remove workflows with fundamental errors. Specifically, remove samples that don't t incorporate code, don't I utilize the given APIs, or violate parameter con- straints of APIs. More details of quality control protocol can be found in Appendix C
Finally, we derive a synthesized dataset of 91, 992 instances, which is combined with the initially collected data to form the final Bench. It contains 106, 763 instances with 1, 503 APIs across 83 applications, which are used to train WorkflowLlama. The statistics of Workflow Bench listed in Table and the distribution comparisons of workflow categories, APPs, and the number of actions between the collected data and final data are demonstrated in Figure + From the statistical results, we can see that the synthetic data maintains complexity while expanding diversity.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Training Details We fine-tune the annotator and WorkflowL.ama on LLaMA-3.1-8B (Dubey ct al. 2024) for 3 epochs using the AdamW optimizer (Loshchilov & 2019). A linear learning rate scheduler is used with peak learning rate of 2 10 and warm-up ratio of 0.1. Each mini- batch contains 32 examples, and the maximum sequence length is set as 8, 192 tokens.
Baselines To provide comprehensive comparison, select several representative LLMs as baselines for our experiments. These baselines include proprictary models such as GPT-4o-mint and GPT-4o, well as open-source models like Qwen2-7B (qwel 2024), L88m and Llama-3.1- 70B (Dubey et al. 2024). Additionally, we apply in-context learning (ICL. (Dong et al. (022) with one random-sampled instance to these baselines better adapt them for workflow orchestration.
metrics and a model-based evaluation comprehensively evaluate the quality of the generated workflows. For reference-based metrics, we apply CodeBLEU (Ren et al. 2020) with four components:
Metrics the main experiments, we use both reference-code-baee
BLEU measures N-gram overlap for token-level similarity.
Weighted N-Gram Match assigns higher weights to critical code tokens like keywords.
Syntactic AST Match compares the Abstract Syntax Trees (ASTs) assess syntactic accuracy. Semantic Data-Flow Match evaluates logical correctness by comparing data-flow relationships between variables.
Together, these components provide a comprehensive evaluation of both syntactic and semantic aspects of the workflows. We follow Ren et al. (2020), setting the four components to 0.1, 0.1, 0.4, and 0.4, respectively, and calculate a weighted sum to obtain the CodeBLEU score. For model-basee evaluation, elaborately prompt GPT--o-mini as the automatic evaluator evaluate generated workflows' Pass Rate, which indicates the proportion of functionally correct workflows generated using only the provided APIs thus is format irrelevant. The evaluation prompt Appendix B.2
4.2 EFFECTIVENESS OF EVALUATOR
To validate the reliability of the GPT-4o-mini evaluator in terms of Pass Rate, sample 30 task queries and workflow code pairs for each model Table forming a human-cvaluated dataset of 330 instances (30 11 330). First, complete the given tasks only using the provided APIs. Then, human evaluators re-label the sampled data according the same criteria specified in Appendix [B.2] Ultimately, 268 instances labeled consistently by both the GPT-4o-mini evaluator and human evaluators, achieving an agreement rate of 81.2%, demonstrating the reliability and effectiveness of the evaluator.
GPT-4o-mini to label whether each instance could
4.3 MAIN EXPERIMENTS
Settings The main experiments are conducted on the test set of WorkflowBench. Ideally, by scaling both the quantity and diversity of instructions and unique tools within the training data, WorkflowLlama is expected generalize to novel instructions and APIs that are not seen during training. This is particularly important because it enables users to define custom APIs and allows WorkflowL adapt based solely the provided documentation. To evaluate this capability,

---

Published as a conference paper at ICLR 2025
Table 2: Performance comparison of various models on the of Workflow under the unseen instructions (ID) and unseen APIs (OOD) settings (%).
CodeBLEU
Pass Rate
Model
BLEU ID OOD
Weighted N-Gram
AST ID
Data-Flow OOD ID OOD ID
Overall
ID
0OD
OOD
ID
OOD
Proprictary Models
GPT--4omini
0.4
1.5
1.6
29.5
29.5
37.0
36.3
26.8
26.5
54.8
47.5
w/ ICL
0.5
0.5
1.7
1.8
35.3
34.4
35.1
34.2
28.3
27.7
66.0
57.7
GPT-4o
0.5
0.4
1.8
1.7
33.5
31.8
37.3
36.9
28.5
27.7
56.6
47.5
w/ ICL
0.5
0.5
1.8
1.8
37.1
35.3
38.0
36.6
30.2
30.0
67.5
57.6
Open-Source Models
Qwen2-7B
0.4
0.4
1.2
1.3
27.2
27.7
33.2
33.1
24.4
24.5
25.6
22.6
w/ ICL.
0.5
0.5
1.2
1.3
30.2
29.8
32.4
32.9
25.2
25.3
28.2
26.4
Llama-3.1
0.6
0.7
1.2
1.4
31.0
29.6
30.0
30.8
24.6
24.3
33.0
24.5
w/ ICI.
0.7
0.7
1.3
1.4
34.0
32.4
32.6
32.4
25.3
25.2
40.2
32.7
Llama-3.1-70B
0.4
0.4
1.4
1.5
29.9
30.0
37.8
37.6
27.3
27.2
55.4
42.3
w/ICL
0.4
0.4
1.6
1.5
34.1
32.9
39.1
38,4
29.5
28.7
67.6
61.4
WorkflowLlama (8B)
9.4
7.0
11.09
8.3
55.1
48.8
38.0
35.3
39.3
35.1
76.9
70.4
Workflow generalization performance considers an In-Distribution (ID) setting, which involves using the same set of APIs as those in the training data, and (2) Unseen APIs, considers an Out-Or-Distribution (OOD) setting, involving only 50 common APIs required to construct workflows and APIs that are absent from the training data. To minimize API retrieval errors, we use queries and golden APIs as inputs in both ID and OOD settings to assess the workflow orchestration capabilities of different models S
levels: (1) Unseen Instructions,
Main Results
The results are placed in Table from which we derive that:
Although multiple workflows can successfully complete query, there is positive correlation between the reference-free Pass Rate metric and the reference-based CodeBLEU metric. Given that the Pass Rate metric derived from GPT-4o-mini aligns with human evaluations over 80% of the time, CodeBLEU serves reliable proxy for evaluating workflow orchestration capabilities.
2. All models demonstrate certain capacity for workflow orchestration. This may stem from their inherent instruction-folloiing and code- geeegeeneertoonn capabilities. We find that models like GPT-4o and -laama-3.1-700 which perform better generic tasks, also excel in workflow orchestration. In addition, prompting with in-context samples significantly enhances the models' performance. We find that scores on text overlap metrics such as BLEU and weighted N-gram are low for all models. This is because these metrics only measure the degree of overlap between the candidate and reference texts, and are highly sensitive to implementation details.
The workflow orchestrate performance of WorkflowLlama even outperforms powerful closed- source models GPT-4o with ICL by large margin [ Specifically, WorkflowL.lama achieves 39.3% score on CodeBLEU and a 76.9% Pass Rate under ID settings, demonstrating the validity of our proposed WorkflowLLM framework and BarkflowBench dataset.
5. WorkflowL
demonstrates strong generalization capabilities. Even though it has not been trained on the same instructions or APIs, still sigmificantly outperforms the vanilla Llama-3.1 all metrics, ahead of close the more powerful foundation models. Notably, method achieves 35.1% CodeBLEU and 70.4% in Pass Rate, outperforming all strong baselines.
4.4 ANALYSIS OF WORKFLOW COMPLEXITY
To evaluate the models' ability to generate workflows varying complexity, we break down the per- formance of CodeBLEU according to the total number of actions, the number of branches and loops,
Additional experimental results and analysis using an L.M retriever are provided in Appendix D
Additional human-annotated evaluations demonstrate that Workflowl.aam achieves human rate of 71. 9, surpassing GPT-4o with ICI., which attains human pass of 65.9%.
8

---

Published as a conference paper at ICLR 2025
Number Branches
Natted Depth Reforence Code
Figure 5: Performance comparisons based on the number of actions, the number Branch & Loop, and the nested depth of the reference code.
and the nested depth of the reference code. As shown in Figure 5 the performance of all models deteriorates as the number actions or logical complexity increases, indicating the challenge of orchestrating complex workflows. However, across all levels of complexity, WorkflowLlam significantly outperforms all other models. Moreover, the relative performance of WorkflowLlam improves the complexity of the workflow increases, which demonstrates fine-tuning with Work- flowBench significantly enhances the model's ability to handle more complex workflows.
4.5
OUT-OFFDISTRIBUTTOO GENERALIZATION TO T-EVAL (CHEN ET AL., 2024
Settings To further evaluate the generaliza- tion capability of WorkflowLlamss we conduct experiments using an OOD benchmark, T-Eval, a widely-used benchmark to evaluate the multi- step decision-making capability of LLMs to uti- lize APIs. The original data format in T-Eval based on JSON strings, which differ sig- nificantly from the Python-based format em- ployed in WorkflowBench To ensure the evalu- ation metrics' consistency between ours and the original paper, we convert Workflow into JSON format while preserving the metadata of workflows and the specifics of queries. Sub- sequently, we retrain WorkflowLlama transformed dataset. We employ the F1 Score proposed in the original paper to measure the alignment with the reference API sequences.
Table Comparisons of Fl scores on the PLAN task of T-Eval. (Bold denotes the best score among models the same category.)
Model
F1
Proprictary Models
Claude2
84.9
GPT-3.5
86.6
GPT-4
86.7
Open-Source Models
Qwen-BB
63.1
Mistral-7B
64.9
the
Llama-3.1-88
68.2
Qwen- 14B
69.7
Llama-2-13BB
65.1
Vicuna- 13B
54,0
The results are shown in Table As observed. Workflow Llama demonstrates strong OOD generalization performance the T-Eval benchmark, despite being trained on differ- domains and tasks using different APIs. Notably, Workflowl.lam significantly outper- forms the vanilla Ilaama.1-88 as well as larger open-source models like Llama-2-70B and Qwen highlighting that fine-tuning with Workflow/Bench enhances the model's out-of- distribution planning ability.
Results
Baichuan2 13B
52.1
WizardLM-
42.7
Llama-2-70B
63.1
Qwen-72B
73.4
WorkflowLlama (8B)
77.5
4.6 ABLATION STUDY
Settings To assess the efficacy of WorflowBench's components, we conduct an ablation study under the settings of unseen instructions (i.e., the IDD setting).
Results Table[ppreeett the performance results when the model is trained under different condi- tions: without synthetic data, without the task plan P. without action-level comments C, and without both C and P. The experimental results reveal two key findings. First, the two types natural lan- guage thoughts enhance the reasoning capabilities of the model. Removing either type of thought

---

Published as a conference paper at ICLR 2025
Table 4: Ablation study results of Natural Language Thoughts on Workflow Orchestration (%).
CodeBLEU
Model
BLEU
Weighted N-Gram
AST
Data-Flow
Overall
WorkflowLlans
9.4
11.1
55.1
38,0
39.3
w/o Task Plan
9.1
10.7
53.9
36.6
38.2
w/o Comment
9.1
10.8
54,9
35.3
38.1
w/o Task Plan Comment
8.8
10.2
53.7
35.1
37.4
w/o Synthetic Data
7.8
9.4
53.5
35.4
37.3
Uses
a
colloctionnn atiplane mode gioonwrrd.ord ersuring smooth ext from the process everyting completed?
Hollucination
APls
Smpari tise
aalerr Heau MiJersrin, pwwestt www.eaaa--attttaaa....... ""aaa"" oatir satuar"
sontitas actsira gebllitaardn
that CISAAO JEWAA
Redundant Conditions
thane counee corriteaattion antionn, seteett peeemnteriittpppooeee
latactes Jaan JAIDET:
tae Nowpt13000
Peiiia Daancas Settent and 170ng de ClEENe F 26 JOEEEER
astiuete sirataso
17 sheer catte sor dececeed predivaty. this stae slack in insecte
a ramben rooter ponrfle aetian Winttorttttt
ooun adtact canter free contact to taiter decected
arlacted porte
wwctootteeseoooeooooooooo wugeeeeeaaaaaaa Lake
reprartting dftir interdo
ateatesd vertatte dctoutther
actuvttie balliin
anlyy tine coaa
1122t
Action Omission
Art stoot COCAEEL ON el crenciions poopls 2y3te.
In.weat www.ertoeeeeeeee.ooooooo
Figure 6: Case study of generated code between vanilla Llama-3.1 1.1-8- and WorkflowLlame
leads to a decline CodeBLEU performance. Second, training on large-scale synthetic data further improves performance, highlighting the effectiveness of the WorkflowBench expansion process.
4.7
CASE STUDY
To further illustrate the effect of fine-tuning Werkh, present typical example in Figure 16 In this case, the vanilla Llama-3.1 model exhibits two types of errors. First, the model does not adhere the given instructions for workflow orchestration, using APIs outside the pro- vided list, i.e., hallucination APIs. Specifically, uses the time a sleep () function instead of is www.fllwwww..ooo act ions.nnttto ceres_delas delay(
a timer. Second, due to its relatively weak workflow orchestration capabilities, the model fails to complete all user instructions. Specifically, does func-
activate airplane mode using the w.worrffowwwcce. iittioon_aiiie wtw..iiieaaaeemmeee...o. tion. Fine-tuning on WorkflowBende effectively alleviates these two issues. However, we observe that fine-tuning also introduces redundant actions. For instance, WorkflowLlama repeats the parsing check of the clipboard's content. We will address this redundancy problem in future work. To fur- ther demonstrate the large-seale node structure and complex logical framework of WorkflowBench, also include a generated workflow code with long sequences in Appendix G
CONCLUSION
In this paper, we present WorkflowBBech to enhance the capability large language models in workflow orchestration. By fine-tuning Llama-3.1-8B on WorkflowBench, we derive WorkflowL- lama which can achieve superior performance on the workflow orchestration task exceeding all comparable baselines. Moreover, we adapt our WorkflowLlams perimental results reveal the generalization ability of our constructed WorkflowBench However, Workflow has limitations, including the use of APIs limited shortcuts and the ab- sence of evaluation through actual execution (refer to Appendix H.. Nevertheless, we believe that the dataset have constructed holds significant potential to contribute advancements in APA.
the T-Eval dataset and the ex-
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
We sincerely thank all the anonymous reviewers for their valuable comments and constructive sug- gestions. This work was supported by The National Natural Science Foundation of China (No. 62376273), Beijing Nova Program (No. 20240484568), the Postdoctoral Fellowship Program of CPSF (Grant No. G2B20230343) and the China Postdoctoral Science Foundation (Grant No. 2023M741945).
REFERENCES
Qwen2 technical report. 2024.
Simone Agostinelli, Andrea Marrella, and Massimo Mecella. Towards intelligent robotic process automation for bpmers. arXiv preprint vrXiv:2001.00880 2020.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et Do as can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, et al. T-eval: Evaluating the tool utilization capability of large language models step by step. In Proceedings the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 9510-9529 2024.
Liying Cheng, Xingxuan Li, and Lidong Bing. Is gpt-4 good data analyst? arXiv preprint arXiv:230S.15033, 2023.
Andrzej Cichocki, Helal A Ansari, Marek Rusinkiewicz, and Darrell Woelk. Workflow and process automation: concepts and technology, volume 432. Springer Science & Business Media, 1997.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey in-context learning. arXiv preprint wrXiv2230.0002:: 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.22 1783, 2024.
Deborah Ferreira, Julia Rozanova, Krishna Dubba, Dell Zhang, and Andre Freitas. On the evaluation of intelligent process automation. arXiv preprint wwXiv:2001.02633 2020.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and prigram-aided
Graham Neubig. PAL: program aided language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICMI. 2023, 23-29 July 2023, Honolulu, Hawaï, USA, vol- ume 202 of Proceedings of Machine Learning Research, pp. 10764-10799. PMLR, 2023. URL https: httppppoccingg...c// mwr. wrwss/ /v0esss//v///////////// v202 gatfff/aa3ff...oooo/ooooooolll.. html
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6894-6910, 2021.
Lukas- Valentin Herm, Christian Janiesch, Alexander Helm, Florian Imgrund, Kevin Fuchs, Adrian Hofmann, and Axel Winkelmann. A consolidated framework for implementing robotic process automation projects. In Business Process Management: 18th International Conference, BPM 2020, Seville, Spain, September 13-18, 2020, Proceedings 18, pp. 471-487 Springer, 2020.
Peter Hofmann, Caroline Samp, and Nils Urbach. Robotic process automation. Electronic markets, 30(1):99-106, 2020.
Tian Huang, Chun Yu, Weinan Shi, Zijian Peng, David Yang, Weiqi Sun, and Yuanchun Shi. Promptrpa: Generating robotic process automation on smartphones from textual prompts. arXiv preprint 022:2240.002455. 2024.

---

Published as a conference paper at ICLR 2025
Christian Hummert and Georgina Louise Humphries. Property Lists, pp. 157-165. Springer Interna- tional Publishing. Cham, 2022. ISBN 777-3-0030994667-0 doi: 0010///9788300--4467777 URL URLhttps: https://tooss http::::/ddo..oooo/ wtt. 978 0001:/977----008----------- 08400-0--0------
Lucija Ivančić, Dalia Suša Vugee, and Vesna Bosilj Vukšić. Robotic process automation: system- atic literature review. In Business Process Management: Blockchain and Central and Eastern Europe Forum: BPM 2019 Blockchain and CEE Forum, Vienna, Austria, September 1-6, 2019, Proceedings 17, Pp. 280-295.. Springer, 2019.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 2024.
Zelong Li, Shuyuan Xu, Kai Mei, Wenyue Hua, Balaji Rama, Om Raheja, Hao Wang, He Zhu, and Yongfeng Zhang. Autoflow: Automated workflow generation for large language model agents. arXiv preprint arXiv:2407. 12821,, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL htt.hhttt///peeeeeeeee ew net. forum? d=
Reiichiro Nakano, Jacob Hilton, Suchir Balaji. Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted 2021.
question-answering with human feedback. ArXiv preprint,
OpenAl. OpenAI:
Introducing ChatGPT, 2022.
URL bttps://openai.com homp:/lo..ccmmoooo...
chatgpt.
OpenAl. Gpt-4 technical report, 2023.
Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. Chat- dev: Communicative agents for software development. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Pupers), ACL. 2024, Bangkok, Thailand, August 11-
16, 2024, pp. 15174-15186, Association for Computational Linguistics, 2024. URL https: 7/aclanthology: 2tw.:aattooooo.oooo//222....... acl-
810
Yujia Qin. Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang, al. Webepm: Interactive web search for chinese long-form question answering. arXiv preprint arXiv:2305.06849 2023a.
Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, et al. Tool learning with foundation models. arXiv preprint arXiv:2304.08334, 2023b.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xi- angru Tang. Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language mod- els to master 16000+ real-world apis. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https::/opeenreviee net/ www.eoorriiiiiii ForummiddddHgg200055
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.1022
2020.
Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves use tools. Advances in Neural Information Processing Systems, 36, 2024.
Haiyang Shen, Yue Li, Desong Meng, Dongqi Cai, Sheng Qi. Li Zhang, Mengwei Xu, and Yun Ma. Shortcutsbench: A large-scale real-world benchmark for api-based agents. arXiv preprint a3X 2024.
12

---

Published as a conference paper at ICLR 2025
Qwen Team. Qwen2.5: A party of foundation models, September 2024. /ttptt::://qwwnnln github.io/blogg / (ww.oo//wwwee......
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint urXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint S.Xiv:2337.0928800 2023b.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neu- ral Information P Processing Systems, 33:5776-5788. 2020.
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Ex- ecutable code actions elicit better LLM agents. CoRR, abs/2402.01030, 2024a. doi: 10.48550/ ARXIV.2402.01330. URL https:
httth:tts://eeee /ttp::://do..oo http://i..oogg///////
0..02.01030
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Exe- cutable code actions elicit better Ilm agents. In Forty-first International Conference on Machine Learning, 2024b.
Zilong Wang, Yuedong Cui, Li Zhong, Zimin Zhang, Da Yin, Bill Yuchen Lin, and Jingho Shang. Officebench: Benchmarking language agents across multiple applications for office automation. CoRR, abs/2407. 19056, 2024c. doi: 10.48550/AXXX12240..99966 10. 415448855aaii.... 2407. 19056
org/
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023.
Judith Wewerka and Manfred Reichert. Robotic process automation-a systematic literature review and assessment framework. arXiv preprint arXiv:2012.11951, 2020.
Michael Wornow, Avanika Narayan, Krista Opsahl-Ong, Quinn McIntyre, Nigam H Shah, and Christopher Re. Automating the enterprise with foundation models. arXiv preprint arXiv:2405.03710, 2024.
Chenfei Wu, Shengming Yin, Weizhen Qi. Xiaodong Wang. Zecheng Tang, and Nan Duan. Vi- sual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671, 2023.
Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: A benchmark for real-world planning with language agents. arXiv preprint arXiv:2402.01662, 2024.
Linyao Yang. Hongyang Chen, Zhao Li, Xiao Ding, and Xindong Wu. Chatgpt is not enough: En- hancing large language models with knowledge graphs for fact-aware language modeling. arXiv preprint arXiv:2306. /489, 2023.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du Izhak Shafran. Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. ArXiv preprint, abs/2210.03629, 2022.
Yining Ye. Xin Cong, Shizuo Tian, Jiannan Cao, Hao Wang, Yujia Qin, Yaxi Lu, Heyang Yu, Huadong Wang. Yankai Lin, al. Proagent: From robotic process automation agentic pro- cess automation. arXiv preprint arXiv:2311 2...:2311.1173 2023.
Zhen Zeng, William Watson, Nicole Cho, Saba Rahimi, Shayleen Reynolds, Tucker Balch, and Manuela Veloso. Flowmind: automatic workflow generation with 1lms. Proceedings of the Fourth ACM International Conference Al in Finance, pp. 73-81, 2023.
13

---

Published as a conference paper at ICLR 2025
Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. Toolqa: A dataset for 11m question answering with external tools. Advances in Neural Information Processing Systems, 36, 2024.
14

---

Published as a conference paper at ICLR 2025
A ALGORITHM OF TRANSCRIBING SHORTCUTS
Algorithm 1: Recursive Parsing of Property List to Construct Abstract Syntax Tree
Data: Shortcut file to be transcribed
Result: Abstract syntax tree of the actions
Initialize empty tree with root node and set current.node to root foreach action action list do
Determine action_tti ictiootype and mode from action
action. tope is Conditional then
HandleCondiiional(mdde, action)
else if action_type is RepeatEach then
HandleLoop(moce, action)
else if act.inn.ype is RepeatCount then
HandleL.opp(mode, action)
else if action.type is AtoossFrooMMent then
HandeeMMaeeiCass@mmdde,,, action)
else
HandleDefaul((action)
Function HandleConditionall mode, action):
if mode == 0 (start if) then
AddNode(action)
node else if mode == (else) then
Set current mode
Move wwwrent.rroeeoeeoo to parent node AddNode(action)
Set current_node to new node
else if mode == (end if) then
Move current node to parent node
Function (oode,,, action):
mode == (start loop) then
AddNode(action)
Set current_node to new node
else if mode (end loop) then Move current_node to parent node
Function HandleMatchCase mode, action):
mode == 0 (start match) then
AddNode(action)
Set current_nooe new node
else if mode (start case) then
current_node match node then
AddNode(action)
Set _ode to new node
else
Move current. node to parent match node AddNode(action)
Set current -noe new node
else if mode (end match) then
Move wwwrent.nodeeeo to parent node
Function (andte
AddNode(action)
Function AddNode(action):
Create new node with action
Append new node to current current.node mooe. children
Set parent new node to current.node
15

---

Published as a conference paper at ICLR 2025
B PROMPT DESIGN
All of our prompts were manually designed and iteratively refined during early experiments.
B.1 WORKFLOW ORCHESTRATION PROMPT
You are a very helpful AI assistant who can write corresponding Python main code based on user's query and usable Pythen function interface.
Please generate python main code based on the following query (query)
You can start by using natural language to plan your tool call strategy, and then generate the complete code. For example, Thought:
<tool call strategy>
Code:
python
<main code>
Note that your output should always include 'Code:
"python <main code>
".,, formatted accordingly.
Here are some useful function interface you may use: (apis_docs}
B.2 EVALUATOR PROMPT
You are a kindly code reviewer, I will provide you with a query, a list of allowed apis and piece of code to be reviewed, you help me to check if the code to be reviewed is compliant. with speci ficatiniaaions.
The requirements are as follows:
1. You **should return True even if the code implements additional
functionality not required in the query**, as long as it roughly implements the requirements code readability
the query.
2. We don't impose any requirements conventions. You **should return True as long as the reviewed code doesn't use disallowed function and reasonably accomplishes what is asked in the query in general terms** There's need to get strictly hung up the details.
naming
3. Return False if the code fails to fulfill the requirement in the query. e.g. if it is proposed in the query to turn down the battery level of the phone and the brightness of the screen, it is a failure to fulfill only any one of the functions.
4. Built-in python syntax such as 'if', "loop", "input ()", and () are allowed. Return False if the code uses **any external functions or apis** not allowed apis list and not built-in function such as input (), print For example, if function, this should be used. Any use of any other library like requests etc. is False.
I provide the is_workflow_opennzz
query: (query)
list of allowed apis: (apis)
code to review: (code)
16

---

Published as a conference paper at ICLR 2025
Your answer: [True or False with interpretation)
B.3 COMMENT GENERATION PROMPT
Shortcut is sequence of actons, where each action is APT call, to execute user-provided queries.
As a user-friendly and patient assistant, your task is provide a set of description of each line of the code scrippet. To save time, have retrieved all the lines exclusive of blank lines of the code snippet and listed as a dicticnary below the code.
Your answer should be in the json format as follows: json
"line x": "ddstripttooroo--lee==""" "line xt1": "desoription----linee x+1>", "...": " "..." "line xtn": of x
The code is (code)
The lines are (lines)
B.4 TASK PLAN GENERATION PROMPT
Based on this line by line description of the code, generate flowchart of a workflow by natural language.
This is the code: (code)
B.5 TASK QUERY GENERATION PROMPT
As a helpful assistant, please help me craft a query. This query, formatted as question, should describe the task user wants to complete and adhere to the following criteria:
1. One of the solution to the task described in the query could be the python code below.
2. It should be close to real-world problems requests.
3. It should include major parts of the code.
4. The query should not specify python.
For example, the code is: (ICL [CC_e
And the expected output query should be similar to: (ICL_query)
Please craft query based the examples and the following code: (code)
B.6 QUERY EXPANSION PROMPT
17

---

Published as a conference paper at ICLR 2025
You are exceptionally skilled at crafting real-world queries given some apis. Here are examples: (examples). Please gain inspiration from the following api docs to create high- quality realworld query.
Api docs for inspiration.
ppython (apis_string)
Please refer the above examples and craft new one!
Requirements: API name is strictly prohibited from appearing in the generated query. Each query should be complicated enough and can be solved using all apis above. The query **should be centered around (category) theme** and should not be spread out into unrelated pieces.
B.7 QUALITY CONFIRMATION PROMPT
You are exceptionally skilled at polishing tool calling plan (i.e thought) and python code given task.
Given task: (query)
Old tool calling plan: (thought)
Old code: (code)
Used API doc: (apis)
Here are examples for you refer: (ICI:IIICeotttt
Please make sure the code is logically correct and operational.
Requirements:
[1] Ensure that both plan and code respond correctly to the task and that code calls match the plan, which you can do by tweaking, embellishing, and modifying both plan and code. Plan does not have to be one-to-one correspondence of code; plan can be abbreviated.
[2] Please ensure that the code conforms to python syntax. Ensure that all python code is complete and runnable. You can add code when necessary.
[3) Every Line of code should be preceded by comment marked with "4" When modi modifying the code, please modify the in-line comments accordingly.
[4] Ensure that all function parameter calls are correct and you can change the code in case of errors.
[5] Thought and code should as concise while keeping the meaning intact.
[6] If there are cases including invalid binary code, replace them with reasonable text, delete them, replace them with reading operation on a file (especially when the binary code is encoded image)
Respond strictly with JSON.
18

---

Published as a conference paper at ICLR 2025
B.8 VARIABLE RENAME PROMPT
You are a helpful assistant for renaming variable names in a code snippet.
The follewing code snippet is part of program, and variables are named in format 'variablex_', Your task is to rename these variables so that they conform to the programming
srecification and have some semantic meaning, which can be infered by relative function calls And your output should only be a dictionary containing the old name-new name key value pair The definition of some functions are not included, and you shouldn
't modify them.
Following the code, there's s a dictionary that contains short description of the uuid-named variable. And you can take it reference.
Note that while the description might be the same, but the actual meaning is different across different variables. So you should not just copy the short description. Instead you'd better conprehensively consider the description, names of called functions, and the general logic.
The code is follows:
(code)
The dictionary is as follows: [description]
save time, have retrieved all the variables that requires to be renamed:
(variables)
C QUALITY CONTROL PROCESS
The quality control process consists of three key steps aimed ensuring the integrity and consis- tency of the generated code samples. First, GPT-4o-mini with in-context learning is employed to refine the workflows generated by the annotator. The primary objectives of this refinement minimize Python syntax errors, ensure that each line of code is accompanied by an explanatory com- ment, and eliminate meaningless binary encoding strings. Manual sampling was conducted assess the effectiveness of this refinement, with results showing that 94.2% of the samples demonstrated improvements in at least one area, such as Python syntax, code clarity, and the correspondence between the code and the associated queries. The second step addresses the issue of empty code outputs, which arise when the input or output task plan is excessively lengthy. These instances are identified and excluded from the dataset, leading to the removal of approximately 3% of the total samples. The final step involves ensuring consistency with expected function and API definitions by using the Python interpreter automatically detect syntax errors. Test functions constructed according to the API specifications, and any samples that fail to execute correctly due to violations of parameter constraints discarded. This step results in the exclusion of approximately 18% of the samples from the dataset.
D ANALYSIS OF API RETRIEVER
In our main experiments, the primary objective is to demonstrate that WorkflowLlama can generalize effectively when faced with Diitti-Diitributioo (OOD) APIs, i.c., APIs that were not seen during training. To further validate the robustness of WorkflowLlama, we conduct experiments in which golden APIs are not provided during inference. In the workflow orchestration task, a single query may correspond dozens of APIs. Selecting the correct APIs for given task requires advanced reasoning capabilities, and simple models such as dual-tower semantic matchers (Gao et al. 2021) may fail capture the nuances necessary for accurate API selection. Therefore,
in-context
19

---

Published as a conference paper at ICLR 2025
learning (ICL) to prompt large language models (LLMs) to extract the appropriate APIs for a query. The specific prompt used for API retrieval as follows:
You are an API retriever capable of identifying the APIs zequired to complete a query, based on both the user query and an API list.
API list:
MAPIs with descriptions
Here are some examples:
[example queries and API mappings\)
For the following ng query, which APTs are needed to complete the task? Please return 1ist of required APIs without any explanations.
query: Mqquer\) APIs:
To enhance the capabiliiiies of API Retriever, select the most 50 similar queries from the training set using the Minil model (Wang et al.) 2020) to generate appropriate ICI. samples. The performance of the API retriever is evaluated based precision and recall, and the results are listed in Tabel 5 Although the retriever's precision and recall are not particularly high, manual inspection of 50 randomly selected cases revealed that 84% of the retrieved APIS successfully fulfilled the tasks specified in the queries.
API Retriever
Pass Rate
API Retriever
Precision
Recall
Golden
70.4%
GPT-4o-mini Qwen2.5-72B
42.5% 40.6%
36.4% 40.7%
GPT-4o-mini
66.7%
Qwen2.5-72B
69.0%
Table 5: Precision and Recall for different API retrievers.
Table Pass Rate WorkflowLlama with different API retrievers.
To further validate this insight, we conduct workflow orchestration experiments using APIs retrieved by LLMs in the OOD setting. The results of these experiments, comparing the pass rates of Work- flowa using different API sources, are summarized in Table The results show that Work- flowLlama, trained with golden APIs, experiences only a minor performance decrease when using APIs retrieved by LLMs. Notably, even when APIs are retrieved by the open-source model Qwen2.5 Team 2024). WorkflowLlams achieves comparable or better performance than GPT-4o-mini.
In conclusion, the main takeaway from these additional experiments is that WorkflowLlama remains effective in OOD settings, even in the absence of golden APIs during inference. This emphasizes the flexibility and robustness of our framework for automating workflow orchestration in real-world scenarios, where APIs may not always be available at training time.
E CASE STUDY OF SHORTCUTS
We provide real-world shorteut example, which includes the following three presentation forms: the raw property list configuration file, the Python code after transcription and variable renaming, and the visual interface on MacOS.
The raw property list configuration file is presented below. For the sake of brevity, we have omitted the middle portion containing the actions.
WWWookkfLooclienneeesssoo "105000. www.okkii.oottppttconeeeeetttte
[
20

---

Published as a conference paper at ICLR 2025
"MCContasttonnteentteem",,, "MEPhhonekimbbrrCotttentttt"""" "MERAcchTeettContennttttm""" "#FFstriingCCnteenttteemmm
MMNoorkefLooCCleeeneeeeeess"" "3.0",
WFFrortfL.owiiinnnn..Leeeessssrr 900, "WWWorkkfloowccco"
"#Evoccffooo<conssttttooooo"""" 4292093695,
"##rorrklooottcondlppllllmee""" 59814
"ParameterKey": "KFCallContact", "Category": "Parameter",
"ActionIndex": 12,
"Text": "PreSetup CallNumber for FastStart. AnAn"
"ParameterKey": "#FeextactionTex"" "Category"; "Parameter",
"ActionIndex": 13, "Text"; "The call will be Ended after 30min! (1800 sec \nnnnnn, "DefaultValue": "1800"
WEWoz:kFFLWActtinnn":: [
"
"is wes.wolkflw..oo actions www.ee..p.oeeeeeeeeeeeee phonenumber",
wwW.rrtflooocciiioooaaraaaeeeeee "WFInput":
"Value":
"string": "
"attachmenttsSylaagee" {
"(0, 1)": "Type": "ExtensionInput"
},
"FFserialiizationtppe""" oarr
"OUID": 1BC2888-88C--00--71---5530000000
"is wss.woofooo..cccooossstttoooooo. ( "JUID": 7339888--880-4999-8-----11166177
"#FWorkfl wiw.:ookk..otccsioonneeiiieee""
".s. www.wwwwwww.oooo actions.cininn...m
wwt.wooo........oon...eeo detect. "trteect.phnnnuuber...
wwwrooecELooocccioPaaaamttttttr {
"WFInput".
"Value":
21

---

Published as a conference paper at ICLR 2025
"string": "
"attachmentts/Ragge": {
"(0, 1)":
"OutputUUTD"; 03119807-3071-499------------
"Type": "ActionOutput", "OutputName": "Zwischenablage"
"WFSerializationTyee "#FTexttokenString"
"OUID": "F3E902F4- 1198 - 1338-99060-C48522372223
"is. www.wwow.w....oon...sss cow.://onns.oooiiiiooo.llla...
"WFInput":
"Type": "Variable", "Variable":
"Value":
"OutputOUID":
"Type": "ActionOutput",,
"OutputName": "Telefonnummern"
1,
"#Fserializationyyee *
WF 0, "Groupiigg.dentifieee"" "#FCondition": 101
"MFKorttflooActiinnleettitee"""" Wwrrocrkkloooecctooomaramatttts" ()
w.sssorkflow..cciinnss
wtffreefffoottttuuuunteeeeeeetes "900",
WWWorkkfowwypps":
"WatchKit"
"ActionExtension"" "NCWidget"
can be observed that this configuration file employs non-semantic hexadecimal strings to rep- resent variables and uses keywords such is workfloo actions. conditional and GroupingIdent ifier to implement logic controls like conditions, making it inherently dif- ficult read and comprehend. Consequently, have converted it into Python-like code format. The Python code, after transcription, variable renaming, and commenting, is shown as follows:
Calls function that detects a phone number based sateee_umbbbrrrreee
input and
stores the result the variable phons_nunber______e_eephp cuuberr_input is workflow workflov_action cections_detect________e__ WFInput=f' input ("Please enter the value: ") ())
22

---

Published as a conference paper at ICLR 2025
Calls function retrieve the current content from the clipboard and stores the variable ceapboard_contenne clipboard_content
()
Detects a phone number from the clipboard content and assigns the resuit to detected_phone__ntee
is ss_workflow_sst #FInput=f prr httpippaadd_ctteet]_
Checks if a phone number was detected from the clipboard content. if not paaieed_hhoe__uuueee
Starts nested conditional that checks again if phone number was detected.
if not panbtteed_hhne__umbeee
# Retrieves the clipboard content again and stores it in clipboard content retrieval for processing. clipboard wtt__eeetettteeeeeeeee
sattfloooacacissoooomaaooo is_ ss_workflox. actions actiinss_eeeeliiiiiieciee getclipboard ()
Calls a function to convert the retrieved clipboard content
into a phone number and assigns it to solected volectod_phone_nneee phone_number. sclected__hone_ nnmbee http://mmmoorooooaoooooottttteee If a phone number was detected previously, this else block is
executed.
else:
Calls function to allow the user to select a phone number from contacts if number was detected.
selected phone number is workflow actions selectcontacts () Displays an alert message to the user indicating that the call will end after 30 minutes.
is is_
WeAlertAct.onNssaee""!
Sets the title of the alert inform the they are starting call to the selected phone number.
The Call ends after 30 min''', WFAlertActionTTtle==="" Start calling "[cocrce_variable (w whwpeedppppppphhheeeeeeeeee _umbeenummerrrrrr oatrrcacccccssssPPPPPPoeeeeereeo
Initiates a call to the selected phone number using a specific function to handle mobile phone calls.
com hom_apple_mcbl wammmillphone_aaal Stores string representing the delay time in seconds (1800 seconds 30 minutes) the variable delay_time. delay time "118001!
Calls a function to impose a delay in the execution of the subsequent code, using the delay time variable converted to float. is wwwoooofoowacctions.oooo
WFDelayTime-float (delay_time) Activates airplane mode by calling function with OnValue set to to enable it, and operation set indicating that it should be set. is www.wwowwwwc.tt..ooooo wtwp:aattiis_airllaeeeeeee ( OnValue=1, "peration=""!st""" Calls delay function again with specified delay time of 3 seconds before proceeding.
woorkflow_actions.
( WFDeLayTime=3.0
Calls the function to set airplane mode again but without changing the value, effectively keeping it enabled. is saankfllowactiomsaarppp wtt.arreeenemmeesseee.... Onvalue-l)
Calls function exit the script workflow gracefully.
is wwrkflowwactione
We can clearly see that the transcribed Python code is of high quality, with strong readability, making it well-suited for training LLMs that have been pre-trained extensive Python code.
For easy understanding, we also present the visual interface of this shorteut Figure which corresponds to the transcribed Python code line-by-line basis.
23

---

to Published as a conference paper at ICLR 2025
F CASE STUDY OF WORKFLOWBENCH
In this section, we provide a workflow example of Workflow Bench based on the shorteut listed in Appendix E We omit the annotated workflow code that has been shown in Appendix E
The query Q is:
What steps can I take to streamline a process that involves using a phone number from my clipboard to make a call, setting a 30- minute timer for that call, activating airplane mode afterward and ensuring a smooth exit from the process once everything is completed?
The list of provided APIs is:
" wcw.ehhceeoo..cttooooseeeeeleleo vccm..ppleemmbiieghhoneec.."""
" "is. workffooo..accttoos.exxii",
"is worifloo..aciionssaalett""
"is wex.ffoov..cccoooe.ooooeeeeeeee.
"is. www.://oo...tttoo...cttteppeeeee
"is www.:elooo..atttoo..aappaaaaaeee set",
"is. "is.wocccoov.actoooseeetoopppppp
]
The task plan P is:
1. *Start**
Begin the process.
2. Aeet User Input**
Prompt the user: "Please enter the value:"
Call function 'is 'is_work"" detect phone number from input.
Store the result in ghhone_number_inptt"
3. **Get clipboard Content** Call function www.oooooooo_ctttooo_eeeeeeeeeee current clipboard content.
to retrieve
Store it in
content'.
4. **Detect Phone Number from Clipboard**
Call .clipboard_content to detect a phone number.
with
Store the result in
5. **Check if Phone Number Detected**
If ddetected_phorte p_tpteed__pheee_mmeerr numberr is not found:
1. **Retrieve Clipboard Again**
Call "is wwo.kflowwactiios_geeeeeeeeee in "alpphoarrlLootent__ertteaall
and store result
2. **Convert Clipboard Phone Number**
- Call coaraarddcotttteeeeeeetree
with
Store the resul: in deeeeetee_ehhoeeeeeeeeee_eeeeeee phone_number"
Else:
1. **Select from Contacts**
Call hit..ooww.oo..atttooo.eeeeetttee
to select a
phone number.
Store it in "selected_phone selletted_phon__nmmbe""." _umberr_mmbbrreee
6. **lert User**
Call
alertss_alert" to display an alert indicating the call will end after 30 minutes.
24

---

Published as a conference paper at ICLR 2025
1. **Set Alert Message**
Alert messaqe; "The Call ends after 30 min". Alert title: "Start calling [selected [mone_number]] 7. **Initiate Call** Call cco__appl__mbbilephonee..ec
with "selecttad_phone__umerr
to start the call.
**et Delay Time** Store "1800" in 'delay_time'.
Call impose a 30-minute delay.
_eeay""_ with "float (delay_tine) to
9. **Enable Airplane Mode**
Call
with "OnValue=1"
(to enable).
10. **ee Airplane Mode Enabled** with delay of seconds.
Call
Call 1s__wokfllo_actiie poo__atttooss__ppaaaemedddee set' with 'OnValue=1"
to maintain airplane mode.
11. **Exit Workflow**
Call "is
to terminate the script The process completes.
gracefully.
12. **End**
G CASE STUDY OF GENERATED WORKFLOWS BY WORKFLOWLLAMA
Task query:
What steps would I need to take to develop automated messaging system that alerts users when their message bank is depleted and enables them to dispatch pre-defined messages?
Additionally, how can this system monitor for updates to its automation script and notify users accordingly?
The list of provided APIs
"is oag.wccckkkoooucttoooonooofftii.
" "is_wor2tfooosacctioss....aa_"""
" ri..weaiifooo.atttt.naaaaaerr"""
" iig..oocttioo..aatiioaasoooon"""
" oaa_cccceecoog.cc11t.ooooooooor,
" htg.ao/ffoollcc.....o...........
"is ort.:pphhho.ccccoooocc.....eaaar
"ig_Nociffloooccttionsnttttt,"""
iia_workfloow.cctionn_....
]
The generated task plan P is:
25

---

Published as a conference paper at ICLR 2025
1. **Start**
Initialize the process.
2. *Deeiine Metadata**
Create a dictionary named "mssage_bai__mttdase containing:
Name: "Message Bank"
Author: "Gar0512"
Version: "1.0"
Last Updated: "01/17/2020"
Change Leg: "Initial release"
Reutinefuub ID: "3967"
3. **ssiggn Metadata to Variablex*
Set "Info" equal to
4. *Retrieve Contacta** cooo_aactios_ootteeeee () to get list of
Call ss_wwrrflowweee contacts and store it in contacts_lisst.
11. **End** Complete the workflow.
The generated workflow code A is:
Defines dictionary named grssage_bank_metateee that contains metadata about the message bank including its name, author, version, last updated date, change log, and RoutineHub ID. message Blank_metadate = NName I'Message Bank "A'uhh :1: """ar0512"',, " "": ''01/17/2020"1 'Version''': "11.0" LLatt Updated "Channe Log!": (01/17/2020: Initial release), I'RoutineHuu ID"": "'3367''1)
Assigns the message_bank _ank_metadata dicticnary to variable named Info for easier reference later in the code.
Info message bank_
Calls a function
bortffoo_acttons_.. (ontoctsc that retrieves a list of contacts and assigns it to contacts_list list.
contacts_liss
fs__worllow_act ams_contactss ()
Prompts the user for input and stores the value entered in user_input_value
user_input_value f f 'input ("Please enter the value: ")"" 4 Counts the number of items in cts__jnnut__vallla using the w_w.ookkloo_actioossooo iten_count.
function and stores the result in
item_count iss_wrrkloowsaciioosocoomm
WFCountType= " "Itemm"""""""""t""""""""""
Input=user_ipput (itee__ipptt_allee
Checks if titem_count is greater than to proceed with further actions.
if item_count. 0.0:
# If the condition is true, prompts the user for input again and stores the value in user_input usr_input_value_2, user_input_value_2
f ("Please enter the value: ")" # Counts the number of items in iser_input_value_ and stores the result in sten_count_2 2.
iitee_counttt = wt__orrflow_ccttonss.
WECountType=" IItea
.., Ingut=uuse__iput__vaa
# Checks if item_count_2 is greater than 1 to determine if the user has entered multiple values. if item_count_? > 1.0:
I If the condition is true, prompts the user for input again and stores the value in user pet__nnppttlllll user sser_iaput_value_ g'r'input ("Please enter the value: ")
26

---

text= Published as a conference paper at ICLR 2025
# Counts the number of items in user iser_input_valuee and stores the result in item [tem_count_3
item count__cut__ is ww_rorlllwww.ccc cettonss_couee WFCountZype=!'! penut=uuee_ipu__valllee # Checks if item_count is greater than 2 to confirm the user has entered enough values for processing.
Items"',
if item_count_3 2.0:
# Displays an alert to the user indicating the maximum number messages that can be sent at a time.
is may send a maximum of messages at a time. Please choose your message. "", WFAlertActionTitle 'Maximum Number of Messages'",
VFALertacttionVessaeee
Prompts the user to choose a message from the contacts list and stores the chosen message in uaer_input_messaee.
user_input_message f 'input ("Please choose your message: ")""
Splits the text the chosen message into parts and stores the result in split_text.
split_text =
user_ jser_input_message
Retrieves the message to send based on the user's choice from split_text and stores it in message_to_send.
message to to_send is www.wwrklwwwwoc actions getitemfromlist htt.:pooootttooeeiteeeeettomtttt WFInput=split_text "FItemIndx= WFTtemSpeci cecteeeciffeeeeea fier=!! At Index!
is_workflow_actions
(134 1ines code)
H LIMITATIONS
While the framework proposed in this paper represents notable progress in workflow orchestration, it also has certain limitations that warrant discussion. First, the APIs used in our work are exclu- sively derived from Apple Shorteuts application, resulting in lack of coverage across more diverse fields and potentially limiting the generalizability of the dataset to broader application contexts. Second, our approach lacks evaluation through actual execution. This limitation arises due the complexities involved in executing workflows, such as the need for intricate user registration and permission acquisition. Moreover, the APIs are subject to frequent changes as applications continue to evolve, making it challenging to implement consistent executable evaluation. Consequently, our evaluation limited to static analysis.
I ETHICAL STATEMENT
this study, the dataset construction process was fully automated using LLMs and algorithms for data annotation, eliminating the need for human annotators and thereby avoiding concerns related to annotator compensation and working conditions. The data utilized was collected through web scraping from publicly accessible sources, with strict adherence the Terms of Service (ToS) of the respective websites. Scraping was avoided on platforms where such activity is explicitly pro- hibited, ensuring compliance with ethical standards. Additionally, mation (PII) or private user data was collected at any stage of the research process. All data was anonymized protect privacy and mitigate any potential ethical concerns related informa- tion.
personally identifiable infor-
27

---

Published as a conference paper at ICLR 2025
FonCall Limiter
Receive
more input from
Sheee
there's is no input:
Continue
Shorteut
clipboard
Phone Numbers
does
Phone Numbers
does not have any value
Oet clipboard
Phone Number
Select contact
Show More
End
End If
Show alert
Show More
The Call ends after 30 min
Text
1800
Wait Text
Set Airplane Mode
action supported
Wait seconds
Set Airplene Mode
This action not suoported Mec.
Stop
Figure 7: The visual interface of the shorteut Rout ineHub
CallDuration Limite.
28


Reviews:
[{"review_id": "cADGH5KHDH", "reviewer": "Reviewer_x9fK", "summary": "The paper argues that state-of-the-art models like GPT-4o face challenges in effectively handling complex workflows. To address this, the paper introduces WorkflowLLM, a data-centric framework designed to enhance the workflow orchestration capabilities of LLMs. Central to this framework is WorkflowBench, a large-scale fine-tuning dataset comprising 106,763 workflows, 1,503 APIs, and 83 applications across 28 categories. WorkflowBench is constructed through a three-phase pipeline: data collection from sources like RoutineHub, query expansion using ChatGPT to create diverse and complex workflows, and workflow generation leveraging a trained annotator model for high-quality workflow synthesis. The dataset is enriched with Python-style code, comments, and hierarchical thought structures to improve LLM learning.\n\nBased on WorkflowBench, the paper fine-tunes Llama-3.1-8B, resulting in WorkflowLlama, which demonstrates superior performance compared to existing models, including GPT-4o, on workflow orchestration tasks. WorkflowLlama achieves notable success in generalization to unseen APIs and tasks, evaluated using metrics such as CodeBLEU and Pass Rate. Additionally, WorkflowBench exhibits robust zero-shot generalization on the T-Eval benchmark, achieving an F1 plan score of 77.5%.", "questions": "Q1)The “pass rate” is an important metric to evaluate the performance in section 4.3. Could you elaborate more on how it is calculated? What are the reasons to choose it? And how is related to training/fine-tuning phases?\n\nQ2)Could you  contextualize the size and the quality of the generated dataset   in terms of  how significant it is in comparison with SOTA or related works? Are they general enough for RPA for just for applications around Apple Shortcuts and RoutineHub?\n\nQ3) The papers also mention about fine-tuning Worfflow Lalama and annotator in a very short paragraph in section. Could you elaborate more about setup, why such training parameters are chosen?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "S1)The observation and problems are interesting and relevant to the conference\n\nS2) The dataset might have potential uses\n\nS3)The experiments involve many systems", "weaknesses": "W1) The scientific or technical contributions are limited as the key contributions of paper are around data curation effort. And the significance of data size and quality  is not clear (see Q2). \n \nW2) Section 3 mentions many steps to generate the needed data for its workflow using ChatGPT but the Quality control protocol is not very clear. Even the author provides some examples and algorithms in appendix, the details are very descriptive . Also section4.2  mentions using  human evaluator to re-label the sampled data,It’d be better to provide more details towards the quality control protocol for this human-driven process as well.\n\nW3) Many technical details are not very clear (see questions)\n\nW4) Code is provided in supplemental material but there is no document how to reproduce the experiment results ( also see Q3)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G7C2o1htGv", "reviewer": "Reviewer_WEy1", "summary": "This paper proposes a framework that leverages LLMs to automate workflow orchestration. It extends the real world dataset by creating synthetic data through LLMs, to train the model. It conducts experiment study to compare the proposed framework with several proprietary and open source models, using CodeBLEU and Pass Rate as the metrics.", "questions": "Are the prompts used in the framework that drive the LLMs usage automatically generated?  If yes, how to ensure the quality of the prompts? If not, how to scale up?\n\nFor each step from data generation to model evaluation, how to optimize the prompts?\n\nThe description on quality confirmation is a bit vague. The example given about the issues is not clear. How are such issues detected? How to prompt ChatGPT to refine A’ and P’, and how to ensure the quality of the refinement? How to perform the rule-based filtering, e.g., how to automatically detect if parameter constraints are violated or not?\n\nFor OOD settings, how are the other models evaluated? Do they also be given the same input like the one for Workflow Llamma, i.e., providing the list of required APIs? \n\nGiven synthetic data takes the large portion of the final benchmark (91k out of 11k) but the quality is unsure, it would be interesting to also see how Workflow Llama performs on the real-word dataset, i.e., without the synthetic data, compared to other models? \n\nThe proposed framework outperforms all the other models by an extremely large margin in the first four columns in Table 2 (such as 9.4% vs 0.5). This seems to be suspicious and needs further examination and explanation. \n\nIn Table 2, both CodeBLEU and Pass Rate are used, one examines the similarity of code and one examines if the code is executable. It would be interesting to combine both of them, such as setting a threshold on the CodeBLEU score when checking the Pass Rate to compare models, since Pass Rate itself doesn’t necessarily indicate the success of the workflow orchestration.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and easy to follow. Examples are given to illustrate the important steps. \n\nThe extended dataset improves the diversity and complexity of the samples in the training data, allowing the models to adapt to the situations that are closer to real-world applications. \n\nThe experimental results show that the proposed framework outperformed the existing models by a large margin.", "weaknesses": "The paper mainly relies on prompting LLMs to generate the results of the key steps, such as data generation, commenting actions, and plan generating. It seems that there is a large space of prompt engineering to improve the performance but left to be undone. \n\nThe proposed framework bypasses the API selection when handling the OOD setting by directly serving the required APIs as the input. This greatly simplifies the problem of automating workflow orchestration. \n\nThe paper relies heavily on LLMs for the entire process, including generating the dataset, training the model, and evaluating the models (only less than 82% accuracy in a small sample with 330 instances in total). This weakens the technical contribution and the reliability of the experiment results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "45qx9rbs4b", "reviewer": "Reviewer_yKyu", "summary": "The paper presents a detailed explanation of the construction and evaluation of the WorkflowBench dataset, which contains many examples of workflow descriptions. The dataset is later used to fine-tune the open-source LLM Llama-3.1-8B.\nThe dataset creation follows a well-established and potentially reproducible approach. It starts with data gathering, expands to increase data diversity, and generates a final workflow dataset, enhancing real collected data with synthetic data.\nThe fine-tuning intends to overcome the limitations of existing LLMs in workflow orchestration and provide a consistent improvement in process automation by proposing an agentic approach.\nThe fine-tuned LLM called WorkflowLllama is detailed, and its capabilities are evaluated, showing a solid capacity to orchestrate complex workflows. Its performance was compared with commercial and open-source LLMs using the test set of Workflowbench.", "questions": "I am curious about the choice of ChatGPT to run both the query expansion phase of the dataset creation (section 3.2) and the experiments (sections 4.1 and 4.2). Altough I think it was a good idea, why rely on a proprietary model (the GPT-4o) with proprietary and non-disclosed prompt engineering behind it? Did the authors try to use directly the GPT-4o model? Did they try other models to run this phase? I assume that the \"ease-of-use\" of the ChatGPT is a fair point in favor of its use, but it should be evident in the paper.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-presented and well-written overall. The authors identified a clear pain point in achieving an agentic LLM approach for process automation: lacking LLMs capable of correctly describing complex workflows. Although partially addressed by other works, as pointed out by the authors in the related work section (section 2), their approach is innovative in dealing with fairly more complex workflows than previous solutions.\n\nTheir approach to overcoming this limitation is sound and clever. They provide a fine-tuned LLM specialized to handle this kind of task.\nTo do so, they craft a well-defined dataset, and it is possible to highlight the process of dataset creation as one of the paper's main contributions, as crucial as the constructed dataset and the fine-tuned model.\nAfter the data collection, the query expansion phase is especially interesting because it uses another LLM (e.g., ChatGPT) to help overcome the lack of diversity and complexity of the gathered initial data with syntactic data. The same applies to the clever use of ChatGPT in the evaluation phase.\nCompared with other LLMs solving the same kind of problem, the presented results show the potential use of the author's solution to deal with the initial established problem, i.e., the automation of complex tasks.\nWorkflowLlama outperforms other LLMs when the workflow demands generalization to unseen instructions and unknown APIs.\n\nI also highlight the good approach to evaluating the effectiveness of the LLM-based evaluator (section 4.2), which strengthens the paper's arguments.", "weaknesses": "- The background on automation's relevance (lines 36-38) might be condensed, as automation's importance is widely recognized.\n- In section 3.1, the author can better explain the $\\mathcal{D}$. The other elements were presented before, but  $\\mathcal{D}$ is directly introduced on line 248 without prior explanation.\n- Lines 245-247 read (my emphasis):  \"This bottom-up [...], efectively **ensuring content realiability** and **minimizing the risk of hallucination**\". These are pretty strong assumptions without solid evidence, especially if generalized to every LLM. Do the authors know published papers that confirm them (since this paper is not about reducing hallucinations)? If not, the phrase can be removed or rephrased to be more humble.\n- The paper lacks a clearer section on threats to validity. The authors provided some in Appendix E, but they should be incorporated into the main text.\n- The paper also lacks a more evident running example/use case. Although the authors provide a small case study in section 4.7 and some examples in Appendix D, a consistent running example, incorporated throughout sections, could clarify the model's applications and emphasize practical use cases.\n\nMinor issues:\n- Line 52 mentions GPT-4, while the rest of the paper uses GPT-4o(-mini). Maybe it is only a typo, or the work mentioned indeed uses the GPT-4 model. Since they are different models, it can be good to double-check every reference to ensure it always talks about the correct model, maybe highlighting somewhere they are not the same.\n- The size of Figure 2 can be increased to improve readability.\n- Figures 3,5 and 6 could improve accessibility through higher contrast colors.\n- Although well-written, the paper is sometimes repetitive (e.g., sections 1 and 3). Proofreading with this issue in mind may help the authors achieve a better final text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gR3bJXgnec", "reviewer": "Reviewer_mrzv", "summary": "This paper proposes a pipeline for constructing workflow (RPA) data to enhance LLMs' workflow orchestration capabilities. The core motivation stems from the observation that real-world workflows require more complex logic and longer sequences than what current LLMs can directly generate. The primary contribution is the construction of a dataset containing over 100K samples. The main limitations lie in the lack of certain technical details (such as specific ChatGPT versions used for annotation) and the lack of deeper investigation into model capability improvements.", "questions": "1. The performance of generic LLMs in workflow orchestration through agent-like approaches would serve as an interesting baseline for comparison. Through this comparison, we could better understand whether the core challenge lies in the model's reasoning capabilities or in its ability to maintain consistent output formats. This point won't affect the score; it's just for discussion.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. Accurate and valuable problem identification. The paper identifies the limitations of existing work in handling real-world workflows: difficulty in processing long-sequence workflows with complex logical relationships like control flow. This observation directly addresses practical application pain points, making the motivation very solid.\n2. Proposes a systematic data construction method. The paper designs a complete data synthesis process: from real data collection, workflow abstract representation, data synthesis to quality control, with detailed design considerations at each stage. The resulting large-scale dataset provides an important resource for this field.\n3. Comprehensive experimental validation. The effectiveness of the method is strongly supported through multi-dimensional evaluation using CodeBLEU and Pass Rate, cross-domain testing with T-Eval, along with detailed case analysis and ablation studies.", "weaknesses": "1. Important technical details are missing. The paper mentions using ChatGPT for data annotation and optimization multiple times but doesn't specify the model version. For work with data construction as its core contribution, this seriously affects reproducibility and rigor.\n2. Lacks methodological innovation and mechanism analysis. Although data construction is an important contribution, the paper lacks in-depth analysis of how this data enhances model capabilities. Specifically, it doesn't investigate whether the improvements come from enhanced logical reasoning abilities or simply better format matching. Without such analysis, it's difficult to determine if the model has truly learned to reason about complex workflows or has merely memorized patterns from the training data.\n3. Missing critical ablation experiments. The paper doesn't compare the performance difference between the Annotator Model and WorkflowLLM, despite both models using the same training method and data type. This results in question the necessity of the data synthesis strategy and weakens the core argument.\n4. The paper mentions that the Annotator Model can generate workflows with over 70 actions, and theoretically WorkflowLLM should be capable of the same. Given the general limitations of LLMs in long sequence generation, including such long-sequence workflows in the appendix would further demonstrate the work's contribution.\n5. Given that the core contribution is data construction, whether the dataset will be open-sourced directly affects the practical value of the work. The authors should consider open-sourcing the dataset to promote development in this field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)