WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models

## Abstract

Abstract Recent advancements in large language models (LLMs) have driven a revolutionary paradigm shift in process automation from Robotic Process Automation to Agentic Process Automation by automating the workflow orchestration procedure based on LLMs.
However, existing LLMs (even the advanced OpenAI GPT-4o) are confined to achieving satisfactory capability in workflow orchestration.
To address this limitation, we present WorkflowLLM, a data-centric framework elaborately designed to enhance the capability of LLMs in workflow orchestration.
It first constructs a large-scale fine-tuning dataset WorkflowBench with 106 , 763 106 763 106,763 106 , 763 samples, covering 1 , 503 1 503 1,503 1 , 503 APIs from 83 83 83 83 applications across 28 28 28 28 categories.
Specifically, the construction process can be divided into three phases:
(1) Data Collection: we collect real-world workflow data from Apple Shortcuts and RoutineHub, transcribing them into Python-style code.
We further equip them with generated hierarchical thought via ChatGPT.
(2) Query Expansion: we prompt ChatGPT to generate more task queries to enrich the diversity and complexity of workflows.
(3) Workflow Generation: we leverage an annotator model trained on collected data to generate workflows for synthesized queries. Finally, we merge the synthetic samples that pass quality confirmation with the collected samples to obtain the WorkflowBench.
Based on WorkflowBench, we fine-tune Llama-3.1-8B to obtain WorkflowLlama.
Our experiments show that WorkflowLlama demonstrates a strong capacity to orchestrate complex workflows, while also achieving notable generalization performance on previously unseen APIs. Additionally, WorkflowBench exhibits robust zero-shot generalization capabilities on an out-of-distribution task planning dataset, T-Eval.
Our data and code are available at https://github.com/OpenBMB/WorkflowLLM .

## 1 Introduction

Process Automation (PA) (Cichocki et al., ), as a long-standing pursuit of the human race, aims to automate repetitive tasks to minimize human labor and improve efficiency.
Tracing back to the agricultural era, humanity has employed waterwheels and oxen to automate farming practices.
Robotic Process Automation (RPA), the current predominant PA technique, abstracts the repetitive task into a workflow (i.e., a program that can execute automatically) by orchestrating various actions (e.g., functions or APIs) (Ivančić et al., ; Hofmann et al., ; Wewerka & Reichert, ; Agostinelli et al., ; Ferreira et al., ).
While RPA successfully reduces the human labor via automated workflow execution, the process of orchestrating workflows still requires substantial manual effort.
Recently, large language models (LLMs) (OpenAI, ; ; Touvron et al., ; ; Dubey et al., ) have achieved remarkable performance beyond natural language processing (Ahn et al., ; Cheng et al., ; Qian et al., ).
The emergence of LLMs has unveiled a paradigm shift trend, moving from Robotic Process Automation to Agentic Process Automation (APA) (Ye et al., ; Zeng et al., ; Huang et al., ; Wornow et al., ; Li et al., ) which automates the workflow orchestration process by utilizing LLMs to build the workflow.

However, such a paradigm shift trend is constrained by the limited ability of LLMs to orchestrate complex workflows, which in turn leads to two crucial limitations in current APA methods:
(1) Constrained Action Scale: Current LLMs can only orchestrate small-scale workflows with a limited number of actions. The most advanced OpenAI GPT-4 is capable of managing workflows with an average of only 6.1 actions, even when equipped with advanced decision-making mechanisms (Ye et al., ). This falls short of the complexity required to meet real-world demands.
For instance, as a widely-used representative, Apple Shortcuts (Apple, ) involves an average of $70.4$ actions.
(2) Simple Logical Structure: Currently, most existing work mainly focuses on generating sequential actions (Yao et al., ; Qin et al., ; Chen et al., ) while workflows of the real-world applications usually involve intricate logical structures such as branches and loops.
For example, Apple Shortcuts averages $2.6$ nested branch/loop logical structures.
As a result, there is an urgent need to unlock the workflow orchestration capability of LLMs to expedite the paradigm shift in process automation.

Figure: Figure 1: Overview of WorkflowLLM. It first constructs WorkflowBench through a three-phase pipeline and fine-tunes WorkflowLlama, which can generate workflows based on the user’s query (appointment management in this case).
Refer to caption: x1.png

To address these challenges, we propose WorkflowLLM, a data-centric framework including dataset construction, model training, and evaluation to enhance LLMs’ workflow orchestration capabilities (shown in Figure 1).
Specifically, we first construct WorkflowBench, which consists of $106,763$ supervised fine-tuning instances, encompassing $1,503$ APIs across $83$ applications, structured through three primary phases:

- •
Data Collection:
We select shortcuts from RoutineHub as high-quality data sources because they represent a robust RPA application with numerous expert-developed workflows available.
We curate $14,771$ human-annotated, high-quality shortcuts spanning $28$ diverse categories (e.g., Business, Health & Fitness, Productivity), alongside associated metadata including titles, functionality descriptions, and API documentations.
As the raw workflow data is not directly suitable for LLMs to process, and considering that Python allows more convenient parameter passing and control logic (Ye et al., ; Wang et al., ), we transcribe the shortcut source code into Python-like code.
Subsequently, we prompt ChatGPT to generate comments, task plans, and task queries at varying levels of granularity—from fine-grained to coarse-grained—to enrich the data with detailed thought processes and enhance the learning efficacy of LLMs (Wei et al., ).
- •
Query Expansion:
To enrich the diversity and complexity of workflows, we utilize ChatGPT to generate additional task queries.
Specifically, we first sample applications with diverse functionalities and select their APIs, along with built-in APIs, to prompt ChatGPT to generate task queries that leverage these sampled APIs to accomplish specific tasks.
To further ensure workflow complexity, we also sample real-world workflow examples as demonstrations to guide ChatGPT in generating similar workflows.
- •
Workflow Generation:
As existing LLMs even GPT-4o still struggle in workflow generation, we first train a workflow annotator model based on the collected real-world shortcuts.
Then we utilize the trained annotator to generate workflows for the expanded task queries.
To prevent low-quality workflows generated by the annotator model from affecting subsequent training, we perform quality confirmation to ensure dataset integrity.
We first utilize ChatGPT to refine the generated workflows to fix existing minor bugs in them and then use rule-based filtering to remove workflows with logical errors.

To evaluate the capability of LLMs in workflow orchestration, we employ two metrics: the reference-code-based metric CodeBLEU and the model-based metric Pass Rate.
Experimental results demonstrate that WorkflowLlama consistently and significantly outperforms all baselines, including GPT-4o even with the in-context learning technique, across both metrics under unseen instructions and unseen APIs settings.
Furthermore, WorkflowBench demonstrates strong generalization capabilities in out-of-distribution (OOD) scenarios, particularly on the T-Eval benchmark (Chen et al., ), where it achieves an F1 plan score of 77.5%.

## 2 Related Work

#### Process Automation

RPA has gained considerable attention for automating repetitive tasks in various productivity scenarios (Ivančić et al., ; Hofmann et al., ; Wewerka & Reichert, ; Agostinelli et al., ; Ferreira et al., ).
RPA predominantly relies on handcrafted workflows (e.g., programming, recording human behavior), making them highly suitable for automating well-structured, routine processes (Herm et al., ).
However, such approaches require substantial efforts and in-depth domain expertise, resulting in high setup costs and limited adaptability.
Recent advancements in LLMs have spurred interest in integrating these models into RPA to enhance flexibility and reduce dependency on manual workflow creation.
Ye et al. () introduced the concept of APA, which utilizes LLMs to autonomously orchestrate workflows based on human instructions.
Subsequently, several studies have sought to apply APA in various domains, including travel planning (Xie et al., ), smartphone applications (Huang et al., ), enterprise automation (Wornow et al., ), financial question answering (Zeng et al., ), and data analysis (Li et al., ).
Despite relying on advanced LLMs (e.g., GPT-4), these approaches have often exhibited suboptimal performance, highlighting challenges faced by existing LLMs in workflow orchestration.
While Li et al. () made an effort to fine-tune Mixtral-8$\times$7B (Jiang et al., ), it could only orchestrate sequential workflows with an average of $15.6$ actions, remaining insufficient for real-world requirements.
This work addresses a critical gap by proposing WorkflowLLM framework to enhance the workflow orchestration capabilities of LLMs to meet real-world demands.

#### Tool Learning

Workflow orchestration driven by LLMs frequently depends on external tools, such as APIs, to extend their operational capabilities.
Recent studies have demonstrated that LLMs can effectively acquire and utilize external tools by learning from their documentation, thereby solving complex tasks that would otherwise be beyond the model’s native capabilities (Wu et al., ; Schick et al., ; Qin et al., ; ).
This integration enables LLMs to access real-time knowledge and perform specialized operations, particularly for executing intricate processes (Yang et al., ; Nakano et al., ; Qin et al., ; Wang et al., ; Gao et al., ).
To further enhance this capability, several efforts have introduced datasets specifically designed to fine-tune LLMs for tool interactions (Zhuang et al., ; Qin et al., ; Wang et al., ).
However, these datasets are often constrained to limited actions scale, thus limiting their effectiveness for managing complex, real-world workflows.
Compared to tool learning scenarios, orchestrating workflows demands more sophisticated planning and reasoning that current LLMs have yet to fully realize.
In response to these limitations, we present WorkflowLLM to significantly improve LLMs’ capabilities in workflow orchestration.
Besides, Shen et al. () also used Apple’s Shortcuts but aimed to assess LLMs’ tool utilization ability.
In contrast, we emphasize a different scenario, workflow orchestration and aim to enhance the workflow orchestration ability rather than evaluation alone.

Figure: Figure 2: Illustration of our WorkflowLLM which contains three phases to construct WorkflowBench, followed by the supervised fine-tuning phase to derive WorkflowLlama.
Refer to caption: x2.png

## 3 WorkflowLLM

As Figure 2 shows, WorkflowLLM introduces a data-centric framework to enhance the capability of LLMs in workflow orchestration by constructing a high-quality supervised fine-tuning dataset WorkflowBench.
In this section, we outline the dataset construction process, which is carried out in three distinct phases: Data Collection, Query Expansion, and Workflow Generation.

### 3.1 Data Collection

We first give the introduction to Apple Shortcuts and RoutineHub, and describe how we crawl and filter to get high-quality data.
We then convert the shortcuts into Python-style workflow code.
Inspired by Chain-of-Thought (Wei et al., ; Chen et al., ), we prompt ChatGPT to generate hierarchical thoughts, including comments, task plans, and task queries, progressing from fine-grained to coarse-grained details for each shortcut.

#### Apple Shortcuts and RoutineHub

Apple Shortcuts, as a representative application of RPA, is developed by Apple Inc.
This tool facilitates the automation of a series of actions, enabling users to efficiently perform a diverse range of tasks.
The actions within Shortcuts are APIs provided by both built-in Apple applications, such as Safari, and third-party applications like OpenAI. Each application may provide multiple actions.
For instance, OpenAI provides APIs that facilitate voice conversations and text interactions with ChatGPT.
Through a simple drag-and-drop interface, users can construct complex workflows, such as navigating to the nearest coffee shop or downloading watermark-free images from TikTok.

RoutineHub(^1^11https://routinehub.co/) is a prominent community for sharing shortcuts, with a collection of thousands of shortcuts across both iOS and macOS platforms.
All shortcuts on RoutineHub are categorized into $28$ workflow categories (e.g., Business, Health & Fitness, Productivity, etc).
RoutineHub records the metadata of each shortcut (e.g., title, description, iCloud URL), providing valuable information.

#### Crawling and Filtering

For each shortcut, we crawl the title, developer-provided description, and iCloud URL linked to Apple.
As RoutineHub does not provide the source code for these shortcuts, we further crawl it from their iCloud URLs.
Besides, we merge shortcuts collected by ShortcutsBench (Shen et al., ), sourced from platforms like ShareShortcuts(^2^22https://shareshortcuts.com) and MacStories(^3^33https://www.macstories.net/shortcuts), to further expand the scale of our dataset.
However, the source code of these shortcuts lacks detailed information about the involved actions, such as API metadata.
Inspired by ShortcutsBench (Shen et al., ), we extract action information from macOS’s built-in definition files and third-party application interface definition files.
For each API, we record its name, description, parameter names, parameter types, default values, return value types, and return value name, which provides a valuable resource for LLMs to efficiently interpret and utilize these APIs, even in zero-shot scenarios.

To ensure compatibility between the crawled shortcuts and the action interfaces, we implement a stringent filtering mechanism to verify that all API calls are executed correctly.
During this process, we identify that some shortcuts contain non-interpretable binary sequences as API parameters, potentially disrupting the training process of language models.
To maintain data quality, we remove these samples from the dataset. As a result, we curate a final set of $14,771$ high-quality shortcuts, ensuring the reliability of the dataset for subsequent data expansion and model training.

#### Shortcuts Transcription

The original shortcut source codes are written in property lists format (Hummert & Humphries, ), which sequentially encodes logical constructs like branches and loops. This encoding is notably different from the types of data commonly used in the pre-training of LLMs.
To address this gap, we convert the shortcuts into abstract syntax trees (ASTs), apply pre-order traversal to transform them into Python code, with further algorithmic details provided in Appendix A.
Furthermore, the original shortcuts use hexadecimal strings as variable names, leading to reduced semantic clarity.
To improve interpretability, we use ChatGPT to automatically reassign these variables with more contextually meaningful names, thereby enhancing the overall readability and utility of the code for further language model training.
A typical comparison between property lists and Python code can be found in Appendix C.

#### Thought Generation

To provide informative guidance for LLMs in orchestrating workflows, we design a three-level thought hierarchy from fine-grained to coarse-grained:
(1) Low-level comments are intended to clarify the purpose of each action within the workflow.
(2) Median-level plans represent an abstraction over a sequence of actions, outlining the collective goal of these steps.
(3) High-level queries reflect the user’s requirements, specifying the intended outcome without prescribing specific methods to achieve it.
These three levels of thought are generated through a bottom-up approach.
Specifically, given the transcribed workflow $w$, let the set of actions in the workflow $w$ be denoted as $\mathcal{A}$, where each action $a_{i}\in\mathcal{A}$ corresponds to a function calling in the Python code.
For each action $a_{i}$, we generate a corresponding comment $c_{i}$ by prompting ChatGPT.
Subsequently, given the action set $\mathcal{A}=\{a_{i}\}$ and comments $\mathcal{C}=\{c_{i}\}$ of workflow $w$, we prompt ChatGPT to generate the corresponding task plan $\mathcal{P}$.
We combine the task plan $\mathcal{P}$, the comments $\mathcal{C}$, and the action set $\mathcal{A}$ of the workflow $w$ to generate the high-level task query $\mathcal{Q}$.
This bottom-up manner is analogous to the summarization task, effectively ensuring content reliability and minimizing the risk of hallucination.

Figure: Figure 3: Illustration of data field composition in WorkflowBench comprising Task Query, API documentations, Task Plan, and Workflow code with Comments.
Refer to caption: x3.png

Finally, as Figure 3 shows, each workflow $w$ is represented as: $w=\{\mathcal{Q},\mathcal{D},\mathcal{P},\mathcal{A}\}$,
where the workflow $w$ consists of the task query $\mathcal{Q}$, action documentation $\mathcal{D}$ for all involved actions, the task plan $\mathcal{P}$, and all actions represented as annotated Python code $\mathcal{A}$.
A detailed example can be found in Appendix D.

### 3.2 Query Expansion

After performing a comprehensive statistical analysis on the collected data, we find that the data exhibits significant complexity, with an average of 70.4 actions and 12 branches, surpassing the complexity of existing workflow-related benchmarks. However, the diversity of the data is relatively low. Specifically, 40.3% of the workflows fall under the Utilities category, and over 99% of the APIs used are Apple’s built-in APIs (i.e., those classified as is_workflow_actions APP).

Therefore, we intend to expand the dataset by focusing on two key aspects:
(1) Diversity: making up for the lack of diversity in real data and covering a broad range of APIs and workflow categories to enhance the model’s utility and robustness; (2) Complexity: matching the action scale and logical complexity of the real-world data to ensure that they can effectively represent real-world problems and orchestrate nodes accordingly.
To this end, we sample APIs from diverse applications and multiple workflows with representative logical structures (e.g., whether they contain branches or loops) to synthesize additional queries.

To ensure that the number of APIs in the synthesized dataset aligns with real-world usage, we sample $n$ APIs based on real-world distributions. Approximately $\lfloor n/2\rfloor$ are drawn from Apple’s built-in API set (e.g., openurl or sendemail), with the remainder from third-party applications (e.g., OpenAI). The total number of built-in and external APIs is thus $n$.

To ensure that the sampled APIs can interact coherently, we do not sample directly from the entire API set.
Instead, we first randomly select 1-5 applications and then choose all APIs from these selected applications.
This method ensures that the selected APIs are functionally compatible and capable of representing real-world workflows.

The prompt used for ChatGPT to synthesize queries consists of four components: (1) a general prompt to describe the task query generation task, (2) documentations for the sampled APIs, (3) in-context examples from the collected data for reference, and (4) the workflow category to which the query belongs to.
By controlling the workflow category and in-context examples, we can ensure the diversity and complexity of the generated data. As seen from Figure 4, the synthesized query has a more balanced category distribution and uses more third-party APIs. Although most of the APIs used are still built-in APIs, this is reasonable considering that they carry necessary operations.

### 3.3 Workflow Generation

Figure: Figure 4: Comparison of the distributions across workflow categories, APPs, and action counts in the collected data and the final dataset. The upper section shows the original data collected from Apple Shortcuts and RoutineHub, while the lower section presents the expanded dataset distributions.
Refer to caption: x4.png

**Table 1: Detailed statistics of WorkflowBench. Seed. refers to the collected data from Shortcuts. Train. and Test. refers to the training set and the test set of WorkflowBench respectively.**
| Statistics | Seed. | Train. | Test. |
| --- | --- | --- | --- |
| Num. of Instances | 14,771 | 105,573 | 1,190 |
| Num. of APPs | 71 | 83 | 31 |
| Num. of APIs | 584 | 1,503 | 324 |
| Num. of Categories | 28 | 28 | 28 |
| Avg. Action | 70.4 | 78.5 | 41.7 |
| Avg. IF | 12.0 | 7.4 | 7.9 |
| Avg. LOOP | 0.7 | 0.5 | 0.5 |
| Avg. Nested Depth | 2.6 | 2.7 | 2.1 |

To annotate the corresponding workflows of the synthesized queries effectively, we train an annotator model based on the collected shortcuts data to support more diverse applications and categories, while ensuring consistency with the real-world data as much as possible.

#### Annotator Training

First, we construct the supervised fine-tuning (SFT) dataset based on the collected human-labeled shortcuts.
Specifically, each workflow data point comprises a query $\mathcal{Q}$, the corresponding action documentation $\mathcal{D}$, the task plan $\mathcal{P}$, and the workflow represented as annotated Python code $\mathcal{A}_{\text{commented}}$.
During the SFT process, as shown in Figure 3, we take the query $\mathcal{Q}$, the corresponding action documentation $\mathcal{D}$ as the input to guide the model to generate a task plan $\mathcal{P}$, followed by the step-by-step generation of the current thought (i.e., the comment $c_{i}$) and the corresponding action $a_{i}$, which includes the action name and its associated parameters.
We use the trained annotator to generate workflows $\mathcal{A^{\prime}}$ from synthesized queries.

#### Quality Confirmation

Due to the limited accuracy of the annotator model, the generated workflows may contain errors to some extent.
For example, we identify issues in $\mathcal{A^{\prime}}$ (e.g., extraneous branches not relevant to the query and incorrect function call formats).
To enhance the overall quality, we prompt ChatGPT with in-context samples to refine both $\mathcal{A^{\prime}}_{\text{commented}}$ and $\mathcal{P^{\prime}}$, ensuring that the workflow accurately addresses the query.
Then, we use rule-based filtering to remove workflows with fundamental errors.
Specifically, we remove samples that don’t incorporate code, don’t utilize the given APIs, or violate parameter constraints associated with those APIs.

Finally, we derive a synthesized dataset of $91,992$ instances, which is combined with the initially collected data to form the final WorkflowBench.
It contains $106,763$ instances with $1,503$ APIs across $83$ applications, which are used to train WorkflowLlama.
The statistics of WorkflowBench are listed in Table 1 and the distribution comparisons of workflow categories, APPs, and the number of actions between the collected data and final data are demonstrated in Figure 4.
From the statistical results, we can see that the synthetic data maintains complexity while expanding diversity.

## 4 Experiments

### 4.1 Experimental Setup

Training Details  We fine-tune the annotator and WorkflowLlama on LLaMA-3.1-8B (Dubey et al., ) for 3 epochs using the AdamW optimizer (Loshchilov & Hutter, ). A linear learning rate scheduler is used with a peak learning rate of $2\times 10^{-5}$ and a warm-up ratio of $0.1$. Each mini-batch contains $32$ examples, and the maximum sequence length is set as $8,192$ tokens.

Baselines
To provide a comprehensive comparison, we select several representative LLMs as baselines for our experiments. These baselines include proprietary models such as GPT-4o-mini and GPT-4o, as well as open-source models like Qwen2-7B (qwe, ), Llama-3.1-8B, and Llama-3.1-70B (Dubey et al., ).
Additionally, we apply in-context learning (ICL) (Dong et al., ) with one random-sampled instance to these baselines to better adapt them for workflow orchestration.

Metrics  In the main experiments, we use both reference-code-based metrics and a model-based evaluation to comprehensively evaluate the quality of the generated workflows.
For reference-based metrics, we apply CodeBLEU (Ren et al., ) with four components:

- •
BLEU measures N-gram overlap for token-level similarity.
- •
Weighted N-Gram Match assigns higher weights to critical code tokens like keywords.
- •
Syntactic AST Match compares the Abstract Syntax Trees (ASTs) to assess syntactic accuracy.
- •
Semantic Data-Flow Match evaluates logical correctness by comparing data-flow relationships between variables.

Together, these components provide a comprehensive evaluation of both syntactic and semantic aspects of the workflows.
We follow Ren et al. (), setting the four components to 0.1, 0.1, 0.4, and 0.4, respectively, and calculate a weighted sum to obtain the CodeBLEU score.
For model-based evaluation, we elaborately prompt ChatGPT as the automatic evaluator to evaluate the Pass Rate of the generated workflows.

### 4.2 Effectiveness of Evaluator

To validate the reliability of the ChatGPT evaluator in terms of Pass Rate, we sample 30 instruction-response pairs (i.e., task queries and their corresponding workflow codes) for each model in Table 4.2, forming a human-evaluated dataset of $330$ instances ($30\times 11=330$). First, we use GPT-4o-mini to label whether each instance could complete the given tasks only using the provided APIs. Then, human evaluators re-label the sampled data according to the same criteria. Ultimately, $268$ instances are labeled consistently by both the ChatGPT evaluator and human evaluators, achieving an agreement rate of $\textbf{81.2}\%$, demonstrating the reliability and effectiveness of the evaluator.

**Table 2: Performance comparison of various models on the test set of WorkflowBench under the unseen instructions (ID) and unseen APIs (OOD) settings (%).**
| Model | F1 |
| --- | --- |
| Proprietary Models |  |
| Claude2 | 84.9 |
| GPT-3.5 | 86.6 |
| GPT-4 | 86.7 |
| Open-Source Models |  |
| Qwen-7B | 63.1 |
| Mistral-7B | 64.9 |
| Llama-3.1-8B | 68.2 |
| Qwen-14B | 69.7 |
| Llama-2-13B | 65.1 |
| Vicuna-13B | 54.0 |
| Baichuan2-13B | 52.1 |
| WizardLM-70B | 42.7 |
| Llama-2-70B | 63.1 |
| Qwen-72B | 73.4 |
| WorkflowLlama (8B) | 77.5 |

### 4.3 Main Experiments

Settings
The main experiments are conducted using the test set of WorkflowBench. Ideally, by scaling both the quantity and diversity of instructions and unique tools within the training data, WorkflowLlama is expected to generalize to novel instructions and APIs that are not seen during training.
This is particularly important because it enables users to define custom APIs and allows WorkflowLlama to adapt based solely on the provided documentation.
To evaluate this capability, we assess WorkflowLlama’s generalization performance at two levels: (1) Unseen Instructions, considers an In-Distribution (ID) setting, which involves using the same set of APIs as those in the training data, and (2) Unseen APIs, considers an Out-Of-Distribution (OOD) setting, involving only 50 common APIs required to construct workflows and APIs that are absent from the training data.
Since WorkflowBench contains a comprehensive set of APIs, which poses a substantial challenge for LLMs in terms of API comprehension and selection, we provide the correct APIs directly as input. It allows us to focus on the workflow orchestration, bypassing the issue of API selection.

Main Results
The results are placed in Table 4.2, from which we derive that:

- 1.
Although multiple workflows can successfully complete a query, there is a positive correlation between the reference-free Pass Rate metric and the reference-based CodeBLEU metric. Given that the Pass Rate metric derived from ChatGPT aligns with human evaluations over 80% of the time, CodeBLEU serves as a reliable proxy for evaluating workflow orchestration capabilities.
- 2.
All models demonstrate a certain capacity for workflow orchestration. This may stem from their inherent instruction-following and code-generation capabilities.
We find that models like GPT-4o and Llama-3.1-70B, which perform better on generic tasks, also excel in workflow orchestration.
In addition, prompting with in-context samples significantly enhances the models’ performance.
- 3.
We find that scores on text overlap metrics such as BLEU and weighted N-gram are low for all models. Even the fine-tuned WorkflowLlama only achieves $8.2\%$ and $9.7\%$ on these two metrics.
This is because the reference codes consist mainly of workflows with function names and arguments, and contain few Python-related keywords, making exact matching challenging.
In contrast, models achieve better scores on syntactic AST match and semantic data-flow match.
- 4.
After fine-tuning, WorkflowLlama shows a significant improvement in its ability to orchestrate actions. The performance of WorkflowLlama even outperforms powerful closed-source models GPT-4o with ICL by a large margin. Specifically, WorkflowLlama achieves a $\textbf{39.3}\%$ score on CodeBLEU and a $\textbf{76.9}\%$ Pass Rate under ID settings, demonstrating the validity of our proposed WorkflowLLM framework and WorkflowBench dataset.
- 5.
WorkflowLlama demonstrates strong generalization capabilities. Even though it has not been trained on the same instructions or APIs, it still significantly outperforms the vanilla Llama-3.1 on all metrics, ahead of or close to the more powerful foundation models.
Notably, our method achieves $\textbf{35.1}\%$ in CodeBLEU and $\textbf{70.4}\%$ in Pass Rate, outperforming all strong baselines.

### 4.4 Analysis of Workflow Complexity

Figure: Figure 5: Performance comparisons based on the number of actions, the number of Branch & Loop, and the nested depth of the reference code.
Refer to caption: x5.png

To evaluate the models’ ability to generate workflows of varying complexity, we break down the performance of CodeBLEU according to the total number of actions, the number of branches and loops, and the nested depth of the reference code. As shown in Figure 5, the performance of all models deteriorates as the number of actions or the logical complexity increases, indicating the challenge of orchestrating complex workflows.
However, across all levels of complexity, WorkflowLlama significantly outperforms all other models. Moreover, the relative performance of WorkflowLlama improves as the complexity of the workflow increases, which demonstrates fine-tuning with WorkflowBench significantly enhances the model’s ability to handle more complex workflows.

### 4.5 Out-of-distribution Generalization to T-Eval (Chen et al., 2024 )

**Table 3: Comparisons of F1 scores on the PLAN task of T-Eval. (Bold denotes the best score among models of the same category.)**
| Model | F1 |
| --- | --- |
| Proprietary Models |  |
| Claude2 | 84.9 |
| GPT-3.5 | 86.6 |
| GPT-4 | 86.7 |
| Open-Source Models |  |
| Qwen-7B | 63.1 |
| Mistral-7B | 64.9 |
| Llama-3.1-8B | 68.2 |
| Qwen-14B | 69.7 |
| Llama-2-13B | 65.1 |
| Vicuna-13B | 54.0 |
| Baichuan2-13B | 52.1 |
| WizardLM-70B | 42.7 |
| Llama-2-70B | 63.1 |
| Qwen-72B | 73.4 |
| WorkflowLlama (8B) | 77.5 |

Settings
To further evaluate the generalization capability of WorkflowLlama, we conduct experiments using an OOD benchmark, T-Eval, a widely-used benchmark to evaluate the multi-step decision-making capability of LLMs to utilize APIs.
The original data format in T-Eval is based on JSON or strings, which differ significantly from the Python-based format employed in WorkflowBench. To ensure the evaluation metrics’ consistency between ours and the original paper, we convert WorkflowBench into JSON format while preserving the metadata of workflows and the specifics of queries. Subsequently, we retrain WorkflowLlama on the transformed dataset.
We employ the F1 Score proposed in the original paper to measure the alignment with the reference API sequences.

Results  The results are shown in Table 3. As observed, WorkflowLlama demonstrates strong OOD generalization performance on the T-Eval benchmark, despite being trained on different domains and tasks using different APIs. Notably, WorkflowLlama significantly outperforms the vanilla Llama3.1-8B as well as larger open-source models like Llama-2-70B and Qwen-72B, highlighting that fine-tuning with WorkflowBench enhances the model’s out-of-distribution planning ability.

### 4.6 Ablation Study

Table: Table 4: Ablation study results of Natural Language Thoughts on Workflow Orchestration (%).

### 4.7 Case Study

Figure: Figure 6: Case study of generated code between vanilla Llama-3.1-8B and WorkflowLlama.
Refer to caption: x8.png

To further illustrate the effect of fine-tuning on WorkflowBench, we present a typical example in Figure 6. In this case, the vanilla Llama-3.1 model exhibits two types of errors.
First, the model does not adhere to the given instructions for workflow orchestration, using APIs outside the provided list, i.e., hallucination APIs. Specifically, it uses the time.sleep() function instead of is_workflow_actions_delay() to set a timer.
Second, due to its relatively weak workflow orchestration capabilities, the model fails to complete all user instructions. Specifically, it does not activate airplane mode using the is_workflow_actions_airplanemode_set() function.
Fine-tuning on WorkflowBench effectively alleviates these two issues. However, we observe that fine-tuning also introduces redundant actions. For instance, WorkflowLlama repeats the parsing check of the clipboard’s content. We will address this redundancy problem in future work.

## 5 Conclusion

In this paper, we present WorkflowLLM to enhance the capability of large language models in workflow orchestration.
In WorkflowLLM, WorkflowBench is constructed covering $106,763$ workflows with $1,503$ APIs across $83$ applications through a three-phase pipeline.
By fine-tuning Llama-3.1-8B on WorkflowBench, we derive WorkflowLlama which can achieve superior performance on the workflow orchestration task exceeding all comparable baselines including the most advanced OpenAI GPT-4o.
Moreover, we adapt our WorkflowLlama on the T-Eval dataset and the experimental results reveal the generalization ability of our constructed WorkflowBench.
We believe that our constructed dataset has the potential to contribute to advancements in APA.

## Appendix A Algorithm of Transcribing Shortcuts

Figure: Algorithm 1 Recursive Parsing of Property List to Construct Abstract Syntax Tree

## Appendix B Prompt Design

### B.1 Workflow Orchestration Prompt

### B.2 Evaluator Prompt

### B.3 Comment Generation Prompt

### B.4 Task Plan Generation Prompt

### B.5 Task Query Generation Prompt

### B.6 Query Expansion Prompt

### B.7 Quality Confirmation Prompt

### B.8 Variable Rename Prompt

## Appendix C Case Study of Shortcuts

We provide a real-world shortcut example, which includes the following three presentation forms: the rwa property list configuration file, the Python code after transcription and variable renaming, and the visual interface on MacOS.

The raw property list configuration file is presented below. For the sake of brevity, we have omitted the middle portion containing the actions.

It can be observed that this configuration file employs non-semantic hexadecimal strings to represent variables and uses keywords such as is.workflow.actions.conditional and GroupingIdentifier to implement logic controls like conditions, making it inherently difficult to read and comprehend. Consequently, we have converted it into a Python-like code format. The Python code, after transcription, variable renaming, and commenting, is shown as follows:

Figure: Figure 7: The visual interface of the shortcut RoutineHub · Buy Kindle Book.
Refer to caption: x9.png

We can clearly see that the transcribed Python code is of high quality, with strong readability, making it well-suited for training LLMs that have been pre-trained on extensive Python code.

For easy understanding, we also present the visual interface of this shortcut in Figure 7, which corresponds to the transcribed Python code on a line-by-line basis.

## Appendix D Case Study of WorkflowBench

In this section, we provide a workflow example of WorkflowBench based on the shortcut listed in Appendix C. We omit the annotated workflow code that has been shown in Appendix C.

The query $\mathcal{Q}$ is:

The list of provided APIs is:

The task plan $\mathcal{P}$ is:

## Appendix E Limitations

While the framework proposed in this paper represents notable progress in workflow orchestration, it also has certain limitations that warrant discussion. First, the APIs used in our work are exclusively derived from Apple Shortcuts application, resulting in a lack of coverage across more diverse fields and potentially limiting the generalizability of the dataset to broader application contexts. Second, our approach lacks evaluation through actual execution. This limitation arises due to the complexities involved in executing workflows, such as the need for intricate user registration and permission acquisition. Moreover, the APIs are subject to frequent changes as applications continue to evolve, making it challenging to implement a consistent executable evaluation. Consequently, our evaluation is limited to static analysis.

## Appendix F Ethical Statement

In this study, the dataset construction process was fully automated using LLMs and algorithms for data annotation, eliminating the need for human annotators and thereby avoiding concerns related to annotator compensation and working conditions. The data utilized was collected through web scraping from publicly accessible sources, with strict adherence to the Terms of Service (ToS) of the respective websites. Scraping was avoided on platforms where such activity is explicitly prohibited, ensuring compliance with ethical standards. Additionally, no personally identifiable information (PII) or private user data was collected at any stage of the research process. All data was anonymized to protect privacy and mitigate any potential ethical concerns related to user information.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 1
questions: Q1)The “pass rate” is an important metric to evaluate the performance in section 4.3. Could you elaborate more on how it is calculated? What are the reasons to choose it? And how is related to training/fine-tuning phases?

Q2)Could you  contextualize the size and the quality of the generated dataset   in terms of  how significant it is in comparison with SOTA or related works? Are they general enough for RPA for just for applications around Apple Shortcuts and RoutineHub?

Q3) The papers also mention about fine-tuning Worfflow Lalama and annotator in a very short paragraph in section. Could you elaborate more about setup, why such training parameters are chosen?
soundness: 2
strengths: S1)The observation and problems are interesting and relevant to the conference

S2) The dataset might have potential uses

S3)The experiments involve many systems
summary: The paper argues that state-of-the-art models like GPT-4o face challenges in effectively handling complex workflows. To address this, the paper introduces WorkflowLLM, a data-centric framework designed to enhance the workflow orchestration capabilities of LLMs. Central to this framework is WorkflowBench, a large-scale fine-tuning dataset comprising 106,763 workflows, 1,503 APIs, and 83 applications across 28 categories. WorkflowBench is constructed through a three-phase pipeline: data collection from sources like RoutineHub, query expansion using ChatGPT to create diverse and complex workflows, and workflow generation leveraging a trained annotator model for high-quality workflow synthesis. The dataset is enriched with Python-style code, comments, and hierarchical thought structures to improve LLM learning.

Based on WorkflowBench, the paper fine-tunes Llama-3.1-8B, resulting in WorkflowLlama, which demonstrates superior performance compared to existing models, including GPT-4o, on workflow orchestration tasks. WorkflowLlama achieves notable success in generalization to unseen APIs and tasks, evaluated using metrics such as CodeBLEU and Pass Rate. Additionally, WorkflowBench exhibits robust zero-shot generalization on the T-Eval benchmark, achieving an F1 plan score of 77.5%.
weaknesses: W1) The scientific or technical contributions are limited as the key contributions of paper are around data curation effort. And the significance of data size and quality  is not clear (see Q2). 
 
W2) Section 3 mentions many steps to generate the needed data for its workflow using ChatGPT but the Quality control protocol is not very clear. Even the author provides some examples and algorithms in appendix, the details are very descriptive . Also section4.2  mentions using  human evaluator to re-label the sampled data,It’d be better to provide more details towards the quality control protocol for this human-driven process as well.

W3) Many technical details are not very clear (see questions)

W4) Code is provided in supplemental material but there is no document how to reproduce the experiment results ( also see Q3)

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Are the prompts used in the framework that drive the LLMs usage automatically generated?  If yes, how to ensure the quality of the prompts? If not, how to scale up?

For each step from data generation to model evaluation, how to optimize the prompts?

The description on quality confirmation is a bit vague. The example given about the issues is not clear. How are such issues detected? How to prompt ChatGPT to refine A’ and P’, and how to ensure the quality of the refinement? How to perform the rule-based filtering, e.g., how to automatically detect if parameter constraints are violated or not?

For OOD settings, how are the other models evaluated? Do they also be given the same input like the one for Workflow Llamma, i.e., providing the list of required APIs? 

Given synthetic data takes the large portion of the final benchmark (91k out of 11k) but the quality is unsure, it would be interesting to also see how Workflow Llama performs on the real-word dataset, i.e., without the synthetic data, compared to other models? 

The proposed framework outperforms all the other models by an extremely large margin in the first four columns in Table 2 (such as 9.4% vs 0.5). This seems to be suspicious and needs further examination and explanation. 

In Table 2, both CodeBLEU and Pass Rate are used, one examines the similarity of code and one examines if the code is executable. It would be interesting to combine both of them, such as setting a threshold on the CodeBLEU score when checking the Pass Rate to compare models, since Pass Rate itself doesn’t necessarily indicate the success of the workflow orchestration.
soundness: 2
strengths: The paper is well-written and easy to follow. Examples are given to illustrate the important steps. 

The extended dataset improves the diversity and complexity of the samples in the training data, allowing the models to adapt to the situations that are closer to real-world applications. 

The experimental results show that the proposed framework outperformed the existing models by a large margin.
summary: This paper proposes a framework that leverages LLMs to automate workflow orchestration. It extends the real world dataset by creating synthetic data through LLMs, to train the model. It conducts experiment study to compare the proposed framework with several proprietary and open source models, using CodeBLEU and Pass Rate as the metrics.
weaknesses: The paper mainly relies on prompting LLMs to generate the results of the key steps, such as data generation, commenting actions, and plan generating. It seems that there is a large space of prompt engineering to improve the performance but left to be undone. 

The proposed framework bypasses the API selection when handling the OOD setting by directly serving the required APIs as the input. This greatly simplifies the problem of automating workflow orchestration. 

The paper relies heavily on LLMs for the entire process, including generating the dataset, training the model, and evaluating the models (only less than 82% accuracy in a small sample with 330 instances in total). This weakens the technical contribution and the reliability of the experiment results.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: I am curious about the choice of ChatGPT to run both the query expansion phase of the dataset creation (section 3.2) and the experiments (sections 4.1 and 4.2). Altough I think it was a good idea, why rely on a proprietary model (the GPT-4o) with proprietary and non-disclosed prompt engineering behind it? Did the authors try to use directly the GPT-4o model? Did they try other models to run this phase? I assume that the "ease-of-use" of the ChatGPT is a fair point in favor of its use, but it should be evident in the paper.
soundness: 3
strengths: The paper is well-presented and well-written overall. The authors identified a clear pain point in achieving an agentic LLM approach for process automation: lacking LLMs capable of correctly describing complex workflows. Although partially addressed by other works, as pointed out by the authors in the related work section (section 2), their approach is innovative in dealing with fairly more complex workflows than previous solutions.

Their approach to overcoming this limitation is sound and clever. They provide a fine-tuned LLM specialized to handle this kind of task.
To do so, they craft a well-defined dataset, and it is possible to highlight the process of dataset creation as one of the paper's main contributions, as crucial as the constructed dataset and the fine-tuned model.
After the data collection, the query expansion phase is especially interesting because it uses another LLM (e.g., ChatGPT) to help overcome the lack of diversity and complexity of the gathered initial data with syntactic data. The same applies to the clever use of ChatGPT in the evaluation phase.
Compared with other LLMs solving the same kind of problem, the presented results show the potential use of the author's solution to deal with the initial established problem, i.e., the automation of complex tasks.
WorkflowLlama outperforms other LLMs when the workflow demands generalization to unseen instructions and unknown APIs.

I also highlight the good approach to evaluating the effectiveness of the LLM-based evaluator (section 4.2), which strengthens the paper's arguments.
summary: The paper presents a detailed explanation of the construction and evaluation of the WorkflowBench dataset, which contains many examples of workflow descriptions. The dataset is later used to fine-tune the open-source LLM Llama-3.1-8B.
The dataset creation follows a well-established and potentially reproducible approach. It starts with data gathering, expands to increase data diversity, and generates a final workflow dataset, enhancing real collected data with synthetic data.
The fine-tuning intends to overcome the limitations of existing LLMs in workflow orchestration and provide a consistent improvement in process automation by proposing an agentic approach.
The fine-tuned LLM called WorkflowLllama is detailed, and its capabilities are evaluated, showing a solid capacity to orchestrate complex workflows. Its performance was compared with commercial and open-source LLMs using the test set of Workflowbench.
weaknesses: - The background on automation's relevance (lines 36-38) might be condensed, as automation's importance is widely recognized.
- In section 3.1, the author can better explain the $\mathcal{D}$. The other elements were presented before, but  $\mathcal{D}$ is directly introduced on line 248 without prior explanation.
- Lines 245-247 read (my emphasis):  "This bottom-up [...], efectively **ensuring content realiability** and **minimizing the risk of hallucination**". These are pretty strong assumptions without solid evidence, especially if generalized to every LLM. Do the authors know published papers that confirm them (since this paper is not about reducing hallucinations)? If not, the phrase can be removed or rephrased to be more humble.
- The paper lacks a clearer section on threats to validity. The authors provided some in Appendix E, but they should be incorporated into the main text.
- The paper also lacks a more evident running example/use case. Although the authors provide a small case study in section 4.7 and some examples in Appendix D, a consistent running example, incorporated throughout sections, could clarify the model's applications and emphasize practical use cases.

Minor issues:
- Line 52 mentions GPT-4, while the rest of the paper uses GPT-4o(-mini). Maybe it is only a typo, or the work mentioned indeed uses the GPT-4 model. Since they are different models, it can be good to double-check every reference to ensure it always talks about the correct model, maybe highlighting somewhere they are not the same.
- The size of Figure 2 can be increased to improve readability.
- Figures 3,5 and 6 could improve accessibility through higher contrast colors.
- Although well-written, the paper is sometimes repetitive (e.g., sections 1 and 3). Proofreading with this issue in mind may help the authors achieve a better final text.

---

confidence: 4
contribution: 1
initial_rating: 5
presentation: 2
questions: 1. The performance of generic LLMs in workflow orchestration through agent-like approaches would serve as an interesting baseline for comparison. Through this comparison, we could better understand whether the core challenge lies in the model's reasoning capabilities or in its ability to maintain consistent output formats. This point won't affect the score; it's just for discussion.
soundness: 2
strengths: 1. Accurate and valuable problem identification. The paper identifies the limitations of existing work in handling real-world workflows: difficulty in processing long-sequence workflows with complex logical relationships like control flow. This observation directly addresses practical application pain points, making the motivation very solid.
2. Proposes a systematic data construction method. The paper designs a complete data synthesis process: from real data collection, workflow abstract representation, data synthesis to quality control, with detailed design considerations at each stage. The resulting large-scale dataset provides an important resource for this field.
3. Comprehensive experimental validation. The effectiveness of the method is strongly supported through multi-dimensional evaluation using CodeBLEU and Pass Rate, cross-domain testing with T-Eval, along with detailed case analysis and ablation studies.
summary: This paper proposes a pipeline for constructing workflow (RPA) data to enhance LLMs' workflow orchestration capabilities. The core motivation stems from the observation that real-world workflows require more complex logic and longer sequences than what current LLMs can directly generate. The primary contribution is the construction of a dataset containing over 100K samples. The main limitations lie in the lack of certain technical details (such as specific ChatGPT versions used for annotation) and the lack of deeper investigation into model capability improvements.
weaknesses: 1. Important technical details are missing. The paper mentions using ChatGPT for data annotation and optimization multiple times but doesn't specify the model version. For work with data construction as its core contribution, this seriously affects reproducibility and rigor.
2. Lacks methodological innovation and mechanism analysis. Although data construction is an important contribution, the paper lacks in-depth analysis of how this data enhances model capabilities. Specifically, it doesn't investigate whether the improvements come from enhanced logical reasoning abilities or simply better format matching. Without such analysis, it's difficult to determine if the model has truly learned to reason about complex workflows or has merely memorized patterns from the training data.
3. Missing critical ablation experiments. The paper doesn't compare the performance difference between the Annotator Model and WorkflowLLM, despite both models using the same training method and data type. This results in question the necessity of the data synthesis strategy and weakens the core argument.
4. The paper mentions that the Annotator Model can generate workflows with over 70 actions, and theoretically WorkflowLLM should be capable of the same. Given the general limitations of LLMs in long sequence generation, including such long-sequence workflows in the appendix would further demonstrate the work's contribution.
5. Given that the core contribution is data construction, whether the dataset will be open-sourced directly affects the practical value of the work. The authors should consider open-sourcing the dataset to promote development in this field.

Meta-review: The paper introduces WorkflowLLM, a framework aimed at improving workflow orchestration using large language models (LLMs). A key contribution is WorkflowBench, a large-scale dataset containing over 106,000 workflows, enriched with Python-style code, comments, and hierarchical structures to facilitate learning. WorkflowBench is created through a systematic three-phase process involving data collection, query expansion, and synthetic data generation via an annotator model. Using this dataset, the authors fine-tune Llama-3.1-8B, producing WorkflowLlama, which demonstrates state-of-the-art performance on workflow orchestration tasks compared to both open-source and commercial models, achieving significant improvements on metrics like CodeBLEU and Pass Rate, as well as robust zero-shot generalization on unseen APIs and tasks.

Strengths
+ The paper identifies a valuable and practical challenge: the limitations of current LLMs in handling complex workflows with long sequences and logical dependencies.
+ A systematic and reproducible data construction pipeline is proposed, producing a dataset that enhances diversity and complexity, which is critical for advancing LLM capabilities in this domain.
+ Extensive experiments validate the effectiveness of the proposed framework, showing significant performance gains across multiple metrics (e.g., CodeBLEU, Pass Rate) and strong generalization capabilities.
+ The paper is well-written, easy to follow, and includes illustrative examples and case studies to support key claims.
+ WorkflowLlama demonstrates the potential to handle out-of-domain (OOD) tasks and unseen APIs, outperforming competing models in these challenging scenarios.

Weaknesses
+ The technical contributions of the paper are limited, with a heavy reliance on data curation rather than novel methodological advances or architectural innovations.
+ The quality control process for the dataset, including human evaluation and LLM-based annotation, is not well-detailed, which raises concerns about reproducibility and reliability.
+ Critical technical details, such as the specific version of ChatGPT used and the exact evaluation setups, are missing, limiting the clarity and rigor of the proposed approach.
+ The experimental results, especially the extremely high performance margins (e.g., in Table 2), appear suspicious and require further justification or ablation studies to confirm their validity.
+ The paper lacks a comprehensive discussion of threats to validity and could benefit from a more detailed, consistent running example to illustrate the model's real-world applicability.

Some concerns have been addressed by the authors during the rebuttal period.

Decision: Accept (Poster)