ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents

## Abstract

Abstract Recent advancements in integrating large language models (LLMs) with application programming interfaces (APIs) have gained significant interest in both academia and industry. These API-based agents, leveraging the strong autonomy and planning capabilities of LLMs, can efficiently solve problems requiring multi-step actions. However, their ability to handle multi-dimensional difficulty levels, diverse task types, and real-world demands through APIs remains unknown.
In this paper, we introduce ShortcutsBench , a large-scale benchmark for the comprehensive evaluation of API-based agents in solving tasks with varying levels of difficulty, diverse task types, and real-world demands. ShortcutsBench includes a wealth of real APIs from Apple Inc.’s operating systems, refined user queries from shortcuts, human-annotated
high-quality action sequences from shortcut developers, and accurate parameter filling values about primitive parameter types, enum parameter types, outputs from previous actions, and parameters that need to request necessary information from the system or user.
Our extensive evaluation of agents built with 5 5 5 5 leading open-source (size >= 57B) and 4 4 4 4 closed-source LLMs (e.g. Gemini-1.5-Pro and GPT-3.5) reveals significant limitations in handling complex queries related to API selection, parameter filling, and requesting necessary information from systems and users. These findings highlight the challenges that API-based agents face in effectively fulfilling real and complex user queries. All datasets, code, and experimental results will be available at https://github.com/eachsheep/shortcutsbench .

## 1 Introduction

Large language model based agents (LLM-based agents) (Wang et al., ; Xi et al., ) built on application programming interfaces (APIs) (Qin et al., ; Huang et al., ) have recently gained significant interest in both academia (Shen et al., ; Wang et al., ) and industry (Microsoft, ; OpenAI, ). By integrating LLM with APIs, these agents can access real-time information (OpenAI, ; Microsoft, ), reduce hallucination with external knowledge (Li et al., ; Gao et al., ), and automatically plan and complete complex tasks that need multi-step actions (Gravitas, ; Pan et al., ). Many of these agents (OpenAI, ; Microsoft, ; Gravitas, ) have also demonstrated commendable performance on simple tasks involving only a few actions such as “Check the weather ① and tell me ②” (OpenAI, ). These impressive performances raise an important question:
Do these API-based agents truly possess the capability to generate complex action sequences for real demands with real APIs?

Some existing benchmarks / datasets (Huang et al., ; Qin et al., ; Patil et al., ; Tang et al., ; Li et al., ; Xu et al., ; Zhuang et al., ; Schick et al., ; Hao et al., ) have attempted to evaluate API-based agents. However, they have three limitations:
First, the APIs (a.k. tools available to the agent) lack richness, and the queries (a.k. the task to the agent) lack complexity (Table 1). They either involve a limited number of APIs (Li et al., ; Xu et al., ; Zhuang et al., ) or cover only a few numbers of
apps (Patil et al., ; Tang et al., ; Schick et al., ; Hao et al., ) (an app may have one or multiple APIs), or the difficulties of the queries are limited in a narrow range, with the average action length ranges from $1$ to $5.9$ (Table  1).
This lack of richness and complexity makes it difficult to effectively distinguish the capabilities of different agents, particularly on larger and smarter LLMs like Gemini-1.5 Pro DeepMind () and GPT-3.5-turbo OpenAI ().
Second, the APIs lack realism as they may be manually crafted, and the queries fail to reflect actual user demands since they may be either created by hand or generated directly by ChatGPT OpenAI () without verifying real user demands (Table 1).
Moreover, they only cover the evaluation of API selection, lacking a study on API parameter filling (Table 1). Efficient and accurate parameter filling is essential for an agent to complete tasks successfully.
Thirdly, they don’t adequately evaluate the agent’s ability to ask systems or the users for the necessary information to resolve the missing information for solving the queries (Table 1). This is crucial as a user’s query may be implicit or may not provide all the information an agent needs to solve the task (Qin et al., ; Qian et al., ) effectively.

In this paper, we present ShortcutsBench. To our best known, ShortcutsBench is the first large-scale real API-based agent benchmark considering APIs, queries, and corresponding action sequences.
ShortcutsBench provides rich real APIs, queries with various difficulties and task types, high-quality human-annotated action sequences by shortcuts developers, along queries from real user demands. Moreover, it also provides precise values for parameter filling, including primitive data types, enum types, and the use of output from previous actions for parameter values, as well as evaluations of the agent’s awareness in requesting necessary information from the system or user.
Furthermore, the scale of APIs, queries, and the corresponding action sequences in ShortcutsBench is comparable or even better to benchmarks and datasets created by LLM or modified by existing datasets. The overall comparison between ShortcutsBench and existing benchmarks / datasets is listed in Table 1.

We conducted extensive evaluations of API-based agents from $9$ leading LLMs on  ShortcutsBench, including the evaluation of API selection, parameter value filling, and recognition of the need for input from the system or the user. The chosen LLMs including four closed-sourced LLMs like Gemini-1.5-Pro (DeepMind, ) and GPT-3.5-turbo (OpenAI, ), and five open-source LLMs like LLaMA-3-70B (Meta, ) and QWen-2-70B (Qwen, ).
Our findings highlight the limitations of these agents in addressing real, rich, and complex user queries.
In summary, this paper makes the following key contributions:

- •
To our best known, we have built the most realistic, rich, and comprehensive API-based agent benchmark. This benchmark is even comparable in scale to existing benchmarks / datasets built using LLMs (Table 1).
- •
We evaluated nine most advanced and mainstream LLM-based agents on all operations required to complete user queries, including API selection, parameter filling, and their awareness to request necessary information from the system or user when needed.
- •
We obtained massive interesting conclusions such as (1) The performance gap between open-source and closed-source LLMs has become very small; (2) Existing LLMs still have significant shortcomings in multi-step reasoning; (3) Extracting necessary parameters from queries is the most challenging task in parameter filling; (4) There is a substantial lack of awareness in agents when it comes to requesting necessary information.

**Table 1: ShortcutsBench has a great advantage in the realness, richness, and complexity of APIs, queries, and corresponding action sequences, the validity of action sequences, accurate parameter value filling, the awareness for asking information from the system or the users, and the overall scale.**
| Resource |  | Shortcuts<br>Bench |  | Meta<br>Tool |  | Tool<br>LLM |  | API<br>Bench |  | Tool<br>Alpaca |  | API<br>Bank |  | Tool<br>Bench |  | Tool<br>QA |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| (Ours) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Real API? |  | ✓ |  | ✓ |  | ✓ |  | ✓ |  | ✓ |  | ✗ |  | ✗ |  | ✗ |
| Demand-driven Query? |  | ✓ |  | ✗ |  | ✗ |  | ✗ |  | ✗ |  | ✗ |  | ✗ |  | ✗ |
| Human-Annotated Act.? |  | ✓ |  | ✗ |  | ✗ |  | ✗ |  | ✗ |  | ✗ |  | ✗ |  | ✗ |
| Multi-APIs Query? |  | ✓ |  | ✓ |  | ✓ |  | ✗ |  | ✗ |  | ✗ |  | ✗ |  | ✓ |
| Multi-Step Act.? |  | ✓ |  | ✓ |  | ✓ |  | ✗ |  | ✓ |  | ✓ |  | ✓ |  | ✓ |
| Prec. Val. for Para. Fill? |  | ✓ |  | ✗ |  | ✗ |  | ✗ |  | ✗ |  | ✗ |  | ✗ |  | ✗ |
| Awareness for Ask Info? |  | ✓ |  | ✗ |  | ✗ |  | ✗ |  | ✗ |  | ✗ |  | ✗ |  | ✗ |
| # Apps |  | 88 |  | 390 |  | 3451 |  | 3 |  | 53 |  | 400 |  | 8 |  | 13 |
| # APIs |  | 1414 |  | 390 |  | 16464 |  | 1645 |  | 53 |  | 400 |  | 232 |  | 13 |
| # Queries |  | 7627 |  | 21127 |  | 12657 |  | 17002 |  | 3938 |  | 274 |  | 2726 |  | 1530 |
| # Avg APIs |  | 9.62 |  | 1.0 |  | 2.3 |  | 1.0 |  | 1.0 |  | 2.1 |  | 5.4 |  | 13.0 |
| # Avg Actions |  | 21.62 |  | 1.0 |  | 4.0 |  | 1.0 |  | 1.0 |  | 2.2 |  | 5.9 |  | 1.0 |

## 2 Related Work

##### API-based agents.

API-based agents treat APIs as tools (Huang et al., ; Qin et al., ; Patil et al., ; Tang et al., ; Li et al., ; Xu et al., ; Zhuang et al., ; Schick et al., ; Hao et al., ; Zhu et al., ; Gravitas, ; AgentGPT, ). They accept queries, generate action sequences based on queries and provided APIs, and generate next action depends on the history actions (Yao et al., ). Related work about API-based agents can generally be categorized into $3$ types depending on the objective: (1) Task-specific enhancement focuses on improving the agent’s ability to solve a specific type of task like game and question-answering (Hao et al., ; Zhu et al., ; Gravitas, ; AgentGPT, ). (2) Data-driven workflows emphasize the importance of data by researching how to construct workflows to get action sequences, enabling generated data to fine-tune the model (Qin et al., ; Patil et al., ; Tang et al., ; Xu et al., ; Zhuang et al., ; Schick et al., ). (3) Agent evaluation studies the assessment of agents (Huang et al., ; Li et al., ).

##### Code-based agents.

Code-based agents use code generated as a medium for interaction with the external environment (OpenAI, ; Wang et al., ; OpenInterpreter, ; Wu et al., ). They accept queries, generate scripts in programming languages such as Python (OpenAI, ), JavaScript (Wang et al., ), or Shell (OpenInterpreter, ), and then input the code into interpreters. The results of the interpreter are then returned to the agent, which is used to help determine the next action in code generation. Currently, these approaches primarily focus on enhancing agent performance in specific tasks by incorporating additional knowledge (Wang et al., ; Wu et al., ), increasing feedback (OpenAI, ; Wang et al., ; OpenInterpreter, ; Wu et al., ), and decomposing tasks (OpenAI, ; Wang et al., ; OpenInterpreter, ; Wu et al., ).

##### Digital Automation Platforms (DAPs).

DAPs (Abdou et al., ; Coze, ; Rahmati et al., ; Chakraborti et al., ) refer to software tools or services designed to optimize workflows through automation. DAPs leverage technologies such as robotic process automation (RPA) (Chakraborti et al., ) and low-code/no-code development tools to achieve the goals. DAPs like Zapier (Abdou et al., ; Rahmati et al., ), Make (Make, ), and IFTTT (Abdou et al., ; Rahmati et al., ) offer extensive APIs that enable users to create automated workflows. Similarly, DAPs such as Microsoft Power Automate (Abdou et al., ) and Tasker (Dias, ) are primarily used to build workflows on Azure and Android, respectively. Recently, with the rise of LLM-based agents, platforms like Coze (Coze, ) and Dify (Dify, ) have emerged as “agent construction platforms”. Functionality like “workflow” in these platforms can also help manually build workflows, but they have been specifically optimized for integration with LLMs.

Shortcuts app (formerly Workflow) (Apple, ) is an app developed by Apple for building workflows through a graphical interface, available on Apple’s operating systems (iOS/iPadOS and macOS). Shortcuts app can be seen as the DAP of Apple. It allows users to create workflows (known as shortcuts (Apple., )) that execute specific tasks on their devices and share them online via iCloud (Apple, ). Users can also download curated shortcuts from the Gallery of the Shortcuts app. However, the shortcuts available in the Gallery are very limited, with only a few dozen options. To access more shortcuts, users must either collect them from third-party sharing sites like Shortcuts Gallery (Gallery, ) and SSPAI (SSPai, ) or create their own. Shortcuts can be triggered through the Shortcuts app, widgets, the share sheet, and Siri, and they can also be automated to run upon specific events.

Shortcuts are composed of multiple API calls (actions). An agent can use the shortcut as a whole API or utilize the individual APIs involved in the shortcut. This paper treats the various APIs within the shortcuts as APIs available to the agent, aiming for the agent to automatically construct workflows of API calls.

## 3 Dataset

In this Section, we first introduce the acquisition of the dataset (Section 3.1). Then, we outline the ShortcutsBench’s construction process (Section 3.2). Finally, we outline the setup for evaluation tasks to evaluate the agent’s ability to handle tasks of varying difficulty (Section 3.3.1), including the ability to select suitable APIs, the ability to do parameter filling (Section 3.3.2), and the awareness in requesting additional information from the system or user (Section 3.3.3).

### 3.1 Dataset Acquisition

Figure: Figure 1: (a) illustrates the data acquisition process. (b) shows the dataset acquisition of existing work (Table 1). APIs in existing benchmarks / datasets are created by hand or modified from existing datasets, or API hubs, and queries and action sequences are constructed using templates or semi/fully automated methods with LLMs.
Refer to caption: x1.png

Figure 1 shows the data acquisition process. We first use search engines to identify popular public shortcut-sharing sites ①. We totally find $14$ sites such as Shortcuts Gallery (Gallery, ) and SSPai (SSPai, ). Then we crawled these sites to obtain fields such as “shortcut name”, “function description”, “shortcut type”, and “iCloud link” ②. After deduplicating based on “iCloud link” (Apple, ), we got the source files of all $8675$ shortcuts (Apple, ) ③. Subsequently, we extracted “app name” using the  field WFWorkflowActionIdentifier in the source file like com.openai.chat.AskIntent, and then downloaded related apps ④ from various sources: (1) third-party apps from the “macOS App Store” or the official website of the app, (2) non-system apps (uninstallable) like Keynote from path /Applications/ on macOS, (3) system apps like Reminders (uninstallable) from path /System/Application/ on macOS, (4) iOS apps from the “iOS App Store”, (5) Shortcuts app itself from path /System/Library/PrivateFrameworks/WorkflowKit.framework/ on macOS. During the downloading, we also excluded some legacy apps and $12$ paid apps. For more details about the whole acquisition process, please refer to Appendix A.1.

Then we managed to extract APIs from the downloaded apps ⑤. The APIs are mainly from intent definition file ${filename}.actionsdata from AppIntent (Apple-Inc., ; ; ) framework, ${filename}.intentdefinition from SiriKit (Apple-Inc., ; ; ; ) framework, and WFActions.json from system path /System/Library/PrivateFrameworks/WorkflowKit.framework/ on macOS. We extracted all APIs involved in the app’s shortcuts. During the extraction, we perform deduplication of APIs based on manually crafted rules. Finally, as shown in Table 1, we get $88$ apps from various categories such as “Health & Fitness”(iTunes App Store, ) and “Developer Tools” (iTunes App Store, ). These apps include $1414$ APIs, including all of $556$ APIs involved in $7627$ shortcuts. For more details about the extraction, please refer to the Appendix A.1.

### 3.2 Dataset Construction

As shown in Figure 2, existing benchmarks / datasets consist of two parts: (1) APIs; (2) queries and corresponding action sequences.

##### APIs

(“a” in Figure 2) include the “API description” (“a.4”), “API name” (“a.1”), “parameter names” (“a.2”), “parameter types” (“a.1”), “default value” (“a.2”), “return value type” (“a.3”), and “return value name” (“a.3”). The field names in [...] in Figure 2 more details about ${filename}.actionsdata, ${filename}.intentdefinition, and WFActions.json, please refer to the Appendix A.2.
In existing benchmarks / datasets (Table 1), the “parameter types” (“a.1”) and “return value types” (“a.3”) are composed of primitive data types such as int and string. In addition to primitive data types, APIs in ShortcutsBench also include “enum” or “advanced data types”. Enum is composed of “the class name” and “the possible value”, with each value equipping a “value name”. We also provide the agent with a description of the “enum” in the API information. Advanced data types, such as the model (“a.1”) in app chatGPT (OpenAI, ), include three String types named identifier, title, and subtitle. We can comprehend them through their “type name” and “type description”.

Figure: Figure 2: The construction of ShortcutsBench. (a) shows the information of API com.openai.chat.AskIntent extracted from the app ChatGPT’s ${filename}.actionsdata. We provide this API description to the LLM, expecting it to call the API at the appropriate time. The API information shown in (a) includes the API functionality description (a.k. (a.1)~(a.4)) as shown in (d), and the user-friendly natural language description of the API (a.k. (a.4.3)) seen by shortcut developers during programming, as shown in (c). (e) presents the shortcut name and functionality description from the shortcut sharing-sites. (b) shows the simplified prompt fed to GPT-4o, instructing it to generating queries based on demands indicated by shortcuts by integrating the info from (c), (d), and (e). Different colors indicate different information sources.
Refer to caption: x2.png

##### Query and action sequence.

A query is a user command, such as “Tell me what the weather will be like tomorrow.” The action sequence (aka. shortcut) is the series of API calls to complete the query, with each API call referred to as an action. The action sequence identifies the steps needed to complete a query. As shown in Figure 1.b, Existing benchmarks / datasets (Table 1) collect APIs first and then use them, either fully automatically or semi-automatically, to construct query and action sequences through LLMs. In contrast, action sequences in ShortcutsBench are all human-annotated (①②③ in Figure 1). The shortcut developers are our annotators. APIs in ShortcutsBench (④ in Figure 1) are also all real-world.

Generating queries. As shown in Figure 1, existing works construct query and action sequences based on available APIs. In contrast, we construct queries based on existing action sequences and APIs. When constructing a query for a specific action sequence, we need to understand the functional description of the action sequence (“e” in Figure 2) and detailed information about the involved APIs (“a” in Figure 2). With this information, we can generate higher-quality queries. To ensure the quality of the generated queries, we also leverage the unique advantage of shortcuts: the natural language workflow descriptions (“b.7” / “c” in Figure 2). By inputting these intuitive natural language descriptions into an LLM, we can generate more accurate queries. When generating queries, we also require the model to naturally include primitive data type parameters and enum data type needed for API calls in generated queries. This helps us evaluate the agent’s ability to fill in primitive parameters in Section 3.3.2. To ensure the quality of generated queries, we use the state-of-the-art LLM, GPT-4o (OpenAI, ), to generate the queries. The prompt templates we used to generate queries can be found in the Appendix A.2.

### 3.3 Task Definition and Metrics

We aim to address $3$ research questions regarding the performance of existing agents built using leading LLMs on ShortcutsBench with varying difficulties: (1) How do they perform in API selection? (2) How do they handle API parameter value filling, including parameters for primitive data types, enums, and outputs from previous actions? (3) Can they recognize when input is required for tasks that need system or user information?

**Table 2: Final evaluation set with varying difficulties.**
| $\bm{|aseq_{i}|}$ | (0, 1] | (1, 5] | (5,15] | (15,30] | Overall |
| --- | --- | --- | --- | --- | --- |
| # Queries | 706 | 2169 | 1571 | 774 | 5220 |
| # Avg APIs | 1.17 | 3.43 | 8.30 | 13.76 | 6.60 |
| # Avg Acts | 1.00 | 3.19 | 9.60 | 21.58 | 8.34 |

Preliminaries. ShortcutsBench consists of a set of queries $Q=\{q_{1},q_{2},...,q_{n}\}$, corresponding "golden" action sequences $ASeq=\{aseq_{1},aseq_{2},...,aseq_{n}\}$, and all available APIs $APIs=\{api_{1},api_{2},...,api_{m}\}$. For each query $q_{i},1\leq i\leq n$, the corresponding “golden” action sequence is $aseq_{i}=\{a_{1},a_{2},...,a_{\left|aseq_{i}\right|}\}$, where the length of the action sequence is $\left|aseq_{i}\right|$. Each app $app_{j}$ has a set of APIs $apis_{j}=\{api_{1},api_{2},...,api_{\left|apis_{j}\right|}\}$. The action sequences generated by the agent for each query $q_{i}$ are referred to as $bseq_{i}$.

Prepare available APIs for each query. For each query $q_{i}$, we provide the LLM with a certain number of usable APIs to simulate real-world scenarios where APIs can be input into the LLM’s context. Following existing work (Meta, ; Qin et al., ; Tang et al., ; Xu et al., ; Schick et al., ; Hao et al., ), we equip each $q_{i}$ with a specific number of APIs. For each $aseq_{i}$, let $\left|APIs_{i}\right|$ represent the number of APIs involved. In addition to these $\left|APIs_{i}\right|$ APIs, we equip each query with extra APIs calculated as $max(min(x\times\left|APIs_{i}\right|,20-\left|APIs_{i}\right|),0)$, where $x\in\{3,4,5\}$. We do this because it is impractical to input all APIs into the context simultaneously. When dealing with a large number of APIs, additional retrieval is often required (Qin et al., ), which we do not consider in this work.

Further Processing.
Considering the context limitations of LLMs, we excluded shortcuts longer than $30$ and parts using the API is.workflow.actions.runworkflow to call other shortcuts. While these shortcuts remain in our open-source dataset, they will not be included in the subsequent evaluation. We aim to study the performance of agents on queries of varying difficulties. As shown in Table 2, we categorize ShortcutsBench into $4$ difficulty levels and $8$ task types based on $\left|aseq_{i}\right|$ and “shortcut type” (Section 3.1), respectively. For more details, please refer to the Appendix A.3. When calculating the length, for branching actions like is.workflow.actions.conditional, we consider the longest branch as the length. Additionally, we ignore the lengths of looping actions like is.workflow.actions.repeat.count and special actions such as is.workflow.actions.comment. Due to the presence of branching actions, the average number of APIs involved when $p=1$ is greater than one, specifically $1.17$. For a detailed process, please refer to the Appendix A.3. The number of shortcuts in each level is denoted as $n_{p}$. Each query and action sequence is referred to as $q_{p,i}$ and $aseq_{p,i}$, with $1\leq p\leq 4$ and $1\leq i\leq n_{p}$.

#### 3.3.1 Performance about API Selection

Following existing work (Huang et al., ; Patil et al., ; Li et al., ; Xu et al., ; Schick et al., ; Hao et al., ), we use the accuracy of API selection as the metric. The accuracy is calculated as the number of correct API selections $m_{p}$ divided by $n_{p}$. Specifically, each time we predict an action $b_{j},1\leq j\leq|aseq_{i}|$, we provide the agent with all the correct historical actions $\{a_{1},a_{2},...,a_{j-1}\}$. We then require the agent to predict the next action. All actions predicted by the agent form the prediction sequence $bseq_{p,i}$. This method is similar to the next token prediction (NTP) in LLMs, effectively preventing a cascade of errors in subsequent action predictions due to a single incorrect prediction. During the prediction, when encountering special actions such as branching and looping, we skip predicting these actions and directly add them to the historical actions. For more details, please refer to Appendix A.4. We chose API selection accuracy over the final result for the following two additional reasons:

- •
ShortcutsBench contains numerous APIs such as opening the “All Shortcuts Folder” in the Shortcuts app that do not have a return value. This makes it challenging to evaluate using existing metrics that measure the success rate of solving queries (Qin et al., ; Tang et al., ; Xu et al., ; Schick et al., ; Xu et al., ; Hao et al., ).
- •
ShortcutsBench includes numerous APIs with complex input and output types, such as PDFs and Rich Text. Converting these formats into text that an LLM can process presents a significant challenge (Naveed et al., ), as LLMs struggle to serialize them into text. Consequently, it becomes difficult to ascertain the correctness of the final results. However, measuring API selection accuracy is straightforward.

#### 3.3.2 Effectiveness of API Parameter Value Filling

In this part, we aim to investigate the performance of agents in API parameter value filling, including parameters for “primitive data types” and “enums” and filling output from previous actions. For each input parameter of every action in ShortcutsBench, we expect the agent to fill in the following parameters correctly:

- •
Static Parameters Preset: These are static parameters that users provide as default inputs of the action. These static parameters typically include primitive data types such as String and Integer, as well as custom Enum defined by app developers. When the query explicitly specifies a parameter that can be used as a static parameter, we expect the agent to accurately fill in the parameter values according to the user’s query and the API’s definition. When generating queries, we have already required the LLM to naturally include primitive and enumerated data types (Section 3.2). To further ensure that the corresponding parameters are indeed included in the queries during evaluation, we used the LLM to filter these parameters further, ensuring their presence in the queries. Detailed prompts can be found in the Appendix A.5.
- •
Outputs from Previous Actions: An action may either have no output or, if it does have an output, the output may be used by the following actions. In shortcuts, In ShortcutsBench, outputs that are difficult to input directly into the LLM are represented by a unique identifier (UID) and an output name (OutputName), which can be input into the LLM for processing. The agent should have the ability to correctly use the output values of previous actions.

For the static parameters preset, we evaluate using the overall parameter fill rate. Let $sppa_{i}$ be the total number of parameters that need to be filled in $aseq_{i}$, $1\leq i\leq n_{q}$, where $n_{q}$ is the number of queries. If the agent correctly fills $sppt_{i}$ parameters in the generated action sequence $bseq_{i}$, then the static parameter preset accuracy can be calculated as $Acc_{spp}=\sum_{i=1}^{n_{q}}sppt_{i}/\sum_{i=1}^{n_{q}}sppa_{i}$.
Similarly, for outputs from previous actions, the accuracy can be calculated as $Acc_{ofpa}=\sum_{i=1}^{n_{q}}ofpat_{i}/\sum_{i=1}^{n_{q}}ofpaa_{i}$.

#### 3.3.3 Recognition of Need for Input

In this section, we aim to investigate the ability of existing API-based agents to ask systems or users for necessary information to resolve the missing information. This missing information can come from the system like clipboard (Clipboard), input files (ExtensionInput), and the current date (CurrentDate) or from the user (Ask) (Apple-Inc., ). For example, a parameter named tags is usually represented in a shortcut as "tags":{"Value":{"Type": "Ask"}}, where "Type": "Ask" indicates that the parameter will prompt the user for input. For more details, please refer to Appendix A.6.
We use the proportion of correctly identified parameters to evaluate the agent’s ability to recognize the need for input from the system or the user. Let $n_{s}$ be the number of queries, $aska_{i}$ be the number of times the need from the system or the user appears in $aseq_{i}$, $askt_{i}$ be the number of times the need from the system or the user appears in $bseq_{i}$. The accuracy of ask for necessary information can be calculated as $Acc_{afni}=askt_{i}/aska_{i}$.

## 4 Evaluation

### 4.1 Setup

Model. Referencing existing work (Huang et al., ; Qin et al., ; Patil et al., ; Tang et al., ; Li et al., ; Xu et al., ; Zhuang et al., ; Schick et al., ; Hao et al., ), considering the performance of mainstream LLMs, we selected and tested $9$ most advanced LLMs to construct API-based agent. The chosen model including $4$ closed-sourced LLMs like Gemini-1.5-Pro (DeepMind, ), Gemini-1.5-Flash (DeepMind, ), GPT-3.5-turbo (OpenAI, ), and ChatGLM-4-Air (ChatGLM, ), and $5$ open-source LLMs like LLaMA-3-70B (Meta, ), QWen-2-70B (Qwen, ), QWen-2-57B Qwen (), Deepseek-2-Chat (236B) (DeepSeek, ), and Deepseek-2-coder (236B) (DeepSeek, ). Among them, Gemini-1.5-Pro, LLaMA-3-70B, QWen-2-70B, Deepseek-2-chat, and Deepseek-2-coder are LLMs benchmarked against GPT-4 OpenAI (), while Gemini-1.5-Flash, ChatGLM-4-Air, and QWen-2-57B are benchmarked against GPT-3.5-turbo performance. We did not evaluate smaller LLMs like LLaMA-3-8B (Hugging Face, ) or Vicuna-7b-v1.5 (LMSYS, ) because we found that agents built on them can only handle simple tasks such as single API selection (Huang et al., ) and they cannot handle well on advanced tasks like parameter filling. Agents built with such models often fail to produce the required JSON actions correctly and frequently generate nonsensical outputs.

Prompt Template. Following existing work (Huang et al., ; Qin et al., ; Patil et al., ; Tang et al., ; Li et al., ; Xu et al., ; Zhuang et al., ; Schick et al., ; Hao et al., ), we slightly modified the ReACT (Yao et al., ) templates to construct the API-based agents. Specifically, we added prompts related to shortcuts, such as the types of fillable parameters and the meanings of special statements like branches. For all $3$ research questions (RQs), we use the same prompt templates. An agent should correctly select APIs, fill in parameters, and be aware of the need to request necessary information from the system or user at appropriate times. Please refer to Appendix A.7 for more details.

### 4.2 Result Analysis

Figure: Figure 3: The API selection accuracy on queries with different complexity levels.
Refer to caption: x3.png

Figure: Figure 5: The API selection accuracy of each task type on $9$ API-based agents.
Refer to caption: x5.png

Through the results of API selection accuracy (Section 3.3.1), we get the following conclusions:

- •
Agents built using open-source LLMs now perform comparably to closed-source models on lower-difficulty tasks but still lag on higher-difficulty tasks. From Figure 4 we know that open-source LLMs $>=70$B match the performance of closed-source LLMs from the first $3$ difficulty tasks, significantly outperforming GPT-3.5-turbo. However, they still lag behind closed-source LLMs in handling complex tasks at the $4$-th level. Moreover, the price of open-source LLMs is significantly lower than that of GPT-3.5-turbo. For more details, please refer to Appendix A.8.
- •
Existing LLM-based agents still perform poorly on tasks requiring multi-step reasoning, even Gemini-1.5-Pro level LLMs struggle with high-difficulty tasks. From Figure 4 we know that almost all LLMs handle well in API selection tasks at the level of (0,1], but only more advanced models like Gemini-1.5-Pro and QWen-2-72B can do well in higher-difficulty tasks of (1,5]. As tasks become more complex, the accuracy drops sharply. The average accuracy dropped by $19\%$ as task difficulty rose from (0,1] to (1,5], ranging from a $9\%$ decrease (Deepseek-2-chat) to a $44\%$ (ChatGLM-4-Air). From (0,1] to (5,15], accuracy fell by $46\%$, with drops from $38\%$ (Gemini-1.5-Pro) to $58\%$ (ChatGLM-4-Air).
- •
Agents built with the same LLM show significant performance variations across different types of tasks. From Figure 5 we know that the performance difference of agents built with different LLM ranges from $18.04$% (ChatGLM-4-Air) to $36.70$% (Gemini-1.5-Pro).
- •
Existing API-based agents perform well on tasks in daily life such as Lifestyle & Social but show poorer performance on professional tasks like Development & API. From Figure 5 we know that Lifestyle & Social exhibit the highest average accuracy, surpassing the lowest category, Development & API by approximately $18\%$.

Based on the results of API Parameter Value Filling (Section 3.3.2), we draw following conclusions:

Figure: (a) Accuracy of primitive data types & enum data types (upper) and outputs from previous actions (lower).
Refer to caption: x6.png

- •
Compared to the API selection, for existing most intelligent LLM like Gemini-1.5-Pro, increased task difficulty has a much smaller impact on the accuracy of parameter filling, especially on using outputs from previous actions. As shown in Figure 6(a), the precision of API parameter filling of the existing most intelligent LLM like Gemini-1.5-Pro and QWen-2-72B remains similar across tasks of varying difficulty in both the upper and lower figures. This indicates that the greatest limitation of existing API-based agents in addressing user queries lies in the reasoning and planning capabilities implied by API selection.
- •
Compared to API selection, the performance of API parameter filling remains a bottleneck for existing cost-effective LLMs like GPT-3.5-turbo and ChatGLM-4-Air. As shown in Figure 6(a), the performance of these LLMs in API parameter filling significantly decreases as task difficulty increases.
- •
Compared to using the outputs of previous actions, extracting relevant parameters from the user’s query and filling them according to the query and API description is more challenging. As shown in Figure 6(a), the colors in the top plot (filling primitive data types and enum data types) are generally lighter than those in the bottom plot (filling the outputs of previous actions as parameters). The accuracy drop ranges from $2.55$% (GPT-3.5-turbo) to $15.39$% (Deepseek-2-Chat).
- •
For existing cost-effective LLMs like GPT-3.5-turbo and ChatGLM-4-Air, errors mainly stem from incorrect output formats and wrong API selections. Figure 6(b) shows error types for tasks requiring outputs from previous actions. It can be seen that powerful LLMs like  Gemini-1.5-Pro rarely make format errors, whereas the most cost-effective models frequently make mistakes in both output format and API selection.

The results from Recognition of Need for Input (Section 3.3.3) lead us to the following conclusions:

- •
All agents perform poorly at recognizing necessary system and user inputs when required. Overall, all agents have weak recognition capabilities, with accuracy ranging between $30.55$% (GPT-3.5-turbo) and 55.18%(Deepspeed-2-coder). Larger LLMs such as Deepspeed-2-chat (236B) still demonstrate better recognition accuracy.

**Table 3: The accuracy of recognition of the need for input from the system or the user.**
| Levels |  | Gemini |  | QWen |  | Deeps. |  | Deeps. |  | LLaMA |  | Gemini |  | QWen |  | GPT |  | ChatGLM |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | 1.5-Pro |  | 2-72B |  | 2-chat |  | 2-coder |  | 3-70B |  | 1.5-Flash |  | 2-57B |  | 3.5-turbo |  | 4-Air |  |
| (0, 1] |  | 33.33 |  | 37.78 |  | 64.29 |  | 62.71 |  | 47.62 |  | 62.79 |  | 22.22 |  | 28.89 |  | 47.62 |
| (1, 5] |  | 45.95 |  | 50.40 |  | 55.50 |  | 60.08 |  | 44.08 |  | 53.99 |  | 37.24 |  | 37.70 |  | 48.06 |
| (5, 15] |  | 51.85 |  | 36.42 |  | 40.76 |  | 49.44 |  | 35.71 |  | 40.65 |  | 28.37 |  | 20.33 |  | 48.42 |
| (15, 30] |  | 46.67 |  | 25.00 |  | 27.59 |  | 43.14 |  | 22.22 |  | 44.64 |  | 8.11 |  | 17.14 |  | 48.89 |
| Overall |  | 46.59 |  | 41.97 |  | 47.90 |  | 55.18 |  | 49.89 |  | 40.71 |  | 30.74 |  | 30.55 |  | 48.28 |

## 5 Conclusion

In this paper, we introduce ShortcutsBench, a benchmark for evaluating API-based agents. To the best of our knowledge, ShortcutsBench is the most realistic, rich, and comprehensive benchmark of its kind. Our findings indicate that for agents built on the most advanced LLMs, the primary bottleneck is API selection. For the most cost-effective LLMs, there is considerable room for improvement in both API selection and parameter filling. Additionally, we identified a significant deficiency in the agents’ awareness of requesting necessary information.

## Appendix A Appendix / supplemental material

### A.1 Dataset Acquisition Process

In this section, we introduce more details about the dataset acquisition.

Regarding data acquisition, we first use search engines to identify popular public shortcut-sharing sites. We found a total of $14$ sites. These sites include:

| # | Site Name | URL | Count |
| --- | --- | --- | --- |
| 1 | Matthewcassinelli | https://matthewcassinelli.com | 1535 |
| 2 | Routinehub | https://routinehub.co | 6860 |
| 3 | MacStories | https://www.macstories.net/shortcuts | 4993 |
| 4 | ShareShortcuts | https://shareshortcuts.com | 2395 |
| 5 | ShortcutsGallery | https://shortcutsgallery.com | 4269 |
| 6 | iSpazio | https://shortcuts.ispazio.net | 115 |
| 7 | Jiejingku | https://jiejingku.net | 3347 |
| 8 | SSPai | https://shortcuts.sspai.com | 145 |
| 9 | Jiejing.fun | https://jiejing.fun | 84 |
| 10 | Kejicut | https://www.kejicut.com | 37 |
| 11 | RCuts | https://www.rcuts.com | 133 |
| 12 | Sharecuts | https://sharecuts.app | 2395 |
| 13 | Siri-shortcuts | https://www.siri-shortcuts.de | 15 |
| 14 | Reddit | https://www.reddit.com/r/shortcuts | 100 |
| Total (After Deduplication): | 8675 |  |  |

In addition to collecting iCloud Links (URLs) from shortcut-sharing sites, these sites also provide information such as the shortcut’s name (NameInStore), functional description (DescriptionInStore), category (CategoryInStore), number of downloads (Downloads), favorites (Favorites), reads (Reads), and ratings (Rates). Most shortcuts include NameInStore and DescriptionInStore (except for a few obtained from Reddit), while the availability of other fields varies slightly depending on the specific shortcut-sharing site.

After deduplicating based on “iCloud link” (Apple, ), we attempt to get the source files of all shortcuts (Apple, ). We can share shortcuts using the Shortcuts app on Apple devices. The first method is through iCloud links, which do not provide access to the shortcut’s source file. The second method involves sharing the shortcut’s source file, with the .shortcut suffix, but these files are signed by Apple devices and cannot be easily decrypted. Through our efforts, we discovered that when shortcuts are imported into the Shortcuts app, they are displayed in an easily understandable PLIST file format. Users can obtain the source files of any shortcut using our decryption shortcuts available at [https://www.icloud.com/shortcuts/b04412850b9f4f74ad16f2f15ef09a3f](https://www.icloud.com/shortcuts/b04412850b9f4f74ad16f2f15ef09a3f) and [https://www.icloud.com/shortcuts/8fa07dea82cf413c81732dca5f15323f](https://www.icloud.com/shortcuts/8fa07dea82cf413c81732dca5f15323f). To facilitate the large-scale acquisition of shortcut source files, we analyzed the network traffic of the Shortcuts app. We found that we could retrieve shortcut metadata from [https://www.icloud.com/shortcuts/api/records/${unique_id}](https://www.icloud.com/shortcuts/api/records/%24%7Bunique_id%7D), where ${unique_id} is a string like b04412850b9f4f74ad16f2f15ef09a3f. Using the download link indicated by the metadata at ["fields"]["shortcut"]["value"]["downloadURL"], we could obtain the shortcut source files. This method enabled us to acquire all shortcut source files. The retrieved metadata includes the downloadURL and the name field of the shortcut. In our subsequent processing, we use this name to replace NameInStore, as it directly corresponds to the name of the shortcut when imported into the Shortcuts app.

Subsequently, we extracted “app name” using the  field WFWorkflowActionIdentifier in the shortcut source file like com.openai.chat.AskIntent, and then downloaded related apps from various sources. Shortcuts are composed of a series of shortcut API calls (Actions). A typical shortcut is shown in Figure 7. Each shortcut API call is identified by a name, which typically consists of the name of the shortcuts app like com.openai.chat and the name of the Intent like AskIntent. For most API names, the part before the last dot is the app name, and the part after is the Intent name. We semi-automatically extracted all app names to facilitate downloading the apps.

Figure: Figure 7: An example of a shortcut: Ask ChatGPT.
Refer to caption: x8.png

Most app downloads can be accomplished using the ipatool tool, which supports downloading iOS / iPadOS versions of apps. However, there are two categories of apps that ipatool cannot download: (1) Some apps are available only in macOS and do not have iOS / iPadOS versions; (2) Most first-party Apple apps are not available for download from the App Store, and even those that do not have API definition files. For category (1), we manually downloaded these apps. For category (2), we found most of the apps in the /Applications/ and /System/Applications/ directories on macOS, and located most of the API definition files in /System/Library/PrivateFrameworks/WorkflowKit.framework/. In summary, we adopt the following approach to choose apps:
(1) We select the macOS version for apps from Apple like the “Shortcuts” app.
(2) For apps that have an iOS version, we prioritize the iOS version.
(3) For apps that only have a macOS version, we choose the macOS version. Then we conducted app and shortcut filtering to remove apps that had no definition file and shortcuts that had APIs with missed API descriptions. Our choice here is made solely for the convenience of extracting API definition files. For most apps, selecting iOS / iPadOS or macOS versions does not impact the retrieval of API definition files. We found that only a few apps, such as Chrome, have different APIs on macOS compared to iOS / iPadOS. These differences will be filtered out in subsequent steps.

Then we managed to extract APIs from the downloaded apps. The APIs are mainly from intent definition file ${filename}.actionsdata from AppIntent (Apple-Inc., ; ; ) framework, ${filename}.intentdefinition from SiriKit (Apple-Inc., ; ; ; ) framework, and WFActions.json from system path /System/Library/PrivateFrameworks/WorkflowKit.framework/ on macOS. SiriKit was introduced in 2016 with iOS 10 (Apple-Inc., ), allowing applications to integrate with Siri for voice command interactions. AppIntents, launched with iOS 16 in 2022 (Apple-Inc., ), offers a more modern and flexible way to define and handle app intents, facilitating integration with Siri, Shortcuts, widgets, and more. Apple is actively encouraging developers to adopt AppIntents by providing migration tools from SiriKit (Apple-Inc., ; ). However, some apps still use the SiriKit framework.
When developing with the SiriKit framework, ${filename}.intentdefinition files are used, while the App Intent framework uses ${filename}.actionsdata files. These files define the APIs corresponding to actions in shortcuts. An app may contain only ${filename}.intentdefinition files, only ${filename}.actionsdata files, or both, leading to potential redundancy in API definitions. We strive to reduce the number of API definition files and have established a set of rules to ensure API uniqueness in ShortcutsBench. Finally, as shown in Table 1, we get $88$ apps from various categories such as “Health & Fitness”(iTunes App Store, ), “Developer Tools” (iTunes App Store, ), and “Lifestyle” (iTunes App Store, ). These apps in total include $1414$ APIs, including all of $556$ APIs involved in $7627$ shortcuts.

### A.2 Dataset Construction

The API definition files extracted from the app exist in two forms: the ${filename}.intentdefinition files as indicated by the Sirikit framework and the ${filename}.actionsdata files as indicated by the App Intent framework. Additionally, Apple’s first-party apps provide a third type of definition file, WFActions.json. All three file formats provide “API description”, “API name”, “parameter names”, “parameter types”, “default value”, “return value type”, and “return value name”, but differ in their file format. We give a sample from each of the three different file formats, as shown in Figure 8.

Figure: Figure 8: We randomly selected three samples from three different definition files, as shown in the upper (${filename}.actionsdata), middle (WFActions.json), and lower (${filename}.intentdefinition) figures. The content in brackets represents different field names. In practice, there are various details to handle, such as name prefixes and missing fields. For complete details, please refer to our open-source code.
Refer to caption: x9.png

We construct queries based on existing action sequences and APIs. To ensure the quality of these queries, we utilize the natural language workflow descriptions unique to shortcuts. When generating queries, we require the model to naturally include primitive data type parameters and enum data types needed for API calls. This helps us evaluate the agent’s ability to handle primitive parameters. We do not require the inclusion of complex data types in the queries, as they are difficult to convert to text and challenging to evaluate. To ensure high-quality query generation, we use the state-of-the-art LLM, GPT-4o (OpenAI, ). The prompt templates used for generating queries are provided in Figure 9.

Figure: Figure 9: System and user prompt templates for query generation based on a shortcut

### A.3 Task Definition and Metrics

Considering the context limitations of LLMs, we excluded shortcuts longer than $30$ and parts using the API is.workflow.actions.runworkflow to call other shortcuts. While these shortcuts remain in our open-source dataset, they will not be included in the evaluation. We aim to study the performance of agents on queries of varying difficulties. As shown in Table 2, we categorize ShortcutsBench into $4$ difficulty levels and $8$ task types based on $\left|aseq_{i}\right|$ and “shortcut type” (Section 3.1), respectively.

In calculating the length of shortcut commands, we do not simply count the number of actions within the shortcut. Instead, we apply a specialized approach. Initially, certain actions that do not contribute meaningful operations, such as is.workflow.actions.comment and is.workflow.actions.alert, which are akin to comments in programming, are excluded. Furthermore, we disregard the length of certain control flow statements, including is.workflow.actions.conditional, is.workflow.actions.choosefrommenu, is.workflow.actions.repeat.count, is.workflow.actions.repeat.each. For branching statements, we consider the length of the longest branch, rather than the cumulative length of all branches.

When categorizing shortcuts, we first analyzed all available categories from the CategoryInStore field in the collected data. We then classified the shortcuts into $8$ categories, referencing with the classification of apps on the Apple App Store (). The categories are as follows:

- 1.
Productivity & Utilities
- 2.
Health & Fitness
- 3.
Entertainment & Media
- 4.
Lifestyle & Social
- 5.
Education & Reference
- 6.
Business & Finance
- 7.
Development & API
- 8.
Home & Smart Devices

Subsequently, I employed a language model to categorize all shortcuts using the prompt shown in Figure 10.

Figure: Figure 10: System and user prompt templates for categorizing shortcuts based on their functionalities

### A.4 Performance about API Selection

Following existing work (Huang et al., ; Patil et al., ; Li et al., ; Xu et al., ; Schick et al., ; Hao et al., ), we use the accuracy of API selection as the metric. The accuracy is calculated as the number of correct API selections $m_{p}$ divided by $n_{p}$. Specifically, each time we predict an action $b_{j},1\leq j\leq|aseq_{i}|$, we provide the agent with all the correct historical actions $\{a_{1},a_{2},...,a_{j-1}\}$. We then require the agent to predict the next action. All actions predicted by the agent form the prediction sequence $bseq_{p,i}$. This method is similar to the next token prediction (NTP) in LLMs, effectively preventing a cascade of errors in subsequent action predictions due to a single incorrect prediction. During the prediction, when encountering special actions such as branching and looping, we skip predicting these actions and directly add them to the historical actions.

Specifically, when calculating the precision of API selection, we do not consider the contributions of control statements such as branches and loops. This avoids the unreasonable requirement for the agent to invoke “branch APIs” or “loop APIs” in the next action. The agent should inherently possess the ability to correctly understand and act according to the conditions dictated by branches and loops. In addition to excluding the contributions of these control statements, we also disregard contributions from is.workflow.actions.comment and is.workflow.actions.alert, effectively removing these non-operative commands from the history of actions provided to the agent.

### A.5 Effectiveness of API Parameter Value Filling

To further ensure that the corresponding parameters are indeed included in the queries during evaluation, we used the LLM to filter these parameters further, ensuring their presence in the queries. Detailed prompts can be found in Figure 11.

Figure: Figure 11: System and user prompt templates for classifying parameters based on user queries and API definitions

### A.6 Recognition of Need for Input

In the shortcut, a parameter can be set to ExtensionInput, indicating that the parameter requires a file provided by the user, or CurrentDate, indicating that the parameter needs to retrieve the date from the system. Similarly, Clipboard indicates that the parameter should obtain content from the clipboard, and DeviceDetails implies that the parameter needs to access certain information about the user’s device. Lastly, Ask denotes that the parameter requires user authorization or essential information from the user. A typical example is shown in Figure 12, where the action uses the is.workflow.actions.getmyworkflows API. The Folder parameter is set to Ask, indicating that this parameter requires information provided by the user.

Figure: Figure 12: An example of Ask parameter.

### A.7 Setup

Following existing work (Huang et al., ; Qin et al., ; Patil et al., ; Tang et al., ; Li et al., ; Xu et al., ; Zhuang et al., ; Schick et al., ; Hao et al., ), we slightly modified the ReACT (Yao et al., ) templates to construct the API-based agents. The templates used in our experiments are as shown in Figure 13.

Figure: Figure 13: System and user prompt templates for executing API calls based on user queries

### A.8 Result Analysis

**Table 5: Pricing, Testing Instances, and Actual Costs of Popular AI Models. (07-22-24). Except for gemini-1.5-pro, which was randomly tested on 800 instances due to cost considerations, all other LLMs were tested across all datasets. However, the number of successful tests varied slightly due to factors such as context length, safety reviews, and etc. The cost of testing primarily stems from inputs, as we continuously feed historical actions into the LLM for evaluation, and all historical conversations are billed repeatedly (OpenAI Community, ).**
| Model Name | Price / 1M tokens | Instances | Estimate Cost ($) |
| --- | --- | --- | --- |
| gemini-1.5-pro | $3.50 / $10.50 | 801 | 592 |
| gemini-1.5-flash | $0.35 / $1.05 | 5295 | 391 |
| qwen2-72b-instruct | $0.70 / $1.40 | 5216 | 800 |
| qwen2-57b-a14b-instruct | $0.49 / $0.98 | 5368 | 580 |
| gpt-3.5-turbo | $0.50 / $1.50 | 5463 | 500 |
| deepseek-chat | $0.14 / $0.28 | 5319 | 90 |
| deepseek-coder | $0.14 / $0.28 | 5317 | 90 |
| GLM-4-Air | $0.14 / $0.14 | 5330 | 110 |
| Total Cost |  |  | 3153 |

Among them, gemini-1.5-pro (tested with 801 instances) and gemini-1.5-flash (tested with 5,295 instances) incurred a total cost of $801, with gemini-1.5-flash accounting for approximately $391 and gemini-1.5-pro approximately $592. The costs for qwen2-72b-instruct (tested with 5,216 instances) were about $800, qwen2-57b-a14b-instruct (tested with 5,368 instances) around $580, gpt-3.5-turbo (tested with 5,463 instances) approximately $500, and the combined expenses for deepseek-chat (tested with 5,319 instances) and deepseek-coder (tested with 5,317 instances) were roughly $180. GLM-4-Air cost about $110.

The cost analysis indicates a notable range in efficiency and value for money. Models like deepseek-chat and deepseek-coder show excellent cost-effectiveness, particularly suitable for high-volume, low-cost deployments. In contrast, models like gemini-1.5-pro and gemini-1.5-flash reflect higher costs, but they offer superior performance.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: Please refer comments in above fields.
soundness: 4
strengths: This paper makes a notable contribution by creating a comprehensive benchmark for API-based agents, utilizing data extracted from Shortcuts. Compared to other API-based benchmarks, it offers several benefits, including a focus on the agents' ability to request necessary input from either the assistant or user and diverse difficulty of tasks. It covers a range of tasks, from simple ones to those involving complex APIs, queries, and action sequences. Additionally, the paper ensures quality by involving human verification, with shortcut developers serving as annotators.

The evaluations demonstrate insightful findings, especially regarding the challenges agent models face in reasoning and planning capabilities as indicated by API selection, as well as the difficulties weaker LLMs encounter in API parameter filling. These insights are valuable for the development of more advanced agent models.
summary: This paper introduces a comprehensive, large-scale benchmark designed to evaluate API-based agents in real-world scenarios. It addresses the shortcomings of existing benchmarks and datasets, which often lack the richness and complexity needed to thoroughly assess different LLM-based agent models. To overcome these limitations, the authors have developed a high-quality benchmark encompassing 88 applications, 1,400 APIs, and an average of 21 actions per API, each covering various aspects of real-world deployments.

In the evaluation, the authors tested 10 LLMs on several key tasks: API selection, parameter filling, and the models' ability to recognize when to request additional input from either the system or the user. The experimental results show interesting insights, such as the challenges these models face with multi-step reasoning and understanding when external input is necessary.
weaknesses: 1. Although the paper emphasizes that the benchmark includes high-quality human-annotated action sequences from shortcut developers and queries derived from real user demands, it only mentions the shortcut developers are our annotators. Further details in this area would be beneficial.

2. In section 3.2, the paper describes using GPT-4o to simulate user queries. However, it would be helpful to include the steps taken to verify the correctness and ensure the diversity of these user queries.

3. The evaluation primarily features selected proprietary models like GPT-4o and Gemini. For open-source LLMs, it mainly compares with general LLMs such as Qwen-2-70B and LLaMA-3-70B. Considering comparisons with more robust, specifically developed AI Agent models such as AgentLM (70B from https://github.com/THUDM/AgentTuning) and xLAM (8x7b or 8x22b from https://github.com/SalesforceAIResearch/xLAM) could provide more insights.

4. The source of the reported numbers in Table 1 could be more clearly specified. There is uncertainty regarding whether models like Qwen-2.5-7B and LLaMA-3-8B consistently achieve more than 90% on ToolBench and over 80% on ToolLLM. Furthermore, given that MetaTool (https://github.com/HowieHwong/MetaTool) does not provide comprehensive details or code for model evaluation and metrics, more information is necessary to verify the accuracy of the table data.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: - Why is all API-based agents the authors evaluate based on the ReACT framework, it would be good if they could provide additional evaluation/analysis on other frameworks, such as CodeAct.
soundness: 3
strengths: - The authors introduce ShortcutsBench, which is a more holistic benchmark that contains real APIs, well-designed queries and actions. This could contribute to better evaluation of current agents's API calling capabilities in solving real-world tasks.
- The authors provide example instances from ShortcutsBench in the appendix, which helps understanding the types of tasks in this benchmark.
- The authors provide detailed analysis based on the evaluation results of several API-based agents.
summary: This paper introduces a new benchmark ShortcutsBench, which evaluates agents' capabilities in solving tasks through API calling. They compare multiple API-based benchmarks, and showcase why ShortcutsBench contributes to the assessing agents' API calling abilities. The paper evaluates several API-based agents constructed based on the ReACT framework. The authors perform detailed analysis on the evaluation results.
weaknesses: - Section 3 is not well-elaborated. readers will benefit from clearer description of this process. For example, for (2), the authors say 'after duplicating based on icloud link, ....', it is not very clear what is duplicated and why this step helps. It would be good if the authors could refine their descriptions on their methodology.
- The authors cite each work too many times in the paper, for example, a research paper is is cited five times in one paragraph in Section 2. Referencing previous works is good practice, but referencing too many times affects readability. It would be good if they could remove repetitive references.
- The paper does not make it very clear what is their most important finding in the abstract/intro/conclusion. It would be good if they could highlight their most important findings in the abstract/intro/conclusion. For example, they could discuss how open source models perform comparably to closed source models on simpler tasks but not harder tasks.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: 1. A notable feature of this benchmark seems to be scalability -- extracting real-world APIs and action sequences from Shortcut apps and sharing sites seems relatively easy. Providing quantitative details on dataset construction—such as time taken for each step and the actual extent of manual effort involved—would be nice.
soundness: 2
strengths: - Interesting approach: mining APIs/existing action sequence from Shortcut Apps makes a lot of sense, which is a resource previous works haven't tapped into.
- Comprehensive evaluation: the authors evaluated a wide range of LMs across Open and close source models.
summary: This paper introduces ShortcutsBench, which mines real-world APIs/action sequences from iOS shortcut app. They then synthesize tasks with language model, and test a range of model's ability on it. Results show the task remains challenging to today's LMs.
weaknesses: 1. Limited evaluation: The paper primarily assesses the model’s ability to choose correct actions based on ground-truth sequences but doesn’t evaluate its end-to-end task success rate (as done in, for example, AppWorld [1]). Experiments linking these aspects are missing.
2. No human validation: Given the synthetic nature of the benchmark, it’s uncertain whether all tasks are truly solvable or what the benchmark’s upper bound is. Including human performance as a reference would add clarity.

[1] AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: I suggest to submit to a more relevant conference, rather than to ICLR, as mentioned above.
soundness: 2
strengths: [1] Analysis results in 4.2 are insightful and confirm the ranking of closed- and open-source LLMs on traditional general knowledge and reasoning benchmarks like MMLU.

[2] The artifacts are provided and well-organized.
summary: The paper introduces ShortcutsBench, a benchmark to assess LLMs’ ability to call external APIs and create structured text (ex: JSON). The dataset was collected from the Shortcuts tool available on Apple devices. The authors perform the analysis of performance of a number of open and closed-source LLMs against ShortcutsBench.
weaknesses: [1] It is a technical report that lacks a scientific component. It should go to a software engineering conference, for example, International Conference on Software Engineering (ICSE) or IEEE/ACM International Conference on Automated Software Engineering (ASE). The paper can also go to Datasets and Benchmarks tutorials at AI conferences. Whereas the work is technically impressive and valuable for the software engineering community, there is no scientific value in the assessment of how good LLMs are able to use APIs. The problem at hand is purely technical and as a clear indicator for this is that over two thirds of references are links to web pages rather than scientific studies.

[2] The analysis of the performance of GPT 4o with the structured output capability was not performed.
https://openai.com/index/introducing-structured-outputs-in-the-api/

[3] The used language appeals to emotions rather than states the scientific value (ex: “We made great efforts to evaluate…”).

[4] Several references are outdated, pointing to arxiv whereas the paper has been already published at a conference (ex: ToolLLM).

Meta-review: The paper "ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents" introduces a novel benchmark dataset, ShortcutsBench, designed to evaluate the capabilities of API-based agents in handling real-world complex tasks. The benchmark leverages real APIs from Apple's iOS Shortcuts app, coupled with refined user queries, human-annotated action sequences, and detailed parameter filling values. The authors evaluate a diverse set of large language models (LLMs), including both open-source (e.g., Qwen-2-70B, LLaMA-3-70B) and closed-source (e.g., Gemini-1.5-Pro, GPT-4o-mini) models, on tasks involving API selection, parameter filling, and recognizing the need for additional input. The findings reveal significant limitations in current API-based agents, particularly in multi-step reasoning and handling complex queries. The paper emphasizes the challenges these agents face in real-world scenarios and provides all datasets, code, and experimental logs openly for reproducibility.

The primary contributions of the paper are:
1. **ShortcutsBench Dataset**: A comprehensive, large-scale benchmark for evaluating API-based agents in real-world scenarios.
2. **Extensive LLM Evaluation**: A detailed analysis of 21 LLMs, including both general and agent-specific models, highlighting their capabilities and limitations.
3. **Insights into Agent Limitations**: Valuable findings on the challenges API-based agents face in complex tasks, particularly in parameter filling and recognizing the need for additional input.

Weaknesses

1. **Dataset Construction Details**: Some reviewers noted a lack of detailed description regarding the dataset construction process, particularly the steps taken to ensure query correctness and diversity. While the authors provided additional explanations and open-sourced the process logs, initial clarity was lacking.

2. **Limited Scope of Evaluation Metrics**: The paper primarily focuses on API selection and parameter filling, with limited emphasis on end-to-end task success rates. This was acknowledged by the authors, who noted plans for future work involving dynamic runtime environments.

3. **Presentation Issues**: Some reviewers criticized the initial presentation, citing repetitive citations and unclear phrasing. The authors addressed these concerns in the revised version, but initial readability issues may have impacted the overall reception.

4. **Human Validation Absence**: The absence of human performance baselines was noted as a limitation, as it leaves uncertainty about the benchmark's upper bound and the solvability of tasks. While the authors explained the practical challenges, this remains a valid concern for validating the benchmark's design.

Decision: Accept (Poster)