ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents
Authors: Haiyang SHEN
Yue Li
Desong Meng
Dongqi Cai
Sheng Qi
Li Zhang
Mengwei Xu
Yun Ma
Venue: ICLR 2025 Poster (2025)
Keywords: Benchmark
Agent
LLM
Shortcuts
Primary area: datasets and benchmarks

Abstract: Recent advancements in integrating large language models (LLMs) with application programming interfaces (APIs) have gained significant interest in both academia and industry. Recent work demonstrates that these API-based agents exhibit relatively strong autonomy and planning capabilities. However, their ability to handle multi-dimensional difficulty levels, diverse task types, and real-world demands remains unknown. 
In this paper, we introduce \textsc{ShortcutsBench}, a large-scale benchmark for the comprehensive evaluation of API-based agents in solving real-world complex tasks. \textsc{ShortcutsBench} includes a wealth of real APIs from Apple Inc., refined user queries, human-annotated 
high-quality action sequences, detailed parameter filling values, and parameters requesting necessary input from the system or user. We revealed how existing benchmarks~/~datasets struggle to accommodate the advanced reasoning capabilities of existing more intelligent LLMs. Moreover, our extensive evaluation of agents built with $5$ leading open-source (size $\geq$ 57B) and $5$ closed-source LLMs (e.g. Gemini-1.5-Pro and GPT-4o-mini) with varying intelligence level reveals significant limitations of existing API-based agents in the whole process of handling complex queries related to API selection, parameter filling, and requesting necessary input from the system and the user. These findings highlight the great challenges that API-based agents face in effectively fulfilling real and complex user queries. All datasets, code, experimental logs, and results are available at https://github.com/EachSheep/ShortcutsBench

Published as a conference paper at ICLR 2025
SHORTCUTSBENCH: A LARGE-SCALE REAL-WORLD BENCHMARK FOR API-BASED AGENTS
Haiyang Shen', Yue Li Desong Meng', Dongqi Cai², Sheng Qi², Li Zhang', Mengwei Xu², Yun Ma
Institute for Artificial Intelligence, Peking University
Schooo of Computer Science, Peking University
School of Software & Microelectronies, Peking University
"School of Electronics Engineering and Computer Science, Peking University
Beijing University of Posts and Telecommunications
(hyshen ) @stu. pku. edu. cn, (mayun) 0pku. cntsr
ABSTRACT
Recent advancements in integrating large language models (LLMs) with appli- cation programming interfaces (APls) have gained significant interest in both academia and industry. Recent work demonstrates that these API-based agents exhibit relatively strong autonomy and planning capabilities. However, their ability to handle multi-dimensional difficulty levels, diverse task types, and real-world demands remains unknown. In this paper, introduce SHORTCUTSBENCH, a large-scale benchmark for the comprehensive evaluation of API-based agents solving real-world complex tasks. SHORTCUTSBENCH includes wealth of real APIS from Apple Inc., refined user queries, human- annotttdd high-quality action sequences, detailed parameter filling values, and parameters requesting necessary input from the system or user. We revealed how existing bench- marks datasets struggle to accommodate the advanced reasoning capabilities of existing more intelligent LLMs. Moreover, our extensive evaluation of agents built with leading open-source (size 2 57B) and closed-source LLMs (e.g. Gemini- 1.5-Pro and GPT-4o-mini) with varying intelligence level reveals signif- icant limitations of existing APl-based agents in the whole process of handling complex queries related to API selection, parameter filling, and requesting essary input from the system and the user. These findings highlight the great challenges that API-based agents face effectively fulfilling real and complex user queries. All datasets, code, experimental logs, and results are available at wwt.://www....eeeceeeeeeeeeeee/e htwp:e/sSheeeeeSSttttttteeee.e
INTRODUCTION
Large language model based agents (LLM-
Table 1: Less intelligent LLMs (even 3B) exist- ing benchmarks dataset demonstrated excellent results with the same prompt in Section 4.1
based agents) (Zhao et al. 2025 Chen et al.) 2025, Wang et al.) 2024b Xi et al. 2023) built on application programming interfaces (APIs) have gained significant interest in academia and industry. By integrating LLM with APIs, LLMs access real-time information QQin ct al. 2024), reduce hallucination with external knowledge (Gao al.
Meta ToolL L. ToolBench 2024b
Ace. (%)
2024
2024
LLaMAA332-33
89.64
72.92
79.47
0Wen-2.5-3B
88.29
77.86
91.35
2024), well as plan
LLaMA-3-8B
89.00
78.31
93.57
and complete complex tasks that need multi-
step actions (Gravitas) 2024 Many of these
QWen-2.5-7B
92.50
82.69
94.26
agents (OpenAl) 2024a) have already demon-
GPT-40-mini
88.31
84.50
89.90
strated commendable performance on simple
tasks involving only few actions such as "Check the weather ① and tell me @", This impressive performance raises an important question: Are these API-based agents truly capable of generating action sequences for real and complex demands?
"Corresponding author

---

Published as a conference paper at ICLR 2025
Some existing benchmarks / datasets have attempted to evaluate API-based agents. However, they have three limitations (please refer to Table for all details); First, the APIs (a.k.a tools available to the agent) lack richness, and the queries (a. the task to the agent) lack complexity. They either involve a limited number of APIs, cover small numbers of apps (an app may have 2 1 APIs), or the difficulties of the queries are limited in narrow range, with the average action length ranges from to 5.9. This lack of richness and complexity makes it difficult to effectively distinguish the capabilities of different agents, even on less intelligent LLMs like OWen 2 5-BB (Alibaba| 2024), let alone more intelligent LLMs like Gemini-2 5-P Google 2024). Our evaluation on API selection of these less intelligent LLMs on 3 representative benchmarks datasets (Table 1 shows that even 3B LLMs can achieve impressive results. There is almost no difference in accuracy across LLMs of varying intelligence levels. Therefore, existing benchmarks / datasets struggle to accommodate more intelligent L.LMs and to differentiate the intelligence levels among various LLMs. Second, the APIs lack realism as they may be manually crafted, and the queries fail to reflect actual user demands since they may be either created by hand or generated directly by LLMs without verifying real user demands. Moreover, they only cover the evaluation of API selection, lacking a study on API parameter filling. Efficient and accurate parameter filling is essential for agent to finish the whole process of completing queriess Third, they don't I adequately evaluate the agent's 's ability to request systems the users for the necessary input to resolve the missing information for solving the queries. This is crucial as a user's 's query may be implicit or may not provide all the input an agent needs to solve the task effectively.
In this paper, we innovatively propose to use data extracted from existing Digital Automation Platforms (DAPs), Apple Shortcuts, to construct a high-quality benchmark for API-based agents, i.c. SHORTCUTSBENCH. To the best of our knowledge, API-based agent benchmark considering APIs, queries, and action sequences. SHORTCUTSBENCH provides rich and real APIs, queries with various difficulties and task types, high-quality human- annotated action sequences, and queries from real user demands. Moreover, it also provides precise values for parameter filling, including primitive data types, enum types, and the use of output from previous actions for parameter values, as well as evaluations of the agent's 's awareness in requesting necessary input from the system or user. Furthermore, the scale of APIs, queries, and the corresponding action sequences is comparable or even better to benchmarks / datasets created by LLM modified by existing datasets. The overall comparison between SHORTCUTSBENCH and existing benchmarks/ / datasets is listed in Table 2
ENHORTCUTSBENCE is the first large-scale real
To demonstrate SHORTCUTS advantages, we do extensive evaluations of API-based agents from 10 leading open-source and close-source LLMs, covering varying intelligence levels. To our best known, this is the most comprehensive evaluation considering the API selection, parameter value filling, and recognition of the need for input from the system or the user, covering all key processes of API-based agent. The evaluation results highlight great limitations of existing API-based agents.
In summary, this paper makes the following key contributions:
We identified problems of the existing benchmarks / datasets, specifically that they struggle to accommodate the advanced reasoning capabilities of existing more intelligent LLMs, and have conducted experiments to validate the problem.
We innovatively extracted data from Shortcuts, to build a high-quality benchmark for API-based agents. To our best knowledge, SHORTCUTSBENCH is the most realistic, rich, comprehensive, and large-scale henchmark for API-based agents. We hope this approach to dataset construction will inspire more researchers.
We made efforts to evaluate 10 advanced LLM-based agents with varying intelligence levels on the whole process required to complete user queries, including API selection, parameter filling, and their awareness of requesting necessary input from the system user.
We obtained massive interesting conclusions such as (1) Open-source LLM agents now match
closed-source ones on simpler tasks but still lag behind on complex ones; (2) Extracting necessary
parameters from queries is the most challenging task in parameter filling; (3) There is substantial
lack of awareness in agents when it comes to requesting the necessary input;
datasets as "benchmarks" and fine-tuning datasets as "datasets". MMetaTonk uses the native GPT API, while ToolBench and TooILLM have the longest average action length and the largest scale with real-world API, respectively.
We refer to the evaluation-specie
2

---

Published as a conference paper at ICLR 2025
Table SHORTCUTSBENCH has great advantage in the Trealness and richness, @the complexity of APIs, queries, and corresponding action sequences, @the validity of action sequences, @detailed parameter value filling, @the awareness for asking necessary input, and @the overall scale.
Shortcuts Bench
Meta
Tool
API
Tool
API
Tool
Tool
Tool
Resource
Tool
LLM
Bench
Alpaca
Bank
Bench
QA
Lens
(Ours)
2024b
2024
2024
2023
2023
2024
2024
2024
Real API?
X
X
X
Demand-driven Query?
X
X
X
X
X
X
Human-Annotar Act.?
X
X
X
X
X
X
X
X
Multi-APIs Query?
V
X
X
V
Multi-Step Act.?
V
X
V
Prec. Val. for Para. Fill?
X
X
X
X
Awareness for Ask Info?
X
X
X
X
X
X
X
X
# Apps
88
N/A
3451
3
N/A
N/A
8
N/A
N/A
# APIs
1414
390
16464
1645
53
400
232
13
464
# Queries
7627
21112
12657
17002
3938
274
2726
1530
18770
# Avg APIs
9.62
1.02
2.3
1.0
1.0
2.1
5.4
3.5
2.65
# Avg Actions
21.62
1.02
4.0
1.0
1.0
2.2
5.9
3.9"
2.67
denotes estimation.
We have fully open-sourced all the datasets, code, experimental logs, and results, and provided detailed documents. We hope our research opens new directions for the real-world deployment of existing I.LM-based agents.
RELATED WORK
API-based agents. API-based agents treat APIs as tools. They accept queries, generate action sequences based on queries and provided APIs, and generate next action depends on the history actions (Wang et al. 2024b Yao et al. 2023). Related work about API-based agents can generally be categorized into types: (1) Task- specific enhancement focuses on improving the agent's ability
like using the model (Shen et al. 2024 Zhong et 2024). (2) Data-driven workflows emphasize the importance of data by researching how to construct action sequences, enabling generated data to 2024 Patil et al. 2024). (3) Agent evaluation studies the assessment
fine-tune the model (Qin et al.
of agents (Huang et al. 2024b ct al. 2023).
Code-based agents. Code-based agents code generated for interaction with the external envi- ronment. They accept queries, generate scripts in programming languages such as Python (OpenAl 2024b Wang et al. 2024c), JavaScript (Wang al. 2024a Zheng al.) 2023) or Shell (Openin terpreter [ Sladić et al. 2024). and then input the code into interpreters. The execution results then returned the agent, which is used to help determine the code generation. Currently, these approaches primarily focus on enhancing agent performance in specific tasks by incorporating additional knowledge (Wang et 2024a] Wu et al., (023), increasing feedback (OpenInterpreter 2024; Huang et al. (024a). and decomposing tasks (Huang et al.) 20233 Prasad et al.) (2024). In
addition to work on optimization methods, numerous efforts have emerged to evaluate code-based
agents (rivedi et al. 2024 Liu et al. 2024
Digital Automation Platforms (DAPs). DAPs Abbou et al. 2021 refer software tools or services designed to optimize workflows through automation. DAPs leverage technologies such as robotic process automation (RPA) (Chakraborti et al. (020)) and low-code no-code development tools achieve the goals. DAPs like Zapier (Zapier 2024), Make (Make 2024), and IFTTT (Rahmati et al. 2017) offer extensive APIs that enable users to create automated workflows. Similarly, DAPs such Microsoft Power Automate (Microsofti. 2024) and Tasker (Dias 2024) primarily used to build workflows on Azure and Android, respectively. Recently. with the rise based agents, platforms like Coze (Coze 2024) and Dify (Dify 2024) have emerged "agent construction

---

Published as a conference paper at ICLR 2025
platforms". Functionality like "workflow" in these platforms can also help manually build workflows, but they have been specifically optimized for integration with LLMs.
Shortcuts app (formerly Workflow) (Apple) 2024) is an app developed by Apple for building workflows through a graphical interface, available on Apple's s operating systems (iOS iPadOS and macOS). Shortcuts app can be seen the DAP of Apple. It allows users to create workflows (known as shortcuts (Apple 2024c)) that execute specific tasks on their devices and share them online via iCloud link (024).) Users can also download curated shortcuts from the Gallery of the Shortcuts app. However, the shortcuts available in the Gallery are very limited, with only a few dozen options. To access more shortcuts, users must either collect them from third-party sharing sites like Shorteuts Gallery (Gallery 2024) or create their own. Shortcuts can be triggered through the Shortcuts app. widgets, the share sheet, old Siri (Apple) 2024a), new Siri of Apple Intelligence (Apple 2024b). and they can also be automated to run upon specific events.
Shorteuts are composed of multiple API calls (actions). An agent can use the shortcut as whole API or utilize the individual APIS involved in the shortcut. This paper treats APIs within the shortcuts as APIs available to the agent, aiming for the agent to automatically construct workflows of API calls.
DATASET
In this Section, we first introduce the acquisition of the dataset (Section B.J). Then, we outline the SHORTCUTSBENCIT construction process (Section 3.2 Finally, we outline the setup for evaluation tasks to evaluate the agent's ability to handle tasks of varying difficulty, including the ability to select suitable APIs (Section[3...l the ability do parameter filling (Section 3.3.2). and the awareness in requesting additional input from the system or user (Section 3.3.3).
3.1 DATASET ACQUISITION
anice
APC4 For macCo
phrd-perty xpps
Figure Shows shows the data acquisition process. We first search engines to identify popular pub- lie chortcut-sharingg sites C. We totally find 14 sites (Table 55. Then crawled these sites to obtain fields such as "shortcut name", "function description", "shortcut type", and "iCloud link" @. Then we downloaded the shortcut source file by "iCloud link" and then perform deduplicating based on iCloud link itself and the actual short- cut content (i.e., the action sequences) @ Sub- sequently, we extracted "app name" using the field WFWorks f owAct i Iddetiffiii e in the shorteut source file, and then downloaded associated apps 4 These apps may come from various sources. (1) apps from the ma- cOS or iOS App Store, (2) apps like Keynote from path /Applicat iactiions. ions/ and /System /Applicas impplicaiion/ on macOS, (3) third-party apps from the the official website of the app. During the downloading, we also excluded some legacy apps and paid apps.
Deduplicato
Deduplicate Fittoring
ppp
Download
Deemlood
+ Essac
3 ICloud di Src File
8 Apps
A
Link
(Query AActonal
CSharing
WFActensjonn sap Shorous
ShortcutsBench
TooiiLM
Prompi
APiBench
ToolAppaca
Hulb
Toolbenoh
API-Bank
By
File
Moaily
(Query Actiens)
Tooetormer
TookenGGPT
(b) Existing Benchmark Datasets
Figure 1: (a) illustrates the data acquisition process. (b) shows the dataset acquisition of existing work. APIs existing work are collected from API hubs, created by hand, or modified from existing datasets. The queries and action sequences are constructed using templates or semi / fully automated methods.
Then we managed to extract APIS from the downloaded apps @. The APIs are mainly from intent definition file $ (flemee))) . a act i onsdata from Appintent Inc. 2024b) framework and $ (fflle [filename) inteentdeii inniinitiii t in from SiriKit (Apple- (0024) framework. We extracted all APIS involved the apps. During the extraction, we perform deduplication of APIS based on manually crafted rules as an app may have multiple duplicate API definition files with the same API definition. This process also involves significant manual filtering. Additionally, for app Shortcuts, which deeply integrated with Apple's operating system, need to obtain their API defini- tion files WPActions json from system path / /System/////// weeeem// LLibraryyyyrrrr /Pry//riii /tt///ivvate pivatte?" Parteeorksssss Work f11 owKi t. framework) on macOS, instead of extracting it from the app itself. Subsequently,
(202cc)
4

---

Published as a conference paper at ICLR 2025
a. API Information
NLP desc davs
d. Funcs API
()) www.pppni.catt.aallaaaa.e (pnergt: String, CowC Boolons modil ModdEniity omninueu:: Brolom) - hooGe String
Parameters [paraneter idefuut vulse).
Descrition]:
(2.1) prempt:
Messgee
ChuidPr
AB Cheche Uniigg
(2.3 cheChat new Indiares
should be somee
deffuit: Model Medel
continusus Contintous chat. Whetas
Sorth complete Shorccut inmedaately
Voldd
Chace
41300-0 Chrrart
Contruass
Value
couulValuNName. diylayTypeNaanz
ChaiGPT Nonz
docriptin actionSummary::
Ask ChaGPT
sisgle mevsgge with CharGP roppnse.
Condle
actionSumtary. ChaGPT (pecmpp)
b. Generating User Queries using Info from (c), (d), (e)
Siieddy and palcect you
provided shortcet
APtc Relined conespondng Parameeess Returz Valte....
NLP deseriptions developers
π
ChaGPTS
Open SChoCiPTS
ChuGPT; Discritiiolhitee with ChalPP.
NameinStore: Ask ChatGPT,
DescriptionlnStorr Chat with ChatGPT. Descs from Sharing-site
The construction of SHORTCCTSSENCCH (a) shows the information of API com openai. chat. Ask Intent extracted from the app ChatGPT's $ fffil ename ) act i onsdata. We provide this API description to the LLM, expecting it to call the API the appropriate time. The API information shown in (a) includes the API functionality description (a.k. (a.1)-(a.4) as shown in (d), and the user-friendly natural language description of the API (a.k. (a.4.3)) seen by shortcut developers during programming, as shown in (c). (e) presents the shortcut name and functionality description from the shorteut sharing-sites. (b) shows the simplified prompt fed GPT-4o, instructing it to generating queries based on demands indicated by shortcuts by integrating the info from (c), (d), and (e). Different colors indicate different information sources.
Figure 2:
we further filtered the shortcuts @ based on criteria such as whether the associated apps is paid app. whether the apps were outdated, and whether the APIs were deprecated. Additionally, we imported all shortcuts into the macOS Shortcuts app to ensure they were functional. Finally, as shown in Table 2 we get 88 apps from various categories such as "Health & Fitness" and "Developer Tools". We finally get 1414 APIs involved in 7627 shortcuts.
As the acquisition process involves specific knowledge about shortcut-sharing sites and Apple's operating system, detailed explanations omitted here due to space constraints. For more details about the whole acquisition process, please refer to Appendix A.
3.2 DATASET CONSTRUCTION
As shown in Figure 2, existing benchmarks datasets consist of two parts: (1) APIs; (2) queries and corresponding action sequences.
APIs (Figure a) include the "API description" (a.4), "API name" (a.1), "parameter names" (a.2). "parameter types" T (). "default value" (a.2), "return value type" (a.3), and "return value name" (a.3). The field names in square brackets [] represent the original field name in the shortcut source file. For more details about $ ffilename, and WFActions json, please refer to the Appendix A.2 In existing benchmarks / datasets, the "parameter types" and "return value types" are composed of primitive data types such as int and string. In addition primitive data types, APIs in SHORTCUTSBENCH also include "enum" or "advanced data types". Enum is composed of "the class name" and "the possible value", with each value equipping "value name". We also provide the agent with description of the "enum" in the API information. Advanced data types, such as the model (a.1), include three Stri in types named ident i fier, title, and subtitle. We can comprehend them through their "type name" and "type description".
3(filename: intentdefinin ion,
Query and action sequence. A query is a user command, such as "Tell me what the weather will be like tomorrow." The action sequence (aka. shorteut) is the series of API calls to complete the
5

---

Published as a conference paper at ICLR 2025
query, with each API call referred to as action. The action sequence identifies the steps needed to complete a query. As shown in Figure existing benchmarks / datasets collect APIs first and then use them, either fully automatically through L.MMs. In contrast, action sequences in SHORTCUTS BENCH are all human-annotated. The shorteut developers we ensured the quality of action sequences by filtering shortcuts based on criteria such as whether the associated apps were paid, outdated, or relied on deprecated APIs. We also imported all shortcuts into the macOS Shortcuts app to verify their functionality.
semii-utomatically.. to construct query and action sequences
our annotators. APIs in SHORTCUTS Bench are also all real-world. Moreover,
Generating queries. As shown in Figure [T] existing works construct query and action sequences based on available APIs. In contrast, we construct queries based on existing action sequences and APIs. When constructing a query for specific action sequence, we need to understand the functional description of the action sequence (Figure 2e) and detailed information about the involved APIs (Figureßla). With this information, we can generate higher-quality queries. To ensure the quality of the generated queries, we also leverage the unique advantage of shortcuts: the natural language workflow descriptions (Figure 2 b.7 Figure 2c). By inputting these intuitive natural language descriptions into an LLM, can generate more accurate queries. When generating queries, require the model to naturally include primitive data type parameters and enum data type needed for API calls in generated queries. This helps us evaluate the agent's ability to fill in primitive parameters in Section 3.3.2 To ensure the quality of generated queries, we use the state-of-the-art LLM, GTT-4oTTTTTT (OpenAl [2024) generate the queries. The prompt templates we used generate queries can be found in the Appendix A.2
also
To ensure the quality of queries generated, follow existing work (Qin et al. 2024), we conducted a preliminary experiment using 3 LLMs: G-,- GPT-3.5, and Gemini- of 100 samples. The results showed that human evaluators rated 40 generated queries the highest, outperforming the other 2 LLMs. GPT-4o demonstrated superior performance by accurately capturing required parameters and providing clear query descriptions, meeting our criteria in 94/100. This superior performance can largely be attributed to the natural language workflow descriptions.
on a dataset
3.3 TASK DEFINITION AND METRICS
We aim to address 3 research questions regarding the performance of existing agents built using leading LLMs SHORTCUTSBENCH with varying difficulties: (1) How do they perform in API selection? (2) How do they handle API parameter value filling, including parameters for primitive data types, enums, and outputs from previous actions? (3) Can they recognize when input is required for tasks that need system or user input?
Preliminaries. SHORTCUTSBENCH consists of a set of queries Q = {q1, 92, In corre- sponding "golden" action sequences ASeq {aseqi, asE02. .... aseqn J. and all available APIs APIs = (api1, apiz..... ap.i For each query 9:, 1 S < n, the correspond- ing "golden" action sequence is aseg; {a), (2,,... (laseq.)). ). where the length of the action sequence is |aseqs]. Each app app, has a set of APIs apis, {apin, apiz...., api apisy/h. The action sequences generated by the agent for each query qs are referred as bseq.
Table Final evaluation set with varying difficulties.
lasessl (0, (1, 51 (5,15) (15,30) Overall
# Queries
706 2169 1571
774
5220
# Avg APIs
1.17 3.43 8.30
13.76
6.60
# Avg Acts 1.00 3.19 9.60
21.58
8.34
Prepare available APIs for each query. For each query q., we provide the LLM with a certain number of usable APIs to simulate woal-world scenarios where APIs can be input into the LLM's context. Following existing work (Qin et al.) 2024 Tang et al.) 2023 Xu et al. 2024], we equip each qi with a specific number of APIs For each aseg, let AP7s; represent the number of APIs involved. In addition to these [AP]s;] APIs, we equip each query with extra APIs calculated as max(min (x x |AP1s;] 1 20 [AP1s))) () 0). where x E {3, 4, 5,5. We do this because it is impractical input all APIs into the context simultaneously. When dealing with large number of APIs, additional retrieval is often required (Qin et al. 2024 Qu et al. 2024), which we do not consider in this work.
Further Processing, Considering the context limitations of LLMs, we excluded shortcuts longer than 30 and parts using the API is work f1 ow, actions ons. runwork I ow to call other shortcuts.

---

Published as a conference paper at ICLR 2025
While these shortcuts remain in our open-source dataset, they will not be included in the subsequent evaluation. We aim to study the performance of agents on queries of varying difficulties. As shown in Table Bl we categorize SHORTCUTSBENCH into difficulty levels and 8 task types based on Jasega and "shortcut type", respectively. For more details, please refer to the Appendix A.. When calculating the length, for branching actions like is work fow..wwww. consider the longest branch as the length. Additionally, we ignore the lengths of looping actions like iesissswwrr www.wwokll...... warrriooo.aationss........ actions repeat. count and special actions such as is workflow, actions. comment. Due to the presence of branching actions, the average number of APIs involved when p=
condi ionall we
greater than one, specifically 1.17. For a detailed process, please refer to the Appendix A.3 The number of shortcuts in each level is denoted as 1p. Each query and action sequence is reterred to as Op.; and ase4pp.. with 1 S p S and i Upp
3.3.1 PERFORMANCE ABOUT API SELECTION
Following existing work (Huang al. 2024b Patil 2024 Xu et al. 2024), we use the accuracy of API selection as the metric. The accuracy is calculated as the number of correct API selections mn divided by np. Specifically, each time we predict an action by. S S |aseg;], we provide the agent with all the correct historical actions {a1. Q2 ..... 0--1 0... }. We then require the agent to predict the next action. All actions predicted by the agent form the prediction sequence bseqp.s. This method is similar to the next token prediction (NTP) in LLMs, effectively preventing a cascade of errors in subsequent action predictions due to a single incorrect prediction. During the prediction, when encountering special actions such as branching and looping, we skip predicting these actions and directly add them to the historical actions. For more details, please refer to Appendix|A.4. We chose API selection accuracy over the final result for the following two additional reasons:
SHORTCUTS BENCH contains numerous APIs such as opening the "All Shortcuts Folder" in the Shortcuts app that do not have a return value. This makes it challenging to evaluate using existing metrics that measure the success rate of solving queries (Qin et al. 2024 Xu et al. 2024 ?).
SHORTCUTSBENCH includes numerous APIs with complex input and output types, such as PDFs and Rich Text. Converting these formats into text that LLM can process presents signifi- cant challenge (Naveed et al., 2003), as LLMs struggle to serialize them into text. Consequently, it becomes difficult to ascertain the correctness of the final results. However, measuring API selection accuracy is straightforward.
3.3.2 EFFECTIVENESS OF API PARAMETER VALUE FILLING
In this part, we aim to investigate the performance of agents in API parameter value filling, including parameters for "primitive data types" and "enums" and filling output from previous actions. For each input parameter of every action in SHORTCUTSBENCH, we expect the agent to fill in the following parameters correctly:
Static Parameters Preset: These are static parameters that users provide as default inputs of the action. These static parameters typically include primitive data types such as String and Integer, well as custom Enum defined by app developers. When the query explicitly specifies a parameter that can be used as a static parameter, we expect the agent to accurately fill in the parameter values according to the user's query and the API's definition. When generating queries, we have already required the LLM to naturally include primitive and enumerated data types (Section 3.2). To further ensure that the corresponding parameters are indeed included in the queries during evaluation, we used the LLM to filter these parameters further, ensuring their presence in the queries. Detailed prompts can be found in the Appendix A.5
Outputs from Previous Actions: An action may either have output or, if it does have an output, the output may be used by the following actions. In SHORTCUTSBENCH outputs that are difficult to input directly into the LLM are represented by a unique identifier (UID) and an output name Namee, which can be input into the LLM for processing. The agent should have the ability to correctly use the output values of previous actions.
For the static parameters preset, we evaluate using the overall parameter fill rate. Let sppa; be the total number of parameters that need to be filled in aseg, 1 S i S ng where n, is the number of queries. If the agent correctly fills sppl, parameters in the generated action sequence bseg, then
7

---

Published as a conference paper at ICLR 2025
the Static parameter preset accuracy can be calculated as Accepp Σim sppts/ Ei sppa;- Similarly, for Outputs from previous actions, the accuracy can be calculated as Accofpa = E ofpats/ Eis ofpaa,.
3.3.3 RECOGNITION OF NEED FOR INPUT
In this section, we aim to investigate the ability of existing API-based agents to ask systems or users for necessary input to resolve the missing information. This missing information can come from the system like clipboard (C1 illpbbard), input files (ExtensionInpul (CurrentDate) or from the user (Ask) (Apple-Inc. 2024a). For example, a parameter named tags is usually represented in a shorteut as "tags": Va 1 ue : ("Type":
and the current date
"Ask")) ), where "Type": "Ask" " indicates that the parameter will prompt the user for input. For more details, please refer to Appendix A.6] We use the proportion of correctly identified parameters to evaluate the agent's ability recognize the need for input from the system or the user. Let n, be the number of queries, aska;, askt, be the number of times the need from the system or the user appears in aseq, bseq;, respectively, The accuracy of ask for necessary information can be calculated as AcCa fmi askts/aaae askaa
4 EVALUATION
4.1 SETUP
Model. Referencing existing work (Huang et al., 2024b; Qin et al.) 2024, Li et al. 2023). considering the performance of existing L.LMs, we selected 10 most advanced LLMs to construct API-based agent. The chosen model includes 5 closed-sourced and 5 open- source LLMs, covering varying intelligence levels. Among them, Semini-1.5-Pre LLAMA-3-70B, QMen-2-7CB, and Deepseek-2-chat/ coder are LLMs bench- marked against GPT-40-2024-05, while Gemini-1 5-Plash, ChatGLM-4-Air, and and 5-turbo. GPT-4o, 1.1aMA-3.1-708 405B, and
QWen-2-57B are benchmarked against GPT-4o-mini-2244- We did not evaluate peT-ol-preview/miiie 5 mainly due to limited access, high costs, and the fact that LLaMA and QWen are minor version improvements with recent releases.
We also compare work with specialized API-calling fine-tuned LLMs, such as AgentLM (Zeng ctt al. 2024) and xLAM (Zhang et al. 2024), uncovering intriguing findings. Due to space limitations, the results presented in Appendix A.9 for interested readers.
Prompt Template. Following existing work (Huang et al., 2024b; Qin et al. 2024 Tang et al. 2023 Zhuang al. 2024), slightly modified the ReACT Yao et al. 2023) templates construct the API-based agents. For all 3 research questions (RQs), we use the same prompt templates. An agent should correctly select APIs, fill in parameters, and be aware of the need to request necessary input from the system or user at appropriate times. Please refer to Appendix A.. for more details.
4.2
RESULT ANALYSIS
100
Oversil
01:1
<5:35)
115.30)
API Selection Accuracy 196)
100
API Selection Accuncc
8
80-
5XT4 11
gemmel P.P
Owten
Dowpsner
Cocater
Liut 3-708
Gemtel Faah
Ottma
GPT
UPT Sttrro
2:748
Divected
Corter teay
Latt
Gramini
GFT
Figure 3: The API selection accuracy on queries with Figure 4: The API selection accuracy dif- different complexity levels.
ference of each LLM across task types. From Figure 3 we can see that for tasks with a lower difficulty level, both less intelligent LLMs and more intelligent L.LMs perform well. This is similar the conclusion drawn from Table 1 However,
8

---

Published as a conference paper at ICLR 2025
Productivity Utllides
tealth Fimaes
Ententainment & Nedia
Lifestyle &a Social
100
100
60-
60
API Selectionn
Development
20
20
Figure The API selection accuracy of each task type on 10 API-based agents.
for tasks with a higher difficulty level, only the more intelligent LLMs like Gemini-1.4. -- Deepseek, and QWin2 perform adequately.
The superiority of SHORTCUTSBENCH. Combined with Table SHORTCUTSBENCH effec- tively distinguishes between different levels of intelligence, making it superior benchmark.
Through the results of API selection accuracy (Section 3.3.1 we get the following conclusions:
Agents built using open-source LLMs perform comparably to closed-source LLMs lower-difficllty tasks but still lag on higher-difficulty tasks. From Figure 3 we know that open-source LLMs >= 70B match the performance of closed-source LLMs from the first difficulty tasks, significantly outperforming they still lag behind closed-source LLMs in handling complex tasks the 4-th level. For more details, please refer to Appendix A.8
44-mmin and GPT-3 5- 5-turbo However,
Existing L.MM-baeed agents still perform poorly on tasks requiring multi-step reasoning, even more intelligent LLMs like Gemini-1. 5-Pro struggle with high-difficulty tasks. From Figure 3 we know that almost all LLMs handle well in API selection tasks at the level of (0, 1], but only more advanced models like Gemini-i 1 5- Pro and QWen 2-72B can do well in higher-difficule tasks of (1, 5]. As tasks become more complex, the accuracy drops sharply. The average accuracy dropped by 19% task difficulty rose from (0, to (1, 5]. ranging from 9% decrease (Deepseek 1oposeeek--cciie 2-chat) to a 44% (Chat r-.ir). From (0, 1] to (5, 15), accuracy fell by 46%, with drops from 38% (Gemin ni 1 5-P 58% (ChatGTM-4422)
Agents built with the same LLM show significant performance variations across different types of tasks. From Figure 5 we know that the performance difference of agents built with different LLM ranges from 15.94% (GPT-4o-mini) to 36.70%
Existing API-based agents perform well on tasks in daily life such as Lifestyle Social but show poorer performance on professional tasks like Development & API. From Figure 5 we know that Li Li feffeyyle Social exhibit the highest average accuracy, surpassing the lowest category, Devel opment & API by approximately 18%.
Based the results of API Parameter Value Filling (Section 3.3.2 draw following conclusions:
API selection and parameter filling both impact the agent's performance. However, API selection has greater effect. As shown in Figure ba| for existing more intelligent LLM like Gemini-1.. 5-Pro, increased task difficulty has much smaller impact on the accuracy of parameter filling, especially on using outputs from previous actions. This indicates that the greatest limitation of existing API-based agents addressing user queries lies in the reasoning and planning capabilities implied by API selection.
The performance of API parameter filling remains a bottleneck existing less intelligent LLMss As shown in Figure bal the performance of less intelligent LLMs like GPT-4o-mini in API parameter filling significantly decreases as task difficulty increases.
Compared to using the outputs of previous actions, extracting relevant parameters from the user's query and filling them is more challenging. As shown in Figure [a] the colors in the top plot (filling primitive data types and enum data types) are generally lighter than those in the

---

Published as a conference paper at ICLR 2025
FIIl
10
03.33
90.24
66.44
89.34
84.48
91.74
63.96
94.29
03.78
89.00
81.90
82.1 13
81.8 31.86
81.41
75.18
83. 92
71.97
33.24
70.59
76.65
90
75.53
73.68
73.92
75.07
63.28
72.12
64.72
75.13
65.28
66.97
Primitive
Ct5
Overall
65.63
59.48
55.33
57.51
80
73,22
65.52
Prev, Actonss FHI
(2.5)
94.17
84.76
91.12
89.74
86.58
76.22
84.88
79.52
75.45
70
(15 (5.15)
88800 05
79.51
83.47
82.94
79. 00
78.81
64.53
72.56
66.12
64.54
30)
60
Overal
89.14
81.47
84.70
85.06
82.01
82.03
67.78
75.13
70.33
68.07
Gemini
OWen
DeepseekDeepseek 2-chat 2-coder
LLaMA 3-70B
Gemini 1.5-Flash
OWen
GPT do-mini
GPT
ChatGLM 4-Air
(a) Accuracy of primitive data types & enum data types (upper) and outputs from previous actions (lower).
Error Rate (%)
20
No Prediction
Format Error
Choose Error
10
Geminni 1.5-Pro
owen 2-728
Deepseek
2-chat Deepseck
2-coder LLaMA
Gemini
1.5-Flash owen
GPT 4 4o-mini
GPT 3.5-turbo
ChatGLM
3-70B
2-57B
4-Air
(b) The error rates for action parameter value filling.
Table 4: The accuracy of recognition of the need for input from the system or the user.
Gemini 1.5
QWen
Deep Deep
LLaMA
Gemini
QWen
GPT
GPT
Chat
Levels
2
seek2 seek2
3
1.5
4o
3.5
GLM4
Pro
72B
chat
coder
70B
Flash
57B
mini
turbo
Air
(0, 1]
33.33
37.78
64.29
62.71
47.62
62.79
22.22
37.14
28.89
47.62
51
45.95
50.40
55.50
60.08
44.08
53.99
37.24
40.55
37.70
48.06
(5, 15]
51.85
36.42
40.76
49.44
35.71
40.65
28.37
29.71
20.33
48.42
(15,
46.67
25.00
27.59
43.14
22.22
44.64
8.11
38.89
17.14
48.89
Overall
46.59
41.97
47.90
55.18
49.89
40.71
30.74
36.71
30.55
48.28
bottom plot (filling the outputs of previous actions as parameters). The accuracy drop ranges from 2.55% (- to 15.39% (Deepseek-2-Chet)
For existing less intelligent LLMs errors mainly stem from incorrect output formats and wrong API selections. Figure 6b shows error types for tasks requiring outputs from previous actions. It can be seen that powerful LLMs like Gemini-1. --rroo rarely make format errors, whereas the less intelligent models frequently make mistakes in output format and API selection.
The results from Recognition of Need for Input (Section 3.3.3 lead us to the following conclusions:
All agents perform poorly at recognizing necessary system and user inputs when re- quired. Overall, agents have weak recognition capabilities, with accuracy ranging between 30.55% (GPT-3.55 (--urboo and 55. (D.e8%Deeeeeeee 2-coder). Larger LLMs such as frepspeed-2-chaee (236B) still demonstrate better recognition accuracy.
CONCLUSION
In this paper, we introduce SHORTCUTSSENCI,, a benchmark for evaluating API-based agents. To the best of our knowledge, SHORTTCTTSBNNC is the most realistic, rich, and comprehensive benchmark of its kind. Our findings indicate that for agents built on the most advanced LLMs, the primary bottleneck is API selection. For the most cost-effective LLMs, there is considerable room for improvement in both API selection and parameter filling. Additionally, we identified a significant deficiency in the agents' awareness of requesting necessary input.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
App store categories. categories Accessed: date-of-access.
URL
Mohammed Abdou, Abdelrahman M Ezz, and Ibrahim Farag. Digital automation platforms com- parative study. 2021 4th International Conference on Information and Computer Technologies
(ICICT), pp. 276-286. IEEE, 2021.
Alibaba.
Dwwnnt.5-b-instruutt 5 3B Instruct, 2024. Accessed: 2024-06-10.
httras///auggiiaffeee./////e////
Apple. Shortcuts app. 2024. URL /ttpa///////////aa..aa../oooo//r
1d915249334 Accessed: 2024-05-09,
Apple. Use shortcut iphone.
htte:://suppprt..pplle.ccc/////a
2024a. Accessed: [Date of Access].
Apple. Apple worldwide developers conference 2024. httpes://develpprr.pppl...co
wwdc247] 2024b. Accessed: [Date of Access].
Apple. icloud upi shortcut, 2024a.
URL http:://wwww....ddddd.o//////tta
Reccords/cc22Bbbeeees a 94 6a 9b2000955222aaaaaa 93220223
Accessed: 2024-005-09.
Apple. icloud shorteut, 2024b.
URL
Accessed: 2024-05-09.
Apple. Which is shortcut, 2024c. 2024c.
URL http://///tta///ooot..pppaaapo.p Accessed: 2024-05-09.
Shortcuts/weelcone/iis
Apple-Inc. Use the ask each time variable in shortcut on iphone or ipad.
attps://support
apple. 2atpa///oooomkkk/oe/e///rrr/r//2 05-15.
2024a. Accessed: 2024-
Anele-Ine. Appintent documentation, 2024b. URL
Dbcumeemtatioon//pp..ntettt//s/ ntent. Accessed: 2024-05-09.
Anc.e-Inc. Sirikit documentation, 2024c. URL bttps://develleerr.ppee...o
documentattioon/iiiiiks Accessed: 2024-05-09.
Tathagata Chakraborti, Vatche Isahagian, Rania Khalaf, Yasaman Khazaeni, Vinod Muthusamy, Yara Rizk, and Merve Unuvar. From robotic process automation to intelligent process automa- tion: -emerging trends- Business Process Management: Blockchain and Robotic Process Automation Forum: BPM 2020 Blockchain and RPA Forum, Seville, Spain, September 13-18, 2020,
Proceedings 18, pp. 215-228. Springer, 2020.
Zhiyang Chen, Yun Ma, Haiyang SHEN, and Mugeng Liu. Weinfer: webGPU on LLM inference web browsers. In THE WEB CONFERENCE 2025, 2025. URL
Unleashing the power of
ILac2
Coze. Coze.
2024. Accessed: 2024-05-10.
João Dias. Tasker for android. https://tasker: joacapps. com/ 2024. Accessed: 2024-11 11-
24.
Dify. Dify. https://ciff..et 2024. Accessed: 2024-05-10.
Shortcuts Gallery. Shorteuts gallery, 2024. URL [ICpss://hoorcuusssaleeyyooo Ac- cessed: 2024-05-09.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. augmentee generation for large language models: survey, 2024. URL TRR.https::/ce http://aawwv..org//aa/1//211111/

---

Published as a conference paper at ICLR 2025
Google.
Gemini-1.5-pp 2024.
URL
http:r//eeeennn...ooo/////hhhooo
gemini/ Accessed: 2024-05-17.
Significant Gravitas.
Autogpt,
2024. Accessed: 2024-05-09.
URL https://github.com
Di Huang, Ziyuan Nan, Xing Hu, Pengwei Jin, Shaohui Peng, Yuanbo Wen, Rui Zhang, Zidong Du, Qi Guo, Yewen Pu, and Yunji Chen. ANPL: Towards natural programming with interactive decomposition. In seirth- Conference on Neural Information Processing Systems, 2023.
URL
net Formm RITOOum
Dong Huang, Jie M. Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation, 2024a. URL https
//arxiv. org/ abs/ 2212. 3010
Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun. Metatool benchmark for large language models: De- ciding whether to use tools and which to use. In The Twelfth International Conference on Learning Representations, 2024b. URL http::t:/pp.eeeeeeeeeeeeee http:/ewwweeeeeeeeeeeeeeeeeeeeee wttprr/wwwww..etteeooreeeeoe... Leaeet//ormmm1--R222tttaatt
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang. and Yongbin Li. API-bank: comprehensive benchmark for tool-augmented LLMs. In Houda Bouamor, Juan Pino. and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 3102-3116 Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18633/v1 http22222...2maaaiii.aom 187. URL wwwp://2oooorr./2222222// emnlp ma iiaiin.. 187
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang. Shudan Zhang, Xiang Deng, Aohan Zeng. Zhengxiao Du Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating LLMs as agents. In The Twelfth International Conference on Learning Representations, 2024. URL
Make. Make. https:: www wwk.aaakk..mm com/ 2024. Accessed: 2024-05-10
Meta. Llama 3, 2024. URL
meta..eet.... www.ta....mlaaaa Accessed: 2024-05-09.
Microsoft. Microsoft power automate process automation platform.
https
7/ w/www.micronm wiw www.oo......ocommmmmmmmmmmccf... com /en eut///nnuuuueeee http://wwrrppppce
2024. Accessed: 2024-11-24.
Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saced Anwar, Muhammad Usman, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435, 2023.
OpenAl. Hello gpt-4o, 2024. URL hwt.tpppp/pppaaa..coo////eee//ls cessed: 2024-05-17.
Ac-
OpenAI. Introducing gpts, 2024a.
URL
https://operai.com//nees
int roducingg Accessed: 2024-05-09,
OpenAI. Openai code interpreter documentation.
assistants/tools / http:///toooossooeeeeeeeeeeeeeee 2024b. Accessed: 2024-05-10.
OpenAl
Community.
Conversation
context
and
quadratic
billing,
2023.
URL
https://comnung openai.......................... com/t/
conversation wen.ttiiionnnnnttt.oommo wwwtext-tttooooo wnweeeett-nd-uuuuuuddaee www.reewwtta-d--aaaaa------llll-
Accessed: 2023-07-
22.
OpenInterpreter.
Open interpreter.
open nterpreter 2024. Accessed: 2024-05-10.
12

---

Published as a conference paper at ICLR 2025
Shishir G Patil, Tianjun Zhang, Xin Wang. and Joseph E. Gonzalez. Gorilla: connected with massive APIs. In The Thirty- Annual Conference on Neural Information Processing Systems, 2024. URL
Large language model
Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. ADaPT: As-needed decomposition and planning with language mod- els. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 4226 42522 Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. /1553/11222244.4 http://////22....innnnnnnnnaaeae findings- URL ht..pss//aclannhclooy.y. wwwph//olll.o..g///oo/e 2024. f www.iiinnnnnnnaaaam
264
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong. Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun. TooILLM: Facilitating large language models to master 16000+ real-world apis. The Twelfth International Conference on Learning Representations, 2024.
Changle Qu Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. Towards completeness-oriente tool retrieval for large language models. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, CIKM '24, pp. 1930-1940, New York, NY. USA, 2024. Association for Com- puting Machinery. ISBN 9798400704369.
doi: 10.1 10.1145/3627763333799888 URL https
Qwen. Qwen2-72b-instruee. http://///////ggggoaacacc/////// 2024. Accessed: 2024-06-10.
Instrrructt
Amir Rahmati, Earlence Fernandes, Jaeyeon Jung, and Atul Prakash. Ifttt vs. zapier: A comparative study of trigger-action programming frameworks. arXiv preprint 1xXiv:1709.02785 2017.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yucting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances Neural Information Processing Systems, 36, 2024.
Muris Sladié, Veronica Valeros, Carlos Catania, and Sebastian Garcia. Lim in the shell: Gen- erative honeypots. In 2024 IEEE European Symposium on Security and Privacy Workshops (EuneSkamp:ammpPPW) pp. 430 435. IEEE, July 2024. doi: 10. 10.110//erooop11112224000000 /00 wtt.:////...1111//////////1111
httLhttpp//dx.ooo
312. 2024 00054
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. Toolalpaca: General- ized tool learning for language models with 3000 simulated cases. arXiv preprint ar Xww.:xi2:330000000000 2023.
Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. AppWorld: A controllable world of apps and people for benchmarking interactive coding agents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume Long Papers), pp. 16022-16076, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 186538/ 1/222/4./-o22//.oooggg URL httthht/s/accclaa http:://aaaaaathhoo..co... 2024 acl long. .80
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large lan- guage models. Transactions on Machine Learning Research, 2024a. ISSN 2835-8856. URL http://ppwweeeeeeeeeeeeeeeeeeeee
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6), March 2024b. ISSN
doi: 10. 10.100774111744--244403333 10. 1007007//0 -24-441----2235333
URL http: : / //dxx. boi.org/
13

---

Published as a conference paper at ICLR 2025
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better LLM agents. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.). Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine ngs
Learning Research, pp. 50208-50232. PMLR, 21-27 Jul 2024c. wttptttts///ppeeee www.ppees/eee v235 wwtp////////2222.
wang 4h. html
Yue Wu, So Yeon Min, Shrimai Prabhumoye, Yonatan Bisk, Ruslan Salakhutdinov, Amos Azaria, Tom Mitchell, and Yuanzhi Li. SPRING: Studying papers and reasoning to play games. In seieth- Conference on Neural Information Processing Systems, 2023. URL https?
RMDER
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang. Changhao Jiang. Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongx- iang Weng, Wensen Cheng, Qi Zhang, Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xuanjing Huang, and Tao Gui. The rise and potential of large language model based agents: A survey, 2023. URL httttt::///artttt
org/ abs/ 22t..2309..2 07864
Qiantong Xu, Fenglu Hong. Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. On the tool manipulation capability of open-sourced large language models, 2024. URL https
//ppenrvviw net. forumfiiioo 10
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023. URL BRK.httts:///penreeeee eew..wwww net/ wwt//eertoumooooo
WE vluYUL-X
Zapier. Zapier- automate your work with no code, 2024. URL cessed: 2024-11-24.
com/ Ac-
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. AgentTuning: Enabling generalized agent abilities for LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computationnal Linguistics: ACL 2024, pp. 3053-3077, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/ 112024.fnndigss-cll.
URL https: /tLLttte/////aaaaalloooooooor http:///222....inggggg.e.. acl
181
Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, Zhiwei Liu, Yihao Feng, Tulika Awalgaonkar, Rithesh Murthy, Eric Hu, Zeyuan Chen, Ran Xu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Silvio Savarese, and Caiming Xiong. xlam: A family of large action models empower agent systems, 2024. URL https:://aaaat
org/ abs 2409 03215
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang, Zican Dong, Yifan Du Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian Yun Nie and Ji-Rong Wen. A survey of large language models, 2025. URL
wwwpaaawww.....aaaao..a /bs/
2303. 18223
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufci Xue, Lei Shen, Zihan Wang. Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '23, pp. 5673-5684, New York. NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701030. doi: 10. 01114///558000533599999 URL/attss://Gae http://oooooogg////// wwt. 1ee.11111133331 3599790
Wanjun Zhong, Lianghong Guo, Qiqi Gao. He Ye, and Yanlin Wang. Memorybank: Enhancing large language models with long-term term memory. In AAAI, pp. 19724-19731, 2024. URL (tthtss
7/doi. www./org/.... 609 a 206117..9944
14

---

Published as a conference paper at ICLR 2025
Yuchen Zhuang, Yue Yu, Kuan Wang. Haotian Sun, and Chao Zhang. Toolqa: A dataset for Ilm question answering with external tools. Advances in Neural Information Processing Systems, 36, 2024.
A APPENDIX SUPPLEMENTAL MATERIAL
A.1 DATASET ACQUISITION
In this section, we introduce more details about the dataset acquisition introduced in Section 3.1
Regarding data acquisition, we first use search engines to identify popular public shortcut-sharing sites (@ in Figure T). We found total of 14 sites. These sites include:
Table 5: 14 shortcut-sharing sites, including names, URLs, and shortcut counts.
# Site Name
URL
Count
Matthewcassinelli
nttp:://natthhewasssineee com
1535
2
Routinehub
attps://routinehub..co
6860
3
MacStories
http:://ww..mccsoorees..ee
4993
ShareShortcuts
attps//tharessoorttttts.coo
2395
5
SholteutsGallery
httee://thorttntssellleeyyooo
4269
6
iSpazio
nttps//sthortcutts.ssppi.s
115
7
Jiejingku
httpe://jllaiinkko.net
3347
8
SSPai
http:r//chortcutss.spp....
145
Jiejing.fun
nttps://leejing.ffu
84
10 Kejicut
https://ww..ej..tt.ccom
37
11 RCuts
http::/www.rcuts.coom
133
12 Sharecuts
http:://sharecuts..ap
2395
13 Siri-shortcuts
http://www.sirrrssorrtttsss
15
14 Reddit
com
100
Total (After Deduplication):8675
We obtain shortcuts from these sites. Specifically, each dataset includes the "shortcut name" (NameInStore), "function description" (DescriptionInstor) CGateoorIInStore
"shortcut type" and most importantly, the "iCloud link" (Apple) 2024a). Addition- ally, it includes less important data such as the number of downloads (Downloads), favorites (Favorites), reads (Reads), and ratings (Rates), All shortcuts include Name InmeonS and Descript ionInStore, while the availability of other fields varies slightly depending on the specific shorteut-sharing site.
We then downloaded the shortcut source file by "iCloud link" and performed deduplication based on both iCloud links and the actual shortcut content (i.c., action sequences) to ensure the uniqueness of each shortcut in the final dataset (@) in Figure D For details on downloading source files via iCloud links, please refer to our open-source code repository. We do deduplication because shortcuts sharing sites store shortcuts as iCloud links, which often results in the same shortcut appearing in multiple sharing-site. Additionally, shortcuts linked by these iCloud links could have identical content, making deduplication essential to ensure that each shortcut in the final dataset was unique.
We then extracted the app name using the field WF Worrkk owAct on on Tdenti fier from the shorteut source file and downloaded the associated apps (@ in Figure T). Shortcuts are composed of a series shortcut API calls, referred to as Actions. An example of a typical shortcut shown in Figure Each shorteut API call is identified by name, which usually includes the app's identifier, such as com. openai. chat, and the Intent name, such as AskIntent. For most API names, the segment before the last dot represents the app name, while the segment after denotes the Intent name. We auniiialomaaiially extracted all app names to streamline the app download process.
We download these apps from various sources:
15

---

Published as a conference paper at ICLR 2025
Ask ChatGPT
Done
API Name
Paras
System Paras
API Paras
"FFWorffooAAccioons"
Ask ChatGPT
What's the
weather like tomorrow?
"WFFootttowAccionddenffee"""" "com. www.mopenai.chtt.ooo Askintent"
WWFWonttowAAciionPrraaasss
"ApplntentDescriloor.
"Applitentidentifee"" "AskIntent",
"Bundieldentfier" "com.opena"" peom.openai.cha"" "Name": "ChatGPT".
Search Actions
"Teamldentifler" "2DC432GLL2"
Scripting
Meda
Sharing
"UUID":
prompt": "What's the weather like tomorrow?"
Open App
Send Message
Figure 7: An example a shortcut: Ask ChatGPT.
Apps from the macOS or iOS App Store: We downloaded a variety of applications directly from Apple's official platforms. This provided with vast selection of apps that are widely used and trusted by users.
System apps like Keynole from paths /Appl iiatiiiiio i and /System/ App 1 cat on on macOS: These are pre-installed applications integral to the operating system. Including them ensured that our dataset covered essential tools commonly used by macOS users.
Third-party apps from the official websites of the apps: To include software available through the App Store, we downloaded apps from their official websites. This allowed us to capture broader range of functionalities offered by third-party developers.
During the downloading process, we also excluded some legacy apps that are no longer maintained and 12 paid apps avoid potential licensing issues and focus on applications readily accessible to the general public.
Then we managed to extract APIs from the downloaded apps (@ in Figure I. The APIs are mainly from intent definition file $ (filename)
iosddtaa from Appintent (Apple- Inc. (024b) framework and $ fi filenn (lleemmm) ) int sentieiiiiiii work. We extracted all APIs involved in the apps. During the extraction, we perform deduplication of
from SiriKit (Apple-Inc, 2024c) frame-
APIs based on manually crafted rules as an app may have multiple duplicate API definition files with the same API definition.
We perform deduplication to streamline API definitions, minimize redundancy, and ensure compati- bility across frameworks, addressing inconsistencies introduced by the coexistence of s Kit and AppIntents. Si rikitt kitt, introduced 2016 with iOS 10, enabled applications to integrate with Siri for voice command interactions. In 2022, Apple launched AppIntents with iOS providing a more modern and flexible approach to defining and handling app intents. AppIntent integration with Siri, Shortcuts, widgets, and more. To encourage adoption, Apple has provided migration tools for developers transitioning from SiriKit. However, some apps still rely on the SiriKit. Under Sirikii developers use $filename int intenttefiiinn
facilitate
files, while the
AppIntents relies on $fi lename. act i inndata files. These files define APIs corresponding actions Shortcuts. Apps may include only $ Sfileeannen ename. nteetdef n i on files, only $fi ffilename.. act actiionee innsdat files, both, potentially leading redundancy in API definitions. To address this, we have implemented a set rules to reduce API definition files and ensure API uniqueness.
Additionally, for app Shortcuts, which are deeply integrated with Apple's 's operating system, we need to obtain their API definition files WFActions. json from system path / /System/ wwtsstemmLibbrrrrr Library/ Pri www.erawwssoooooeeeeeeeee the app itself.
framework/ on macOS, instead extracting from
16

---

Published as a conference paper at ICLR 2025
Subsequently, we further filtered the shortcuts based on criteria such whether the associated apps is paid app, whether the apps were outdated, and whether the APIs were deprecated. Additionally, we imported all shortcuts into the macOS Shortcuts app to ensure they were functional. These steps were repeated multiple times.
Finally, shown in Table 2 get 88 apps from various categories such "Health & Fitness", "Developer Tools", and "Lifestyle". These apps total include 1414 APIs, including all of 556 APIs (Not all APIs have been used in Shortcuts) involved in 7627 shorteuts.
The approximate time spent on each step of the process is outlined below:
Shortcut site collection: Approximately 3 days, completed entirely manually.
Link scraping using Selenium: Around 2 weeks, requiring custom scripts for each site.
Shorteut deduplication, API validity checks, and shorteut functionality validation: Approximately 4 weeks. Deduplication: Automated using iCloud links and content cleaning.
API validity checks:
Performed manually. Shorteut validity checks: A mix of automated and semi-automated methods. Automated filtering was conducted using Apple Scripts to execute shortcuts for preliminary filtering, followed by manual validation through importing shortcuts into the Shortcuts app.
Additional manual and automated checks were conducted throughout the process but are not detailed here.
A.2 DATASET CONSTRUCTION
The API definition files extracted from the app exist in two forms: the S (filennam)) intentdefinitisn files as indicated by the Sirikit framework and the $ (filename) act actiondatt files as indicated by the App Intent framework. Additionally, Apple's first-party apps provide third type of definition file, WFActions. json. All three file formats provide "API description", "API name", "parameter names", "parameter types", "default value", "return value type", and "return value name", but differ their file format. We give a sample from each of three different file formats, shown in Figure
We construct queries based on existing action sequences and APIs. To ensure the quality of these queries, we utilize the natural language workflow descriptions unique to shortcuts. When generating queries, we require the model to naturally include primitive data type parameters and enum data types needed for API calls. This helps evaluate the agent's 's ability to handle primitive parameters. We do not require the inclusion of complex data types in the queries, as they are difficult to convert to text and challenging evaluate. To ensure high-quality query generation, we use the state-of-the-art LLM, GPT- 4o (OpenAl 2024). The prompt templates used for generating queries are provided in Figure
To ensure the quality of shortcuts, we filtered them based on criteria such as whether the associated apps were paid, outdated, or relied on deprecated APIs. Additionally, all shortcuts were imported into the macOS Shortcuts app to verify functionality. Deduplication and error-checking processes were carried out throughout the entire data collection phase.
For ensuring the quality of generated queries, following prior work QQin et al. 2024 we conducted a preliminary experiment with three L.LMs: GPT-4o, GPT-3. 5, and Gemi ni Srrro, on dataset of 100 samples. Human evaluators rated GPT-4c as generating the highest-qualin queries, outperforming the other two models. GPT-4o 4o excelled in accurately identifying required parameters and providing clear query descriptions, meeting our criteria in 94 out of 100 cases. This superior performance can largely be attributed to the natural language workflow descriptions. While we acknowledge that not all queries may fully meet requirements, we believe approach is reasonable. Similar works, such as TooILLM, rely on GPT for large-scale query and action sequence generation without guaranteeing complete accuracy.
17

---

Published as a conference paper at ICLR 2025
(I) com.openui.chat.Adie (prompe: String, parncierDDssciption))
ModelEnntity continaous: Boolean) ChiGPT: String
(2) [paramdeer (default
(2.1) prompt Message. Message to send to ChatGPT
(2.2) Cha Start chat. Indicases whether chat should staree
(2.3) model (default); Model. Model use with the new chat
(2.4) continueus (0): Contimoous chat. Whether enable bock-and aort-frtt chat complete the Shortcur immediately after response
(3) Return Value (reium value ValicNaa, display TypeName]:
(3.1) Ask CharGPT: None
(4) Description [tile description accienSummary::
(4.1) tile: Ask CbatGPT
single message to ChatGiPT and return resporse. (4.3) actionSummary: Search for (query)
(4.2) description: action
(1) www.dllbwaciineeeeriieeeeoooooma (WFHTML: WFStringCoomcentlem)
public.himl
(2) [paramater (dafault valuc): Descriptionlpuu):
(2.1) WFHTML. HTML
(3) Return Value (return value name: DeseriptionResul:]
Rich Text HTME: None
(4) Description [Name Descriptionummas ParamecrSummary.
(4.1) Name: Make Rich Text from HUML.
(4.2) DeseriptioSSmmmary: Takes the inputted HTM MI and tums it into rich text, which can then he converted other formats.
(4.3) PurameterSummary-
text (WPHTML)
(1) com.aly <ompaalyys wwsapsssspppmmc.uu. watersappmma......mmtttiiiooo. Inmerteeetttttttttntttnn (shect: SheetReference (ONjeett), text: String, format: TextForaat (Enum), position: TextPosition (Fnum)) Resalt: None
Parameters [parameter (default value): NimmmmPonmmeeeriilllyyynnn wNw.naaaanennnilllllleal Nome
(2.1) sheet: Sheet. Sheet Reference
(2.2) text: Content.
(2.3) format: None
(2.4) TextPosition: None
(3) Return Value (return value name: NineeilkepomePPracneeeeeiggiiggg
(4) Description, [INinteniTille INtnteniDescription
(4.1) INIntenrTidle: Add Text to Sheet.
INintentDesccipttoon:
existing Ulysses.
(4.3) NMintentParameterC ombination Title: Add (text) (sheet)
We randomly selected three samples from three different definition files, as shown in the upper ($ (filename) act ionsdataa, middle (WFActions, json), and lower ($ filename) names. In practice, there are various details to handle, such as name prefixes and missing fields. For complete details, please refer to open-source code.
Figure 8:
in) figures. The content in brackets represents different field
18

---

Published as a conference paper at ICLR 2025
SYSTEM PROMPT TEMPLATE:
Shortcut consist of a sequence of actions, each is an API call, to execute user-providde nee--poviied queries. As user-friendly and patient inquirer, you need craft query based the provided shortcut. This query, formatted as a question, should describe the task a user wants to complete and adhere to the following criteria:
1. The problem described in the query must be solvable using the shorteut.
2. The query should include all required parameters from the shortcut.
3. The query should be naturally phrased, integrating parameters seamlessly into the question rather than listing them separately.
For each shorteut command, will provide you with five fields:
1. 'RecordName: The name of the shorteut, briefly describing its function.
2. "Description of the Shortcut Workflow': A description of the entire action workflow of the shortcut.
3. 'Comments': Optional. Notes from the shortcut's developer, which may describe its func- tions other features.
4. 'Description in Store': A description of the shortcut's functionality provided in the shortcut store.
5. 'API Description List': Detailed descriptions of the APIs involved in the shorteut.
You should rely primarily the 'Description of the Shorteut Workflow' and API Description List', and refer to 'RecordName', 'Comments", and 'Description in Store' to formulate the final query.
USER PROMPT TEMPLATE:
Below are the five fields I provide to you:
1. 'RecordName': (RecordName)
2. "Deccription of the Shortcut Workflow': DDssrppiiootthshSorruukkkkkooont
3. 'Comments': {Comments}
4. "Description in Store': (DescriptionInStorr)
5. 'API Description List': (APIDescrippiinnList)
Please generate query based on these details. Alongside the query, provide the shortcut's name and a description of its functionality using the following JSON format:
"shortcut_name" "ThisisShortcttNarrd mnhortcut__decrriptiie "hiistsshorrtccttteerrriooooo "query": "ThisIsQuery"
Do not output any other content; your response should only be in this JSON format. Do not simply repeat the shortcut workflow. Parameters surrounded by (U) should not appear in the generated query. Output the ISON directly without using "'json XX" to enclose it.
Note again, you should include all required parameters in the generated query. Please give your answer in English.
Figure 9: System and user prompt templates for query generation based on shorteut
A.3 TASK DEFINITION AND METRICS
Considering the context limitations of LLMs, excluded shortcuts longer than 30 and parts using the API is workflow actions. runworkflow to call other shortcuts. While these shortcuts remain in open-source dataset, they will not be included in the evaluation. We aim to study
19

---

Published as a conference paper at ICLR 2025
the performance of agents queries of varying difficulties. As shown in Table 3 categorize SHORTCUTSBENCH into difficulty levels and 8 task types based on [aseq: and "shortcut type" (Section 3.1 respectively.
In calculating the length of shortcut commands, we do not simply count the number actions within the shorteut. Instead, apply specialized approach. Initially, certain actions that do not con- tribute meaningful operations, such as is. work low. actions comment and is workflow act ions. alert, which are akin comments in programming, excluded. Furthermore, we disregard the length of certain control flow statements, including is workflow actions cenditional, is workflow, actions. whwosefronmer repeat. count, is workflow actions repeat sepeat.each. For branching statements, we consider the length of the longest branch, rather than the cumulative length of all branches.
is workflow actions.
When categorizing shortcuts, we first analyzed all available categories from the Category InStore field in the collected data. We then classified the shortcuts into 8 categories, referencing with the classification of apps on the Apple App Store (app). The categories are as follows:
1. Productivity & Utilities
2. Health & Fitness
3. Entertainment & Media
4. Lifestyle Social
5. Education & Reference
6. Business Finance
7. Development & API
8. Home & Smart Devices
Subsequently, I employed a language model to categorize all shortcuts using the prompt shown in Figure 10
A.4 PERFORMANCE ABOUT API SELECTION
Following existing work (Huang et al. 2024b Patil et al. 2024 Xu et al. 2024). we use the accuracy of API selection as the metric. The accuracy calculated as the number of correct API selections my divided by ng- Specifically, each time we predict an action by. S I Jaseqi we provide the agent with all the correct historical actions (a1, 02 -...- 0-- }. We then require the agent to predict the next action. All actions predicted by the agent form the prediction sequence bseqa.i. This method similar to the next token prediction (NTP) in LLMs, effectively preventing cascade of errors in subsequent action predictions due to single incorrect prediction. During the prediction, when encountering special actions such as branching and looping, we skip predicting these actions and directly add them to the historical actions.
Specifically, when calculating the precision of API selection, we do not consider the contributions of control statements such branches and loops. This avoids the unreasonable requirement for the agent to invoke "branch APIs" or "loop APIs" in the next action. The agent should inherently possess the ability to correctly understand and act according to the conditions dictated by branches and loops. addition to excluding the contributions of these control statements, we also disregard contributions from is work flow act ions. comment and is work fflow act ions. a alert, effectively removing these non-operative commands from the history of actions provided to the agent.
A.5 EFFECTIVENESS OF API PARAMETER VALUE FILLING
To further ensure that the corresponding parameters are indeed included in the queries during evaluation, we used the LLM to filter these parameters further, ensuring their presence in the queries. Detailed prompts be found in Figure
20

---

Published as a conference paper at ICLR 2025
SYSTEM PROMPT TEMPLATE:
Shortcut consist of a sequence of actions, each is an API call, to execute user-provided queries.
As friendly and patient assistant, you need categorize the provided shortcut into of the following cight categories:
Productivity & Utilities
Health & Fitness
3. Entertainment & Media
4. Lifestyle & Social
5. Education & Reference
6. Business & Finance
7. Development API
8. Home & Smart Devices
For each shorteut command, will provide you with five fields:
1. 'RecordName:: The name of the shorteut, briefly describing its function.
2. DDeccription of the Shorteut Workflow': A description of the entire action workflow of the shortcut.
'Comments': Optional. Notes from the shortcut's developer, which may describe its func- tions or other features.
4. 'Description in Store': A description of the shortcut's functionality provided in the shortcut store.
5. 'API Description List': Detailed descriptions of the APIs involved in the shorteut.
You should rely primarily the 'Description of the Shortcut Workflow' and API Description List', and refer to 'RecordName', 'Comments", and 'Description in Store' to give the final category.
USER PROMPT TEMPLATE:
Below the five fields I provide to you:
1. 'RecordName': (RecordName)
2. DDeccription of the Shortcut Workflow': (DeecipiooothhsStoorcuoWWofffoot
3. 'Comments': [Comments]
4. 'Description in Store': (DescriptionlnSStrre)
5. 'API Description List': [APIlescriptionList)
Please give the category on these details. Alongside the category, provide the shortcut's name and description of its functionality in English using the following JSON format:
"category:: "category",
benglish_namer "ThisIsShortcutaamee
"Thistsfunctionalitty""
Do not output any other content; your response should only be in this JSON format.
Output the JSON directly without using "'json XX** enclose it. Please give your answer in English.
Figure 10: System and user prompt templates for categorizing shortcuts based on their functionalities
21

---

Published as a conference paper at ICLR 2025
SYSTEM PROMPT TEMPLATE:
Your task is to classify the parameters provide based on user queries, API information, and API calls (also known actions).
User query describes the task the user wants accomplish.
Information about the API definition includes the API name, parameter names, parameter types, default values, return value names, and return value types. Parameters are identified by 'Parameters' and explained. The return value names and return value types are identified by 'Return Values', The API's brief and detailed descriptions are marked by "Description' The natural language description of the API is marked by ParameterSummary.
Completing the user query requires series API calls, each API call needs the correct and appropriate parameters. We have pre-selected possible parameters that may appear in the query.
Please note, you must classify these pre-selected parameters based on the user query. Each parameter can generally be classified into the following categories:
Precise parameter: Parameters stated by users in the query, or those implicitly indicated in the query but can be accurately inferred by combining the query and the API definition.
2. Not precise parameter: Parameters not stated by users in the query and cannot be accurately inferred even with the combination of the query and the API definition.
Note! Note! Note! all precise parameters must be clearly implicitly specified in the query.
USER PROMPT TEMPLATE:
The user query is: {query)
Information about the API definition is provided below: (api_desc)
The API call (API_call] The pre-selected possible parameters that may appear in the query are listed below: {possible_paras)
Output the classification in the following format:
para_nama para_namel:
: para_typel,
"reason1": The reason
}
para_name2:
para
name2: para_name2: para type2, "reason2"; The reason
}
Do not output any additional content; only output JSON, Do not enclose your output with "'json XXX"".
Note! Note! Note! all precise parameters must be clearly implicitly specified in the query.
Figure 11: System and user prompt templates for classifying parameters based on user queries and API definitions
A.6 RECOGNITION OF NEED FOR INPUT
In the shortcut, a parameter can be set Extension indicating that the parameter requires a file provided by the user, or CurrentDate, indicating that the parameter needs to retrieve the date from the system. Similarly, Clipboard indicates that the parameter should obtain content from the clipboard, and DeviceDetail implies that the parameter needs to access certain information about the user's device. Lastly, Ask denotes that the parameter requires user authorization
22

---

Published as a conference paper at ICLR 2025
essential input from the user. A typical example is shown in Figure 12 where the action the is workfll workflow act ions getmyworkfllooo API. The Folder parameter is set to Ask, indicating that this parameter requires imput provided by the user.
"is. www.wookflow..ctc.omo wtw.ee/rr....cccooooooeeeeooorrr getmyworkflows"
"Folder":
"Value": "Type": "Ask"
,, "WEserializationTyee"" "IFTexttToeenttaacheenn
"OUID": "E5F695A5-9DD3) 000-9-5----00--7-77------------0
Figure 12: An example of Ask parameter.
A.7 SETUP
Following existing work (Huang et al. 2024b Qin et al. 2024 Li et al. 2023), we slightly modified the ReACT (Yao et al., 2023) templates construct the API-based agents. The templates used in our experiments are as shown Figure
A.8 RESULT ANALYSTS
Table 6: Pricing, Testing Instances, and Actual Costs of Popular AI Models. (07-22-24). Except for gemini-1. 5-pro, which randomly tested 800 instances due to cost considerations, all other LLMs were tested across all datasets. However, the number of successful tests varied slightly due factors such context length, safety reviews, and etc. The cost of testing primarily stems from inputs, as we continuously feed historical actions into the LLM for evaluation, and all historical conversations are billed repeatedly (OpenAI Community 2023).
Model Name
Price 1M tokens Instances Estmmat Cost ($)
gemini-1.5-pro
$3.50 / $10.50
801
592
gemini-1.5-flas.
$0.35 $1.05
5295
391
qwen22-7b-insttzue
$0.70 / $1.40
5216
800
gve:2--77b-al4--instrce
$0.49 / $0.98
5368
580
GPT-4o-mini
$0.15 / $0.60
5320
100
$0.50 $1.50
5463
500
deepseek-chat
$0.14 / $0.28
5319
90
deepseek-coder
$0.14 / $0.28
5317
90
GLM-4-Air
$0.14 $0.14
5330
110
Total Cost
3253
Among them, gemini-1.5-pp (tested with 801 instances) and gemini-1.5-fles (tested with 5,295 instances) incurred a total cost of $801, with gemini-1.5-flem
accounting for approximately $391 and comini-1.5-pre approximately $592. The costs for qwwn2-72b-inss - (tested with 5,216 instances) were about $800, qwen2- 1ann2-57b-aan 4b-iisttuutt (lested with 5,368 instances) around $580, and GPT-4o-ming (tested with 5,320 instances) approximately S50. got-3. 5-turbo (tested with 5,463 instances) cost approximately $500. The combined expenses for deepseek-chat (tested with 5,319 instances) and deepseek-codes (tested with 5,317 instances) were roughly $180, while CTM-4-Ais cost about $110.
23

---

Published as a conference paper at ICLR 2025
SYSTEM PROMPT TEMPLATE:
You are AutoGPT. Your task is to complete the user's query using all available APIs.
First, the user provides the query, and your task begins.
At each step, you need to provide your thought process to analyze the current status and determine the next action, with an API call to exccute the step. After the call, you will receive the result, and you will be in new state. Then, you will analyze your current status, decide the next step, and continue... After multiple (Thought- Calll pairs, you will eventually complete the task.
Below all the available APIS, including the API name, parameter names, parameter types, default values, return value names, and return value types. (all_apl ]
For each step, only API. Strictly follow the JSON format below for your output and do include any irrelevant characters.
"Thought"" : "Your analysis of what to do next", "WFWork frrrkkllowe www.kfowowwiiioooo wew.wctionndenteeo
"The API name you call",
"parameter name": "parameter value"
W/Wokkfow.AciioPPrraneec are parameters required for the API call. The parameter value might be:
1. basic data types like string, integer, float, or boolean.
2. output from previous API call.
3. input from the system or the user, including file provided by the user.
4. Previously defined variable names.
5. If the parameter is of type string, you also combine the output of previous action, input from the system or the user, with a string.
6. the output of a previous action is an Object type, or if you need to use input from the system or the user, you utilize specific properties from the previous action's 's output.
USER PROMPT TEMPLATE:
The user query is: (query) The history actions and observations are follows: {history_actions)
Please continue with the next actions based on the previous history. Do not output any other content; your response should only in this JSON format. You should only output action time.
Figure 13: System and user prompt templates for executing API calls based on user queries
The analysis indicates notable range in efficiency and value for money. Models like deepseek-cie chat and deepseek seepseek-coodrr show excellent cost-effectiveeness particularly suit- able for high-volume, low-cost deployments. In contrast, models like gemini-1 5-pro and gemini-1. 111.--flash reflect higher costs, but they offer superior performance.
A.9 ADDITIONAL RESULTS ON EVALUATION ON SPECIALIZED API-CALLING FINE-TUNED LLMS
To comprehensively evaluate the effectiveness of fine-tuning LLMs for agent- based scenarios, extended our comparison to include specialized AI agent models such as AgentLM (Zeng et al. 2024) and xLAM (Zhang al, 2024). alongside general-purpose LLMS like Qwen-2-728 Qwen
24

---

Published as a conference paper at ICLR 2025
Table 7: Performance Comparison Fine-Tuned and Base LLMs
LLaMA-2-7B
LLaMA-2-13
LLaMA-2-70B
Model
AgentLM-7B
AgentLM--13B
AgentLM-700
Base
Fine-Tunee
Base
Fine-Tuned
Base
Fine-Tuned
Context Length
4k
4k
4k
4k
4k
4k
32.75
24.95
38.59
30.57
52.13
43,41
API Selection
12.57
17.43
25.99
35.48
43.08
37.34
23.70
32.78
35.45
40.78
57.76
39.22
51.11
34.88
30.77
31.82
72.86
52.83
API Parameter
32.14
35.71
12.50
40.98
45.05
37.31
Value Filling
0.00
0.00
11.11
40.00
6.67
11.11
7.41
10.53
5.00
5.26
21.74
0.00
Recognition of
12.50
4.17
0.00
3.57
7.69
9.09
Need for Input
0.00
0.00
0.00
0.00
0.00
0.00
Mistral-7B
Mixtral-8x7b
LLaMA-2-70B
Model
XLAM-7b-r
-LAM-8x7b-r
Hamur-70B-Chat-
Base
Fine- Tunee
Base
Fine-Tuned
Base
Fine-Tuned
Context Length
32k
32k
32k
32k
4k
4k
17.03
70.56
70.21
85.29
52.13
52.45
API Selection
3.27
50.82
49.15
51.72
43.08
47.08
0.50
66.33
29.92
29.67
57.76
60.39
40.00
66.67
53.45
68.93
72.86
45.45
API Parameter
40.54
59.32
33.72
46.76
45.05
36.49
Value Filling
11.11
31.82
27.20
42.71
6.67
26.92
0.00
5.56
6.25
11.90
21.74
16.00
Recognition of
0.00
2.56
19.27
19.91
7.69
3.57
Need for Input
0.00
0.00
0.00
0.00
0.00
0.00
(2024) and LLL.M-3-- 3- Meta 2024). This was aimed at understanding the applicability of fine-tuning strategies for APl-based agents.
We conducted experiments with additional 11 LLMs, comprising 6 models fine-tuned specifically for agent scenarios and base models. The fine-tuned models include the Agent LM series (AgentLM)
7B. AgentLM-13B AgentL.M-0BB), the xLAM series (xLAM-7b-r XLAM-8x7b-)), and Lemur (Lemur 70B-Chat- V1). The base models include various configurations of LLaMA-2 and Mistral from Hugging Face repositories.
The fine-tuning processes of these LLMs varied: the Agent series was fine-tuned on 1866 traces of simple code data, Lemur utilized scripting and interpreted languages from The Stack dataset, and the xLAM series was specialized for API-based agents with a predefined format for API calls. To ensure standardized evaluation, outputs were processed using gpt 40 do--innn, with manual verification confirming the accuracy of conversions. Due to context length limitations (4k tokens for most models and 32k tokens for xLAM series), results are reported for the first difficulty levels.
Experimental results are summarized in Table 7
A.9.1 RECOGNITION OF NEED FOR INPUT
All evaluated LLMs performed exceptionally poorly in the recognition need for input, significantly lagging behind the general LLMs mentioned in the ShorteutsBench. Compared the performance of the base LLMs before fine-tun aeeetunnng, neither the AgentLM and Lemur series models fine-tuned on code nor the xLAMM series models fine-tuned specifically for API-based agents showed any improvement
25

---

Published as a conference paper at ICLR 2025
in this aspect. This suggests the need for targeted fine-tuning specifically aimed at enhancing this capability.
A.9.2 API PARAMETER VALUE FILLING
The evaluated LLMS perform significantly worse in API Parameter Value Filling compared to the general LLMs mentioned in ShortcutsBench. The general LLMs achieve accuracies of 94.29%, 83.92%, and 75.53% across three difficulty levels, whereas the best results from the evaluated models only reach accuracies of 72.86%, 59.32%, and 42.71%.
Compared to the base LLMs before fine-tuning, the code-based agents, including the AgentLM and Lemur models fine-tuned code, did not achieve general improvement in purameter-filling accuracy for API-based tasks and even exhibited performance degradation. In contrast, the API-based agents, such the xL.AM series fine-tuned specifically for API-based agents, showed consistent improvement in parameter-filiing accuracy. The existing fine-tuning methods for API-hased agents (xLAM-7b-r xLA.M-8x are effective for parameter filling in short sequences, but accuracy drops sharply as the context length increases.
A.9.3 API SELECTION
Methods fine-tuned specifically for API-based agents, such as the xLAM series models, can signifi- cantly enhance agents' API selection accuracy. The xLAM series achieves reasonable accuracy in API selection, even compared to general L.Mss. However, compared to XLAM-7b-r, the API selec- tion accuracy of xI-AM-8x7b-r is significantly lower, suggesting that existing fine-tuning methods specialized for agents may only achieve notable improvements on models with relatively weaker foundational capabilities.
Additionally, methods fine-tuned on code can still enhance agents' API selection accuracy, as demonstrated by the AgentLM-77/1133 and Lemur-70b-chat-v models. The performance drop of AgentLM-70b may be attributed to the fine-tuning dataset used for the AgentLM series lacking realism and complexity.
26


Reviews:
[{"review_id": "QM0LMeN0DB", "reviewer": "Reviewer_XY2i", "summary": "This paper introduces a comprehensive, large-scale benchmark designed to evaluate API-based agents in real-world scenarios. It addresses the shortcomings of existing benchmarks and datasets, which often lack the richness and complexity needed to thoroughly assess different LLM-based agent models. To overcome these limitations, the authors have developed a high-quality benchmark encompassing 88 applications, 1,400 APIs, and an average of 21 actions per API, each covering various aspects of real-world deployments.\n\nIn the evaluation, the authors tested 10 LLMs on several key tasks: API selection, parameter filling, and the models' ability to recognize when to request additional input from either the system or the user. The experimental results show interesting insights, such as the challenges these models face with multi-step reasoning and understanding when external input is necessary.", "questions": "Please refer comments in above fields.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper makes a notable contribution by creating a comprehensive benchmark for API-based agents, utilizing data extracted from Shortcuts. Compared to other API-based benchmarks, it offers several benefits, including a focus on the agents' ability to request necessary input from either the assistant or user and diverse difficulty of tasks. It covers a range of tasks, from simple ones to those involving complex APIs, queries, and action sequences. Additionally, the paper ensures quality by involving human verification, with shortcut developers serving as annotators.\n\nThe evaluations demonstrate insightful findings, especially regarding the challenges agent models face in reasoning and planning capabilities as indicated by API selection, as well as the difficulties weaker LLMs encounter in API parameter filling. These insights are valuable for the development of more advanced agent models.", "weaknesses": "1. Although the paper emphasizes that the benchmark includes high-quality human-annotated action sequences from shortcut developers and queries derived from real user demands, it only mentions the shortcut developers are our annotators. Further details in this area would be beneficial.\n\n2. In section 3.2, the paper describes using GPT-4o to simulate user queries. However, it would be helpful to include the steps taken to verify the correctness and ensure the diversity of these user queries.\n\n3. The evaluation primarily features selected proprietary models like GPT-4o and Gemini. For open-source LLMs, it mainly compares with general LLMs such as Qwen-2-70B and LLaMA-3-70B. Considering comparisons with more robust, specifically developed AI Agent models such as AgentLM (70B from https://github.com/THUDM/AgentTuning) and xLAM (8x7b or 8x22b from https://github.com/SalesforceAIResearch/xLAM) could provide more insights.\n\n4. The source of the reported numbers in Table 1 could be more clearly specified. There is uncertainty regarding whether models like Qwen-2.5-7B and LLaMA-3-8B consistently achieve more than 90% on ToolBench and over 80% on ToolLLM. Furthermore, given that MetaTool (https://github.com/HowieHwong/MetaTool) does not provide comprehensive details or code for model evaluation and metrics, more information is necessary to verify the accuracy of the table data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mgM2tcB2I9", "reviewer": "Reviewer_AydN", "summary": "This paper introduces a new benchmark ShortcutsBench, which evaluates agents' capabilities in solving tasks through API calling. They compare multiple API-based benchmarks, and showcase why ShortcutsBench contributes to the assessing agents' API calling abilities. The paper evaluates several API-based agents constructed based on the ReACT framework. The authors perform detailed analysis on the evaluation results.", "questions": "- Why is all API-based agents the authors evaluate based on the ReACT framework, it would be good if they could provide additional evaluation/analysis on other frameworks, such as CodeAct.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The authors introduce ShortcutsBench, which is a more holistic benchmark that contains real APIs, well-designed queries and actions. This could contribute to better evaluation of current agents's API calling capabilities in solving real-world tasks.\n- The authors provide example instances from ShortcutsBench in the appendix, which helps understanding the types of tasks in this benchmark.\n- The authors provide detailed analysis based on the evaluation results of several API-based agents.", "weaknesses": "- Section 3 is not well-elaborated. readers will benefit from clearer description of this process. For example, for (2), the authors say 'after duplicating based on icloud link, ....', it is not very clear what is duplicated and why this step helps. It would be good if the authors could refine their descriptions on their methodology.\n- The authors cite each work too many times in the paper, for example, a research paper is is cited five times in one paragraph in Section 2. Referencing previous works is good practice, but referencing too many times affects readability. It would be good if they could remove repetitive references.\n- The paper does not make it very clear what is their most important finding in the abstract/intro/conclusion. It would be good if they could highlight their most important findings in the abstract/intro/conclusion. For example, they could discuss how open source models perform comparably to closed source models on simpler tasks but not harder tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R8eKxIxXed", "reviewer": "Reviewer_2zSS", "summary": "This paper introduces ShortcutsBench, which mines real-world APIs/action sequences from iOS shortcut app. They then synthesize tasks with language model, and test a range of model's ability on it. Results show the task remains challenging to today's LMs.", "questions": "1. A notable feature of this benchmark seems to be scalability -- extracting real-world APIs and action sequences from Shortcut apps and sharing sites seems relatively easy. Providing quantitative details on dataset construction—such as time taken for each step and the actual extent of manual effort involved—would be nice.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Interesting approach: mining APIs/existing action sequence from Shortcut Apps makes a lot of sense, which is a resource previous works haven't tapped into.\n- Comprehensive evaluation: the authors evaluated a wide range of LMs across Open and close source models.", "weaknesses": "1. Limited evaluation: The paper primarily assesses the model’s ability to choose correct actions based on ground-truth sequences but doesn’t evaluate its end-to-end task success rate (as done in, for example, AppWorld [1]). Experiments linking these aspects are missing.\n2. No human validation: Given the synthetic nature of the benchmark, it’s uncertain whether all tasks are truly solvable or what the benchmark’s upper bound is. Including human performance as a reference would add clarity.\n\n[1] AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FLY0mi1HJt", "reviewer": "Reviewer_vqjw", "summary": "The paper introduces ShortcutsBench, a benchmark to assess LLMs’ ability to call external APIs and create structured text (ex: JSON). The dataset was collected from the Shortcuts tool available on Apple devices. The authors perform the analysis of performance of a number of open and closed-source LLMs against ShortcutsBench.", "questions": "I suggest to submit to a more relevant conference, rather than to ICLR, as mentioned above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "[1] Analysis results in 4.2 are insightful and confirm the ranking of closed- and open-source LLMs on traditional general knowledge and reasoning benchmarks like MMLU.\n\n[2] The artifacts are provided and well-organized.", "weaknesses": "[1] It is a technical report that lacks a scientific component. It should go to a software engineering conference, for example, International Conference on Software Engineering (ICSE) or IEEE/ACM International Conference on Automated Software Engineering (ASE). The paper can also go to Datasets and Benchmarks tutorials at AI conferences. Whereas the work is technically impressive and valuable for the software engineering community, there is no scientific value in the assessment of how good LLMs are able to use APIs. The problem at hand is purely technical and as a clear indicator for this is that over two thirds of references are links to web pages rather than scientific studies.\n\n[2] The analysis of the performance of GPT 4o with the structured output capability was not performed.\nhttps://openai.com/index/introducing-structured-outputs-in-the-api/\n\n[3] The used language appeals to emotions rather than states the scientific value (ex: “We made great efforts to evaluate…”).\n\n[4] Several references are outdated, pointing to arxiv whereas the paper has been already published at a conference (ex: ToolLLM).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)