ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents

Authors: Haiyang SHEN, Yue Li, Desong Meng, Dongqi Cai, Sheng Qi, Li Zhang, Mengwei Xu, Yun Ma
Primary area: datasets and benchmarks
Keywords: Benchmark, Agent, LLM, Shortcuts

Abstract:
Recent advancements in integrating large language models (LLMs) with application programming interfaces (APIs) have gained significant interest in both academia and industry. Recent work demonstrates that these API-based agents exhibit relatively strong autonomy and planning capabilities. However, their ability to handle multi-dimensional difficulty levels, diverse task types, and real-world demands remains unknown. 
In this paper, we introduce \textsc{ShortcutsBench}, a large-scale benchmark for the comprehensive evaluation of API-based agents in solving real-world complex tasks. \textsc{ShortcutsBench} includes a wealth of real APIs from Apple Inc., refined user queries, human-annotated 
high-quality action sequences, detailed parameter filling values, and parameters requesting necessary input from the system or user. We revealed how existing benchmarks~/~datasets struggle to accommodate the advanced reasoning capabilities of existing more intelligent LLMs. Moreover, our extensive evaluation of agents built with $5$ leading open-source (size $\geq$ 57B) and $5$ closed-source LLMs (e.g. Gemini-1.5-Pro and GPT-4o-mini) with varying intelligence level reveals significant limitations of existing API-based agents in the whole process of handling complex queries related to API selection, parameter filling, and requesting necessary input from the system and the user. These findings highlight the great challenges that API-based agents face in effectively fulfilling real and complex user queries. All datasets, code, experimental logs, and results are available at https://github.com/EachSheep/ShortcutsBench

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730283905368

thread_type:
official_review

<thread_object>
creator: reviewer_vqjw

<content>
summary:
The paper introduces ShortcutsBench, a benchmark to assess LLMs’ ability to call external APIs and create structured text (ex: JSON). The dataset was collected from the Shortcuts tool available on Apple devices. The authors perform the analysis of performance of a number of open and closed-source LLMs against ShortcutsBench.

strengths:
[1] Analysis results in 4.2 are insightful and confirm the ranking of closed- and open-source LLMs on traditional general knowledge and reasoning benchmarks like MMLU.

[2] The artifacts are provided and well-organized.

weaknesses:
[1] It is a technical report that lacks a scientific component. It should go to a software engineering conference, for example, International Conference on Software Engineering (ICSE) or IEEE/ACM International Conference on Automated Software Engineering (ASE). The paper can also go to Datasets and Benchmarks tutorials at AI conferences. Whereas the work is technically impressive and valuable for the software engineering community, there is no scientific value in the assessment of how good LLMs are able to use APIs. The problem at hand is purely technical and as a clear indicator for this is that over two thirds of references are links to web pages rather than scientific studies.

[2] The analysis of the performance of GPT 4o with the structured output capability was not performed.
https://openai.com/index/introducing-structured-outputs-in-the-api/

[3] The used language appeals to emotions rather than states the scientific value (ex: “We made great efforts to evaluate…”).

[4] Several references are outdated, pointing to arxiv whereas the paper has been already published at a conference (ex: ToolLLM).

questions:
I suggest to submit to a more relevant conference, rather than to ICLR, as mentioned above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730311061126

thread_type:
official_review

<thread_object>
creator: reviewer_2zss

<content>
summary:
This paper introduces ShortcutsBench, which mines real-world APIs/action sequences from iOS shortcut app. They then synthesize tasks with language model, and test a range of model's ability on it. Results show the task remains challenging to today's LMs.

strengths:
- Interesting approach: mining APIs/existing action sequence from Shortcut Apps makes a lot of sense, which is a resource previous works haven't tapped into.
- Comprehensive evaluation: the authors evaluated a wide range of LMs across Open and close source models.

weaknesses:
1. Limited evaluation: The paper primarily assesses the model’s ability to choose correct actions based on ground-truth sequences but doesn’t evaluate its end-to-end task success rate (as done in, for example, AppWorld [1]). Experiments linking these aspects are missing.
2. No human validation: Given the synthetic nature of the benchmark, it’s uncertain whether all tasks are truly solvable or what the benchmark’s upper bound is. Including human performance as a reference would add clarity.

[1] AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents

questions:
1. A notable feature of this benchmark seems to be scalability -- extracting real-world APIs and action sequences from Shortcut apps and sharing sites seems relatively easy. Providing quantitative details on dataset construction—such as time taken for each step and the actual extent of manual effort involved—would be nice.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730583540470

thread_type:
official_review

<thread_object>
creator: reviewer_aydn

<content>
summary:
This paper introduces a new benchmark ShortcutsBench, which evaluates agents' capabilities in solving tasks through API calling. They compare multiple API-based benchmarks, and showcase why ShortcutsBench contributes to the assessing agents' API calling abilities. The paper evaluates several API-based agents constructed based on the ReACT framework. The authors perform detailed analysis on the evaluation results.

strengths:
- The authors introduce ShortcutsBench, which is a more holistic benchmark that contains real APIs, well-designed queries and actions. This could contribute to better evaluation of current agents's API calling capabilities in solving real-world tasks.
- The authors provide example instances from ShortcutsBench in the appendix, which helps understanding the types of tasks in this benchmark.
- The authors provide detailed analysis based on the evaluation results of several API-based agents.

weaknesses:
- Section 3 is not well-elaborated. readers will benefit from clearer description of this process. For example, for (2), the authors say 'after duplicating based on icloud link, ....', it is not very clear what is duplicated and why this step helps. It would be good if the authors could refine their descriptions on their methodology.
- The authors cite each work too many times in the paper, for example, a research paper is is cited five times in one paragraph in Section 2. Referencing previous works is good practice, but referencing too many times affects readability. It would be good if they could remove repetitive references.
- The paper does not make it very clear what is their most important finding in the abstract/intro/conclusion. It would be good if they could highlight their most important findings in the abstract/intro/conclusion. For example, they could discuss how open source models perform comparably to closed source models on simpler tasks but not harder tasks.

questions:
- Why is all API-based agents the authors evaluate based on the ReACT framework, it would be good if they could provide additional evaluation/analysis on other frameworks, such as CodeAct.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731055574780

thread_type:
official_review

<thread_object>
creator: reviewer_xy2i

<content>
summary:
This paper introduces a comprehensive, large-scale benchmark designed to evaluate API-based agents in real-world scenarios. It addresses the shortcomings of existing benchmarks and datasets, which often lack the richness and complexity needed to thoroughly assess different LLM-based agent models. To overcome these limitations, the authors have developed a high-quality benchmark encompassing 88 applications, 1,400 APIs, and an average of 21 actions per API, each covering various aspects of real-world deployments.

In the evaluation, the authors tested 10 LLMs on several key tasks: API selection, parameter filling, and the models' ability to recognize when to request additional input from either the system or the user. The experimental results show interesting insights, such as the challenges these models face with multi-step reasoning and understanding when external input is necessary.

strengths:
This paper makes a notable contribution by creating a comprehensive benchmark for API-based agents, utilizing data extracted from Shortcuts. Compared to other API-based benchmarks, it offers several benefits, including a focus on the agents' ability to request necessary input from either the assistant or user and diverse difficulty of tasks. It covers a range of tasks, from simple ones to those involving complex APIs, queries, and action sequences. Additionally, the paper ensures quality by involving human verification, with shortcut developers serving as annotators.

The evaluations demonstrate insightful findings, especially regarding the challenges agent models face in reasoning and planning capabilities as indicated by API selection, as well as the difficulties weaker LLMs encounter in API parameter filling. These insights are valuable for the development of more advanced agent models.

weaknesses:
1. Although the paper emphasizes that the benchmark includes high-quality human-annotated action sequences from shortcut developers and queries derived from real user demands, it only mentions the shortcut developers are our annotators. Further details in this area would be beneficial.

2. In section 3.2, the paper describes using GPT-4o to simulate user queries. However, it would be helpful to include the steps taken to verify the correctness and ensure the diversity of these user queries.

3. The evaluation primarily features selected proprietary models like GPT-4o and Gemini. For open-source LLMs, it mainly compares with general LLMs such as Qwen-2-70B and LLaMA-3-70B. Considering comparisons with more robust, specifically developed AI Agent models such as AgentLM (70B from https://github.com/THUDM/AgentTuning) and xLAM (8x7b or 8x22b from https://github.com/SalesforceAIResearch/xLAM) could provide more insights.

4. The source of the reported numbers in Table 1 could be more clearly specified. There is uncertainty regarding whether models like Qwen-2.5-7B and LLaMA-3-8B consistently achieve more than 90% on ToolBench and over 80% on ToolLLM. Furthermore, given that MetaTool (https://github.com/HowieHwong/MetaTool) does not provide comprehensive details or code for model evaluation and metrics, more information is necessary to verify the accuracy of the table data.

questions:
Please refer comments in above fields.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731566511423

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank all reviewers for their comments and feedback. 

We summarized the strengths highlighted and categorized the concerns as follows:

The reviewer has affirmed the benchmark, method, evaluation, findings, and artifacts of our paper.
1. **benchmark**: notable contribution by creating a comprehensive benchmark (XY2i);  more holistic benchmark that contains real APIs, well-designed queries and actions (AydN);
2. **method**: interesting approach (2zSS); this benchmark seems to be scalability (2zSS);
3. **evaluation**: detailed analysis (AydN); comprehensive evaluation (2zSS);
4. **findings**: insightful findings (XY2i); analysis results in 4.2 are insightful (vqjw);
5. **open-source**: the artifacts are provided and well-organized (vqjw);

The reviewer expressed concerns about the dataset construction process, raised questions regarding the methodological details, and suggested that the authors conduct additional experiments in certain areas:

1. **concerns about the dataset construction process**: Further details about action sequences would be beneficial (XY2i); including steps taken to verify the correctness and ensure the diversity of user queries would be helpful (XY2i); including human performance as a reference would add clarity (2zSS).
2. **questions regarding the methodological details**: More information is necessary to verify the accuracy of the Table 1 (XY2i); Providing quantitative details on dataset construction—such as time taken for each step and the actual extent of manual effort involved—would be nice (2zSS); why the authors evaluate based on the ReACT framework, and it would be good if they could provide additional evaluation/analysis on other frameworks, such as CodeAct (AydN);
3. **suggestions to conduct additional experiments**: Consider comparisons with more specifically models like AgentLM and xLAM could provide more insights (XY2i); analyse the performance of GPT-4o with the structured outputs. (vqjw).

There are also some suggestions regarding certain phrasing and citations. For more details, we kindly refer you to the specific responses to each reviewer.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper "ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents" introduces a novel benchmark dataset, ShortcutsBench, designed to evaluate the capabilities of API-based agents in handling real-world complex tasks. The benchmark leverages real APIs from Apple's iOS Shortcuts app, coupled with refined user queries, human-annotated action sequences, and detailed parameter filling values. The authors evaluate a diverse set of large language models (LLMs), including both open-source (e.g., Qwen-2-70B, LLaMA-3-70B) and closed-source (e.g., Gemini-1.5-Pro, GPT-4o-mini) models, on tasks involving API selection, parameter filling, and recognizing the need for additional input. The findings reveal significant limitations in current API-based agents, particularly in multi-step reasoning and handling complex queries. The paper emphasizes the challenges these agents face in real-world scenarios and provides all datasets, code, and experimental logs openly for reproducibility.

The primary contributions of the paper are:
1. **ShortcutsBench Dataset**: A comprehensive, large-scale benchmark for evaluating API-based agents in real-world scenarios.
2. **Extensive LLM Evaluation**: A detailed analysis of 21 LLMs, including both general and agent-specific models, highlighting their capabilities and limitations.
3. **Insights into Agent Limitations**: Valuable findings on the challenges API-based agents face in complex tasks, particularly in parameter filling and recognizing the need for additional input.

Weaknesses

1. **Dataset Construction Details**: Some reviewers noted a lack of detailed description regarding the dataset construction process, particularly the steps taken to ensure query correctness and diversity. While the authors provided additional explanations and open-sourced the process logs, initial clarity was lacking.

2. **Limited Scope of Evaluation Metrics**: The paper primarily focuses on API selection and parameter filling, with limited emphasis on end-to-end task success rates. This was acknowledged by the authors, who noted plans for future work involving dynamic runtime environments.

3. **Presentation Issues**: Some reviewers criticized the initial presentation, citing repetitive citations and unclear phrasing. The authors addressed these concerns in the revised version, but initial readability issues may have impacted the overall reception.

4. **Human Validation Absence**: The absence of human performance baselines was noted as a limitation, as it leaves uncertainty about the benchmark's upper bound and the solvability of tasks. While the authors explained the practical challenges, this remains a valid concern for validating the benchmark's design.