StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models (2024)

Abstract: Large Language Models (LLMs) have witnessed remarkable advancements in recent years, prompting the exploration of tool learning, which integrates LLMs with external tools to address diverse real-world challenges. Assessing the capability of LLMs to utilise tools necessitates large-scale and stable benchmarks. However, previous works relied on either handcrafted online tools with limited scale, or largescale real online APIs suffering from instability of API status. To address this problem, we introduce StableToolBench, a benchmark evolving from ToolBench, proposing a virtual API server and stable evaluation system. The virtual API server contains a caching system and API simulators which are complementary to alleviate the change in API status. Meanwhile, the stable evaluation system designs solvable pass and win rates using GPT-4 as the automatic evaluator to eliminate the randomness during evaluation. Experimental results demonstrate the stability of StableToolBench, and further discuss the effectiveness of API simulators, the caching system, and the evaluator system.

heading: 1 Introduction
text: With the rapid developments of Large Language Models (LLMs; Brown et al., 2020; Gemini Team, 2023; OpenAI, 2023; Touvron et al., 2023), tool learning which leverage LLMs to schedule a variety of external tools has attracted enormous attention (Nakano et al., 2022; Yao et al., 2022b; Lu et al., 2023). Previous studies (Hao et al., 2023; Hsieh et al., 2023; Schick et al., 2023; Tang et al., 2023) aim to augment LLMs with tools to enhance performance on conventional natural language processing (NLP) downstream tasks, while recent work (Qin et al., 2023a; Yao et al., 2022a; Cai et al., 2024) primarily focus on solving realworld scenarios that require the use of tools. In
Project: zhichengg.github.io/stb.github.io/ GitHub: THUNLP-MT/StableToolBench
general, tool learning complements the capabilities of vanilla LLMs and bridges the gap to real-world applications.
To assess the capability of LLMs to use tools, a series of tool learning benchmarks have been introduced. Several pioneering studies have heavily relied on human-crafted offline tools (Yang et al., 2023b; Xu et al., 2023) or hand-selected online tools (Li et al., 2023b,a; Chen et al., 2023b). While these tools are high-quality, their scale remains relatively small, thereby limiting their ability to accurately reflect real-world scenarios. To address this limitation, subsequent studies (Tang et al., 2023; Ye et al., 2024; Qin et al., 2023b) have advocated for leveraging extensive collections of online tools that span across various domains. Owing to the increased scale, the automatic evaluation of tool learning has moved closer to real-world scenarios. However, concerns have been raised regarding the stability of these online tools, which has implications for the reproducibility and comparability of benchmark performance over time1. For instance, the well-recognised ToolBench2 (Qin et al., 2023c) has shown performance discrepancies that cannot be reproduced months after its release, as analysed in Section 2.1. This is even more important when faced with a complex environment, where APIs and tools keep changing while the evaluation should maintain its consistency across time.
Existing large-scale benchmarks may struggle to provide stable evaluations for various reasons. We propose several hypotheses for this issue. Firstly, the complexity of tasks involving tool usage makes it challenging for the common automatic evaluator, gpt-3.5, to function effectively as a discriminator. As discussed in Section 2.2, the evaluator cannot reliably determine whether a task is solvable or
1According to its definition, benchmarks should remain stable, and the model performance assessed on them must be comparable over time.
2We use ToolEval2 in ToolBench as the benchmark.
1 11143
unsolvable, leading to variability in model performance due to this capability limitation. Secondly, the stability of API status for a significant portion of online tools (55.6% in ToolBench) is inconsistent. Users may be required to authorise the use of these tools or APIs, and tools provided by developers may be accessible during the initial construction of the benchmark but become unavailable later. This fluctuation further undermines the reliability and reproducibility of model performance assessments over time. This situation results in a problem where the constructed queries in the benchmarks may no longer be completed with their originally referenced tools. Consequently, it is crucial to strike a balance between enhancing the stability of these benchmarks and maintaining their diversity and scope.
To address these issues, we propose a new benchmark named StableToolBench, which incorporates a virtual API system and a stable evaluation system. We first build a virtual API system to replace the real one. As a start, we build a caching system to store the outputs of API calls. This approach ensures the stability and reproducibility of API behaviours. Given the limited number of benchmark questions, our caching system can cover a significant number of API call scenarios. However, relying solely on a cache is insufficient because many APIs remain unavailable. To resolve this problem, we use large language models (LLMs) to simulate the behaviours of these APIs. Specifically, we feed the documentation and few-shot real API calls if available in the cache to LLMs and ask LLMs to mock the behaviour of the APIs given a request. As a result, users can always get responses from APIs in an indistinguishable way as long as the LLMs are accessible. On the whole, our system first tries to find a hit in the cache. Unless there is a cache miss and a real API call is not received, the simulated server will be used.
We then improve the evaluation system to make it more stable. We design two metrics (i.e., SoPR and SoWR) after judging solvable tasks and replace all the automatic evaluators with GPT-4 to mitigate the randomness and indistinguishability during evaluation. Experiments demonstrate that our virtual API system, when combined with the improved evaluation system, can provide stable evaluation against API modifications. Furthermore, our system exhibits significant reliability in terms of realism, diversity, and documentation following accuracy.
The main contributions of our work are summarised as follows:
• A tool-learning benchmark featured a large number of cached stable simulated APIs, wellbalancing stability and reality of the APIs and much more stable evaluation metrics. • Extensive experiments show that our benchmark provides much more stable model performance, robust to various types of API failures. • Besides enhanced stability, our virtual API system exhibits reality, diversity and reliability comparable to that of the real API system.

heading: 2 Stability Analysis on ToolBench
text: In this section, we initiate a comprehensive analysis to reveal the stability of established tool benchmarks, using Toolbench (Qin et al., 2023b) as a case study. We examine the stability of ToolBench by investigating three key dimensions: performance, evaluation, and API status.

heading: 2.1 Stability of Performance
text: Benchmarks are designed to consistently evaluate the performance of various models over time. To test this consistency, we reproduce the model performances and record any variations. Our study employs Chain-of-Thought (CoT; Wei et al., 2023) and Depth First Search (DFS) strategies, leveraging ChatGPT and ToolLLaMA for comparative analysis. We adhere strictly to the configurations detailed in ToolBench, utilising the ChatGPT version gpt-3.5-turbo-0613 and ToolLLaMA-v2. As depicted in Figure 1, we compare the original Pass Rates for the I1-Instruction group reported by ToolBench with the Pass Rates we reproduced for four conventional methods. Our findings indicate a no-
2
table decline in the performance of all methods over time, which raises concerns about the stability of ToolBench as a benchmark.

heading: 2.2 Stability of Evaluation
text: In ToolBench, there are two types of metrics, including Pass Rate (PR) and Win Rate (WR). PR is calculated based on using gpt-3.5-turbo-16k to determine if a task is solvable and whether the generated answer can solve the corresponding task. Figure 2 details the computation process of PR. Specifically, a solvable task results in a pass if the answer is solved, a failure if unsolved, and is randomly determined if unsure. For tasks deemed unsure, a pass is assigned only if the answer is solved; otherwise, a random outcome is chosen. If a task is unsolvable, the result defaults to a pass regardless of the answer status. Moreover, WR is derived from the comparative PR of paired candidates. A candidate’s WR increases by one each time it passes while the other fails. In all other situations, WR relies on gpt-3.5-turbo-16k to automatically evaluate the paired candidates.
To assess the stability of evaluation, we perform both CoT and DFS using gpt-3.5-turbo-0613
on the I1-Instruction group dataset. These analyses are conducted using the provided tools and repeat over three iterations each. The resulting PR and WR are presented in Table 1, with detailed task and answer items. Despite PRs of DFS being generally higher than CoT, the contribution of the task is larger than the answer. However, it is worth noting that the tasks are the same in both CoT and DFS, where their results are expected to be consistent. On the contrary, the discrimination of answers between CoT and DFS is weak, where a considerable proportion are unsolved. Moreover, WR does not reflect the same trend as PR, where the second run of WR in DFS (48.0) is even lower than the first run of CoT. Therefore, all the phenomena reflect that gpt-3.5-turbo-16k can not assume the role of the automatic evaluator in tool learning, which will be discussed in Section 4.6.

heading: 2.3 Stability of API Status
text: We investigate the change of API status in ToolBench. In detail, we scan the original APIs downloaded from ToolBench, and use gpt-4-turbo to automatically write calls via the prompts as shown in Appendix H. According to the keyword in API feedback, we classify these APIs into three categories: success, not availability, and not authorisation3. The API status and the detailed errors of not availability are presented in Figure 3. As can be seen, only 44.4% of API calls are successful, while other API calls are mostly not available with various errors and some are not authorised.
Furthermore, to validate the impact of API call failures on the stability of model performance, we manually make some success tools4 down by returning a special failure call. Specifically, we ran-
3Note that we use the toolbench-key provided in ToolBench to simulate the real running process in the benchmark.
4In ToolBench, a tool is composed of several APIs. For example, a database tool can have two APIs: a writing API and a reading API.
3
domly sample a proportion of success tools containing success APIs found in Figure 3. At testing time, when sampled tools are called, a special response will be thrown: {“error”: “”, “response”: “This API did not return any useful information...”} to simulate the API call failures. We conduct baseline models with different proportions (i.e., 0%, 10%, 20%, and 50%) of sampled APIs on the I1 Instruction set. Due to the issues in evaluation, we use our stable evaluation system proposed in Section 3.2 as the same as our main experiments. For each experiment, we evaluate three times and report the average scores as shown in Figure 4. It can be seen that the performance degrades a lot when the proportion of successful APIs is down, thus the impact of API status on stability is considerable.

heading: 3 StableToolBench
text: Considering that stability is a crucial feature of benchmarking, in this paper, we specifically design a virtual API server and stable evaluation system to improve the stability based on ToolBench, and propose a new benchmark, named StableToolBench.

heading: 3.1 Virtual API Server
text: With real APIs, many of the failures encountered when reproducing its experiments are caused by expired APIs, network issues, or failed authentication. To address this problem, we specifically propose a virtual API server with two components as illustrated in Figure 5, including a caching system and API simulator. Moreover, we design API calling rules to combine these two components to ensure the virtual API server is stable.
Caching System. We first implement a caching system that stores responses of all API callings to ensure consistency. The caching system uses keys composed of their category, tool, API name, and arguments. As a start, we populate the initial cache using the API call history from the training data and the reproduced test data released in ToolBench5. To ensure the quality of cached APIs, only valid records following the rule in Appendix D will be saved. It is worth noting that we will also reserve some APIs with exceptions to keep the reality. In this way, most API responses will be readily available, allowing the benchmark focus on probing the tool usage ability of designed methods with minimal impact on tool availability. Furthermore, the API call in new experiments will also be continuously updated in the cache to ensure scalability. The statistics of cache are shown in Table 2. Additionally, as an extra benefit, this approach re-
duces the latency introduced by interacting with real APIs, and also saves the costs for the API simulator discussed below.
API Simulator. Due to the limited coverage of the caching system, we propose to use LLMs to simulate API responses that are not in the cache and unavailable. Specifically, we ask gpt-4-turbo to simulate the API behaviour based on the original documentation in ToolBench. The API documentation includes the descriptions of the functions and their corresponding parameters. To mitigate the difference between simulated and real APIs, we use real API calls in the caching system as few-shot examples (Brown et al., 2020) for the LLM to better mock the behaviours. We keep the maximum number of examples at five. When less than five examples exist in the cache, all of them will be used. Detailed prompts can be found in Appendix F.
API Calling Rules. Based on the caching system and the API simulator, we create API calling rules to ensure the stability of the virtual API server.
5https://drive.google.com/drive/folders/ 1yBUQ732mPu-KclJnuQELEhtKakdXFc3J
4
When a call request (e, args), where e is the API endpoint and args is the arguments for that endpoint, is received, the system will first search the caching system for (e, args) pair. If a cache hit exists, the cached response will be directly returned. When there is a cache miss, then the system will try to call the real API for a response to maintain the reality of the whole system. If the real API calling is successful, the response will be returned. However, when the caching system does not contain and the real API is not available, the system will finally call the simulated API. The final response whether from the real API or the simulated API will be saved to update in the caching system.

heading: 3.2 Stable Evaluation System
text: In this section, we propose a two-phase evaluation process, including judging solvable tasks and evaluating with two metrics, as shown in Figure 6. Moreover, we replace all the automatic evaluators with gpt-4-turbo.
Judging Solvable Tasks. Considering that both unsolvable and unsure tasks would introduce enormous randomness while the results of tasks fluctuate wildly, we try to obtain a fixed collection of solvable tasks to eliminate the problems. To
achieve this, we use three state-of-the-art LLMs, i.e., gpt-4-turbo, gemini-pro, and claude-2, to determine whether a task is solvable or unsolvable. The prompt is shown in Appendix G. The task will be judged as solvable when more than two models evaluate it as solvable. The statistics of solvable tasks across all test datasets in ToolBench are shown in Table 3.
Metrics. We then report Solvable Pass Rate (SoPR) and Solvable Win Rate (SoWR) based on our obtained solvable tasks. Due to the limitation of gpt-3.5-turbo-16k in tool learning, we uniformly adopt gpt-4-turbo as the automatic evaluator. SoPR is in essence PR with all tasks solvable and only assesses the answers using the same prompt in ToolBench. The evaluator assigns outcomes of answers categorised as Solved, Unsolved, or Unsure, which respectively contribute scores of 1, 0.5, and 0 to the overall SoPR calculation. As for SoWR, when one is solved and the other is unsolved, the solved one wins. Under other circumstances, gpt-4-turbo will be used to make a win-lose decision.
5

heading: 3.5 0613 (D) 60.7 67.3 59.5 63.2 62.1 75.4 64.7

heading: 4 0613 (C) 54.6 58.8 58.2 75.5 60.5 62.3 61.7

heading: 3.5 1106 (C) 47.2 47.7 44.9 50.9 54.0 62.3 51.2

heading: 4 Experiment

heading: 4.1 Performance
text: Following ToolBench, we run gpt-3.5-0613, gpt-4-0613, ToolLLaMA-v2 with CoT and DFS, replenishing with latest models gpt-3.5-1106 and gpt-4-turbo. The results of SoPR and SoWR are shown in Tables 4 and 5. Generally, GPT-4 series models outperform GPT-3.5 models, while ToolLLaMA performs worst with the same inference algorithm. Also, DFS significantly outperforms CoT whichever LLMs are used. These phenomena are consistent with ToolBench. Furthermore, probably thanks to the improved function calling capabilities, newer GPT models performed better.

heading: 4.2 Stability of Virtual API Server
text: Following the same setups as in Section 2.3, we manually make the same success tools not available during the running time. In our design, when a
call is on an unavailable tool, it will be directed to the simulated API immediately. Compared to Section 2.3, the results as shown in Figure 7 are much more stable with our virtual API server. Even when 50% of APIs are not available, changes in performance are still not significant, which is explainable within the range of variance.
Considering we use gpt-4-turbo as the backbone of the API simulator which may change even with the same version number, we ablate different versions and different temperatures of gpt-4-turbo. The results are shown in Table 6. Under different settings of the backbone LLMs, the performance change is still acceptable within the variance of LLM evaluation, indicating the robustness of our API simulators.
6

heading: 3.5 0613 (C) 49.1±1.0 49.0±0.8 52.1±0.5 50.2±0.8

heading: 3.5 0613 (D) 68.1±1.4 67.4±0.9 69.3±1.0 67.9±1.2

heading: 4 0613 (C) 55.4±0.6 56.3±0.7 57.7±0.5 54.5±0.6

heading: 4 0613 (D) 69.7±1.4 70.3±1.0 71.4±0.7 70.4±1.3

heading: 3.5 1106 (C) 52.1±0.7 51.1±0.5 54.5±0.9 52.7±0.6

heading: 4.3 Turing Test of API Simulator
text: To test the effectiveness of API simulators, we design a “Turing Test” (Turing, 2009) between the real APIs and the simulated ones. Note that we believe it is not required for API simulators to exactly output the same answers as those of real APIs, where rationality is more important. For example, when a query asks about the weather today, the API simulator does not need to retrieve the “real” temperature. Instead, the API simulator needs to generate a reasonable temperature number.
To do the test, we first sample 70 available real APIs and their corresponding simulated APIs. Given the API callings and their real and simulated response pairs, we ask three human annotators to determine which response more closely resembles an actual API response overall, based on the given descriptions of the API functions. We ask human annotators to evaluate along three dimensions: Overall, Format Accuracy and Answer Relevance. The annotator first need to answer which response is overall more like a real response. When assessing format accuracy, annotators must determine which response more accurately adheres to the format specifications outlined in the documentation. In evaluating answer relevance, they are tasked with identifying which response more effectively fulfills the instruction in accordance with the documentation’s guidelines. The results are shown in Figure 8. Surprisingly, human annotators cannot distinguish simulated and real APIs very well, where the simulated APIs are judged to act more like real situations. Moreover, the proportion of tie is much larger, indicating that simulated APIs can work very similarly to real APIs.
In addition to the Turing Test mentioned above, we assess the quality of LLM simulations by evaluating the adherence of simulated outputs to their
corresponding documentation. To conduct this evaluation, we randomly sample 50 simulated outputs along with their documentation from the Turing Test dataset. A human evaluator is then tasked with determining whether the LLM simulations reasonably follow the provided documentation. The results indicate that 90% of the simulations are deemed reasonable, 6% are considered unreasonable, and 4% are uncertain. These findings suggest that the LLM is highly capable of generating simulated responses that adhere closely to the provided documentation.

heading: 4.4 Diversity of API Simulator
text: With the LLM simulation, API simulators will not exactly feedback the same as real APIs. Hence, a natural concern is whether the simulated APIs will degrade in diversity in API functionalities. To study the problem, firstly, we explore the distribution of real and simulated API responses. We first use all 246 APIs in the Tool API category from the successful APIs mentioned in Section 2.3. Then, we use the same call arguments to call these real and simulated APIs. All the responses are encoded using S-BERT (Reimers and Gurevych, 2019) and their corresponding embeddings are visualised by UMAP (McInnes et al., 2018). Detailed configuration is shown in Appendix E. The result is shown in Figure 9. As can be seen from the figure, real and simulated APIs occupied similar embedding space, indicating that the diversity of simulated APIs is similar to the real APIs.
Secondly, we try to explore the behaviour when a simulated API is given several calls with different input arguments. We sample 60 APIs from all successful APIs and make 5 different calls to each API, using the same prompt as in Appendix H. We then count the number of APIs that give exactly the same responses in any 2 of the 5 calls. Results show that only 2 of 60 APIs contain such
7
responses, which supports the sufficient diversity of our simulated APIs.

heading: 4.5 Effectiveness of Caching System
text: To show the effectiveness of our caching system in maintaining the stability of the virtual API server, we run several methods and record the cache hit rates. In detail, we run four methods used in ToolBench, gpt-3.5-turbo-0613 and gpt-4-0613 with CoT and DFS. Reproduction data in ToolBench of these methods has been used in the cache. Results are recorded in Table 7, which shows that rerunning these in-domain methods has a very high cache hit rate. This means that most of the call responses are fixed and instability from the API system are much smaller. Nevertheless, models and methods may change over time, and therefore, we further run gpt-3.5-turbo-1106 and gpt-4-turbo-preview with CoT and DFS. As can be seen in the table, although the cache hit rates are smaller with these out-of-domain methods, the scores are still high enough to mitigate the instability significantly.

heading: 4.6 Human Evaluation of Evaluator
text: Considering that GPT-3.5 is limited to evaluating the performance in tool learning, we replace the automatic evaluators with stronger LLMs. In this section, we manually assess the correctness of different automatic evaluators. Here, we sample 100 task-solvable questions, 50 answer-solving questions in the PR / SoPR evaluation, and 50 comparison questions in the WR / SoWR evaluation from the experiments running.
We then collect all the corresponding answers of different LLMs during previous evaluations. These questions are further manually labeled by three human annotators to obtain the ground truth. With the ground truth, we calculate the accuracy scores
of these models and show the results in Table 8. It can be seen that our used LLMs (i.e., Claude 2, Gemini, and GPT-4) are much better than GPT3.5 in ToolBench to determine the solvability of tasks, where the Gemini and GPT-4 outperform by a large margin. In both the evaluation of answers and comparison, GPT-4 significantly outperforms GPT-3.5, especially in the comparison to compute WR. It is worth noting that all the accuracies of GPT-3.5 are lower than 70%, indicating that GPT3.5 cannot assess the performance in tool learning.

heading: 5 Related Work
text: Tool Learning Benchmarks. Recent studies have shed light on the burgeoning capabilities of LLMs in understanding and mastering tools (Li et al., 2023a; Patil et al., 2023; Yang et al., 2023b; Song et al., 2023; Tang et al., 2023; Ye et al., 2024; Xu et al., 2023). Gaining access to external tools endows LLMs with real-time factual knowledge (Yang et al., 2023a), multimodal functionalities (Gupta and Kembhavi, 2023), and specialised skills in vertical domains (Jin et al., 2024). However, few work has been done to explore the stability of the tool environment in specific benchmarks and how it affects the LLMs’ performance in toolaugmented tasks.
Tool Inference Methods. Recent literature has begun to explore various methodologies for integrating tool functionalities within LLMs. Notably, the robust in-context learning prowess of LLMs, as demonstrated in (Brown et al., 2020), has facilitated the augmentation of LLMs with external tools via in-context tool descriptions and demonstrations (Hsieh et al., 2023; Ruan et al., 2023; Mialon et al.,
8
2023). An alternative approach involves the explicit training of LLMs (Patil et al., 2023; Tang et al., 2023; Chen et al., 2023a; Qin et al., 2023c; Huang et al., 2023) using datasets enriched with tool interactions, thereby familiarising models with the nuances of tool usage.
Evaluation in Tool Learning. Evaluating the performance of LLMs in tool-augmented tasks presents unique challenges and opportunities. Numerous works have been developed for the assessment of tool utilisation, primarily emphasising response comparison (Zhuang et al., 2023), tool call accuracy (Patil et al., 2023), or a synthesis of these aspects (Li et al., 2023a). Distinguishing itself, (Qin et al., 2023c) introduces an innovative methodology by integrating a large language model (LLM) as a judge to evaluate the comprehensive solution path. Subsequent research (Wang et al., 2023) focuses on the multi-turn interaction capabilities of LLMs with both tools and user feedback. In a departure from the aforementioned approaches, (Chen et al., 2023b) presents itself as the inaugural benchmark specifically tailored for the fine-grained assessment of tool utilisation capabilities. However, there exists a gap in the literature concerning the exploration of evaluation stability when evaluating the tool usage capabilities of LLMs.

heading: 6 Conclusion
text: In this paper, we propose StableToolBench, a benchmark developed to enhance the stability of ToolBench. Our analysis identified instability issues in the evaluation processes of ToolBench and API status, causing variability in model performance assessments. To address this, we implement a caching system for consistent data availability. We also replace the real API server with an LLM-simulated virtual server for reliable API behaviour simulation. Experiments show that StableToolBench significantly improves the stability of model performance evaluations, with the simulated APIs offering realism and the caching system
contributing greatly to the enhanced stability of the benchmark.

heading: Acknowledgement
text: This work is supported by the National Natural Science Foundation of China (No. 62276152, 61925601). We also extend our gratitude to Jingwen Wu and Yao Li for their assistance with human evaluation and additional suggestions.
Limitations
In this work, we propose StableToolBench, a new tool learning benchmark with increased stability but non-declining reality. However, our work faces the following limitations. Firstly, we used GPT-4 as the automatic evaluator in the evaluation process and as the backbone server, which increase the cost of using our benchmark. Secondly, the GPT-4 backboned server demonstrate strong performance in simulating API behaviours. Nevertheless, the backbone LLM may experience significant upgrades, which may affect the performance. Therefore, we believe that the ultimate solution is to solve the problem with a trained open-source LLM. However, current open-source LLMs are not performant enough to simulate API behaviours well. As a result, closed-source LLMs are the only options. We believe that when open-source LLMs are strong enough to be well suited for this task, In the future, we may turn to open-source LLMs when they are strong enough to be well suited for this task. Thirdly, although the cache hit rates are high with our explored methods, new methods will be developed in the future. Whether this cache will still be effective is unsure. In this regard, we aim to continuously update the cache in the future in a slow pace for both balanced stability and effectiveness.

heading: A Comparison of Reported and Reproduced Performance
text: Detailed comparison scores of reported and reproduced performance are shown in Table 9.

heading: B Statistics of API change information
text: Detailed statistics of API change categories and information are shown in Table 10 and Table 11.

heading: C Stability Test Scores with Virtual API Systems
text: Detailed scores of stability tests of various models are shown in Table 12. Note that in addition to GPT 3.5 Turbo 0613 and GPT 4 0613, we report the performance of newer versions, namely GPT 3.5 Turbo 1106 and GPT 4 Turbo Preview.

heading: D Call Error Identification and Cache Filtering Rule
text: We identify call errors and filter out invalid call to RapidAPI based on keyword occurences. In detail, we identify the following error:
Method Real API Failure Rate
0% 10% 20% 50%
GPT 3.5 Turbo 0613 + CoT 49.1±1.0 48.7±0.9 51.2±1.3 49.0±0.7 GPT 3.5 Turbo 0613 + DFS 68.1±1.4 70.9±1.3 67.5±1.8 67.3±1.3
GPT 4 0613 + CoT 55.4±0.6 55.5±1.0 58.0±0.5 55.2±0.6 GPT 4 0613 + DFS 69.7±1.4 71.4±1.4 71.2±0.9 69.9±0.9
GPT 3.5 Turbo 1106 + CoT 52.1±0.7 52.4±0.8 53.9±0.6 50.2±0.6 GPT 3.5 Turbo 1106 + DFS 69.9±0.7 71.7±0.7 69.4±0.8 71.6±0.9 GPT 4 Turbo preview + CoT 60.8±0.7 62.8±0.5 64.2±0.7 62.4±0.5 GPT 4 Turbo preview + DFS 73.2±1.1 76.7±1.0 76.0±0.8 74.2±1.3
We consider all types of errors when identifying errors. However, when filtering the cache, we do not conside the“Other Errors”.

heading: E Configurations of API Diversity Analysis
text: The configurations of diversity analysis are as follows:
• Embedding model: all-mpnet-base-v2;
12
• UMAP metric (distance metric): correlation;
• Num of neighbours: 15;
• Min distance: 0.5.

heading: F Prompts of API simulation
text: The prompt used to simulate API behaviours is shown in Table 13.

heading: G Prompt to Filter Solvable Task
text: The prompt used to filter solvable tasks is shown in Table 14.

heading: H Prompt Used to Make API Calls
text: The prompt used to construct API calls to scan availables is shown in Table 15.
13
14

heading: Future Work
text: We believe that when open-source LLMs are strong enough to be well suited for this task, In the future, we may turn to open-source LLMs when they are strong enough to be well suited for this task. Thirdly, although the cache hit rates are high with our explored methods, new methods will be developed in the future. Whether this cache will still be effective is unsure. In this regard, we aim to continuously update the cache in the future in a slow pace for both balanced stability and effectiveness