API Pack: A Massive Multi-Programming Language Dataset for API Call Generation

## Abstract

Abstract We introduce API Pack, a massive multi-programming language dataset containing more than 1 million instruction-API call pairs to improve the API call generation capabilities of large language models. By fine-tuning CodeLlama-13B on 20,000 Python instances from API Pack, we enable it to outperform GPT-3.5 and GPT-4 in generating unseen API calls. Fine-tuning on API Pack also facilitates cross-programming language generalization by leveraging a large amount of data in one language and small amounts of data from other languages. Scaling the training data to 1 million instances further improves the model’s ability to generalize to new APIs not used in training. To facilitate further research, we open-source the API Pack dataset, trained model, and associated source code at https://github.com/zguo0525/API-Pack .

## 1 Introduction

Large language models (LLMs) have shown promise in assisting with software engineering tasks [, , , , , , , , ], with a primary focus on code generation [, , , , , , , , , , , , , , , , ]. Our work builds on these advances but targets a time-consuming task developers often face in finding application programming interface (API) call code examples. While finding an appropriate API is relatively easy with search, the challenge lies in generating the specific API call code. This involves understanding the API’s parameters, data types, and required operations, which can be complex and vary significantly across different APIs. Currently, developers typically seek out examples in documentations or API hubs [], sifting through lengthy pages to locate relevant information []. This process which requires developers to manually look for code examples for their specific use case is often cumbersome and tedious. Our work aims to transform this workflow by using LLMs to identify appropriate API endpoints and generate corresponding API calls based on natural language instructions, thus automating the process of searching the appropriate API call code examples.

To achieve this goal, we create API Pack, a dataset designed to improve the API call generation capabilities of LLMs. This multi-programming language dataset, containing over 1 million instances across 10 programming languages, is by far the largest open-source instruction dataset (see Table 1) for single API call generation and API call intent detection. API Pack distinguishes itself from prior works [, , ] in two key aspects: programming language diversity and dataset size. API Pack includes API calls across 10 different programming languages (see Table 1), allowing for a unique assessment of cross-lingual skill transfer - the extent to which improvements in one language apply to others. This important aspect has been under explored in previous studies even though it is crucial for the practical application of LLMs to API call generation use cases. Furthermore, with more than a million instances that span a wider range of real-world APIs and use cases compared to its predecessors (see Table 1), API Pack enables a more rigorous evaluation of generalization capabilities by controlling the volume of training data.

We summarize three key findings from our experiments:

**Table 1: A comparison of API Pack with other instruction datasets for API intent detection and/or API call code generation. The upper section of the table reports the features that each dataset covers, and the bottom section reports the data statistics available (/ means unavailable).**
| Feature | API Pack (this work) | APIBench (Gorilla) | ToolBench | ToolBench (ToolLLM) | API Bank | ToolAlpaca | ToolFormer |
| --- | --- | --- | --- | --- | --- | --- | --- |
| API call intent detection? | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| API call code generation? | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ |
| Multi-programming language? | ✓(10) | ✗(Python) | ✓(Curl, Python) | ✗ | ✗ | ✗ | ✗ |
| Multi-API call scenario? | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Data generation method | custom | self-instruct | self-instruct | custom | custom | custom | custom |
| # of Sources | 4 | 3 | 8 | 1 | 53 | / | 5 |
| # of APIs / Tools | 11,213 | 1,645 | 8 | 16,464 | 53 | 400 | 5 |
| # of API calls | 1,128,599 | 16,450 | / | 37,204 | 568 | 3,938 | 9,400 |
| # of Instances | 1,128,599 | 16,450 | 2,746 | 12,657 | 264 | 3,938 | 22,453 |

- 1.
Fine-tuning CodeLlama-13B on 20,000 Python instances from API Pack yields over 10% and 5% higher accuracy than GPT-3.5 and GPT-4, respectively, for unseen API calls.
- 2.
Cross-lingual API call generation can be facilitated by fine-tuning models on a large amount of data in one language along with small amounts of data from other languages.
- 3.
Scaling the instruction data from smaller instances to 1 million instances improves generalization to new APIs, validating the benefits of a larger dataset.

The paper is organized as follows: Section 2 reviews related works. Section 3 describes the construction of API Pack dataset. Section 4 outlines the experimental design for fine-tuning models on API Pack and evaluating their performance. The key findings are presented in Section 5. We conclude the paper in Section 6, and address limitations and future work plans in Section 7. We discuss ethical considerations related to API Pack’s use in Appendix 9.1. API Pack dataset is under the Creative Commons Attribution 4.0 International License. The associated source code is under an MIT License.

## 2 Related Works

### 2.1 Methods to Generate Instruction Data with LLMs

Manually creating instruction data is labor intensive []. To automate this process, researchers have proposed the use of LLMs. Two popular methods to generate data via an LLM are Self-Instruct [] and Evol-Instruct [], both methods involve instance (instruction - response pairs) generation and filtering. Self-Instruct mutates seed instances using in-context examples and filters them based on ROUGE-L similarity and other heuristics. Evol-Instruct generates instances via targeted prompting and filters them using predefined heuristics. Custom LLM-based pipelines to generate synthetic data, such as the one we followed to create API Pack, also exist. In the case of code datasets, the pipelines typically involve mining high-quality code and using an LLM to generate instructions for each code data sample (e.g., [], []). Post-quality filtering is crucial when using LLM-based synthetic data for fine-tuning. Although human annotators can determine the correctness of the instance [], this kind of revision takes time and is thus not reasonable for large datasets. Therefore, recent work has proposed the use of strong LLMs (i.e., GPT-4 and Mistral AI models families) to score or classify instances [] based on heuristics and examples derived from a small sample of data reviewed by human annotators. A similar approach has also been used to assess instruction complexity [, ].

### 2.2 LLMs for API Call Code Generation and Intent Detection

LLMs for code generation primarily focus on general coding tasks, evaluated by benchmarks like MBPP [], HumanEval [], and its variants [, , ]. Recent research explores LLMs and APIs in two areas: API call intent detection and API call code generation. API call intent detection identifies the appropriate API endpoint for a natural language task. LLMs for this purpose [, , , , ] work in hybrid architectures, where the LLM finds the API endpoint(s) and other components generate the code. These studies explore single and multi-API intent scenarios, with few achieving good results in the latter []. Our work focuses on training LLMs that generate code to invoke API functionality, which is less explored than API call intent detection. Gorilla [] generates API calls for loading pre-trained machine learning models from known model hubs. ToolBench [] benchmarks tool manipulation capabilities of open-source LLMs. Another line of work explores function calling in language models [, , ], including ChatGPT. These models require pre-defined functions before execution, differing from our scenario where developers rely on a language model to guide them to the appropriate function or API.

## 3 API Pack

API Pack is an instruction dataset with over one million instances. Each instance contains an input-output pair plus additional information about the API and respective endpoints. The inputs are instructions for finding an API call to solve a coding task, including a task description in software engineering language and the name of the API. The outputs are API call examples, specifically HTTP request code snippets curated from OpenAPI specification (OAS) files. The data in API Pack is sourced from four hubs that store OAS files: RapidAPI(^1^11https://rapidapi.com/categories), APIGurus(^2^22https://apis.guru/), Swaggerhub(^3^33https://app.swaggerhub.com/search), and IBM’s public API Hub(^4^44https://developer.ibm.com/apis/). To the best of our knowledge, these API Hubs were the only sources with publicly available OAS files at a large scale at the time we collected data. From these sources, we kept all the APIs that passed our filtering criteria (see Section 3.1). Table 2 summarizes the total number of APIs, unique endpoints, and total instances in API Pack.

**Table 2: Final count of data curated per source, an instance has one API call.**
| Source | APIs | Unique Endpoints | Total Instances |
| --- | --- | --- | --- |
| IBM API Hub | 73 | 2,884 | 17,206 |
| APIs Gurus | 1,980 | 37,097 | 495,533 |
| Swaggerhub | 5,045 | 26,747 | 345,765 |
| RapidAPI | 4,115 | 21,525 | 270,095 |
| Total | 11,213 | 88,253 | 1,128,599 |

The construction of API Pack involves four main stages: data pre-processing (Section 3.1), API Database (DB) creation (Section 3.2), instruction generation (Section 3.3), and data validation (Section 3.4). Figure 1 illustrates the overall pipeline.

Figure: Figure 1: Dataset curation pipeline.
Refer to caption: x1.png

### 3.1 Data Pre-processing

This process involves two steps. First, as the presence of non-English metadata (e.g., API description, endpoint description) in the files was insignificant (less than 1%), we filter out OAS files with non-English data. We also remove OAS files with zero endpoints, as it is not possible to generate API calls for these APIs.
Next, we extract relevant information from the remaining OAS files. We extracted information about the APIs, and about the endpoints that each API contains. At the API level, we gather the API name, description, and provider. At the endpoint level, we collect the name, functionality, description, method, and path. To further ensure data quality, we apply a second filter to remove instances with missing crucial data for generating API calls (e.g., method, path, or endpoint name) and instructions (e.g., functionality, description, or endpoint name). This step guarantees that all instances in the dataset contain the necessary information for generating useful API call examples and instructions.

### 3.2 Create API Databases

We build an API database (DB) using the pre-processed data, with each instance containing endpoint information, and API details. Then, we use OpenAPI Snippet(^5^55https://www.npmjs.com/package/openapi-snippet) library to generate API calls in 10 programming languages (cURL, libcurl, Java, Node.js, Python, Go, Ruby, PHP, Swift, JavaScript) for each API DB instance. We followed this process to create API calls with the data from RapidAPI, APIGurus, and OpenAPI. We directly extract API calls from OAS files for IBM API Hub as this source contained API calls in more than one programming language. Two factors influenced our selection of programming languages. First, the 10 programming languages included in API Pack appear in the StarCoder report of programming languages []. Second, given that we used an open-source library (OpenAPI Snippet) to generate API calls, language programming selection is inherently limited to the languages that this library supports. Appendix 9.6 provides details on the API DB instance structure and the programming language diversity for each source.

### 3.3 Instructions Generation

This process involves creating high-quality instruction examples, followed by generating instruction candidates based on these examples. First, we
randomly select three endpoints from each API DB and use their information (e.g., functionality, description, endpoint_name, path) along with the respective API name to fill in a list of predefined instruction templates. This process yields three instruction examples per API DB file.
For the API Gurus and IBM API Hub sources, three authors manually refine the generated instruction examples by correcting grammatical errors, removing unnecessary information, and ensuring that the API name is present in all examples.
To streamline the refinement process of examples for the Swaggerhub and RapidAPI sources, we replace human labor with an LLM (Mistral-7B-Instruct-v0.2(^6^66https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2)). We create a prompt to refine the instruction examples (Prompt 1 in Appendix 9.10) based on the heuristics identified during the manual review. We use this prompt to task the LLM with refining the instruction examples for the Swaggerhub and RapidAPI sources. Second, we use the high-quality instruction examples to generate five instruction candidates for each instance in the API DBs. We prompt an LLM (Mistral-7B-Instruct-v0.2) with the endpoint’s information as input and provide the high-quality instruction examples as in-context examples (see Prompt 2 in Appendix 9.10).
Figure 7 in Appendix 9.8 shows an instance containing one instruction candidate along with its respective API call.
Appendix 9.9 presents all five candidates generated for the same instance.

### 3.4 Data Validation

The data validation process involves three main steps: 1) verifying the validity of API calls as HTTP request examples in a given programming language, 2) assessing the quality of generated instructions, and 3) selecting the highest-quality instruction for fine-tuning.

To verify the validity of API calls, we first compare the API call’s content with the instance data to ensure the correctness of the endpoint_name and the HTTP method (e.g., get, post, put, delete, custom method name). We then use regular expressions to validate the URL format, considering that placeholder strings may exist for the URL domain, path parameters, or query parameters. Additionally, we check that the programming language keywords in the API call match the language string ID assigned to each instance.

To validate the generated instructions for each API call, we randomly select a sample of 121 instances, each containing one API call and five instructions. These instances have unique functionality, description, and endpoint name. One of the authors manually annotates 605 instructions (121 instances x 5 instructions) as good or bad. The analysis of the instructions annotated as bad reveals three common characteristics: 1) containing multiple instructions instead of a single one, b) including unnecessary text before or after the main instruction, and c) failing to accurately use the correct API name. Based on these characteristics, we create three prompts with a fixed set of in-context examples to automatically annotate instructions via an LLM. To select the best prompt, we use an LLM (Mistral-7B-Instruct-v0.2) to annotate the 605 manually classified instructions and compare the results with human-annotated results. We choose the prompt with the best performance (Prompt 4 in Appendix 9.10) and instruct Mistral-7B-Instruct-v0.2 to classify all instructions in our dataset. Instances with fewer than two good instructions are removed from the dataset.

To select the best candidate from the subset of good instructions, we calculate the likelihood of an LLM re-generating the input text used to generate the instruction. We prompt Mistral-7B-Instruct-v0.2 with each instruction candidate (Prompt 3 in Appendix 9.10) and obtain the log probability of each token for the re-generated input text. We calculate the mean of these log probabilities (input_tokens_mean). Appendix 9.9 shows five instruction candidates and their respective input_tokens_mean. We select instructions annotated as good with the best input_token_mean for fine-tuning. Our final dataset contains 1,128,599 instances, each with a valid API call example and at least two high-quality instructions. Appendix 9.2 shows the data instances filtered out at each stage of the pipeline.

## 4 Experiments and Evaluation Framework

To optimize the instruction-following capabilities of the language models, we post-process API Pack into two instruction-tuning templates. The 0-shot-tuning template targets scenarios where the output is expected to be a straightforward inference from the given input. The 3-shot-tuning template, with random selections within the selected API, emphasizes the model’s ability to learn and generate output with in-context learning. The 3-shot-tuning template is available in Appendix 5.

### 4.1 Experimental Settings

A. Selecting a base model: Our first experimental study serves the purpose of selecting a base model for the rest of our experiments. We fine-tune Mistral 7b [], CodeLlama 7b and 13b, as well as Llama 2 13b [] on a subset of API Pack (20,000 instances in Python programming language). We evaluate the performance of resulting models and select the best-performing base model.

B. Inference with retrieval: Our second experiment aims to understand the influence of retrieval augmentation on model generalization. We evaluate the models under four distinct prompt settings during test time:

- •
0-shot: No API examples provided for the model.
- •
3-shot random: Three randomly selected API examples.
- •
3-shot retrieved: Three retrieved relevant API examples.
- •
3-shot retrieved & re-ranked: Five retrieved API examples, with three selected using a re-ranker model in the final prompt.

The 3-shot random inference uses the same prompt template as 3-shot-tuning. We use bge-large-en-v1.5 [] as the embedding model for retrieval and beg-reranker-large [] for re-ranking. Appendix 9.5 illustrates the inference pipelines used to evaluate the models’ performance.

C. Cross-lingual generalization: To test the model’s ability to generalize to new programming languages, we supplement a cURL dataset of 100,000 instances with 1,000 instances from each of the nine additional languages in API Pack: Go, Java, JavaScript, libcurl, Node.js, PHP, Python, Ruby, and Swift. The goal is to determine if a model can generalize to new languages without requiring a large amount of multi-programming language data.

D. Scaling experiment: We conduct a scaling experiment to investigate whether more API data improves a model’s generalization ability for unseen APIs. We fine-tune models on progressively larger API datasets with unique API calls on 10k, 20k, 40k, 80k, 100k, and 1 million instances. Our hypothesis is that exposure to a greater scale and diversity of APIs during fine-tuning will improve the model’s ability to generalize to new, unseen APIs.

### 4.2 Evaluations

To measure the generalization capabilities enabled by API Pack, we establish a comprehensive evaluation framework spanning three levels of complexity for API call generation:

- •
Level 1: Seen APIs and endpoints. This level assesses generalization to new instructions.
- •
Level 2: Seen APIs and new endpoints. This level tests generalization to new endpoints of known APIs.
- •
Level 3: Unseen APIs and endpoints. This level validates performance on entirely new APIs.

The process of data splitting is detailed in Appendix 9.3. Training hyperparameters are in Appendix 9.4. Endpoint and API call accuracy at each level is measured by SequenceMatcher, which identifies the longest matching subsequences while excluding insignificant elements. These elements refer to specific parts of the API requests that do not affect the overall functionality or purpose of the request, such as whitespace and formatting differences (e.g., variations in spaces, tabs, or newlines within the API requests) and specific naming conventions or variable names (i.e., the focus is on the structure and parameters of the API requests rather than the specific names used for variables or functions, as long as they serve the same purpose). A heuristic threshold of 0.9 is applied to compare the generated output with the ground truth Endpoint and API call, where the developers are expected to complete the API call based on the model response above this threshold.

## 5 Experimental Results

### 5.1 Fine-tuned CodeLlama Excels in API call Generation

Table 3 presents the evaluation results for the four models fine-tuned with 20,000 Python instances from API Pack. The fine-tuned CodeLlama-13b model demonstrates superior performance in API call generation for the 3-shot retrieved setting, achieving the highest API call accuracy across all evaluation levels. Models fine-tuned with the 3-shot-tuning template consistently outperform those fine-tuned with the 0-shot-tuning template, showing the importance of using the 3-shot template to improve the model’s in-context learning abilities during training. Furthermore, prompting with 3-shot (retre) at testing time leads to substantial improvements compared to 0-shot, a trend observed across all models and levels. This finding highlights the significance of providing relevant examples to improve the models’ accuracy in generating API calls. Notably, the fine-tuned CodeLlama-13b model with API Pack surpasses the performance of GPT-3.5 and GPT-4 models (not fine-tuned) for Level 3, both 0-shot and 3-shot (retre).

**Table 3: Evaluation for models fine-tuned with 20k Python API dataset, including a comparison with (not-fine-tuned) GPT-3.5 and GPT 4. Note that 3-shot (retre) is short for 3-shot retrieved.**
| Model | Fine-tuning | Testing | Evaluation Accuracy (%) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| template | Level 1 | Level 2 | Level 3 |  |  |  |  |  |
| Intent | API Call | Intent | API Call | Intent | API Call |  |  |  |
| Mistral-7b | 0-shot | 0-shot | 17.2 | 10.9 | 14.1 | 11.4 | 14.3 | 11.2 |
| 3-shot (retre) | 42.0 | 29.7 | 35.4 | 28.7 | 39.1 | 29.1 |  |  |
| 3-shot | 0-shot | 40.5 | 28.5 | 24.0 | 18.3 | 15.2 | 12.1 |  |
| 3-shot (retre) | 64.1 | 55.4 | 49.1 | 42.8 | 50.8 | 42.5 |  |  |
| CodeLlama-7b | 0-shot | 0-shot | 8.1 | 6.1 | 10.0 | 7.0 | 11.0 | 7.8 |
| 3-shot (retre) | 52.6 | 42.6 | 43.6 | 35.9 | 50.2 | 40.1 |  |  |
| 3-shot | 0-shot | 12.1 | 9.3 | 13.7 | 10.2 | 16.8 | 13.0 |  |
| 3-shot (retre) | 60.6 | 52.7 | 54.1 | 47.3 | 55.9 | 49.1 |  |  |
| Llama-2-13b | 0-shot | 0-shot | 9.4 | 6.2 | 11.6 | 9.0 | 10.9 | 8.4 |
| 3-shot (retre) | 44.5 | 33.9 | 45.4 | 35.6 | 46.7 | 39.1 |  |  |
| 3-shot | 0-shot | 15.7 | 10.2 | 14.0 | 11.2 | 11.7 | 9.6 |  |
| 3-shot (retre) | 59.5 | 51.5 | 50.8 | 44.3 | 52.7 | 44.2 |  |  |
| CodeLlama-13b | 0-shot | 0-shot | 9.8 | 6.8 | 10.8 | 8.1 | 12.1 | 8.5 |
| 3-shot (retre) | 55.6 | 44.4 | 50.6 | 43.3 | 52.3 | 44.1 |  |  |
| 3-shot | 0-shot | 14.4 | 10.3 | 15.9 | 13.3 | 14.2 | 8.9 |  |
| 3-shot (retre) | 63.5 | 55.5 | 56.8 | 51.4 | 56.1 | 49.5 |  |  |
| gpt-3.5-1106 | none | 0-shot | - | - | - | - | 1.0 | 0.7 |
| none | 3-shot (retre) | - | - | - | - | 47.2 | 39.5 |  |
| gpt-4-1106 | none | 0-shot | - | - | - | - | 0.2 | 0.1 |
| none | 3-shot (retre) | - | - | - | - | 53.5 | 44.3 |  |

### 5.2 Retrieval Augmentation Improves API Call Generation

Table 4 illustrates the impact of different retrieval methods on 3-shot API call generation. The results demonstrate that at inference time, 3-shot (retre) is the most effective approach for fine-tuned Mistral-7b and CodeLlama-13b models compared to 3-shot (rand) and 3-shot (retre & rerank). These models were fine-tuned using 20,000 Python instances with the 3-shot-tuning template.

**Table 4: Performance comparison for different retrieval methods in 3-shot prompting. Note that 3-shot (retre & rerank) is short for 3-shot retrieved & re-ranked.**
| Model | Testing | Evaluation Accuracy (%) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Level 1 | Level 2 | Level 3 |  |  |  |  |  |
| Endpoint | API Call | Endpoint | API Call | Endpoint | API Call |  |  |
| Mistral-7b | 3-shot (rand) | 54.5 | 41.8 | 48.2 | 41.2 | 45.2 | 37.0 |
| 3-shot (retre) | 64.1 | 55.4 | 49.1 | 42.8 | 50.8 | 42.5 |  |
| 3-shot (retre & rerank) | 63.0 | 53.6 | 49.0 | 42.2 | 51.5 | 43.9 |  |
| CodeLlama-13b | 3-shot (rand) | 49.2 | 38.6 | 49.8 | 43.6 | 50.0 | 41.4 |
| 3-shot (retre) | 63.5 | 55.5 | 56.8 | 51.4 | 56.1 | 49.5 |  |
| 3-shot (retre & rerank) | 61.0 | 52.9 | 55.1 | 49.2 | 55.9 | 49.3 |  |

### 5.3 Cross-lingual Generalizations in API Call Performance

Figure 2 compares three fine-tuning approaches for cross-lingual API call performance: a model fine-tuned exclusively on 100,000 instances of ‘cURL’ data, or cURL Model, three expert models, each fine-tuned on 100,000 samples of ‘cURL’, ‘Python’, and ‘Java’ data separately, and a mixture model
fine-tuned on 100,000 instances of ‘cURL’ data with additional samples of 1,000 instances each for nine different languages.

The graph shows that generalizing to new programming languages without fine-tuning is challenging in 0-shot scenarios. However, the models perform better than expected in 3-shot settings, indicating some in-context learning adaptability. Mixed-language fine-tuning improves performance in both 0-shot and 3-shot scenarios, suggesting that even a small amount of fine-tuning data in various languages can improve the model’s effectiveness in cross-lingual API call tasks. These findings demonstrate that cross-lingual API call generation can be achieved by fine-tuning models on a large amount of data in one language plus small amounts of data from other languages, without requiring an extensive dataset for each target programming language. Figure 3 illustrates the models’ adaptability across ten programming languages, with the Mixture Model performing on par with Expert Models in their perspective programming languages in 3-shot testing, highlighting their potential in multi-lingual programming applications.

Figure: Figure 2: Cross-lingual API call performance for different models in cURL, Python, and Java.
Refer to caption: extracted/5641407/images/multi-ling-0-shot-apicall.png

Figure: Figure 3: Three-shot performance in ten languages.
Refer to caption: extracted/5641407/images/10-ling-3-shot-apicall.png

### 5.4 Scaling Instruction Dataset Helps Generalization

Figure: Figure 4: Scaling instruction dataset on CodeLlama-13b with 0-shot and 3-shot retrieval evaluations. The x-axis represents the size of fine-tuning data from API Pack. The y-axis is Endpoint or API call accuracy.
Refer to caption: extracted/5641407/images/scaling-0-shot-endpoint.png

Fig. 4 clearly demonstrates an upward trend in 0-shot performance that correlates with the fine-tuning dataset size (20k, 40k, 80k, 100k, and 1 million instances from API Pack). This improvement highlights the advantage of larger datasets in providing diverse examples, which is crucial in 0-shot prompting where the model relies solely on its pre-existing knowledge. For 3-shot prompting, the graph also shows improvements as the dataset size increases, although less prominent than in the 0-shot case, particularly for level 3 cases. This suggests that while additional fine-tuning data is beneficial, model pretraining plays a more critical role in in-context learning ability.

### 5.5 Additional Ablation Experiments

**Table 5: Performance comparison for ToolBench and API Pack in different shot settings.**
| Train | Testing | Evaluation Accuracy |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Level 1 Endpoint | Level 1 API Call | Level 2 Endpoint | Level 2 API Call | Level 3 Endpoint | Level 3 API Call |  |  |
| ToolBench | 0-shot | 5.7 | 5.7 | 8.1 | 8.0 | 7.3 | 7.0 |
| 3-shot | 44.5 | 37.8 | 40.7 | 36.4 | 43.7 | 38.1 |  |
| API Pack | 0-shot | 14.4 | 10.3 | 15.9 | 13.3 | 14.2 | 8.9 |
| 3-shot | 63.5 | 55.5 | 56.8 | 51.4 | 56.1 | 49.1 |  |

To demonstrate the benefit of API Pack compared to existing API datasets in the literature,
we evaluated a fine-tuned CodeLlama-13b model on 20k instances of ToolBench dataset, which only contains APIs from RapidAPI. Results in Table 5 show that using APIs from a single source like in ToolBench significantly decreases model performance in both 0-shot and 3-shot settings compared to API Pack, which curates data from four different sources.

**Table 6: Performance comparison for none-filtered and with-filtered datasets in different shot settings.**
| Train | Testing | Evaluation Accuracy |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Level 1 Endpoint | Level 1 API Call | Level 2 Endpoint | Level 2 API Call | Level 3 Endpoint | Level 3 API Call |  |  |
| none-filtered | 0-shot | 10.3 | 8.3 | 12.8 | 10.3 | 12.2 | 8.4 |
| 3-shot | 62.4 | 54.9 | 54.9 | 48.7 | 55.7 | 49.5 |  |
| with-filtered | 0-shot | 14.4 | 10.3 | 15.9 | 13.3 | 14.2 | 8.9 |
| 3-shot | 63.5 | 55.5 | 56.8 | 51.4 | 56.1 | 49.1 |  |

Furthermore, to demonstrate the effectiveness of our data filtering pipeline, we replaced 45% of the filtered instructions with non-filtered ones and fine-tuned CodeLlama-13b with 20k instances of each. The results in Table 6 confirm that the 0-shot and 3-shot performances indeed drop with the introduction of non-filtered instructions.

## 6 Conclusion

In this paper, we introduced API Pack, a multi-programming language dataset containing over 1 million instruction-API call pairs. Our experiments demonstrated that fine-tuning CodeLlama-13B on 20,000 Python instances from API Pack enables it to outperform GPT-3.5 and GPT-4 in generating unseen API calls. We also showed that cross-lingual API call generation is achievable by primarily training on single-language data augmented with a small amount of data from other languages. Furthermore, we found that increasing the training data volume improves the model’s ability to generalize to unseen API calls. These findings highlight the potential of large-scale, diverse datasets in enabling LLMs to effectively identify appropriate API endpoints and generate corresponding API calls based on natural language instructions.

## 7 Limitations and Future Work

The first release of API Pack exhibits a set of core limitations. First, the fact API Pack was not designed for multi-API call scenarios, limits its application to single-API development tasks. Second, current instruction-style requires of the API Name. Third, programming language coverage is inherently limited by the library that we used to generate code. Finally, only few OAS files contained license information. We plan to address these limitations in future versions of the dataset, by integrating multi-API call scenarios and instructions that simply require a task description. We also plan to explore other methods to generate API calls so that we can further increase programming language diversity, and to track down complete and accurate license information for all APIs included in API Pack.

## 8 Broader Impact Statement

Our research has a broader impact on Software Productivity as it aims to accelerate software development workflows by automating routine coding tasks. Alongside promising productivity gains, the integration of advanced LLMs into software development workflows raises sociotechnical concerns (e.g., job displacement, developers’ continuous oversight to ensure accuracy). Therefore, a responsible innovation lens assessing and responding to potential harms remains vital.

## Acknowledgments and Disclosure of Funding

This work was supported by MIT-IBM Watson AI Lab and IBM Research.

## 9 Appendix

### 9.1 Ethical Considerations

Developers and other kind of end-users that plan to use API Pack for LLM fine-tuning should consider the following factors:

- •
Hallucination: API Pack’s code (API calls) was not generated via an LLM. Thus, there is no risk of code errors in the dataset due to LLMs hallucination. That said, hallucination is a general risk associated with the use of LLMs. Therefore, code generated with LLMs that were fine-tuned with API Pack or any other code dataset must be always verified by a developer.
- •
Data Ownership and License Information: The OAS files used to build API Pack come from public and official API data hubs. API data owners published these files on those websites themselves. That said, if a legitimate API code owner requires their data to be removed from API Pack we will do so. Regarding API’s license, only few OAS files contained this information. We are committed to tracking down complete and accurate license information for all APIs included in API Pack in a future version of the dataset.
- •
Sensitive Information: There is no risk of sensitive information being released as API Pack code data was sourced from API documentation that does not include real argument values. Instead, placeholders indicate the need for an argument value (i.e., REPLACE_BASIC_AUTH). In a real application, the users can provide these values through a conversational AI interface (i.e., code assistant).

### 9.2 Data Filtering at each Stage of the Pipeline

Table 7 shows the number of instances from each API source at different stages of our data filtering pipeline. Detailed description for each stage can be found in Section 3.

**Table 7: Data Filtering Progress**
| Source/Instances | Before Data Validation | After Removing Invalid API calls | After Removing Instances without Good Instructions |
| --- | --- | --- | --- |
| IBM API Hub | 27,635 | 17,712 | 17,206 |
| APIs Gurus | 500,160 | 499,250 | 495,533 |
| Swaggerhub | 351,756 | 351,756 | 345,765 |
| RapidAPI | 274,014 | 273,388 | 270,095 |
| Total | 1,153,565 | 1,142,106 | 1,128,599 |

### 9.3 Data Spliting for Level 1, Level 2, and Level 3 Testing

To split the data, we first copy the total dataset into total_training and calculate the occurrence of each API. For the first level of testing data, select up to 1,000 data points from the first 20,000 entries where instruction_test is not "None" and the API appears more than four times. For the second level, iterate through the first 20,000 entries, selecting data points with APIs not yet selected, until 1,000 unique APIs are collected, then remove these from the training dataset. For the third level, from the 80,000th entry onwards, select data points with APIs not used in Level 2, collecting up to 1,000 data points, and remove data points with these APIs from the training dataset.

### 9.4 Hyperparameters for Training

We fine-tune the models using the HuggingFace Transformers library on a cluster consisting of 1 node with 8 NVIDIA H100 80GB GPUs, with Fully Shared Data Parallelism (FSDP). Techniques such as mixed precision, gradient checkpointing, and AdaFactor optimizer are used to improve training efficiency. The key hyperparameters are summarized in Table 8.

**Table 8: Hyperparameters for Training**
| Hyperparameter Name | Value |
| --- | --- |
| Learning rate | $2\times 10^{-5}$ |
| Batch size | 128 |
| Max seq length | 4096 |
| Number of epochs | 2 |
| Warmup ratio | 0.03 |

### 9.5 Testing Pipeline

Figure 5 shows an example of our testing pipeline including zero and few-shot.

Figure: Figure 5: Testing pipeline.
Refer to caption: x2.png

### 9.6 API DB instance

Figure 6 shows and the structure of an API DB instance. We used openapi-snippet (^7^77https://www.npmjs.com/package/openapi-snippet), an open-source package that takes as input an OpenAPI v2.0 or v3.0.x specification file (OAS file) and translates it into an HTTP Archive 1.2 request object, to generate API calls. We generated API calls (api_call) in 10 different programming languages (cURL, libcurl, java, node, python, go, ruby, php, swift, JavaScript) for RapidAPI, API Gurus, and the Swaggerhub. For the IBM Developer Hub API calls were extracted directly from the OAS files. We extracted API calls in eight different programming languages from this source (cURL, java, node, python, go, ruby, php, swift).

Figure: Figure 6: Structure of an API DB instance

### 9.7 Improving Code Models with API Pack

We also evaluated the performance of mixing a subset of 50,000-entries of API Pack with Magicoder dataset and fine-tuned CodeLlama-13b model. The resulting model shows an increase of over 35.3% in API call code generation accuracy for Level 3, specifically with the 3-shot setting. This improvement does not come at the expense of general coding efficiency, as the resulting model still performs well on benchmarks such as HumanEval+ and MBPP. See Table 9 for further details.

**Table 9: Evaluation for Code Generation with CodeLlama-13b**
| Data Mixture | Bench. (pass@10) | Level 3 (3-shot) |  |
| --- | --- | --- | --- |
| HumanEval+ | MBPP | Endpoint |  |
| - | 47.8 | 58.3 | - |
| Magicoder | 60.8 | 66.4 | 17.0 |
| Magicoder + API Pack | 61.3 | 64.3 | 52.3 |

### 9.8 API Pack Dataset Instance Example

Figure 7 shows a fragment of an API Pack data instance. As data is sourced from public OAS files, personally identifiable and sensitive information in API calls is represented by anonymous strings (e.g., REPLACE_BASIC_AUTH), which indicates the need to pass this value as an argument in real execution. In this example, we only show one instruction for clarity, we represent the place for the other 4 instructions generated for this instance with the symbol … .

Figure: Figure 7: Fragment of an API Pack instance in cURL.

### 9.9 Instruction Candidates

Figure 8 shows all the candidates generated for an instance with their respective id and input_tokens_mean.

Figure: Figure 8: Examples of five instruction candidates generated for the same instance.

### 9.10 Prompts

In this section, we share all the prompts used to create instructions. We also explain the process to craft each prompt. When in use, the values within the curly brackets would be replaced by the actual data.

#### 9.10.1 Examples Refinement

We use Prompt 1 to refine the examples for instruction generation. We created three examples per API. Examples were manually refined by a human for the APIs from IBM API Hub and APIs Gurus. Then, we tasked an LLM with refining the examples for the other two sources.

Figure: Listing 1: Prompt for instruction example refinement

#### 9.10.2 Instruction Generation

We use Prompt 2 to generate five instruction candidates for each API Pack instance. We crafted a custom prompt that considers the information a developer may provide to an LLM, while looking for an API call to complete a task.

Figure: Listing 2: Prompt for instruction generation

#### 9.10.3 Back Translation

Prompt 3 seeks to re-generate the API information associated to an instruction by passing an instruction candidate as input. To create this prompt, we took Prompt 2 as reference and simply reversed the task. As a result of using this prompt for inference, we save the input-token-mean of the output return (the API information that was re-generated).

Figure: Listing 3: Prompt for instruction backtranslation

#### 9.10.4 Instruction Annotation

We use Prompt 4 to task an LLM with annotating the instruction candidates as good or bad. First, a human annotator manually reviewed 605 instructions to distill common error patterns among them. Then, we craft a prompt to detect these error patterns via an LLM. Note the prompt includes examples for ’good’ and ’bad’ instructions. These examples were extracted from the manual revision performed by the human annotator.

Figure: Listing 4: Prompt for instruction annotation

#### 9.10.5 Few-shot Fine-tuning and Evaluation

This is the prompt template used for 3-shot-tuning to post-process API Pack data, as well as for 3-shot random, 3-shot retrieved, and 3-shot retrieved & re-ranked inference.

Figure: Listing 5: Prompt for few-shot fine-tuning and evaluation

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: See notes above. I have translated some of the above in question format below:

1. What is the baseline performance of non-fine-tuned models on API Pack?
2. How is multi-lingual generalization being measured in Section 5.3? Is it the same API invocation across different languages?
3. Have you considered training on API Pack and evaluating on other benchmarks like ToolBench and Gorilla, and vice versa, to more thoroughly compare API Pack with existing datasets?
4. What is the performance with an oracle retriever/re-ranker in the context of the simple retrieval discussion?
5. Can you characterize if there are any data leakage concerns and/or what researchers should be mindful of when evaluating with API Pack?
soundness: 4
strengths: - **Useful contribution.** The paper presents API Pack, which is a large-scale and multi-lingual instruction-API calling dataset. This can be used for both training and evaluations in multiple languages. In Table 1, the authors demonstrate how this is better along multiple dimensions than previous works.
- **Several detailed insights.** The paper is packed with insightful experiments. I particularly liked the experiments covering multi-lingual generalization — the designs of control experiments for this are interesting, and show that it is sufficient to train on a lot of data from one language and little data from other languages. There are other experiments, including ablations on RAG strategies and data volume used during training, which will also be valuable to the community.
- **Reproducibility.** Section 3, discussing the creation of API Pack, is extremely detailed and will be of great value to researchers looking to process data for similar tasks.
summary: The paper introduces API Pack, a large-scale dataset with more than a million instruction-API call samples. The authors discuss the aspects of API Pack making it unique and how it was constructed from public sources. Experiments are conducted to demonstrate: (1) Code LLM performance on API Pack under various RAG and evaluation settings (2) strong multi-lingual generalization on API Pack tasks from training a lot on one language and a little bit on others, (3) data-scaling properties hold when training on API Pack data, (4) comparisons with another training dataset, ToolBench.
weaknesses: - **Lack of certain details.**
    - The baseline performance of non fine-tuned models on API Pack is missing (unless I missed it somewhere). Including this will greatly speak to the advantages of fine-tuning on API Pack.
    - For Section 5.3, how is the multi-lingual generalization being measured? Is it the same API invocation across different languages?
    - Could API Pack suffer from data leakage when evaluating public models trained on public sources?
- **Additional experiments.** While there are several experiments conducted, I am curious about a few more experiments. This is not a major concern however. While Section 5.5 does some comparison of API Pack with ToolBench, I believe that a more thorough evaluation requires training on API Pack and evaluation on ToolBench, Gorilla, etc. and also training on ToolBench, Gorilla, etc. and evaluating on API Pack. This would concretely identify where API Pack stands out and lacks relative to other API calling benchmarks out there.
- **Paper writing.** Overall the paper seems to be a bit verbose, and could benefit from making the writing more concise and crisp. For e.g., Section 5.3 is too wordy and a jumble of multiple findings. It would benefit the paper to organize such findings in a clearer way.
- **Evaluation metrics.** While string matching based metrics are being used for evaluations here and have been used in several previous works as well, I think it is time that we move on from these inaccurate metrics. We should resort to more robust metrics relying on static or execution analysis such as pass rate [1] or hallucination rate [2] for measuring correctness of code completions.
- **Nitpicks.**
    - Line 381 - “simple retrieval might be sufficient in this context” — I think it can also be the case that using a better re-ranker might help improve performance. For e.g., what’s the performance with the oracle retriever/re-ranker?
    - Figure 2
        - The legend(s) should be moved to make the results more readable
        - Is the “Expert” model in each subplot different? This should be clarified in the text/caption given all models have the same color
    - Figure 4 - it would be better for comparison if all the subplots had the same y-axis limits

[1] Chen, Mark, et al. "Evaluating large language models trained on code." *arXiv preprint arXiv:2107.03374* (2021).

[2] Jain, Nihal, et al. "On Mitigating Code LLM Hallucinations with API Documentation." *arXiv preprint arXiv:2407.09726* (2024).

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: How do the authors view the trade-offs between a RAG approach (with minimal or no fine-tuning) and the approach proposed here of fine-tuning on a large, static API dataset? Since APIs are continually updated, it seems that a RAG or search engine approach could offer more reliable, dynamic knowledge. How does the static nature of API Pack compare in reliability and practical utility?
soundness: 3
strengths: - The focused problem is quite interesting. This paper fills a gap in LLM fine-tuning resources for cross-language API call generation, which has been under-explored.
- The dataset is robust, sourced from multiple reputable platforms, and undergoes a thorough pre-processing and validation pipeline. The authors’ experiments, conducted with open-source and proprietary models, validate API Pack’s effectiveness across multiple levels of generalization (known APIs, new endpoints, unseen APIs).
- The paper is clearly structured, detailing each stage of dataset creation, the methodology for instruction generation, and the evaluation framework used to measure LLM performance across scenarios. This transparency supports reproducibility and makes the dataset accessible for further research.
summary: The paper introduces "API Pack," a large, multi-programming language dataset designed to enhance the API call generation capabilities of large language models (LLMs). API Pack aims to help LLMs generate API calls based on natural language instructions, addressing a key developer challenge of navigating and generating code from extensive API documentation. Experiments show that API Pack fine-tuning improves LLM performance in unseen API calls, enabling open-source models like CodeLlama-13B to outperform proprietary models GPT-3.5 and GPT-4.
weaknesses: - [Major] Insufficient Related Work: The paper lacks discussion on some relevant studies in API learning. Notable examples include:

    - Zan, Daoguang, et al. "When language model meets private library." arXiv preprint arXiv:2210.17236 (2022).
    - Zhang, Kechi, et al. "Toolcoder: Teach code generation models to use API search tools." arXiv preprint arXiv:2305.04032 (2023).

- [Major] API Dataset Validity: Given the vast scale of the dataset, it’s unclear if all included APIs are usable or up-to-date. Additionally, APIs frequently evolve; fine-tuning on static API datasets could introduce outdated information or "hallucinations," potentially compromising the reliability of API knowledge encoded in the model. The authors should discuss this question.

- [Major] Limited Response Diversity: The paper does not sufficiently discuss the diversity of the code response. Using tools like OpenAPI may lead to highly uniform code outputs, such as the preference for a small set of common libraries for HTTP requests in Python or other languages, potentially constraining the model's generalization capabilities. This limitation might affect the model’s ability to use a broader range of libraries or to produce varied code styles when calling APIs. The authors should add experiments to show the influence of the response diversity.

- [Major] Lack of Baseline Comparison: A critical missing baseline is a model augmented with RAG or a search engine approach (e.g., ToolCoder), **without fine-tuning**, for handling complex API calls. This would provide a practical comparison for API Pack’s utility.

- [Minor] Base Model Selection: The selection of base models seems somewhat detached from practical application. API Pack functions as a specialized fine-tuning stage post-general SFT, targeting API call generation specifically. Therefore, selecting foundational SFT code models, like CodeLlama-Instruct or Magicoder, as discussed in Appendix A.8, would enhance the practical relevance and clarity of the paper’s contributions. Including more experiments with these models would substantiate the paper's claims.

- [Minor] Evaluation on General Code Generation Benchmarks: The paper does not discuss potential **performance trade-offs on general code generation benchmarks**, such as HumanEval, that could drop due to fine-tuning for API-specific tasks.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: * Do the authors have more intuition behind the 0-shot vs 3-shot template results after fine-tuning? The findings seem inconsistent across Levels and data sizes (level-1 0-shot is better than 3-shot but for levels 2/3 3-shot is better in Figure 4)
* Can the authors clarify the metrics used to describe it more formally?
* Did the authors attempt to measure the "naturalness" of the collected instructions (automatically or through human annotation) or evaluate models on human written problems for a subset of the APIs?
soundness: 3
strengths: * **Extensive Dataset with some human-annotation/filtering steps.** The collected dataset sizing over a million instances in unique and large covering a wide array of APIs and programming languages.
* **Experimental insights.** The paper performs API data scaling experiments, retrieval experiments and cross-language experiments.
  * The data scaling experiment highlights interesting observations across the three levels. Particularly, flat curve for Level-3 APIs does highlight a broader concern of out-of-distribution generalization for LLMs
  * The mixture model shows generalization to a new language with few examples solidifying the advantage of the collected large-scale dataset.
summary: The paper introduces API Pack, a large-scale, multi-programming language dataset containing over one million instruction-API call pairs generated semi-synthetically from API database collected from four API endpoints. This dataset is used for training LLMs on instruction API pairs and to demonstrate improvements in unseen code generation, cross-language transference.
weaknesses: * **Novelty.** The paper provides limited novelty. It thoughtfully curates API sets and uses LLMs to generate synthetic training and test sets via prompting (with notably some human-in-the-loop quality assurance rounds). While potentially performed carefully, this following standard set of ideas in a somewhat different domain. Note that this is not necessarily a strict limitation and a carefully done execution can be helpful to the community -- perhaps primarily identified over time via the community. 

* **In-distribution evaluation.** The paper collects a synthetically generated dataset -- thus comprising instructions of limited natural language variance (compared with human written instructions). This dataset is next divided into a train and a test set. Thus the training and test set are precisely collected from the same and a small distribution. This partly challenges the findings from and we might see smaller gains to more "diverse" real-world queries. 

* **Marginal benefits from quality filtering.** The data filtering ablation reports performance improvement but it is notably small, often <1%. This raises concerns in the soundness of this step.

Minor: 

* **0-shot training does not improve 0-shot retre.** For the 20k fine-tuning experiments, the accuracy of 0-shot retrieval after 0-shot fine-tuning is worse than performance after fine-tuning on 3-shots. This is quite surprising -- and continues in Figure 4 as well and could be expanded further, perhaps with a qualitative analysis of the mistakes.

* **Metric.** The metrics used in the paper are not completely clear and should be more formally defined or described with examples or relevant citations.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Model Selection for Quality Assessment: In the 3.4 section, the authors use the Mistral 7B model to validate the generated instruction, which is also the model used to generate the instructions. This raises concerns about the validity of using the same model to evaluate its outputs. It is unclear whether the 7B model can accurately distinguish between high-quality and low-quality instructions. A more robust approach might involve using a larger (>7B), more powerful LLM to assess the quality of the instructions.
2. Definition of Unseen APIs: The term "new API" refers to APIs that the model has not encountered during fine-tuning. However, given the lack of information about the models' pretraining data, it is challenging to verify whether the APIs labeled as "unseen" truly are. This lack of clarity could affect the validity of the experimental results. 
3. Evaluation Metrics for Code Generation: The authors use text-matching-based metrics for evaluating the generated API calls. However, for code generation tasks, metrics based on test case execution are more accurate and reliable. These metrics can provide a more objective measure of the correctness and functionality of the generated code. The authors should consider incorporating such metrics to enhance the evaluation process and provide a more comprehensive assessment of the models' performance.
soundness: 3
strengths: 1. New Data: The authors introduce API Pack, a dataset containing over 1 million instruction-API call pairs. The dataset's multi-programming language nature and rich source diversity enable it to cover various scenarios. It will be useful for further research on improving the API call code generation of LLMs in different scenarios.
2. Performance Enhancement: The paper demonstrates the effectiveness of fine-tuning on API Pack by evaluating multiple models, including CodeLlama-13B, Mistral-7b, and Llama-2-13b. The results show that fine-tuning on API Pack significantly improves the models' ability to generate API call code, highlighting the dataset's value in enhancing model capabilities.
3. Extensive Experiments: The authors have conducted extensive experiments to validate their claims. The thoroughness of these experiments provides strong evidence for the dataset's effectiveness and the benefits of fine-tuning on it.
summary: This paper introduces API Pack, a comprehensive multi-programming language dataset comprising over 1 million instruction-API call pairs designed to enhance the API call generation capabilities of large language models. By fine-tuning CodeLlama-13B on 20,000 Python instances from API Pack, the authors demonstrate that it outperforms GPT-3.5 and GPT-4 in generating unseen API calls. The fine-tuning process also facilitates cross-programming language generalization by leveraging a large volume of data in one language and smaller amounts from other languages. Scaling the training data to 1 million instances further improves the model's generalization of unseen APIs.

Contributions:

1. Introduction of API Pack Dataset: The authors present API Pack, a massive multi-programming language dataset containing over 1 million instruction-API call pairs, which is a significant contribution to the field of API call generation.
2. Model Performance Improvement: Through fine-tuning CodeLlama-13B on a subset of API Pack, the authors demonstrate superior performance in generating unseen API calls compared to GPT-3.5 and GPT-4.
3. Cross-Programming Language Generalization: The study highlights the dataset's ability to facilitate cross-programming language generalization, leveraging large amounts of data in one language and smaller amounts from others.
4. Scalability and Generalization: Scaling the training data to 1 million instances enhances the model's generalization capabilities to unseen APIs not used in training, showcasing the dataset's scalability.
5. Open-Source Initiative: The authors contribute to the research community by open-sourcing the API Pack dataset, the fine-tuned model, and the associated source code, fostering further research and development.
weaknesses: 1. Clarification Needed for Methodology in Section 3.4: The description of the method used to filter high-quality instructions in Section 3.4 is unclear. 
2. Lack of Results for GPT-3.5 and GPT-4: The experimental section does not show results of GPT-3.5 and GPT-4 at Level 1 and Level 2.

Meta-review: The paper presents API Pack, a large-scale dataset comprising over one million instruction-API call samples. The authors highlight the unique aspects of API Pack and detail its construction from publicly available sources. Experiments demonstrate several key findings: (1) the performance of code LLMs on API Pack under various retrieval-augmented generation (RAG) and evaluation settings, (2) strong multilingual generalization on API Pack tasks achieved by training extensively in one language while using limited data from others, (3) the retention of data-scaling properties when training on API Pack data, and (4) comparisons with another training dataset, ToolBench.

The reviewers generally agree that this is a strong paper; however, they raised some concerns regarding the comparisons with related work and the validity of the benchmark. The authors should carefully revise the paper in accordance with the reviewers' comments.

Decision: Accept (Poster)