API Pack: A Massive Multi-Programming Language Dataset for API Call Generation
Authors: Zhen Guo
Adriana Meza Soria
Wei Sun
Yikang Shen
Rameswar Panda
Venue: ICLR 2025 Poster (2025)
Keywords: Instruction Datasets
Tool Datasets
API Datasets
Code Datasets
Synthetic Data Generation
Primary area: datasets and benchmarks
TL;DR: API Pack, a 1M multi-programming language dataset to improve API call generation capabilities of LLMs.

Abstract: We introduce API Pack, a massive multi-programming language dataset containing over one million instruction-API calls for improving the API call generation capabilities of large language models. Our evaluation highlights three key findings: First, fine-tuning on API Pack enables open-source models to outperform GPT-3.5 and GPT-4 in generating code for entirely new API calls. We show this by fine-tuning CodeLlama-13B on 20,000 Python instances from API Pack. Second, fine-tuning on a large dataset in one language, combined with smaller datasets from others, improves API generation accuracy across multiple languages. Third, we confirm the benefits of larger datasets for API generalization, as increasing fine-tuning data to one million instances enhances generalization to new APIs. To support further research, we open-source the API Pack dataset, trained model, and code at https://github.com/zguo0525/API-Pack.

Published as a conference paper at ICLR 2025
API PACK: A MASSIVE MULTI-PROGRAMMING LAN- GUAGE DATASET FOR API CALL GENERATION
Zhen Guo
Adriana Meza Soria
Wei Sun IBM Research sunw@us.. i
MIT EECS
MIT-IBM Watson Al Lab rdr ana meza. sor a @ bm com
zguo0525emit..o t.. edu
Yikang Shen
Rameswar Panda
MIT-IBM Watson Al Lab yikang. shen ibm. com com
MIT-IBM Watson Al Lab rpanda ibm. com
ABSTRACT
We introduce API Pack, a massive multi-progran programming language dataset contain- ing over one million instruction-API calls for improving the API call generation capabilities of large language models. Our evaluation highlights three key find- ings: First, fine-tuning GPT-3.5 and GPT-4 in generating code for entirely new API calls. We show this by fine-tuning CodeLlama- 13B on 20,000 Python instances from API Pack. Second, fine-tuning on large dataset in one language, combined with smaller datasets from others, improves API generation accuracy across multiple languages. Third, we confirm the benefits of larger datasets for API generalization, as increasing fine-tuning data to one million instances enhances generalization to new APIs. To support further research, we open-source the API Pack dataset, trained model, and code at http:://w..aaocccooooooooo//////
API Pack enables open-source models outperform
com
INTRODUCTION
Large language models (LLMs) have shown promise in assisting in software engineering tasks (Li
ct al. 2023a Ozkaya 2023 Fan et al. 2023 Belzner al.) 2023 Ross et al. 2023 Hou al.
2023
Ebert & Louridas 2023: Chang et al. 2024; Wang et al. 2024). with a primary focus on
code generation (Chen al. 202 Xu al. [2022] [Sarsa et al. 2022 Vaithilingam et al. 2022
Poesia et ||222022]aa] et al. [003bb] Shrivasttaaa et al. 2023a]] Liang et al. 2002 Zan et al. 2023
Liu al. [2023a] Shrivastava al. 2023b; Wei al.] 2023] Muennighoff
al.. 2023 Li al.
2023c Huang et al.] 2023 Lozhkov er al.) 2024 Thakur et al. 2024] Mishra et al. 2002).. While
these advances are significant, developers still face a time-consuiing challenge: manually finding
and adapting API code examples from lengthy documentation (Meng et al. 2018). Although prior
research has addressed API intent detection (finding the right API for task), the more complex challenge of generating specific API call code has received limited attention, particularly given the varying syntax of HTTP protocol elements across programming languages.
To achieve this goal, we create API Pack, dataset designed to improve the API call generation capabilities of LLMss. With over million instances spanning programming languages, API Pack represents the largest open-source instruction dataset for API call generation and intent detection (see Table T). Our dataset distinguishes itself from prior work (Zan et al., 2022 Zhang et al.) 2023a Xu al. 2023b Patil et al. 2023: Qin et al. 2023) in two key aspects: programming language diversity and dataset size. By including API calls across multiple languages, API Pack enables the first comprehensive study of cross-language skill transfer in API generation, it examines how improvements in one language generalize to others. Additionally, its extensive collection of real-world API examples allows for the evaluation of generalization capabilities through controlled variations in training data volume.
'These authors contributed equally this work.
WWrr performed while MIT-IBM Watson AT Lab. Currently Apple AIMI..

---

Published as a conference paper at ICLR 2025
Table I: A comparison of API Pack with other instruction datasets for API intent detection and/or API call code generation. The upper section of the table reports the features that each dataset covers, and the bottom section reports the data statistics available means unavailable).
Feature
API Pack (this work)
APIBench (Gorilla)
ToolBench
ToolBench (ToolLLM)
API Bank
ToolAlpaca
ToolFormer
API call intent detection?
API code geacration?
X
Mulii programming language?
(10) X custom
X(Pythou)
/(Curi, Python)
X
Multi API call scenario?
Data generation method
self-imstruct
self-instruct
custom
custom
# of Sources
4
8
1
53
APIs/ /Taols
11,213
16,464
# of API calls
1,128,599
16,450
37,204
568
3,938
9,400
1.128,599
2,746
We summarize three key findings from our experiments:
Fine-tuning CodeLlama- 13B on 20k Python API Pack instances outperforms GPT-3.5 by 10% and GPT-4 by 5% unseen API calls.
2. Cross-language API call generation can be enabled by large amount of data in one programming language plus small amounts of data in others.
3. Increasing fine-tuning data from 0 to million instances improves generalization to new APIs, demonstrating clear benefits larger datasets for API generalization.
This paper is structured follows: We begin with review related work (Section 2). followed by a detailed description of the API Pack dataset construction (Section 3 Our experimental method for model fine-tur tneettnnng and evaluation is presented in Section 4 with key findings discussed in Section 5 We conclude in Section 6 with discussion of limitations and future directions. Additional considerations regarding ethics and broader impact are addressed in Appendices A.. and [.2 respectively. The API Pack dataset is made available under the Creative Commons Attribution 4.0 International License, with accompanying source code released under the MIT License.
RELATED WORK
2.1 METHODS TO GENERATE INSTRUCTION DATA WITH LLMS
The manual creation of instruction datasets is a time-consuming and resource-intensio process Xu ctt al. 2023a). To address this challenge, researchers have developed automated methods leveraging LLMs. Two notable approaches in this domain are Self-Instruct (Wang al. 2023a and Evol- Instruct (Xu et al. 2023a), both focused on generating and filtering Self-Instruct utilizes in-context learning modify seed examples and employs ROUGE-L similarity and heuristies for filtering. In contrast,
pairs.
uses targeted prompting for data generation and
relies on predefined heuristies for filtering.
For code-specific datasets, hybrid LLM--ased pipelines offer an alternative approach. These pipelines typically combine high-quality human code samples with LLM-generated instructions (Patil al. 2023: Xu et al. (2023b). While human verification remains the gold standard Wang et al. (2023a). becomes impractical at scale. Recent work has instead adopted powerful LLMs like GPT-4 and Mistral as automated judges, using example-guide guided heuristics derived from human-reviewwdd reviewed data (Liu
al. 2023c). Similar techniques have been applied evaluate instruction complexity (Chen et
2023
Lu et al. 2023).
2.2
LLMS FOR API CALL CODE GENERATION AND INTENT DETECTION
LLMs for code generation have predominantly focused on general coding tasks, as evaluated by benchmarks like MBPP (Austin al. 2021), HumanEval (Chen al. 202 and their variants (Liu
et al. 20236; Zheng et al. 2023) Peng et al. 2024). Recent work has expanded into two specific API-related domains: API call intent detection and API call code generation.
2

---

Published as a conference paper at ICLR 2025
API call intent detection focuses matching natural language tasks appropriate API endpoints. LLMs designed for this purpose (Zan et al. 2022 Zhang et al. 2023a Qin et al. 2023 Li et al.
2023b Tang al., 2023 Yang al. [023] Schick et al. 2023) often operate in hybrid architectures, where the L.M identifies the relevant API endpoint(s) while other components handle the code generation. These studies have explored both single-API and multi-APl intent detection. Achieving strong performance in the latter remains a challenge Qin et al. (2023). At present, this is likely the way most function calling LLMs have been trained operate (Srinivasan et al., 2023 Yuan al. 2024 Chen et al. 2024).
Our work, however, focuses on training LLMs for API call code generation, which remains under explored compared to API call intent detection. To the best of our knowledge, only two projects have previously worked on this challenge: Gorilla (Patil al. 2023 which generates API calls load pre-trained machine learning models from known model hubs, and ToolBench (Xu et al. 2023b), which focuses on improving L.M''s tool manipulation capabilities for few applications.
API PACK
API Pack is an instruction dataset with over one million instances. Each instance contains an input-oulput pair plus additional information about the API and respective endpoints. The inputs are instructions for finding an API call to solve coding task, including a task description in software engineering language and the name of the API. The outputs are API call examples, specifically HTTP request code snippets curated from OpenAPI specification (OAS) files. The data in API Pack sourced from four hubs that store OAS files: RapidAP APIGarus Swaggerhut and IBM's public API Hut To the best of our knowledge, these API Hubs were the only sources with publicly available OAS files at large scale the time collected data. From these sources, kept all the APIs that passed our filtering criteria (see Section (. Table Table[[summarizes summarizes the total number of APIs, unique endpoints, and total instances that part of API Pack.
Table 2: Final count of data curated per source, where each instance contains one API call
Source
APIs
Unique Endpoints
Total Instances
RapidAPI
4,115
21,525
270,095
APIs Gurus
1,980
37,097
495,533
SwaggerHub
5,045
26,747
345,765
IBM API Hub
73
2,884
17,206
Total
11,213
88,253
1,128,599
The construction of API Pack involves four main stages: data pre-processing (Section 3.1). API Database (DB) creation (Section 3.2). instruction generation (Section 3.3 and data validation
(Section 3.4). iillustaaee the overall pipeline.
3.1 DATA G
This process involves two main steps. First, filter out OAS files with non-English metadata (less than 1%) and those with zero endpoints, as they cannot be used to generate API calls. Next, we extract key information from the remaining OAS files. the API level, we collect the name, description, and provider. At the endpoint level, we extract the name, functionality, description, method, and path. To ensure data quality, we remove instances missing critical information needed for generating API calls (e.g.. method, path, endpoint name) or instructions (e.g., functionality, description). This step ensures that all dataset instances include the necessary details for generating accurate API call examples and instructions.
httpr/ggg....oooooeeeeee//////aa Patt-Da
2 httpp::/rapidap..co/ccee com/categoriiss
3 "httss///ppis.. guru/
4 *hutps//app..a www.sswww....woweeeeeo...
5 hhtps://deeeloperrimm.cco htwpaaappcccc////////ta/aatiaitt
3

---

Published as a conference paper at ICLR 2025
Generate API calls </>
Rule-based, templete- based, REGEX
1. Instruction examples
Instruction condidates
RepidAPi
2
3
APiGunas
Pre- processing
Create API DBs
Generate Instructions
Data Validation
OperAPi
Remove OKS flles: non non-englsh, zero endpoints
fittering
</>
Extract
</> Remove instances with emply values for créiicll date
Hub
Instruction fittering
Figure I: Dataset curation pipeline.
3.2 CREATE API DATABASES
We construct API database (DB) from the pre-processed data, where each instance contains end- point and API details. Using the OpenAPI Snippe[librarry we generate API calls in 10 programming languages (cURL, libeurl, Java, Node.js, Python, Go, Ruby, PHP, Swift, and JavaScript) for each instance. This process is applied to data from RapidAPI. APIGurus, and OpenAPI. For IBM API Hub, directly extract API calls from OAS files, these already include calls in multiple languages. The selection of programming languages is based on two factors: (1) the 10 languages align with those highlighted in the StarCoder report (Li al.) (023c), and (2) the OpenAPI Snippet library's support for these languages. Appendix A.7 provides details on the API DB structure and language diversity across sources.
3.3 INSTRUCTIONS GENERATION
This process involves creating high-quality instruction examples and generating instruction candidates based on them. First, we randomly select three endpoints from each API DB and use their details (e.g.. functionality, description, endpoint name, path) along with the API name fill predefined instruction templates. This results in three instruction examples per API DB file. For the API Gurus and IBM API Hub sources, three co-authors of this work manually refine the generated instruction examples. The refinement criteria are: 1) correcting grammatical errors, 2) removing unnecessary information, and 3) ensuring the API name is included and correct in the examples. Most annotations were done independently, but annotators discussed non-standard cases to reach consensus. To streamline the refinement process for Swaggerhub and RapidAPI sources, we use an LLM
7B--sruutt vo.27) instead of human effort. We create prompt (see Prompt 2 in Appendix A.11) based manual review heuristics and use it to task the LLM with refining instruction examples for these sources. Next, we use the high-quality instruction examples generate five instruction candidates for each API DB instance. We prompt the LLM with the endpoint's s information and provide the high-quality instruction examples as in-context examples (see Prompt in Appendix [A.1I]. Figure 7 in Appendix A.8 shows an instance with one instruction candidate and its corresponding API call. Appendix A.9 presents all five candidates generated for the same instance.
3.4
DATA VALIDATION
The data validation process involves three main steps: 1) verifying the validity of API calls HTTP request examples in a given programming language. 2) assessing the quality of generated instructions, and 3) selecting the highest-quality instruction for fine-tuning.
First, to verify API calls, we ensure that the endpoint name and HTTP method (e.g., GET, POST) in the API call match the instance data. We use regular expressions to validate the URL format, accounting for placeholders in the domain, path parameters, or query parameters. Additionally,
7 http::/whngiiigftccccccmmi/ii/it
4

---

Published as a conference paper at ICLR 2025
check that the programming language keywords in the API call correspond to the language ID assigned to each instance.
Second, to assess instruction quality, randomly sampled 121 instances, each with API call and five instructions, totaling 605 instructions. co-author with expertise in software development manually classified these instructions good or bad, with second annotator providing feedback. Bad instructions typically: 1) contain multiple instructions instead of one, 2) include unnecessary text, or 3) incorrectly use the API name. Based on these traits, created three prompts with fixed in-context examples to automatically annotate instructions using an LLM. We used the LLM Bestral-7B-Instruuee
to annotate the 605 instructions and compared the results with human annotations. We selected the prompt with the best performance (see Prompt in Appendix (A.II) and used to classify all instructions in our dataset. We then removed instances with fewer than two good instructions.
Third, to select the best instruction candidate from the good ones, we calculated the likelihood that an LLM would regenerate the input text used to create the instruction. We prompted Mistral- IB-Instruct--O.2 with each instruction candidate (see Prompt the log probability of each token for the regenerated input text. We computed the mean log prob- ability iinput.tokens input tokens mean for fine-tuning, Our final dataset includes 1,128,599 instances, each with a valid API call and at least two high-quality instructions. Appendix A.3 details the data instances filtered each pipeline stage.
Appendix (A.IT) and obtained
and selected the instruction annotated as good with the highest
EXPERIMENTS AND EVALUATION FRAMEWORK
To optimize the instruction Instruction-followinng following capabilities the language models, we post-process API Pack into two instruction-tune iattion templates. The O-shot-tuning template targets scenarios where the output expected be straightforward inference from the given input. The 3-shot-tuning template, with random selections within the selected API, emphasizes the model's ability to learn and generate output with in-context learning. The 3-shot-tuning template available Appendix Model generation parameters follow the BigCodeBench (Zhuo et al. 2024).
4.1 EXPERIMENTAL SETTINGS
A. Selecting a base model: Our first experimental study serves the purpose of selecting base model for the rest our experiments. We fine-tune Mistral (Jiang et al. 2023), Code 7b and 13b, as well as Llama 2 13b (Touvron et al.) 2023) on subset of API Pack (20,000 instances in Python programming language). We evaluate the performance of resulting models and select the best-performing base model.
B. Inference with retrieval: Our second experiment aims to understand the influence retrieval augmentation model generalization. We evaluate the models under five distinct prompt settings during test time:
0-shot: No API examples provided for the model.
3-shot random: Three randomly selected API examples.
3-shot retrieved: Three retrieved relevant API examples.
3-shot retrieved & re-ranked: Five retrieved API examples, with three selected using re-ranker model in the final prompt.
3-shot oracle: Three retrieved relevant API examples. Replacing one of the randomly selected retrieved API examples with the oracle example.
The 3-shot random inference uses the same prompt template as 3-shot-tuning, We use bge-large- en-v1.5 (Zhang al. 2023b) the embedding model for retrieval and bge-reranker-large (Xiao) et al. 2023) for re-ranking the API examples. Appendix in6lids more details of the inference pipeline used to evaluate the models' performance.
C. Cross-language generalization: To test the model's ability to generalize to new programming languages, supplement a cURL dataset of 100,000 instances with 1,000 instances from each of the
5

---

Published as a conference paper at ICLR 2025
9 additional languages in API Pack: Go, Java, JavaScript, libeurl, Node.js. PHI, Python, Ruby, and Swift. The goal is to determine if model can generalize to new languages without requiring large amount of multi-proot programming language data. Notably, these instances across different languages invoke the same API functionality, allowing us to evaluate how well models can translate the same API invocation across different programming languages.
D. Scaling experiment: We conduct a scaling experiment to investigate whether more API data improves a model's generalization ability for unseen APIs. We fine-tune models progressively larger API datasets with unique API calls on Ok, 10k, 20k, 40k, 80k, 100k, and million instances. Our hypothesis is that exposure
greater scale and diversity of APIs during fine-tuning will improve the model's 's ability to generalize to new, unseen APIs.
4.2 EVALUATIONS
To measure the generalization capabilities enabled by API Pack, we establish a comprehensive evaluation framework spanning three levels of complexity for API call generation:
Level Seen APIs and endpoints. This level assesses generalization to new instructions.
Level 2: Seen APIs and new endpoints. This level tests generalization to new endpoints of known APIs.
Level Unseen APIs and endpoints. This level validates performance on entirely new APIs.
The process of data splitting is detailed in Appendix A.4 and training hyperparameters are provided in Appendix A.5 evaluate the accuracy of the generated Endpoints and API calls, we use the SequenceMatcchr class from Python's di ffl i library. This algorithm computes a similar- ity ratio between two sequences by finding the longest contiguous matching subsequences while considering certain elements insignificant. The similarity ratio is calculated as:
2M
where M is the total number of matching characters (excluding insignificant elements), and the total number of characters in both sequences (Black 2021). Insignificant elements refer parts of the API requests that do not affect functionality, such as whitespace, formatting differences (e.g.. spaces, tabs, newlines), and variable naming conventions. generated output considered correct if the similarity ratio meets or exceeds heuristic threshold of 0.9.
This evaluation focuses structural and semantic correctness rather than exact syntactic matches. While execution-bart based metrics like pass rate (Chen et al., 202 or hallucination rate (Jain et al.. 2024) could offer more robust evaluation, our dataset prioritizes privacy by removing personally identifiable information (PII) from the API database, making execution based evaluation challenging. We discuss these evaluation limitations and potential future directions in Section
EXPERIMENTAL RESULTS
5.1 FINE-TUNED CODELLAMA EXCELS IN API CALL GENERATION
We conducted a comprehensive evaluation comparing models fine-tuned on 20,000 Python instances from API Pack against GPT-3.5 and GPT-4. The evaluation assessed fine-tuned open-source models across all three difficulty levels, while GPT-3.5 and GPT-4 were evaluated only on Level 3 due to fine-tuning cost constraints. Table presents these results, demonstrating that open- soun-soure LLMs fine-tuned on API Pack can match or exceed the performance of leading proprietary models in generating unseen API calls.
Our analysis reveals three key findings:
1. Advantage of CodeLlama-1311 The fine-tuned CodeLlama- 13b model consistently out- performs other models, achieving the highest API call accuracy across all evaluation levels, particularly in the 3-shot retrieved setting. For Level 3, it achieves 49.5% accuracy, compared to 42.5% 5% for Mistral-7b and 44.2% for Llama-2-13b.

---

Published as a conference paper at ICLR 2025
2. 3-shot vs. 0-shot Training: Models fine-tuned with the 3-shot-tuming template consistently outperform those trained with the O-shot-tuning template. For example, CodeLLama- 13b's Level 3 accuracy improves from 44.1% 1% (O-shot training) to 49.5% (3-shot training) when using 3-shot retrieved prompts.
Our fine-tuned CadeL 13b model surpasses both GPT- 3.5 and GPT-4 on Level 3 evaluations. In the 3-shot retrieved setting, Codel lama-13b achieves 49.5% API call accuracy, compared to 39.5% for GPT-3.5 and 44.3% for GPT-4. This 5.2 percentage point improvement over GPT-4 demonstrates the effectiveness of our fine-tuning approach with API Pack.
Outperforming GPT Models:
Table 3: Evaluation for models fine-tuned with 20k Python API dataset, including comparison with (not-fine- fine-tuned) CodeLlama- 13b, GPT-3.5, and GPT-4. Levels and are not reported for GPT-3.5 and GPT-4 due to proprietary fine-tuning processes. Note that 3-shot (retre) is short for 3-shot retrieved.
Evaluation Accuracy (%)
Model
Fine-tuning
Testing
Level
Level 2
Level
template
Intent
API Call
Intent
API Call
Intent
API Call
0-shot
17.2
10.9
14.1
11.4
14.3
11.2
O-shot
3-shot (retre)
42.0
29.7
35.4
28.7
39.1
29.1
Mistral-7h
0-shot
40.5
28.5
24.0
18.3
15.2
12.1
3-shot
3-shot (retre)
64.1
55.4
49.1
42.8
50.8
42.5
0-shot
10.0
7.0
11.0
7.8
0-shot
3-shot (retre)
35.9
50.2
CodeLlama-76
0-shot
12.1
10.2
16.8
13.0
3-shot
3-shot (retre)
60.6
52.7
54.1
47.3
55.9
49.1
O-shot
11.6
10.9
8.4
0-shot
3-shot (retre)
44.5
33.9
45.4
35.6
46.7
39.1
Llama-2-13b
0-shot
14.0
11.2
11.7
9.6
3-shot
3-shot (retre)
44.3
52.7
44.2
(--hot
9.8
10.8
12.1
8.5
0-shot
3-shot (retre)
55.6
44.4
50.6
43.3
52.3
O-shot
15.9
13.3
14.2
8.9
CodeLLlama- 13b
3-shot
3-shot (retre)
63.5
56.8
51.4
56.1
49.5
(I-hhot
0.1
0.2
0.0
0.1
0.0
none
3-shot (retre)
49.2
46.2
36.3
34.4
40.7
38.5
gpt-3,5-1106
none
(I-hhot
1.0
0.7
none
3-shot (retre)
47.2
39.5
gpt-4-1106
none
(I-hot
0.2
0.1
none
3-shot (retre)
53.5
44.3
5.2 RETRIEVAL AUGMENTATION IMPROVES API CALL GENERATION
Table 4: Performance comparison for different retrieval methods in 3-shot prompting. Note that 3-shot (retre & rerank) short for 3-shot retrieved re-ranked.
Evaluation Accuracy (%)
Model
Testing
Level I
Level
Level 3
Endpoint
Endpoint
API
Endpoint
API
3-shot (rand)
54,5
41.8
48.2
41.2
45.2
37.0
3-shol (retre)
64.1
55.4
49.1
42.8
50.8
42.5
Mistral-7b
shot (retre & rerank)
63.0
53.6
49.0
42.2
51.5
43,9
s-ohot (oracle)
75.8
68.4
62.5
56.8
60.2
53.1
3-shot (rand)
49.2
38.6
49.8
43.6
50.0
41.4
3-shot (retre)
63.5
56.8
51.4
49.5
CodeL_lama- 13b
3-shot (retre & rerank)
61.0
52.9
55.1
49.2
55.9
49.3
3-shot (oracle)
78.5
71.2
69.8
64.1
65.9
59.3
Table shows how different retrieval methods affect 3-shot API call generation. Our analysis highlights four key findings:
7

---

Published as a conference paper at ICLR 2025
1. Advantages of Retrieved Examples: 3-shot (retre) consistently outperforms 3-shot (rand) for both Mistral-7b and CodeLlama- 13b models across all evaluation levels.
2. Limitations of Generic Re-ranking: Interestingly, 3-shot (retre & rerank) does not con- sistently improve upon 3-shot (retre) and sometimes performs slightly worse (e.g.. Level drops from 63.5% to 61.0% for Cadel_lama- 13b) Our 3-shot (oracle) results show that better example selection can improve performance. However, the current re-ranker fails to capture these gains, suggesting that a fine-tuned re-ranker adapted to API characteristies might be more effective.
3. Performance Gap Across Levels:
We observe notable performance gap between retrieval methods for Level 1, which narrows for Levels and 3. This trend suggests that retrieval is particularly effective seen APIs and endpoints (Level 1).
4. Endpoint vs. API Call Accuracy: Across all conditions, Endpoint accuracy consistently higher than API Call accuracy. This indicates that identifying the correct endpoint is easier than generating the full API call, which requires accurately specifying more parameters, their types, and any required formatting.
These results demonstrate that inference time, 3-shot (retre) is the most effective approach for fine-tuned Mistral-7b and CodeLlama- 13b models compared to 3-shot (rand) and 3-shot (retre & rerank).
5.3 CROSS-LANGUAGE GENERALIZATIONS IN API CALL PERFORMANCE
Figure 2 compares three fine-tuning approaches for cross-language API call performance: a model fine-tuned exclusively 100,000 instances of 'cURL' data, CURL MODEL, three EXPERT MODELS, each fine-tuned on 100,000 samples 'cURL', 'Python'. and 'Java' data separately, and MIXTURE MODEL fine-tuned 100,000 instances of 'cURL' data with additional samples of 1,000 instances each for nine different languages. Since all these instances invoke the same API functionality across different languages, this comparison directly evaluates the models' ability to translate equivalent API calls across programming languages.
0.4
CURL Model (0-shot)
Level 3 API Call Acc.
0.8
CURL Model (3-shot)
Level 3 API Call
Mixture Model (O-shot)
Mixture Model (3-shot)
0.2
Expert Model (0=shot)
0.6
Expert Model (3-shot)
0.0
0.4
CURL
Python
Java
CURL
Python
Java
Figure 2: Comparison O-shot and 3-shot API call performance for different models cURL,
Python, and Java. Note that the EXPERT MODELS are specific to each programming language.
Love aR Call ADC
0.75
CURi Model (3-shor)
Mixture Rodel (3-shot)
Expert Model (3-shot).
0.50
curl
python
java
javascript
libcurl
node
php
ruby
swift
Figure Three-shot performance ten languages across different models.
The comparsion shows that without fine-tuning, models struggle to generalize to programming languages in 0-shot scenarios. However, 3-shot settings, performance improves significantly, demonstrating the benefits of in-context learning.
Figure iilusstrats that the MIXTURE MODEL performs comparably to EXPERT MODELS across ten programming languages in 3-shot testing. This demonstrates the MIXTURE MODEL'S 's ability to generalize effectively without significant performance loss, highlighting its robustness and versatility for multi-language programming applications.
8

---

Published as a conference paper at ICLR 2025
Overall, these findings indicate that cross-language API call generation is achievable by fine-tuning models with a large dataset in one language and smaller datasets in others, eliminating the need for extensive data in each target programming language.
5.4
SCALING INSTRUCTION DATASET HELPS GENERAL DENERALIZATION
Propuent Meeth Arean
Lovel shoc)
Level 10 shot)
Endyont Natth Accurey
AR Cal Natch Accurm anic
Level (0-shrg shot)
Level 2 10 shoti
0.8
Level 213-shot)
0.8
Level 2 (3-shot)
shot)
Level sbb
Level (3-ahos)
Level 3 (3-shot)
0.4
0.6
0.2
0.2
0.0
0.0
10C2SHOE 10C6
Figure 4: Scaling instruction dataset, with 3-shot fine-tuning template. on CodeLlama- 13b with O-shot and 3-shot retrieval evaluations. The x-axis represents the log scale size of fine-tuning data from API Pack. The y-axis is Endpoint or API call accuracy.
Figure 4 shows that O-shot performance increases with the fine-tuning dataset size (ranging from Ok for the con-finetnne baseline to 1 million instances from API Pack using 3-shot fine-tuning template). We observe a roughly linear rise in both endpoint match accuracy and API call match accuracy as the dataset grows. This trend consistent across evaluation levels (1, 2, and 3), indicating that larger datasets enhance the model's 's ability to generate API calls without examples, likely due to exposure to more diverse API patterns and structures. For 3-shot prompting, the graph also shows performance gains the dataset size increases, though these gains smaller than in the O-shot case, especially for level cases.
For Level (seen APIs and endpoints), there is similar linear improvement as in the O-shot but with a gentler slope. This suggests that even with in-context examples, larger fine-tuning datasets further improve performance familiar APIs. However, for Levels 2 (new endpoints of known APIs) and (unseen APIs), performance initially improves but then plateaus as fine-tuning data approaches million instances. The plateau most evident for Level our most challenging evaluation scenario. These findings indicate that there may be diminishing returns from scaling instruction data, particularly for the most difficult generalization tasks (Level 3).
5.5 ADDITIONAL EXPERIMENTS
We conducted two experiments assess the benefits of API Pack: First, to evaluate the impact of using multiple API sources, we fine-tuned CodeLlama-133 on 20k instances from ToolBench, which only includes APIs from RapidAPI. Second, to test our data filtering pipeline, we created Non-filtered API Pack by replacing 45% of the filtered instructions by the LLM with non-fil con-filtered ones, and fine-tume tne-ttned CodeLlama- 13b on 20k instances.
Table Performance comparison for ToolBench, Non-filtered API Pack, and API Pack in different
shot settings.
Evaluation Accuracy (%)
Training Dataset
Testing
Level
Level
Level
Endpoint
API Call
Endpoint
API Call
Endpoint
API Call
0-shot
5.7
5.7
8.1
8.0
7.3
7.0
ToolBench
3-shot
44.5
37.8
40.7
36.4
43.7
38.1
0-shot
10.3
8.3
12.8
10.3
12.2
8.4
Non-filterdd API Pack
3-shot
62.4
54.9
54.9
48.7
55.7
49.5
0-shot
14.4
10.3
15.9
13.3
14.2
8.9
API Pack
3-shot
63.5
55.5
56.8
51.4
56.
49.1
Table presents the results of these experiments alongside our API Pack performance. Key findings from these experiments include:

---

Published as a conference paper at ICLR 2025
1. Multi-source Advantage: Training with APIs from single source (ToolBench) reduces performance in both O-shot and 3-shot settings compared to API Pack, which aggregates data from four sources. This demonstrates the benefit of our multi-source approach.
2. Data-filtering Importance: Introducing non con-filtered instructions (Non-filtered API Pack) leads to a slight performance drop compared to the fully filtered API Pack. This indicates that while our filtering process contributes to dataset quality, its overall impact on performance appears limited, suggesting room for simplification or refinement in future iterations.
We also evaluated the performance of mixing subset of 50,000-entries of API Pack with Magicoder dataset and fine-tuned CodeLlama- 13b model. The resulting model shows an increase of over 35.3% in API call code generation accuracy for Level 3, specifically with the 3-shot setting. This improvement does not come at the expense of general coding efficiency, as the resulting model still performs well on benchmarks such as HumanEval+ and MBPP. See Table further details.
Table 6: Evaluation for Code Generation with CodeLlama-133
Bench, (pass @ 10)
Level 3 (3-shot)
Data Mixture
HumanEyal+
MBPP
Endpoint
47.8
58.3
Magicoder
60.8
66.4
17.0
Magicoder API Pack
61.3
64.3
52.3
CONCLUSION AND FUTURE WORK
We introduced API Pack, multi-programming language dataset containing over 1 million instruction- API call pairs transform software development workflows by enabling LLMS to accurately generate API calls from natural language instructions. Our experimental results demonstrated three key achievements: First, fine-tuning CodeLlama-11 13B on 20,000 Python instances from API Pack enables it to outperform GPT-3.5 and GPT-4 in generating unseen API calls. Second, we established that model performance continues to improve as training data scales to 1 million instances, assisting generalization to unseen APIs. Third, we showed that cross-language API call generation can be achieved by combining extensive training data in one programming language with smaller amounts from other languages.
We acknowledge that the current API Pack exhibits several limitations. Our evaluation method, while systematic, could benefit from execution- based metrics like pass rates or hallucination rates, but focus on privacy protection and PII removal currently makes such testing difficult. Additionally, API Pack was not designed for multi-API call scenarios, and its requirement for explicit API name specifi- cation may limit application complex software development scenarios. Programming language coverage is constrained by code generation library limitations, potentially excluding languages of interest to end-users. Furthermore, while API-gurus operates under a CCO-1.0 license, the limited availability of API-level license information may restrict use in proprietary development contexts.
Looking ahead, we plan to address these challenges through several initiatives. We will explore crivacy-reeerving inccution-based evaluation methods using synthetic endpoints or sanitized envi- ronments. Our roadmap includes expanding the dataset to support multi-API scenarios and task-based instructions, while also increasing programming language diversity. Finally, we aim to improve API-level license documentation to better serve the needs of all potential users.
REFERENCES
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Do- han, Ellen Jiang, Carrie Cai, Michael Temry, Quoe Le, and Charles Sutton. Program Synthesis with Large Language Models, August 2021. URL htllhtttp//axx// http:///www.....///////./....../ arXiv:2108.07733 [es].
07732
10

---

Published as a conference paper at ICLR 2025
Lenz Belzner, Thomas Gabor, and Martin Wirsing. Large language model assisted software engi- neering: prospects, challenges, and a case study. In International Conference on Bridging the Gap between Al and Reality, pp. 355-374. Springer, 2023.
Paul E. Black. Ratclifflobershelp pattern recognition. https://wwwt nist.. govlsttgoo//... gov/dads////gogooooo.. dadd/HTML/ HTMLL/ ratcli ffObershelpppppppmmm html| January 8 2021. In Dictionary of Algorithms and Data Structures [online], Paul E. Black, ed. (Accessed DATE).
Yupeng Chang. Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1-45, 2024.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. AlpaGasus: Training Better Alpaca with Fewer Data, November 2023. URL pattp://aree wwt.o/ggggg..oooooo.o// MEXiv:2307.08701 [es].
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating Large Language Models Trained on Code, July 2021a. URL arxiv. org/ hbtg/ab//22/ 2107 arXiv:2107 .03374 [cs].
03374
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021b.
Wei Chen, Zhiyuan Li, and Mingyuan Ma. Octopus: On-device language model for function calling of software apis. arXiv preprint arXiv:2404.01549 2024.
Christof Ebert and Panos Louridas.
Generative Al for Software Practitioners. IEEE ISSN 1937-4194. doi: 10.1 10.1100/MS.022
Software, 40(4):30-38,
July 2023.
3265877.
URL https:
SFttac
Fw1 BTB.B///ww............T...////.. 1ot.nn100YpPp__pp_00 30 -l-3cchbyy-bocy Conference Name:
IEEE Software.
Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang, Large language models for software engineering: Survey and open problems. arXiv preprint arXiv:2310.03533, 2023.
Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. Large Language Models for Software Engineering: A System- atic Literature Review, September 2023. URL http:://axxii.orrree/// arXiv:2308. 10620 [es].
Qing Huang, Zhenyu Wan, Zhenchang Xing, Changjing Wang, Jieshan Chen, Xiwei Xu, and Qinghua Lu. Let's chat to find the apis: Connecting human, Ilm and knowledge graph through ai chain. In 2023 38th IEEE/ACM International Conference Automated Software Engineering (ASE), pp. 471-48 IEEE, 2023.
Nihal Jain, Robert Kwiatkowski, Baishakhi Ray, Murali Krishna Ramanathan, and Varun Kumar. On mitigating code 1lm hallucinations with api documentation, 2024. URL https: // / arxiv.org/ abs/ /407.447..00 09726
11

---

Published as a conference paper at ICLR 2025
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud. Marie-Ame Lachaux. Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William EI Sayed. Mistral 7B, October 2023. URL http:
06825 arXiv:2310.06825 [cs].
Hongxin Li, Jingran Su, Yuntao Chen, Qing Li, and ZHAO-XIANG ZHANG. Sheetcopilot: Bringing software productivity to the next level through large language models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neu- ral Information Processing Systems, volume 36, pp. 4952-4984. Curran Associates, Inc.,
2023a. URL httppsss/prrooeeedggggsseeeee neur cc wttppesr_ecpeppppeeeeeeeeeeee httppppppee..ieeeapppp/////333/
E
Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. API- Bank: A Benchmark for Tool-Augmented L.MMs, April 2023b. URI. htt.htttp///xxiv.ooom
abs/ /2004. 08244 arXiv:2304.08244 [cs].
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Olch Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang. Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luceioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding. Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dalan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. StarCoder: may the source be with you!, December 2023c. URL
org/ / abs/ //s/////// /2002 06161 61. arXiv:2305.061 [es].
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Flo- rence, and Andy Zeng. Code as Policies: Language Model Programs for Embodied Control. In 2023 IEEE International Conference on Robotics and Automation (ICRA). pp. 1499-9550, May 2023. doi: 10.1099ICRA448990.2223100000
URL https
Leee org htt.://w....a.ooo.ooooo...oooooa
V2iHcM_ BLLLec__ggEKaa22CCRDbeLLLL......
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and LINGMING ZHANG. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.). Advances in Neural Information Processing Systems, volume 36, pp. 11555-22157.. Curran Associates, Inc., lacpaapeerrreeeeeeeeeeeee fwles//appr//22222 file/ f2112//66677766777753533666-0000
2023a. URL httpt//s////rrocceinnnnnneeeeeee
Jiawei Liu, Chunqiu Steven Xia. Yuyao Wang, and Lingming Zhang. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation, October 2023b. URL BtRt:htt:://axxiii http://wwiiiirraaaaa////o/ abs/ /205. 01210 arXiv:2305.0122 [es].
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning, December 2023c. (RRllhtt:///rkk http:://axxvv.ogg/aeae abs /aa/bs/// 15685 arXiv:23 15685 [cs].
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poiries Nouamane Tazi, Ao Tang. Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.1917. 2024.
Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jin- gren Zhou #InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Models, August 2023. URL :ttp::/ arxiv. wttp://v..r/////////.ea/e . 07074. AXXi:2308807074 [cs].
12

---

Published as a conference paper at ICLR 2025
Michael Meng, Stephanie Steinhardt, and Andreas Schubert. Application Programming Interface Documentation: What Do Software Developers Want? Journal of Technical Writing and Commu- nication, 48(3):295-33333 July 2018. ISSN 0047-2816. doi: Publisher: SAGE Publications Inc.
URL
https:://do www. 117..1177110 11w.01111170001110 71221853
Mayank Mishra, Matt Stallone, Gaoyuan Zhang, Yikang Shen, Aditya Prasad, Adriana Meza So- ria, Michele Merler, Parameswaran Selvam, Saptha Surendran, Shivdeep Singh, et al. Gran- itc code models: A family of open foundation models for code intelligence. arXiv preprint 2rXiv:2405504324, 2024.
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. OctoPack: Instruction Tuning Code Large Language Models, August 2023. URL URLhttp::// /tL.ttt:://rxxv.orr/// http:/wwwwww.g//////ooooooo arXiv:2308.07/ 0.2124 [es].
Ipek Ozkaya Application of large language models to software engineering tasks: Opportunities, risks, and implications. IEEE Software, 40(3):4-8, 2023.
Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large Language Model Connected with Massive APIs, May 2023. URL attp::/arxiv.oorraabb/// 15334 arXiv:2305.1525 153334 [es].
Qiwei Peng, Yekun Chai, and Xuhong Li. Humaneval-xt: multilingual code generation benchmark for cross-lingual natural language generalization. arXiv preprint arXtv:2402.16694, 2024.
Gabriel Poesia, Oleksandr Polozov, Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. Synchromesh: Reliable code generation from pre-trained language models. arXiv preprint arXiv:2201. 1227, 2022.
Yujia Qin, Shihao Liang, Yining Ye. Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. TooILLM: Facilitating Large Language Models to Master 16000+ Real-world APIs, July 2023, URL EXXiv23307.16788 [cs].
Steven I Ross, Fernando Martinez, Stephanie Houde, Michael Muller, and Justin D Weisz. The programmer's ' assistant: Conversational interaction with a large language model for software development. In Proceedings of the 28th International Conference on Intelligent User Interfaces, pp. 1991-14,220 2023.
Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. Automatic generation of programming exercises and code explanations using large language models. In Proceedings of the 2022 ACM Conference on International Computing Education Research- Volume pp. 27 27-43, 2022.
Timo Schick, Jane Dwivedi-Yu. Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language Models Can Teach Themselves to Use Tools, February 2023, URL htttp://arxiv.orr//aat arXiv:2302.04761 [cs].
Disha Shrivastava, Hugo Larochelle, and Daniel Tarlow. Repository-level generatio for large language models of code. In International Conference on Machine Learning, pp. 31693-31715. PMLR, 2023a.
Disha Shrivastava, Hugo Larochelle, and Daniel Tarlow. Repository-Level Prompt Generation for Large Language Models of Code. In Proceedings of the 40th International Conference on Machine Learning, pp. 31693-31715, PMLR, July 2023b. URL ISSN: 2640-3498.
html
Venkat Krishna Srinivasan, Zhen Dong, Banghua Zhu, Brian Yu, Damon Mosk-Aoyama Kurt Keutzer, Jiantao Jiao, and Jian Zhang Nexusraven: a commercially-pemiine permissive language model for function calling. In NeurlPS 2023 Foundation Models for Decision Making Workshop. 2023.
13

---

Published as a conference paper at ICLR 2025
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. ToolAl- paca: Generalized Tool Learning for Language Models with 3000 Simulated Cases, September 2023. URL arxiv. org/ als atc//////////5 2ttpp//336...333
arXiv:2306.0555 05300 [cs].
Shailja Thakur, Baleegh Ahmad, Hammond Pearce, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri, and Siddharth Garg. Verigen: A large language model for verilog code generation. ACM Transactions on Design Automation of Electronic Systems, 29(3): 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodrigue7, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama Open Foundation and Fine-Tuned Chat Models, July 2023, URL http:tttp/aaae /ttp:://axxiv.../ org/ abs /2007 09288 arXiv:2307.092 [cs].
Priyan Vaithilingam, Tianyi Zhang, and Elena Glassman. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In Chi conference on human factors in computing systems extended abstructs, pp. 2022.
Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. Software testing with large language models: Survey, landscape, and vision. IEEE Transactions on Software Engineering. 2024.
Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-Instruet: Aligning Language Models with Self-Generated Instructions, May 2023a. URL
org/ abs 2212. 200..22221.111000
#rXiv:2212.10566 [cs].
Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. CodeT5+: Open Code Large Language Models for Code Understanding and Generation, May 2023b. URL http::
org/ abs/2305 07922 arXiv:2305.07922 [es].
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source Code Is All You Need, December 2023. URL arXiv:2312.021100 [es].
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-Pack: Packaged Resources To Advance General Chinese Embedding, December 2023. URL http:t::aaa....or////////
arXiv:2309.07597 [cs].
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardLM: Empowering Large Language Models to Follow Complex Instructions, June 2023a. URL
10e...2244 12244 arXiv:2304.122 12244 [cs].
Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pp. 1-10, 2022.
Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. On the Tool Manipulation Capability of Open-source Large Language Models, May 2023b. URL https
//arxiv. //axxv..rc/abb//223 200...22555.0 16504v1
Rui Yang. Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. GPT4Tools: Teaching Large Language Model to Use Tools via sestrunntr,c,non, May 2023. URL http:
7/ rxivv wtgg/bb///30000 18752 N.ii:2305.18755 [es].
14

---

Published as a conference paper at ICLR 2025
Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Yongliang Shen, Ren Kan, Dongsheng Li, and Deqing Yang. Easytool: Enhancing llm-based agents with concise tool instruction. arXiv preprins arXiv:2401.06201, 2024.
Daoguang Zan, Bei Chen, Zeqi Lin, Bei Guan, Yongji Wang, and Jian-Guang Lou. When language hwtp:g//bb//2//
model meets private library, 2022. URL
Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Wang Yongji, and Jian-Guang Lou. Large Language Models Meet NL2Code: A Survey. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 7443-7464, Toronto, Canada, July URL
2023, Association for Computational Linguisties. doi: 10. N0.188533v 1/265//o//22....cooooog-44/ httpps://aclannhooooryy. http://el..tttoooooo.orrrr///a.d
411
Kechi Zhang, Huangzhao Zhang, Ge Li, Jia Li, Zhuo Li, and Zhi Jin. Toolcoder: Teach code generation models to use api search tools, 2023a. URL https: http:///aawvv../////////////
04032
Peitian Zhang, Shitao Xiao, Zheng Liu, Zhicheng Dou, and Jian- Yun Nie. Retrieve Anything To Augment Large Language Models, October 2023b. URL fttt:://arxiv.xxt arXiv:2310.07554 [cs].
07554
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufci Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su. Zhilin Yang, and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x, 2023.
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari. Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint 2rXiv:2406.15877 2024.
A APPENDIX
A.1 ETHICAL AND CONSIDERATIONS
Developers and other kind of end-users that plan to use API Pack for LLM fine-tuning should consider the following factors:
API Evolution and Validity: APIs included in API Pack may evolve over time, potentially rendering some data outdated. Users should be aware that fine-tuning models static API datasets might introduce deprecated or incorrect API calls. We recommend periodically updating the dataset or employing eetrieval-augmente generation techniques ensure the model has access to the most recent API specifications.
Hallucination: API Pack's 's code (API calls) was not generated via an LLM. Thus, there is a low risk of code errors in the dataser due to LIMSs hallucination. That said, hallucination is a general risk associated with the use of L.MMs. Therefore, code generated with LLMs that were fine-tuned with API Pack or any other code dataset must be always verified by developer.
Data Ownership and License Information: The OAS files used to build API Pack come from public and official API data hubs. API data owners published these files on those websites themselves. If legitimate API code owner requires their data be removed from API Pack we will do so. Regarding API's license, only few OAS files contained this information. We are committed to tracking down complete and accurate license information for all APIs included in API Pack in a future version of the dataset.
Sensitive Information: There no risk of sensitive information being released as API Pack code data was sourced from API documentation that does not include real argu- ment values. Instead, placeholders indicate the need for an argument value (i.e., RE- In a real application, the users can provide these values through a conversational AI interface (i.c., code assistant). Some API calls do include public and restricted API URLs, however these where directly released by the API owners.
15

---

Published as a conference paper at ICLR 2025
A.2 BROADER IMPACT
API Pack enables the integration of code LLMs specialized in API call generation into software development processes, which could have significant broader impacts:
Enhanced Software Productivity: The presented advancements promise significant soft- ware development acceleration by automating routine coding tasks. This could enable faster iterations, reduced cost, and potentially increase the quality of products. However, reliance auto-gent auto-generated code risks reduced developer accountability and control. The oversight of human eye to ensure accuracy still required.
Sociotechnical Considerations and Responsible Innovation: Although intelligent cod- ing tools enhance productivity, sociotechnical concerns exist regarding job displacement, ownership. and digital inequality. We design API Pack to leverage the creation of tools that increase software productivity by keeping humans in the loop, developers still decide whether or not using the API calls generated with models fine tuned on API Pack. Regarding code ownership, we consider API Pack not to have negative effect, data is sourced from documentation that is already publicly available. Towards reducing digital inequality, we have decided to open- our dataset, code, and models.
A.3 DATA FILTERING FACH STAGE OF THE PIPELINE
Table 7 shows the number of instances from each API source at different stages of our data filtering pipeline. Detailed description for each stage can be found in Section
Table 7: Data Filtering Progress
After Removing Invalid
After Removing Instances without Good Instructions
Source/Instances
Before Data Validation
APT calls
IBMAPI Hub
27,635
17,712
17,206
APIs Gurus
500,160
499,250
495,533
Swaggerhub
351,756
351,756
345,765
RapidAPI
274,014
273,388
270,095
Total
1,153,565
1,142,106
1,128,599
A.4 DATA SPLITING FOR LEVEL 1, LEVEL 2, AND LEVEL TESTING
To split the data, first copy the total dataset into total training and calculate the occurrence of each API. For the first level of testing data, select up to 1,000 data points from the first 20,000 entries where instruction_ten _est is not "None" and the API appears more than four times. For the second level, iterate through the first 20,000 entries, selecting data points with APIs not yet selected, until 1,000 unique APIS are collected, then remove these from the training dataset. For the third level, from the 80,000th entry onwards, select data points with APIS not used in Level 2, collecting to 1,000 data points, and remove data points with these APIs from the training dataset.
16

---

Published as a conference paper at ICLR 2025
A.5 HYPERPARAMETERS FOR TRAINING
We fine-tune the models using the HuggingFace Transformers library on cluster consisting of node with 8 NVIDIA HI00 80GB GPUs, with Fully Shared Data Parallelism (FSDP). Techniques such as mixed precision, gradient checkpointing, and AdaFactor optimizer are used to improve training efficiency. The key hyperparameters are summarized Table g
Table Hyperparameters for Training
Hyperparameter Name
Value
Learning rate
2 x 10
Batch size
128
Max seq length
4096
Number of epochs
2 0.03
Warmup ratio
A.6 TESTING PIPELINE
Figure outlines testing pipeline for 0-shot and few-shot learning. The retriever uses embedding model with cosine similarity to find Top-k semantically similar examples from the example collection. A reranker then refines and re-orders these examples. The LLM generates the appropriate API call based on the user input and retrieved examples. For 0-shot inference, no 3-shot approach, incorporating
examples are included in the prompt. For few-shot learning, the three most relevant examples. The prompt template is in Listing 6
<cINPUT>>
is supported for specific card brand using the Adyan
Examples collection
odrgubo; Give example of how to fees cost estimete with the
Adyen Binloskup API
BinLcokup API, For 3D Secure 2, will also provide device fngerprinting keys.
ecOutputoor curl -request POST -urf http:///i..etaddee..co//l/laaaaa Birl.okkupi harp:k/piiippooegeeeosessieeee
Examples Retriver
<EXAMPLE 200
OUTTUUT>
zero-shot
-request POST
-ur https://palttstt.eee wwt.e/awww...oooppplleeeeeiiiidd
getSdsavaillbiltty
-header AAthorization; Basic REPLACE REPLACE_BASIC. BASIC_AUTH"
-header 'content-type: application/sson)
ddata ('additionalData" ""poaaaa""""pprttt"""""""" ""prppprtt""""srrinn""" Parpinggtprrootttttttt "ppppprtty2""'/01 "rrrin""""
Figure 5: Testing pipeline.
17

---

Published as a conference paper at ICLR 2025
A.7 API DB INSTANCE
Figure 6 shows the structure of an API DB instance. We used openapi-snippet an open-source package that takes input OpenAPI v2.0 v3.0.x specification file (OAS file) and translates into an HTTP Archive 1.2 request object, to generate API calls. We generated API calls (api_call) in 10 different programming languages (cURL, libcurl, java, node. python, go, ruby, php. swift, JavaScript) for RapidAPI, API Gurus, and the Swaggerhub. For the IBM Developer Hub API calls were extracted directly from the OAS files. We extracted API calls in eight different programming languages from this source (cURL, java, node, python, go, ruby, php, swift).
"api name" "The name of the API the endpoint belongs to", "api__escripp"""p" deserrpptiinn: "The description of the API the enpoint
+ belongs to",
aapi_provider' "The API's provider name", "endpoint name": : "The name of the function to call", "functionality"; "A brief description of endpoint's
- funetionality",
"description": "A long description of endpoint's functionality", "path": "The enpoint's name plus specific versioning as it appears in the API call's URL", "method": "HTТР method used in the API call (e.g., get, post, put, + delete)",
"api_call"" call": "НТТР request code example to invoke an endpoint", "lang": "The programming language in which the API call is written",
Figure 6: Structure of an API DB instance
A.8 API PACK DATASET INSTANCE EXAMPLE
Figure shows a fragment of an API Pack data instance. As data sourced from public OAS files, personally identifiable and sensitive information in API calls is represented by anonymous strings (e.g., REPLACE BASTC, which indicates the need to pass this value as an argument real execution. In this example, we only show one instruction candidate for clarity.
pnassrurtion_ceddddeee..
"dxx:
"candidate": "I'd like confirn 10 3D Secure is supported for specific
G cord brand or BIN using the Adyen Sintookup API. For 35 Secure 2, it will provide device ringerprint keys.",
"input_ Lekeus___kkesssen" 0.5497539341557309
"apl_call"; "ourl zequest POST
-uurl https: / //palppaaaae www.et.dddde..o....
/servlet Searvlet/Rint.on http:///iittooko/////////aaaaaaa Hoy N.
header 'Author ization: Basic RBPIACE DASIC_AUTH""
heeder
ccontenttyyee application/ json"
["additionallssa"" ("propestyl" "stringg, "rroperty2":"strinn
"branda": "etranda"(""ttrii"" ("string"),
"cardNumber" "string",
"merchantáccount":: "sttinng"
htteeeeeeaaaaeeeeeeeeeese "atring"" "shopperReleren" "sstiing""""""""""""""""""""""""
Figure 7: Fragment of API Pack instance in cURL..
18

---

Published as a conference paper at ICLR 2025
A.9 INSTRUCTION CANDIDATES
Figure 8 shows all the candidates generated for an instance with their respective id and in- put_tokens_mean
"instruction candidates":
"idx"; 1,
"candidate": "I'd like to confirm if 3D Secure is supported for a specific card brand or BIN using the Adyen
BinLookup API. For 3D Secure 2, it will also provide device fingerprinting keys.",
"input_tokens bookkens_mean:: -0. 0054497533441557999
"idx"; 2,
"candidate": "To confirm 30 Secure is enabled for a specific fic card brand or BIN number using the Adyen BinLookup API, you can make use of the bost--gettavvillaiiiiy
endpoint. This API call will
provide you with information about the availability of Secure, along with device fingerprinting keys for 3D Secure 2 transacticns. Let me know if you need assistance in forming the request handling the response.", "input_token" mean": 0.5588801283222344
"idx": 3,
"candidate": "To confirm if specific card brand BIN " supports 3D Secure and retrieves device fingerprinting - keys for 30 Secure 2, please utilize the Adyen Binlookkp endpoint.",
API's 's beat-get:dssvvillbbll Sinput_tokens m -00:::00055550000 00.554441166113355
"idx": 4,
"candidate": "To confirm if 3D Secure is supported for specific card brand BIN number using the Adyen
BinLookup API, please make a POST request to the endpoint pesst-getasavvalaaall.
This API will return whether 3D Secure is enabled and for 3D Secure 2, it will also provide device fingerprinting keys.", "input_token" mean": -0.5466722001055577
"idx": 5,
"candidate": "To confirm if 3D Secure is supported by Adyen for a given card brand or BIN number, you can utilize the Adyen BinLookup API. Simply send POST request to the pesttggetdssvvailabbll
endpoint with the required card details. If 3D Secure is available, the response will include device fingerprinting keys for 3D Secure 2.", "input_tokens m -0.5726057469887477
Figure 8: Examples of five instruction candidates generated for the same instance.
19

---

Published as a conference paper at ICLR 2025
A.10 QUALITY ASSESSMENT COLLECTED INSTRUCTIONS
We recognize that utilizing smaller, open-sourced model like Mistral-7B for data generation might raise concerns about the quality of the instructions produced. To address these concerns, we conducted an evaluation demonstrate the generality and reliability of our collected datasets. Specifically, used GPT-4o-mini to assess the quality of the instructions in our API pack test dataset.
In this evaluation, used detailed prompt (Listing 1h to guide gpt-4o-mini instructions into High, Medium, or Low quality based on their naturalness and clarity. The assessment revealed that the majority of instructions dataset's suitability for training purposes.
mini in categorizing the
rated either High Medium quality, affirming the
Figure iillstrrtte the distribution of instruction quality across the three different levels. As shown, each level maintains a substantial proportion of high-quality instructions, with Level exhibiting the highest percentage of High-quality ratings. The distributions of Medium and Low-quality instructions provide insights into areas where further refinements can be made.
versing eemerto
Sad
Lead
Figure 9: Instruction data quality distribution for Level 1, and test set.
Listing 1: Instruction Quality Evluation Prompt for gpt-4o-mini
**Prrmpp:**
Evaluate the naturainess the "instruction in the given data. Assign a quality label **High+*, M*deium or **Low** based on how naturally a user would ask the question when seeking to use the specified AP1 endpoint. Include detailed reasoning for your conclusion. Use clean, simple, and concise language.
* ExExmllamss+
*Example
(High Quality) ***
**.aata:**
json
"api name"; "Neather Service",
"api call data":
"fuucttion ffncticnality": "Get current weather by city", "éescription"; "Retrieve the current weather information for specified city.", "path": //weather/aurrent/aiii
centrruciio_ttstt"! "How can I get the current weather for specific city using the Weather Service?"
*+ Iotiaalationn:
**labbll *** High
20

---

Published as a conference paper at ICLR 2025
Reascning::* The instruction is clear and naturally phrased. It directly asks how to get current weather information using the API, making it easy to understand.
** Example (Righ Quality) :**
*aaaa****
json
"api_name"; "Library AP1",
"api_call_data""
"functicnality"; "Search books by auther",
"description": "Find books written by specific author.", "path": toook///uthhor/tutttttaaa
"instruction_tsst"; "How do search for books by particular author using the Library API?"
*>Evaluation:** ++Iabel:** High
**Reascning:** The instruction naturally worded and clearly conveys the user's s intent to search for books by author using the API.
**Exampee (Medium Quality) :**
**Data:**
json
"api_named "Movie Database", "api_call_data"" {
"functional "functionality"" "Get movie details",
"description" "Retrieve details about movie by its ID.", "path": "/novies/ [movield)"
"instruction test"; "Explain how to get details of movie using Movie Database.
* **Evaluation:ee
Label Medium
**Reasoning:* The instruction understandable but slightly awkward. It lacks the article "a" before "movie" and doesn't mention using the API explicitly.
Example 4 (Medium fuality):**
**Data*** json
"api_name": "Flight Info Service", "api_call_data""
"functicnality": "Check flight status", "description"; "Get the status of flight using its flight number." "path":
21

---

Published as a conference paper at ICLR 2025
"instruction test": flight?"
"How to Flight Info Service to check status of
**Evaluation:**
**label:** Medium
**Reasoning:** The instruction is somewhat clear but could be more natural. It misses the article "the" before "status" and "a" before flight, naking it slightly less fluent.
Example 5 (Low Quality) :..
**Data:**
jsson
"api_name" "Music Streaming API",
"api_call_data""
"functionality": "play a song", "description' "Stream a song by its ID.", "path": hatss/ppaa//ssnggtiiee
"instruction ; "Need play seng Music Streaming API."
*Bvaluaticn: *+Iabel:** Low
=*Rasccing::** The instruction is ungrammatical and lacks proper sentence structure, making it unnatural and hard to understand.
#xxmmpl 6 (low Quality) :**
*ibata:** json
"api name": "Stock Market API", app_ccal_datta;;
"funeticnality"; stock price", "dcscription"; "Retriove the current price of a stock by its ticker symbol.", "path": "/stocks/ http://eeeeeybbl/piiiiici
innstruction_tett"" "Price stock get API."
#vvaluation Label:** Low
Reasoning: A The instruction is fragmented and lacks coherence. It doesn't form complete, underatandable question.
* Now, evaluate the following data:**
Data:*** json
"api name"; " "lapi name)",
"api call data":
22

---

Published as a conference paper at ICLR 2025
"functicnality": / "description"; "[description)", "path": "(path)"
"instruction_test" "(instruction_test)
#Bvaluation:**
*abbbl:**
:e:essnnin:***
A.11
PROMPTS
In this section, we share all the prompts used to create instructions. We also explain the process to craft each prompt. When in use, the values within the curly brackets would be replaced by the actual data.
A.11.1 EXAMPLES REFINEMENT
We use Prompt 2 to refine the examples for instruction generation. We created three examples per API. Examples were manually refined by human for the APIs from IBM API Hub and APIs Gurus. Then, we tasked LLM with refining the examples for the other two sources.
Listing Prompt for instruction example refinement
Your task is refine and enhance user query that involves a specific API. The original query you'll work with includes key details about the API's functionality, description, endpoint, and name. Focus on these essential aspects when revising the query:
1. Integration of API Details:** Make the revised query includes relevant details about the API's functionality, description, and name without directly mentioning the endpoint.
2. GGaamarr and Syntax Correcticn:* Analyze the original query for grammatical mistakes sucn as improper verb forms (e.g., 'can' followed by 's' or 'es") or misplaced punctuation (like commas or colons). Correct these to improve clarity and professionalism.
3. Relevance and Conciseness:** Eliminate any extranecus information from the original query. Strive for brevity while ensuring all critical details are included.
4. User-Centrric +ewrite:**
Rewerk the query to reflect user's perspective, focusing on their specific needs and how the API can address those needs.
For each query, you will receive:
Inpuut An original user query with API details. Yourr Task Revise the query based on the guidelines above.
Example for Practice:
II Input:
Functionality: Search Ecards
Description: Allows searching the inventory system for ecards.
Endpoint: searchECards
API: eCards Search API
User query to refine: "Please tell me how to searches ecards with the eCards Search API."
### Output (refined user query):
23

---

Published as a conference paper at ICLR 2025
"Can you guide how to search for ecards using the eCards Search API
Another Example:
### Input:
Funationality: RPI of realized sales
Descripticn: Provides KPIs for documents issued, requires ACIs from instances, and uses the "typeRpi' parameter.
Endpoint: getKPIs
API: Blackbird Analytics
User query to refine: "I need to certificates shows kpis related to all documents issued through blackbird. to use it you must have pass 1ist of acls retrieved by /instances and specify which kpi using typekpi' parameter."
#FF Output (refined user qucry):
"How can I access KPIs for documents issued by Blackbird Analytics, and what are the required 'typeøpi' parameters?"
Remember, the goal is to modify the user query to be clear, effective, and grammatically correct, fully showcasing how the user can leverage the specific API.
Now, here your actual task: IHF Input:
Funct functionality: (functionality)
Description: (description)
Endpoint: (endpoint)
API: (api_name)
User query to refine: [template generated instruction)
### Output (refined query)
A.11.2 INSTRUCTION GENERATION
We use Prompt 3 to generate five instruction candidates for each API Pack instance. We crafted custom prompt that considers the information developer may provide an LLM, while looking for an API call to complete task.
Listing 3: Prompt for instruction generation
Your task is to create a user query that effectively utilizes a specific API. The API's functionality, deseription, and name will be provided to you. Your query should be designed in way that makes the best use this API's unique capabilities, when crafting your query, focus on:
1. **API Name Integration:** Clearly include the API's nare in your query ensure relevance.
2. *Specificity:** Replace broad or vague terms with precise, concrete details relevant to the API's purpose.
3. * *Conciseness: Keep your query as brief as possible while still
fully conveying the needed information. Avoid unnecessary verbosity.
4. Excluding API Endpoint:** Do not include the API's endpoint in your query; focus only on the user's ' need and how the API fulfills it.
Create query that a user might realistically use when interacting with the given API. Think about typical scenarios problems that the API designed to solve and formulate your query accordingly.
Examples for practice:
Inpun:
24

---

Published as a conference paper at ICLR 2025
Functicnality: [functionality]
Descripticn: (description)
Endpoint: (endpoint)
API: [api_name)
##Output: {output)
##IInput:
Funstionality: [
Descripticn: (description)
Kndpoint: (endpoint)
API: (api_name)
#HOutppt: (output)
###Input:
Functionality: (functionality)
Description: (description)
Endpoint: [endpoint)
API: (api_name)
##Ouupput: )outppu))
Remember, the goal is demonstrate
would benefit from this specitic API in realistic scenario, using precise and clear language. Here is the actual task for you:
VIInput:
Functionality: (Functionality]
Description: (description)
Endpoint: (endpoint)
API: (api_name)
#HOutput:
A.11.3 BACK TRANSLATION
Prompt 4 seeks to re-generate the API information associated to an instruction by passing an instruction candidate input. To create this prompt, took Prompt 3Jas as reference and simply reversed the task. As result of using this prompt for inference, we save the input-token-mean of the output return (the API information that was re-generated).
Listing 4: Prompt for instruction backtranslation
Your task involves reverse-engineering process where you will analyze user query to infer specific details about an API endpoint. Based on the given user query, you are expected to:
1. **Identify fy the Endpoint's Ident fier:** Derive the endpoint identi fier that aligns with the functionality implied by the user query.
2. *Determine Endpoint Funeticnality:** Interpret the user query to understand and describe the functionality of the endpoint.
3. * Describe the Endpoint:** Previde detailed description of the endpoint based the needs and context presented in the user query.
4. Specify the Name:*. Identify and state name the API which this endpoint belongs, as suggested by the user query.
Your response should clearly articulate these four elements (identifier, functicnality, description, API name) in a manner that reflects accurate understanding the user query. Consider the query a real-world scenario or problem that the endpoint is designed to address.
kxamples for practice:
25

---

Published as a conference paper at ICLR 2025
WIInnpt:
Igenerated instruction)
#Houtput
Funstionality: (Funstionality)
Description: (description)
Endpoint: (endpoint)
API: (api_name)
##Input: (generated instruction)
#Output:
Functionality: (functionality)
Description: (description)
Endpoint: (endpoint)
API: (api_name)
##Ingut:
(generated instruction)
##Output:
Punctionality: (functionality)
rescription: (description)
Endpoint: [endpoint]
API: [api_nanc
The goal is to showcase your ability to connect a user's a needs with the appropriate API endpoint, demonstrating an understanding of how the endpoint's features align with user requirements. Your response should be precise, insightful, and reflective of the query's implications. Here the actual task for you:
##Ingut:
(generated instruction)
#Houtput:
Punstionality: (functionalitty
Description: (description)
indpoint: (endpoint)
API: (api_name)
A.11.4 INSTRUCTION ANNOTATION
We use Prompt to task an LLM with annotating the instruction candidates as good or bad. First, human annotator manually reviewed 605 instructions to distill common error patterns among them. Then, we craft a prompt to detect these error patterns via an LLM. Note the prompt includes examples for goodd' and 'bad' instructions. These examples were extracted from the manual revision performed by the human annotator.
Listing 5: Prompt for instruction annotation
Your Task**: Evaluate the provided instruction from AI assistant and classity its quality as **Good** or **Bad** based on specific criteria.
Criteria for 'Bad' Instruction*::
1. Contains multiple instructions instead single, clear directive.
2. Includes unnecessary additional text before or after the main instruction.
3. Fails to accurately use the specified API name and endpoint.
**Input Structure*s:
You will receive an ++INPT** consisting of three elements:
1. An *insttruttion* generated by the AI assistant.
2. The **APT name** related to the instruction.
3. The **Ap1 endpoint** relevant the instruction.
26

---

Published as a conference paper at ICLR 2025
**ottut***;
Classify the instruction as **Codd** or **Bad.. with concise justification.
#xxampless:
1. **Input**:
*+ IIIntruccion=:: "Create new message in IBM Event Streams using the REST Producer API." "How do format and send massage body via the Event Streams REST Producer AP1?" "Can you help me construct a message using the 1BM event Streams REST Producer?" "Use the 1BM Event Streams REST Producer API to send a message with specific data "What's 's the proper syntax for creating and sending message through the IEM Event Streams REST Producer?"
**API Name**: IEM Event Streams REST Producer
**API Endpoint**: producaMessage
*+ Bad, This instruction is classified as B*Bad* because it combines four separate instructions into one, each asking for different guidance related the IBM Event Streams REST Producer API
2. **Input*::
+=Instruction=*: "I'd like to send new message to IEM Event
Streams. How do fornat the request body to ensure it is correctly processed using the IBM Event Streams REST Producer API?"
+*AAII Name*A; IBM Event Streams REST Producer
-*APPI Endpoint*4: produceMessage
**OutputA*: Good. This instruction is classified G*Good* because it provides a single, clear directive without additional text and correctly uses the API name.
3. IIputt+*:
**Instruction==: "Here's possible user query utilizing the given APT: 'Help me 1ist all the bare metal servers my Virtual Private Cloud account using the Virtual Private Cloud API.!"
**PAII Name**: Virtual Private Cloud API
-**API Endpoint*:: 1ist_bareemetal__ seteres
**Output<*: Bad, This instruction is classified as *BBad* because it contains unnecessary introductory text ("Here's a possible user query utilizing the given API:") before the actual instruction.
4. Input**:
Instructioon "How do retrieve a list of all bare metal servers in region using the Virtual Private Cloud API?"
**API Name**: Virtual Private Cloud API
**API Endpoint**: list_bare_metal _eerr_ssteeerrrsssssseee_e
**Output**: Good. This instruction is classified as **Good*x because it is a singular, straigatiorward instruction without extra text and appropriately uses the API name.
**Your Current Task**:
*IInpu**::
**Instructione:: (candidate)
**APII Name**: (api_name)
**API Endpoint**: (endpoint)
*+Outpu***:
A.11.5 FEW-SHOT FINE-TUNING AND EVALUATION
This is the prompt template used for 3-shot-tuning to post-process. API Pack data, as well as for 3-shot random, 3-shot retrieved, and 3-shot retrieved & re-ranked inference.
27

---

Published as a conference paper at ICLR 2025
Listing 6: Prompt for few-shot fine-tuning and evaluation
aapi_descriptionn _ri__descrptinn
**lang** (programming language)
Given the fellowing examples:
**nntruutiinn**
(instruction)
**output**
(api_cal1)
*instructione*
(instruction]
**output**
(api_call)
*instructions
(instruction)
**output** lapi_call)
Your actual task: reinstructione*
(instruction_testt) **utppt***
### ASSISTANT:
28


Reviews:
[{"review_id": "FQkzAtD8jg", "reviewer": "Reviewer_FEcU", "summary": "The paper introduces API Pack, a large-scale dataset with more than a million instruction-API call samples. The authors discuss the aspects of API Pack making it unique and how it was constructed from public sources. Experiments are conducted to demonstrate: (1) Code LLM performance on API Pack under various RAG and evaluation settings (2) strong multi-lingual generalization on API Pack tasks from training a lot on one language and a little bit on others, (3) data-scaling properties hold when training on API Pack data, (4) comparisons with another training dataset, ToolBench.", "questions": "See notes above. I have translated some of the above in question format below:\n\n1. What is the baseline performance of non-fine-tuned models on API Pack?\n2. How is multi-lingual generalization being measured in Section 5.3? Is it the same API invocation across different languages?\n3. Have you considered training on API Pack and evaluating on other benchmarks like ToolBench and Gorilla, and vice versa, to more thoroughly compare API Pack with existing datasets?\n4. What is the performance with an oracle retriever/re-ranker in the context of the simple retrieval discussion?\n5. Can you characterize if there are any data leakage concerns and/or what researchers should be mindful of when evaluating with API Pack?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- **Useful contribution.** The paper presents API Pack, which is a large-scale and multi-lingual instruction-API calling dataset. This can be used for both training and evaluations in multiple languages. In Table 1, the authors demonstrate how this is better along multiple dimensions than previous works.\n- **Several detailed insights.** The paper is packed with insightful experiments. I particularly liked the experiments covering multi-lingual generalization — the designs of control experiments for this are interesting, and show that it is sufficient to train on a lot of data from one language and little data from other languages. There are other experiments, including ablations on RAG strategies and data volume used during training, which will also be valuable to the community.\n- **Reproducibility.** Section 3, discussing the creation of API Pack, is extremely detailed and will be of great value to researchers looking to process data for similar tasks.", "weaknesses": "- **Lack of certain details.**\n    - The baseline performance of non fine-tuned models on API Pack is missing (unless I missed it somewhere). Including this will greatly speak to the advantages of fine-tuning on API Pack.\n    - For Section 5.3, how is the multi-lingual generalization being measured? Is it the same API invocation across different languages?\n    - Could API Pack suffer from data leakage when evaluating public models trained on public sources?\n- **Additional experiments.** While there are several experiments conducted, I am curious about a few more experiments. This is not a major concern however. While Section 5.5 does some comparison of API Pack with ToolBench, I believe that a more thorough evaluation requires training on API Pack and evaluation on ToolBench, Gorilla, etc. and also training on ToolBench, Gorilla, etc. and evaluating on API Pack. This would concretely identify where API Pack stands out and lacks relative to other API calling benchmarks out there.\n- **Paper writing.** Overall the paper seems to be a bit verbose, and could benefit from making the writing more concise and crisp. For e.g., Section 5.3 is too wordy and a jumble of multiple findings. It would benefit the paper to organize such findings in a clearer way.\n- **Evaluation metrics.** While string matching based metrics are being used for evaluations here and have been used in several previous works as well, I think it is time that we move on from these inaccurate metrics. We should resort to more robust metrics relying on static or execution analysis such as pass rate [1] or hallucination rate [2] for measuring correctness of code completions.\n- **Nitpicks.**\n    - Line 381 - “simple retrieval might be sufficient in this context” — I think it can also be the case that using a better re-ranker might help improve performance. For e.g., what’s the performance with the oracle retriever/re-ranker?\n    - Figure 2\n        - The legend(s) should be moved to make the results more readable\n        - Is the “Expert” model in each subplot different? This should be clarified in the text/caption given all models have the same color\n    - Figure 4 - it would be better for comparison if all the subplots had the same y-axis limits\n\n[1] Chen, Mark, et al. \"Evaluating large language models trained on code.\" *arXiv preprint arXiv:2107.03374* (2021).\n\n[2] Jain, Nihal, et al. \"On Mitigating Code LLM Hallucinations with API Documentation.\" *arXiv preprint arXiv:2407.09726* (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jAlM2XEzbE", "reviewer": "Reviewer_NcGM", "summary": "The paper introduces \"API Pack,\" a large, multi-programming language dataset designed to enhance the API call generation capabilities of large language models (LLMs). API Pack aims to help LLMs generate API calls based on natural language instructions, addressing a key developer challenge of navigating and generating code from extensive API documentation. Experiments show that API Pack fine-tuning improves LLM performance in unseen API calls, enabling open-source models like CodeLlama-13B to outperform proprietary models GPT-3.5 and GPT-4.", "questions": "How do the authors view the trade-offs between a RAG approach (with minimal or no fine-tuning) and the approach proposed here of fine-tuning on a large, static API dataset? Since APIs are continually updated, it seems that a RAG or search engine approach could offer more reliable, dynamic knowledge. How does the static nature of API Pack compare in reliability and practical utility?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The focused problem is quite interesting. This paper fills a gap in LLM fine-tuning resources for cross-language API call generation, which has been under-explored.\n- The dataset is robust, sourced from multiple reputable platforms, and undergoes a thorough pre-processing and validation pipeline. The authors’ experiments, conducted with open-source and proprietary models, validate API Pack’s effectiveness across multiple levels of generalization (known APIs, new endpoints, unseen APIs).\n- The paper is clearly structured, detailing each stage of dataset creation, the methodology for instruction generation, and the evaluation framework used to measure LLM performance across scenarios. This transparency supports reproducibility and makes the dataset accessible for further research.", "weaknesses": "- [Major] Insufficient Related Work: The paper lacks discussion on some relevant studies in API learning. Notable examples include:\n\n    - Zan, Daoguang, et al. \"When language model meets private library.\" arXiv preprint arXiv:2210.17236 (2022).\n    - Zhang, Kechi, et al. \"Toolcoder: Teach code generation models to use API search tools.\" arXiv preprint arXiv:2305.04032 (2023).\n\n- [Major] API Dataset Validity: Given the vast scale of the dataset, it’s unclear if all included APIs are usable or up-to-date. Additionally, APIs frequently evolve; fine-tuning on static API datasets could introduce outdated information or \"hallucinations,\" potentially compromising the reliability of API knowledge encoded in the model. The authors should discuss this question.\n\n- [Major] Limited Response Diversity: The paper does not sufficiently discuss the diversity of the code response. Using tools like OpenAPI may lead to highly uniform code outputs, such as the preference for a small set of common libraries for HTTP requests in Python or other languages, potentially constraining the model's generalization capabilities. This limitation might affect the model’s ability to use a broader range of libraries or to produce varied code styles when calling APIs. The authors should add experiments to show the influence of the response diversity.\n\n- [Major] Lack of Baseline Comparison: A critical missing baseline is a model augmented with RAG or a search engine approach (e.g., ToolCoder), **without fine-tuning**, for handling complex API calls. This would provide a practical comparison for API Pack’s utility.\n\n- [Minor] Base Model Selection: The selection of base models seems somewhat detached from practical application. API Pack functions as a specialized fine-tuning stage post-general SFT, targeting API call generation specifically. Therefore, selecting foundational SFT code models, like CodeLlama-Instruct or Magicoder, as discussed in Appendix A.8, would enhance the practical relevance and clarity of the paper’s contributions. Including more experiments with these models would substantiate the paper's claims.\n\n- [Minor] Evaluation on General Code Generation Benchmarks: The paper does not discuss potential **performance trade-offs on general code generation benchmarks**, such as HumanEval, that could drop due to fine-tuning for API-specific tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "64Eq738E4d", "reviewer": "Reviewer_bcCN", "summary": "The paper introduces API Pack, a large-scale, multi-programming language dataset containing over one million instruction-API call pairs generated semi-synthetically from API database collected from four API endpoints. This dataset is used for training LLMs on instruction API pairs and to demonstrate improvements in unseen code generation, cross-language transference.", "questions": "* Do the authors have more intuition behind the 0-shot vs 3-shot template results after fine-tuning? The findings seem inconsistent across Levels and data sizes (level-1 0-shot is better than 3-shot but for levels 2/3 3-shot is better in Figure 4)\n* Can the authors clarify the metrics used to describe it more formally?\n* Did the authors attempt to measure the \"naturalness\" of the collected instructions (automatically or through human annotation) or evaluate models on human written problems for a subset of the APIs?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* **Extensive Dataset with some human-annotation/filtering steps.** The collected dataset sizing over a million instances in unique and large covering a wide array of APIs and programming languages.\n* **Experimental insights.** The paper performs API data scaling experiments, retrieval experiments and cross-language experiments.\n  * The data scaling experiment highlights interesting observations across the three levels. Particularly, flat curve for Level-3 APIs does highlight a broader concern of out-of-distribution generalization for LLMs\n  * The mixture model shows generalization to a new language with few examples solidifying the advantage of the collected large-scale dataset.", "weaknesses": "* **Novelty.** The paper provides limited novelty. It thoughtfully curates API sets and uses LLMs to generate synthetic training and test sets via prompting (with notably some human-in-the-loop quality assurance rounds). While potentially performed carefully, this following standard set of ideas in a somewhat different domain. Note that this is not necessarily a strict limitation and a carefully done execution can be helpful to the community -- perhaps primarily identified over time via the community. \n\n* **In-distribution evaluation.** The paper collects a synthetically generated dataset -- thus comprising instructions of limited natural language variance (compared with human written instructions). This dataset is next divided into a train and a test set. Thus the training and test set are precisely collected from the same and a small distribution. This partly challenges the findings from and we might see smaller gains to more \"diverse\" real-world queries. \n\n* **Marginal benefits from quality filtering.** The data filtering ablation reports performance improvement but it is notably small, often <1%. This raises concerns in the soundness of this step.\n\nMinor: \n\n* **0-shot training does not improve 0-shot retre.** For the 20k fine-tuning experiments, the accuracy of 0-shot retrieval after 0-shot fine-tuning is worse than performance after fine-tuning on 3-shots. This is quite surprising -- and continues in Figure 4 as well and could be expanded further, perhaps with a qualitative analysis of the mistakes.\n\n* **Metric.** The metrics used in the paper are not completely clear and should be more formally defined or described with examples or relevant citations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o5EjZHsxwt", "reviewer": "Reviewer_FeUP", "summary": "This paper introduces API Pack, a comprehensive multi-programming language dataset comprising over 1 million instruction-API call pairs designed to enhance the API call generation capabilities of large language models. By fine-tuning CodeLlama-13B on 20,000 Python instances from API Pack, the authors demonstrate that it outperforms GPT-3.5 and GPT-4 in generating unseen API calls. The fine-tuning process also facilitates cross-programming language generalization by leveraging a large volume of data in one language and smaller amounts from other languages. Scaling the training data to 1 million instances further improves the model's generalization of unseen APIs.\n\nContributions:\n\n1. Introduction of API Pack Dataset: The authors present API Pack, a massive multi-programming language dataset containing over 1 million instruction-API call pairs, which is a significant contribution to the field of API call generation.\n2. Model Performance Improvement: Through fine-tuning CodeLlama-13B on a subset of API Pack, the authors demonstrate superior performance in generating unseen API calls compared to GPT-3.5 and GPT-4.\n3. Cross-Programming Language Generalization: The study highlights the dataset's ability to facilitate cross-programming language generalization, leveraging large amounts of data in one language and smaller amounts from others.\n4. Scalability and Generalization: Scaling the training data to 1 million instances enhances the model's generalization capabilities to unseen APIs not used in training, showcasing the dataset's scalability.\n5. Open-Source Initiative: The authors contribute to the research community by open-sourcing the API Pack dataset, the fine-tuned model, and the associated source code, fostering further research and development.", "questions": "1. Model Selection for Quality Assessment: In the 3.4 section, the authors use the Mistral 7B model to validate the generated instruction, which is also the model used to generate the instructions. This raises concerns about the validity of using the same model to evaluate its outputs. It is unclear whether the 7B model can accurately distinguish between high-quality and low-quality instructions. A more robust approach might involve using a larger (>7B), more powerful LLM to assess the quality of the instructions.\n2. Definition of Unseen APIs: The term \"new API\" refers to APIs that the model has not encountered during fine-tuning. However, given the lack of information about the models' pretraining data, it is challenging to verify whether the APIs labeled as \"unseen\" truly are. This lack of clarity could affect the validity of the experimental results. \n3. Evaluation Metrics for Code Generation: The authors use text-matching-based metrics for evaluating the generated API calls. However, for code generation tasks, metrics based on test case execution are more accurate and reliable. These metrics can provide a more objective measure of the correctness and functionality of the generated code. The authors should consider incorporating such metrics to enhance the evaluation process and provide a more comprehensive assessment of the models' performance.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. New Data: The authors introduce API Pack, a dataset containing over 1 million instruction-API call pairs. The dataset's multi-programming language nature and rich source diversity enable it to cover various scenarios. It will be useful for further research on improving the API call code generation of LLMs in different scenarios.\n2. Performance Enhancement: The paper demonstrates the effectiveness of fine-tuning on API Pack by evaluating multiple models, including CodeLlama-13B, Mistral-7b, and Llama-2-13b. The results show that fine-tuning on API Pack significantly improves the models' ability to generate API call code, highlighting the dataset's value in enhancing model capabilities.\n3. Extensive Experiments: The authors have conducted extensive experiments to validate their claims. The thoroughness of these experiments provides strong evidence for the dataset's effectiveness and the benefits of fine-tuning on it.", "weaknesses": "1. Clarification Needed for Methodology in Section 3.4: The description of the method used to filter high-quality instructions in Section 3.4 is unclear. \n2. Lack of Results for GPT-3.5 and GPT-4: The experimental section does not show results of GPT-3.5 and GPT-4 at Level 1 and Level 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)