Gorilla: Large Language Model Connected with Massive APIs
Authors: Shishir G Patil
Tianjun Zhang
Xin Wang
Joseph E. Gonzalez
Venue: NeurIPS 2024 poster (2024)
Keywords: LLM
Tool Use
APIs
Function Calling
Primary area: generative_models
TL;DR: Teaching LLMs to use tools at scale with innvoations in finetuning (RAT) and a novel way to mesasure hallucination using AST.

Abstract: Large Language Models (LLMs) have seen an impressive wave of advances, with
models now excelling in a variety of tasks, such as mathematical reasoning and
program synthesis. However, their potential to effectively use tools via API calls
remains unfulfilled. This is a challenging task even for today’s state-of-the-art
LLMs such as GPT-4 largely due to their unawareness of what APIs are available
and how to use them in a frequently updated tool set. We develop Gorilla, a
finetuned LLaMA model that surpasses the performance of GPT-4 on writing API
calls. Trained with the novel Retriever Aware Training (RAT), when combined
with a document retriever, Gorilla demonstrates a strong capability to adapt to
test-time document changes, allowing flexible user updates or version changes.
It also substantially mitigates the issue of hallucination, commonly encountered
when prompting LLMs directly. To evaluate the model’s ability, we introduce
APIBench, a comprehensive dataset consisting of HuggingFace, TorchHub, and
TensorHub APIs. The successful integration of the retrieval system with Gorilla
demonstrates the potential for LLMs to use tools more accurately, keep up with
frequently updated documentation, and consequently increase the reliability and
applicability of their outputs. Gorilla’s code, model, data, and demo are available
at: https://gorilla.cs.berkeley.edu

Gorilla: Large Language Model Connected with Massive APIs
Shishir G. Patil
Tianjun Zhang
Xin Wang
Joseph E. Gonzalez'
UC Berkeley
Microsoft Research
shishirpattllbeexkeeee..ee
Abstract
Large Language Models (LLMs) have seen an impressive wave of advances, with models now excelling in a variety of tasks, such as mathematical reasoning and program synthesis. However, their potential to effectively use tools via API calls remains unfulfilled. This is challenging task even for today's state-of-the-arr LLMs such as GPT-4 largely due to their unawareness of what APIs are available and how to use them in frequently updated tool set. We develop Gorilla, a finetuned LLaMA model that surpasses the performance of GPT-4 on writing API calls. Trained with the novel Retriever Aware Training (RAT), when combined with document retriever, Gorilla demonstrates strong capability to adapt to test-time document changes, allowing flexible user updates version changes. It also substantially mitigates the issue of hallucination, commonly encountered when prompting LLMs directly. To evaluate the model's ability, introduce APIBench, a comprehensive dataset consisting of HuggingFace, TorchHub, and TensorHub APIs. The successful integration of the retrieval system with Gorilla demonstrates the potential for LLMs to use tools more accurately, keep with frequently updated documentation, and consequently increase the reliability and applicability of their outputs. Gorilla's 's code, model, data, and demo are available https: pettps://gorilaae llgggrrllll. cs. berkeley.... edu
1 Introduction
The use of APIs and Large Language Models [10, 3,, 6, 27, 28] has changed what means to program. Previously, building complex machine lcarning software and systems required extensive time and specialized skills. Now with tools like the HuggingFace API, an engineer can set up deep learning pipeline with few lines of code. Instead of searching through StackOverflow and documentation, developers can ask models like GPT for solutions and receive immediate, actionable code with docstrings. However, using off-the-shell LLMs generate API calls remains unsolved because there are millions of available APIs which are frequently updated.
We connect LLM's and massive API's with Gorilla, system which takes an instruction, for example "build me a classifier for medical images", and provides the corresponding API call and relevant packages, along with step-by-step explanation of the pipeline. Gorilla uses self-instruct, fine-tuning, and retrieval to enable LLMs to accurately select from a large, overlapping, and changing set tools expressed using their APIs and API documentation. Further, novel retriever-aware training (RAT) enables the model to adapt to test-time changes of APIs such as evolution in versions and arguments.
With the development of API generation methods comes question of how to evaluate, as many APIs will have overlapping functionality with nuanced limitations and constraints. Thus, we construct
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

"Help find API convert the spoken language recorded audio text using Torch Hub."
<doraln>: Spooch-to-Poxt
<apt provider); Torchilub
(Gorilia)
asz model tocch. hub load! "makers(/alerroomddob "iller_est") result eap model. transcribe (audio _pach_
Speech-to-
API Argument Hallucination
<api, provides>:
<ooda>:
(GPT-4)
aac model torch, hub load "naakrss///aa /aakers/asleroomniiill aee sourcee (Lccal
result asz mo_el. transcribe padioo_ath))
Trannelatiiin proi_peo Pytorch
Incorrect
API Call
<oodad:
(Claude)
inport torchaudio
translation torchaudio ceschaudio.ppppliie MAY2VEC2 ASB PIPELINE ('uudo..va")
Figure I: Examples of API calls. Example API calls generated by GPT-4 [27], Claude [2], and Gorilla for the given prompt. In this example, GPT-4 presents model that doesn't exist, and Claude picks incorrect library. contrast, our Gorilla model can identify the task correctly and suggest a fully- qualified API call.
0-shot
with BM25-Retriever
with GPT-Retriever
with Oracle Retriever
100
100
100
100
Gorilla
Better
90
Gorilla
GPT-3.5
Accuracy
79
70-
GPT-3.5
GPT-3.5
Caude
Claude GPT-4
GPT-3.5
Claude
GPT-4
GPT-4
Goril
GPT-4
LLAMA
LLAMA
Claude
LLAMA
Hallucination
lucination
Hallucination
Hallucination
Figure 2: Accuracy (vs) hallucination in four settings, that is. zero-shot (i.e., without retriever). and with retrievers. Commonly used BM25 and GPT retrievers, and the oracle returns relevant documents with perfect recall, indicating an upper bound. Higher in the graph (higher accuracy) and the left (lower hallucination) better. Across settings, our model, Gorilla, improves accuracy while reducing hallucination.
APIBench (~ 1600 APIs) by scraping large corpus of MI. APIs and developing evaluation framework that uses AST sub-tree matching to check functional correctness. Further, we draw distinction between accuracy and hallucination, and propose an Abstract Syntax Tree (AST) based technique to measure hallucination.
Using APIBench, finetune Gorilla, LLaMA-7B- ---ss-d model with document retrieval and show that it significantly outperforms both open-source and closed-son soseed models like Claude and GPT-4 in terms of API functionality accuracy well reduction in API argument hallucination errors. We show an example output in Fig. 1. Lastly, we highlight Gorilla's is capability to comprehend and reason about user-defined constraints when choosing between APIs, essential requirement for LLMs trained to accomplish tasks.
To summarize, this paper makes the following contributions:
1. We introduce Gorilla, the first system enable large-scale API integration with LLMs, demonstrating state-of of-the-art performance in generating accurate API calls across thousands of functions and libraries.
2. We develop Retriever- Aware Training (RAT), novel technique that enables LLMs to effectively utilize retrieved API documentation at inference time, improving both accuracy and adaptation to API changes.
3. We present APIBench, comprchensive benchmark of ~ 1600 machine learning APIs, along with new baseb evaluation metrics that precisely measure both functional correctness and API hallucination.

---

2 Related Work
By empowering LLMs to use tools [33], we can grant LLMs access to vastly larger and changing knowledge bases and accomplish complex computational tasks. By providing access to search technologies and databases, [24, 39, 35] demonstrated that we can augment LLMs to address a signif- icantly larger and more dynamic knowledge space. Similarly, by providing access to computational tools, [39, 1, 49, 36, 37] demonstrated that LLMs can accomplish complex computational tasks. Consequently, leading LLM providers [27]. have started to integrate plugins to allow LLMs to invoke external tools through APIs.
Large Language Models Recent strides in the field of LLMs have renovated many downstream domains [10, 40, 48, 47], not only in traditional natural language processing tasks but also in program synthesis. Many of these advances are achieved by augmenting pre-trained LLMs by prompting [44, 14] and instruction fine-tuning [11, 30, 43, 15]. Recent open- open-sourced models like LLaMa [40], Alpaca [38], and Vicuna [9] have furthered the understanding of LLMs and facilitated their experimentation. While our approach, Gorilla, incorporates techniques akin to those mentioned, its primary emphasis is on enhancing the LLMs' ability to utilize millions of tools, as opposed to refining their conversational skills. Additionally, we pioneer the study of fine-tuning base model by supplementing it with information retrieval first, to the best of our knowledge.
Tool Usage The discussion of tool usage within LLMs has seen an upsurge, with models like Toolformer taking the lead [33, 19, 20, 24]. Tools often incorporated include web-browsing [32], calculators [12, 39], translation systems [39], and Python interpreters [14]. While these efforts be seen as preliminary explorations of marrying LLMs with tool usage, they generally focus on specific tools. Our paper, in contrast, aims to explore a vast array of tools (i.e., API calls) in an open-ended fashion, potentially covering a wide range of applications.
With the recent launch of Toolformer [33] highlights the exciting potential of using large language models (LLMs) for purposes beyond traditional chatbot applications. Moreover, the application of API calls in robotics has been explored to some extent [41, 4]. However, these works primarily aim at showcasing the potential of "prompting" LLMs rather than establishing a systematic method for evaluation and training (including fine-tuning). Our work, the other hand, concentrates on systematic evaluation and building a pipeline for future use.
LLMs for Program Synthesis Harnessing LLMs for program synthesis has historically been a challenging task [22, 7, 45, 16, 13, 29]. Researchers have proposed an array of strategies to prompt LLMs to perform better in coding tasks, including in-context learning [44, 18, 7], task decomposition [17, 46], and self-debugging [8, 34]. Besides prompting, there have also been efforts to pretrain language models specifically for code generation [25, 21, 26].
DocPrompting [49] looked at choosing the right subset of code including API along with a retriever. Gorilla presents distinct advancements over DocPrompting. First, the way the data-sets are constructed are different, leading to intersting downstream artifacts. Gorilla focuses on model usages where we also collect detailed information about parameters, performance, efficiency, etc. This helps our trained model understand and respond to finer constraints for each API. Docprompting focuses on generic API calls but not on the details within an API call. Second, Gorilla introduces and uses the AST subtree- matching evaluation metric that helps measure hallucination which we find are more representative of code structure and API accuracy compared to traiditional NLP metrics. Finally, Gorilla focuses on instruction-tun -tnning method and has "agency" to interact with users while DocPrompting focuses on building an NLP-to-Code generative model. On equal footing, we demonstrate that Gorilla performs better than DocPrompting in Appendix A.3.
3 Methodology
We first describe APIBench, a comprehensive benchmark constructed from TorchHub, TensorHub, and HuggingFace API Model Cards. We begin by outlining the process of collecting the API dataset and how we generated instruction-answer pairs. We then introduce Gorilla, a novel training paradigm with an information-retriewe incorporated into the training and inference pipelines. Finally, we present our AST tree matching evaluation metric.
3

---

[ JSON
PVTORCH
GORILLA
firsttuct (istucttionAAl
auativel TersorFlow each domain).
Conila-8
Huggingface (Tnp
atilli www.stttbebdeeeeeeeeeeeeeeeeeeee
mant to some carr dewoing In schebraciess"
Information Rear ever
tmaps
sabachhuseppiiiihrrr patsonad) -)
GORILLA
Execution
Zero-shot
Results!
Figure 3: Gorilla: system for enabling LLMs to interact with APIs. The upper half represents the training procedure as described in Sec 3. This the most exhaustive API data-set for MI. to the best of our knowledge. During inference (lower half), Gorilla supports two modes with retrieval, and zero-shot. In this example, it is able to suggest the right API call for generating the image from the user's natural language query.
3.1 Dataset Curation
To curate the dataset, we aggregate all model cards Hub, and TensorFlow Hub. Throughout the rest of the paper, call these HuggingFace, Torch Hub, and TensorFlow Hub respectively for brevity.
from HuggingFace's "The Model Hub", P PyTorch
API Documentation The HuggingFace platform hosts and servers about 203,681 models. However, many of them have poor documentation, lack dependencies, have no information in their model card, etc. To filter these out, pick the top 20 models from each domain. We consider 7 domains in multimodal data, 8 in CV, 12 in NLP, in Audio, 2 in tabular data, and 2 in reinforcement arrive at a total of 925 models from HuggingFace. TensorFlow Hub
learning. Post filtering, versioned into v1 and v2. The latest version (v2) has 801 models in total, and we process all of them. After filtering model cards with little no information,
are left with 626 models. Similar to TensorFlow Hub, we extract 95 models (exhaustive) from Torch Hub. We then convert the model cards for each of these 1,645 API calls into JSON object with the following fields: {domain, framework, functionality, api api_name, api_call, api ppi_arguments, enviromment reriuueentrruurrmmettt example_coo peanp performance, description): J. We provide more information in Appendix A.I. These fields were chosen to generalize beyond API calls within the ML domain, to other domains, including RESTful, SQL, and other potential API calls.
Instruction Generation Guided by the self-instruct paradigm [42], we employ GPT-4 to generate synthetic instruction data. We provide three in-context examples, along with reference API documen- tation, and task the model with generating real-world use cases that call upon the API. We specifically instruct the model to refrain from using any API names or hints when creating instructions. We constructed 6 examples (Instruction-Al) were the only hand-generaan or modified data. For each of our 1,645 API datapoints, we generate 10 instruction-AP pairs by sampling 3 of 6 corresponding instruction examples in each pair (Fig. 3).
pairs) for each of the 3 model hubs. These 18 examples
API Call with Constraints API calls often come with inherent constraints. These constraints necessitate that the LLM not only comprehend the functionality of the API call but also categorize the calls according to different constraint parameters. Specifically, for machine learning API calls, two common sets of constraints are parameter size and lower bound on accuracy. Consider, for instance, the following prompt: "Invoke an image classification model that uses less than 10M parameters, but maintains an ImageNet accuracy of at least 70%." Such prompt presents substantial challenge for the LLM to accurately interpret and respond to. Not only must the LLM understand the user's functional description, but it also needs to reason about the various constraints embedded within the request. This challenge underlines the intricate demands placed on L.MS real-world API calls. is not sufficient for the model to merely comprehend the basic functionality of an API call; it must

---

also be capable of navigating the complex landscape of constraints that accompany such calls. We also incorporate these instructions in our training dataset.
3.2 Gorilla
Our model, called Gorilla, a retriever-aware finetuned LLaMA-7B model, specifically for API calls. As shown in Fig. 3, we employ self-instruct to generate (instruction, API) pairs. To fine-tune LLaMA, we convert this to a user-agent chat-style conversation, where each datapoint a conversation with one round each for the user and the agent. We then perform standard instruction finetuning on the base LLaMA-7B model. For our experiments, we train Gorilla with and without the retriever. We would like to highlight that though we used the LLaMA model, our fine-tuning is robust to the underlying pre-trained model (see Appendinx A.3.5).
Retriever-Aware training (RAT) In retriever-aware training, the instruction-tuned dataset also appends to the user prompt, the relevant retrieved documentation with "Use this API Bitrrieved_AI__do_____
documentation for reference: the retrieved documentation is not necessarily accurate - retrievers have imperfect re-call. By aug- menting the prompt with potentially incorrect documentation, but the accurate ground-truth in the LLM response, we are in-effect teaching the LLM to 'judge' the retriever at inference time. During inference, if the LLM reasons that the retriever presented a relevant API document, it can use the API documentation to respond to the user's 's question, filling in additional details from the user's prompt. However, if after looking at the prompt, the L.LM reasons that the retrieved API document is not relevant to the user's ' prompt, RAT trains the model to not get distracted by irrelevant context. The LLM then relies on the domain- sppciffc knowledge baked-in during RAT training. to provide the user with the relevant API. Through RAT, we aim to teach the LLM to parse the second half of the question (API documentation) to answer the first half (user's query). We demonstrate that this (1) makes the LLM adapt to test-time changes in API documentation, (2) improves performance from in-context learning, and (3) reduces hallucination error.
This critical, because
Surprisingly, we find that augmenting a LLM with retrieval, does not always lead to improved performance, and can at-times hurt performance. We share more insights along with details in Sec 4.
Gorilla Inference During inference, the user provides the prompt in natural language (Fig. 3). This can be for a simple task (e.g.. "I would like to identify the objects in an image"), "), or they can specify a vague goal, (e.g., "I going to the 200, and would like track animals"), Gorilla, similar to training, can be used for inference in two modes: zero-shot and with retrieval. In the zero-shot setting, this prompt (with no additional prompt tuning) is fed to the Gorilla LLM model, which then returns the API call needed to accomplish the task or goal. In retrieval mode, the retriever (either of BM25 or GPT- first retrieves the most up-to-date API documentation stored in the API Database. Before being sent to Gorilla, the API documentation is concatenated to the user prompt along with the message "Use this API documentation for reference." " The output of Gorilla is an API to be invoked. Besides the concatenation as described, we do no further prompt tuning in our system. While we also implemented a system execute these APIs, to help the user accomplish the goal, that is not a focus of this paper.
3.3 Verifying APIS
Inductive program synthesis, where a program is synthesized to satisfy test cases, has found success in several avenues [3. 23]. However, test cases fall short when evaluating API calls, as it is often hard to verify the semantic correctness of the code. For example, consider the task of classifying an image. There are over 40 different models that can be used for the task. Even if we were to narrow down to single family of densenet, there are four different configurations possible. Hence, there exist multiple correct answers and it is hard to tell if the API being used is functionally equivalent to the reference API by unit tests. Thus, to evaluate the performance of our model, we compare their functional equivalence using the dataset we collected. To trace which API the dataset is the LLM calling, we adopt the AST tree-matching strategy. Since we only consider one API call in this paper, checking if the AST of the candidate API call is a sub-tree of the reference API call reveals which API is being used in the dataset.
5

---

bochhhub..oaddpyyoocciiiiiiooo 'densenet121', Tretrained=True)
torch
utils
hub
Tensor
torch
model_2on
load
e
repo:
repo: huggingface/ prtorch-transformers
load
pytorch/vision
Pytorch/vision
model: densenet161
model:
model: densenet201
densenet121
densenet121
optional pretrained True
pretrained:
pretrained: False
Figure 4: AST Sub-Tree Matching to evaluate API calls. On the left is an API call returned by Gorilla. We first build the associated API tree. We then compare this to our dataset, to see if the API dataset has subtree match. In the above example, the matching subtree highlighted in green, signifying that the API call is indeed correct. Pretrained-True an optional argument.
Identifying and even defining hallucinations can be challenging. We use the AST matching process to directly identify the hallucinations. We define a hallucination as an API call that is not sub-tree of any API in the database invoking an entirely imagined tool. This form of hallucination is distinct from invoking an API incorrectly which we instead define an error. So, in our evaluations, error, hallucination. and accuracy add up one.
AST Sub-Tree Matching We perform AST sub-tree matching to identify which API our dataset the LLM calling, Since each API call can have many arguments, we need match on each of these arguments. Further, since, Python allows for default arguments, for each API, we define which arguments match in our database. For example, we check repo_or_da or_dir and model arguments in our function call. In this way, we can easily check if the argument matches the reference API or not. Fig. 4 illustrates an example subtree check for a torch API call. We first build the tree, and verify that it matches subtree in our dataset along nodes torch. hub. load, pytorch/ gytorch/vision and densenet121. We do check for match along leaf node pretrained- since that an optional argument.
Evaluation
When evaluating Gorilla, finetuned APIBench (train ser), aim answer the following questions: How does Gorilla compare to other LLMS on API Bench (lest ser)? (4.1). 1). How well does Gorilla adapt test-time changes in API documentation? ( (4.2). How well can Gorilla handle questions with constraints? (4.3)
We demonstrate that Gorilla outperforms both open-source and close-source models for in-domain function calling. Further, trained with our novel retriever-aware training (RAT) technique, the Gorilla model generalizes APIs that are outside of its training data (out-of-domain). In addition, we assess Gorilla's ability to reason about API calls under constraints. Lastly, we examined how integrating different retrieval methods during training influences the model's final performance.
Baselines We primarily compare Gorilla with state-of sfate-of-the-art language models a zero-shot setting and with 3-shot in-context learning. The models under consideration include: GPT-4 by OpenAI with the gpt-4-0314 checkpoint; GPT-3.5-uuro with the g.t-3.5 turbo-0301 checkpoint, both of which are RLHF-tuned models specifically designed for conversation; Claude with the claude-vi checkpoint, a language model by Anthropic, renowned for its lengthy context capabilities; and LLaMA-7B, state-of-the-art open-source large language model by Meta.
Retrievers The term zero-shot (abbreviated as 0-shot tables) refers to scenarios where no retriever is used. The sole input to the model is the user's natural language prompt. For BM25, consider

---

Torch Hub with GPT-Retriever
Fage with GPT-Retriever
Tensorflow Hub with GPT-Retriever
100
Accuracy (%)
80
60.21
59.13
60.21
61.82
65.59
55.62
64.96
47.34
44.57
41.37
47.45
43.94
40-
20
14.51
10.17
15.62
GPT-3.5 GPT-4 Claude Gonila GPT-3.5 GPT-4 Claude
LLAMA
Gorilla GPT-3.5
Claude
LLAMA
Gorilla
Figure 5: Accuracy with GPT-retriever. Methods to the left of the dotted line are closed source. Gorilla outperforms Torch Hub and Hugging-Face while matching performance on Tensorflow Hub for all existing state-of-the-art LLMs closed source, and open source.
each API as a separate document. During retrieval, we use the user's query fetch the most relevant (top-1) API. This API is concatenated with the user's prompt to query the LLMs. Similarly, GPT- Index refers to state-of-the-art embedding model, text- seert--meedding-ddaaaaae where each embedding is 1,536 dimensional. Like BM25, each API call is indexed an individual document, and the most relevant document, given user query, is retrieved and appended to user prompt. Lastly, we include an Oracle retriever, which serves two purposes: first, to identify the potential for performance improvement through more efficient retrievers, and second, to assist users who know which API to use but may need to help invoking it. In all cases, when retriever is used, is appended to the user's prompt as follows: <user wnser_prompt> Use this API documentati for reference <retrieved
from OpenAI,
The dataset for these evaluations is detailed in Section 3. We emphasize that we have maintained a holdout test set on which we report our findings. The holdout test set was created by dividing the self-instruc dataset's instruction, API pairs into training and testing sets.
4.1 AST Accuracy on API call
We test each model for different retriever settings defined above (Table 1). We report the overall accuracy, the by hallucination and the error by selecting wrong API call. Note that for TorchHub and TensorHub, we evaluate all the models using AST tree accuracy score. However, for Hugging since the dataset cannot be exhaustive given the sheer number of models hosted, for all the models except Gorilla, we only check if they can provide the correct domain names. So this problem reduces to picking one of multiple choices. Across 0-shot and few-shot prompting strategies, Gorilla outperforms close-sourced and open-sourced models (Table 5).
Finetuning without Retrieval In Table we show that lightly fine-tuned Gorilla is able to match, and often surpass performance in the zero-shot setting compared closed-source, and open-source models - 20.43% better than GPT-4 and 10.75% better than GPT-3.5 (ChatGPT). When compared to other open-source models LLAMA, the improvement is as big 83%. This suggests quantitatively. that as a technique to augment information and enforce adherence to syntax, fine-tuning is better than naive retrieval, at-least within the scope of invoking APIs.
Finetuning with Retrieval We now discuss how incorporating retrieval (RAT) during LLM fine- tuning enhances model performance. In this experiment, the base LLAMA model is finetuned with a prompt (instruction generrttdddrrrddd) a reference API document (from a golden-trute truth oracle), and an example output generated by an LLM (GPT-4 in this case). As shown in Table 2, incorporating ground-trutt truth retriever in the finetuning pipeline yields notably improved results 12.37% higher accuracy than training without retrieval in Torch Hub and 23.46% better in HuggingFace. However, at ground-truth
evaluation time, current retrievers show significant performance gap compared retriever: using Gndel at evaluation results in 29.20% accuracy degradation and using BM25 results in a 52.27% accuracy degradation. Despite this, considering the trends across models and re- trievers, findings indicate that finetuning an LLM with effective retrieval integration is preferable to zero-shot finetuning.
Hallucination with LLM One phenomenon we observe is that zero-shot prompting with LLMs (GPT-4/GPT-3.5) to call APIs results in dire hallucination errors. These errors,

---

Table 1: Evaluating LLMss Torch Hub, HuggingFace, and Tensorflow Hub APIs
LLM (retriever)
TorchHub ballu +
HuggingFace hallu
TensorFlow Hub
overall
err
overall
er
overall
hallu
LLAMA (0-shol)
100
97.57
2.43
100
GPT-3.5 (0-shot)
48.38
18.81
32.79
16.81
35.73
47.46
41.75
47.88
10.36
GPT- (0-shot)
38.70
36.55
24.7
19.80
37.16
43.03
18.20
78.65
3.13
Claude (0 shot)
15.59
16.15
Gorilla (0-shot)
59.13
6.98
33.87
71.68
10.95
17.36
83.79
5,40
10.80
LLAMM (BM-25)
8.60
76.88
14.51
3.00
77.99
19.02
8.90
77.37
13.72
GPT-3.5 (BM-25)
54.83
17.26
8.30
74.44
54.16
3.64
42.18
GPT-4 (BM-25) 25)
35.48
11.29
53.22
16.48
15.93
67.59
34.01
37.08
28.90
Claude (BM-25)
39.78
54.83
15.82
69.58
21.16
Gorilla (BM-25)
40.32
4.30
55.37
17.03
6,42
76.55
41.89
2.77
55.32
LLAMA (GPT-Index)
14.51
75.8
9.67
10.18
75.66
14,20
15.62
77.66
6.71
GPT-3.5
GPT-4 (GPT-Index)
59.13
1.07
39,78
44,58
11.18
44.25
43.94
31.53
24.52
Claude (GPT-Index)
60.21
3.76
36,02
41.37
18.81
39.82
55.62
16,20
28.17
Gorilla (GPT-l
61.82
38,17
47.46
8.19
44.36
64.96
2.33
32.70
I.LAMA (Oracle)
16.12
79.03
4.83
17.70
77.10
5.20
12.55
87.00
0.43
GPT-3.5 (Oracle)
66.31
32.08
89.71
3.65
95.03
GPT-4 (Oracle)
66.12
0.53
33.33
85.07
10.62
4.31
55.91
37.95
6.13
Claude (Oracle)
63.44
3.76
32.79
77.21
19.58
3.21
74.74
21.60
3.64
Gorilla (Oracle)
67.20
32.79
91.26
7.08
1.66
94.16
1.89
3.94
Table Understanding the effect of different retrieval techniques used with Gorilla
Gorilla without Retriever
Gorilla with Oracle retriever
zero-shot
BM25
GPT-Index
Oracle
zero-shot
BM25
GPT-Index
Oracle
Hub (overall)
59.13
37.63
40.32
61.82
67.20
HuggingFace (overall)
71 66
11.28
28.10
45.58
0
17.04
47.46
91.26
TensorHub (overall)
83.79
34,30
52.40
82.91
41.89
64.96
94.16
Hub (Hallu)
6.98
11.29
15.59
100
4.30
HuggingFace Face (Hallu) L.
10.95
46.46
41.48
52.77
99.67
6.42
8.19
7.08
TensorHub (Hallu)
5.40
20.43
19.70
13.28
100
2.77
2.33
1.89
while diverse, commonly manifest in erroneous behavior such as the model invoking the AutoModel. fwrm_prrrrrrre portopppeeaaeeeeeee (dir (dir_name) command with arbitrary GitHub repository names. Surprisingly, also found that TorchHub, HuggingFace and TensorFlow Hub, GPT-3.5 has less hallucination errors than GPT-4. This finding is also consistent for the settings when various retrieving methods are provided: 0-shot, BM25, GPT-IIndex and the oracle. This might suggest that RLHF plays central role in turning the model to be truthful. Additional discussion in Appendix A.3.
AST Hallucination Metric We manually execute Gorilla's API generations to evaluate how well AST works as an evaluation metric. Executing every code generated is impractical within academic setting-for example, executing the HuggingFace model needs the required library dependencies (e.g., transformers, sentencepiece, accelerate). correct coupling of software kernels (e.g., torch vision, torch, cuda, cudnn versions), and required hardware support (e.g., A 100 40G gpus). Hence, to make tractable, we sampled 100 random Gorilla generations from our evalualtion set. The accuracy from our AST subtree matching is 78%. consistent with human evaluation of 78% accuracy in calling the right API. All the generations that AST flagged as incorrect, were the same ones that were manually flagged as incorrect. Additionally, Gorilla also generates supporting code call the API which includes installing dependencies e.g., pip install transf ormers [ [sentencepieco ), setting environment variables, etc. When we manually attempt to execute the code, 72% of all code generated executed successfully. It's worth noting that the 6% discrepancy are not semantic errors, but errors that arose due to factors external to the API, and in the supporting code. We have included the full example to illustrate this further in A.3.3. Considering the significant time and effort required for manual validation each generation, the strong correlation between human evaluation and the AST evaluation further reinforces belief in using the proposed AST robust offline metric.

---

Default Response
Update the model
Update the model repository
USER: want auto matically remeve the background from an input in age. cketrieval: 1enuresn:::00.
USER: want automatically remove the background from an input image <Ratrisval resne:11000
USER want automatically remove
background ound from an inpurt mage <Ratrieval: war.oggeeeennnnmllsllse....
GORILLA api. call:
GORILLA aoi call:
GORILL up cal::
"torch hub loadd pytorch/siin,, "en_rresnet50', pretrained Truel"
"oven rub oeetpyytorch/ssion
"nrrc hub IviDIAV
"fon_ fon_resnett01: pretrained Truel"
fen_ressee550",
pretrained-Truse
Figure 6: Gorilla's retriever-aware training enables it to react changes in the APls The second column demonstrates changes in model upgrading FCN's ResNet- 50 backbone to ResNet- 101. The third column demon- strate changes model registry from pytorch/vision WYDDAA/DeepLearmnnnggeeema
tor chhub
Table 4: Evaluating LLMs on constraint-aware API invocations
GPT-3.5
GPT.4
0-shot
BM25
GPT-Index
Oracle
D-shor
BM25
GPT-Index
Oncele
0-shot
BM25
GPT-Index
Orucle
Torch Hub (overall)
62.67
62.67
56.33
57.04
71.83
Torch Huh (Hallu)
19.01
30.98
14.78
14.08
15.49
27.46
14.08
9.15
19.71
39.43
26.05
16.90
Torch Hulh (enr)
7.04
6,33
3,52
4.92
21.83
16.19
14:78
21.83
8.45
3.52
2.11
4.92
Accumey const
43.66
33.80
33.09
69.01
43.66
29.57
29.57
59.15
26.76
67.60
LLAMA
Claude
O-shot
BM25
GPT-ludex
Oracko
0-shot
BM25
GPT-Endex
Oracle
Torch Hub (overall)
0
8.41
11.97
19.71
29.92
81.69
82.39
81.69
Torch Hub (Hallu)
100
91.54
88.02
78.87
67.25
16.19
15.49
13.38
Huh (em)
1.4
2.81
2.11
2.11
4.92
Accuracy const
0
6.33
3.52
17.60
17.25
29.57
31.69
69.71
Table 3: Proposed AST evaluation metric has strong correlation with human evaluation
Accuracy
Gorilla AST metric (proposed)
0.78
Eval by Human
0.78
Code Executable (Eval by Human)
0.72
4.2 Test-Time Documentation Change
The rapidly evolving nature of API documentation presents a significant challenge for the application of LLMs in this field. These documents often updated at frequency that outpaces the re- training or fine-tuning schedule of LLMs, making these models particularly brittle to changes in the information they designed to process. This mismatch in update frequency can lead to decline in the utility and reliability of LLMs over time.
With the introduction of Gorilla's 's retriever-awarr training, the RAT trained LLM readily adapts to changes in API documentation. This novel approach allows the model to remain relevant, even as the API documentation it relies undergoes modifications. This is a pivotal advancement in the field, as it ensures that the LLM maintains its efficacy and accuracy over time, providing reliable outputs irrespective of changes in the underlying documentation.
For instance, consider the scenario illustrated in Fig. 6, where the training of Gorilla has allowed it to react effectively to changes in APIs. This includes alterations such as upgrading the FCN's ResNet-50 backbone ResNet-101, as demonstrated in the second column of the figure. Since the model has encountered ResNet- 101 as backbone with other architectures, it interprets an FCN with a ResNet-101 backbone (unseen during training) as relevant document test time. Conversely, the retriever suggests an FCN with a ResNet-60 backbone, the from RAT-assigns low confidence to this document and defaults back to FCN with ResNet-50. The third column in Fig. 6 further illustrates Gorilla's 's flexibility in adapting to shifts in model registries, torchhub, highlighting its
with ResNet-60
such as from pytorch/vit hearrch/viiionoo to SeaDDD/DDeeLLaannng.ae. ability to accommodate changes in preferred API sources they evolve over time.

---

Table 5: Evaluating Gorilla 0-shot with GPT 3-shot incontext examples
HF (A
HF (Hall J)
(Ace
TH (Hall J)
(Ace 1)
(Hall ()
GPT:335 (0-shot)
16.81
35.73
41,93
10.75
41.75
47.88
GPT:4 (0-shot)
19.80
37.16
54,30
34.40
18.20
78.65
GPT-3.5 (3 incont)
25.77
32.30
73.11
72.58
71.82
11.09
GPT:4 (3 incont)
26.32
35.84
75.80
13.44
77.37
11.97
Govilla (0-shor)
58.05
28.32
75.80
16.12
83.79
5.40
In summary, Gorilla's ability to adapt to test-time changes in API documentation offers numerous benefits. It maintains its accuracy and relevance time, adapts the rapid pace of updates in API documentation, and adjusts to modifications in underlying models and systems. This makes a robust and reliable tool for API calls, significantly enhancing its practical urility.
4.3
API Call with Constraints
We now focus on the language model's capability of understanding constraints. For any given task, which API call to invoke is typically tradeoff between multitude of factors. In the case RESTFul APIs, it could be the cost of each invocation (S) the latency of response (ms), among many others. Similarly, within the scope of ML APIs, is desirable for Gorilla to respect constraints such as accuracy, number of learnable parameters in the model, the size on disk, peak memory consumption, FLOPS,, etc. this section, we present study evaluating the ability of different models zero-shot and in the presence of retrievers to respect given accuracy constraint. if user requests an image classification model that achieves least 80% top-1 accuracy on the ImageNet dataset, then among the classification models hosted by Torch Hub, ResNeXt-100 32x16d, with top-1 accuracy of 84.2%, would be the appropriate model to call, rather than MobileNetV2, which has top-1 accuracy of 71.88%.
For Table 4, we filtered a subset of the Torch Hub component of APIBench, retaining those models that had an accuracy metric defined for at least one-dataset the model was evaluated on, in its model card. We were left with 65.26% of TorchHub dataset from Table 1. We notice that with constraints, understandably, the accuracy drops across all models, with and without retriever. Even in this challenging scenario, Gorilla is able to match the performance of the hest-performing model GPT-3.5 when using retrievals (BM25, Gndex,, and has the highest accuracy in the zero-shot setting. This highlights Gorilla's ability to navigate APIs while considering the trade-offs between constraints.
4.4 Finetuning (vs) Prompting: Gorilla 0-shot (ys) GPT 3-shot
To assess whether finetuning is truly necessary for APIs if prompting alone is sufficient, compare Gorilla a zero-shot setting with three-shot in-context prompting for GPT-3.5 and GPT-4 models. In Table 5, "3-incont" denotes evaluation using three in-context examples, while "HF," "TH," and "TF" represent the HuggingFace, TorchHub, and TensorFlow Hub subsets of APIBench, respectively. Higher accuracy (Acc) and lower hallucination (Hall) rates are preferred. From Table three-shot in-context learning improves the GPT models' ability to generate syntactically correct function calls, even matching accuracy on one subset (TorchHub). However, Gorilla O-shot still outperforms the 3-shot GPT models on average.
Conclusion
LLMs are swiftly gaining popularity across diverse domains. APIs, serving as universal language, essential for enabling LLMs to communicate and operate effectively across diverse systems. In this paper, we introduced Gorilla, state-of ofee---f-thetttt model for API invocation. Our Retriever Aware Training (RAT) approach empowers Gorilla with two essential capabilities: adapting dynamically to API changes at test time and reasoning through user-defined constraints when selecting suitable APIs. We also present APIBench, a comprehensive benchmark for assessing LLMs' function-calling abilities, and propose AST-based hallucination metrics for robust evaluation. Looking forward, we believe this work represents a first step towards transitioning L.LMs from knowledge-boun models into flexible interfaces that interact with the digital world.
10

---

References
[1] Andor, D., Hc, L., Lee, K., and Pitler, E. Giving bert a calculator: Finding operations and arguments with reading comprehension. arXiv preprint arXiv: 19X::0990.00100 2019.
[2] Anthropie. introducing claude.
Claude, 2022.
URL https://www. http///wwwwtttrp......o.... cwt.:/cccccciii.ie /oneiiidx///////
[3] Bavishi, R., Lemieux, C., Fox, R.,, Sen, K., and Stoica, I. Autopandas: neural-backed generators for program synthesis. Proceedings of the ACM on Programming Languages, (OOPSIA)), 2019,
[4] Brohan, A., Chebotar, Y., Finn, C., Hausman, K., Herzog, A., Ho, D., Ibarz, J., Irpan, A., Jang, E.,, Julian, R., et al. Do as can, not say: Grounding language in robotic affordances. In Conference on robot learning. PMLR, 2023.
[5] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., al. Language models are few-shot learners. Advances in neural information processing systems, 2020.
[6] Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, T., Li, Y., Lundberg. S., et al. Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv preprint ar::i::333:333 12712, 2023.
[7] Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03377. 2021.
[8] Chen, X., Lin, M.. Schärli, N., and Zhou, Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023.
[9] Chiang, W.-.., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, E.,, Stoica, I., and Xing, E. Vieuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
[10] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.0231/ 2022.
[11] Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction finetuned language models. Journal of Machine Learning Research, 2024.
[12] Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers solve math word problems. arXiv preprint arXiv:2110. 141668 2021.
[13] Devlin, J., Uesato, J., Bhupatiraju, S., Singh, R., Mohamed, A.-r,, and Kohli, P. Robustfill: Neural program learning under noisy i/o. In International conference on machine learning. PMLR, 2017.
[14] Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. Pal: Program-aie aided language models, In International Conference on Machine Learning. PMLR, 2023.
[15] Iyer, S.. Lin, X. V., Pasumuru. R., Mihaylov, T., Simig, D., Yu, P., Shuster, K., Wang, T., Liu, Q., Koura, P. S., et al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint 22. 12017, 2022.
[16] Jain, N., Vaidyanath, S., Iyer, A., Natarajan, N., Parthasarathy, S., Rajamani, S., and Sharma, R. Jigsaw: Large language models meet program synthesis. In Proceedings of the 44th International Conference on Software Engineering, 2022.
[17] Kim, G., Baldi, P., and McAleer, S. Language models can solve computer tasks. arXiv preprint 230.11303..7 2023.

---

[18] Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Advances in neural information processing systems, 2022.
[19] Komeili, M., Shuster, K., and Weston, Internet-augmented dialogue generation. arXiv preprint arXiv:2107.07666 2021.
[20] Lazaridou, A., Gribovskaya, E., Stokowiec, W., and Grigorev, N. Intereetaugmentedd language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05/15: 2022.
[21] Li. R. Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki. C., Li, J., Chim, J., et Starcoder: may the source be with you! arXiv preprint arXiv:2305.061 2023.
[22] Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., al. Competition- code generation with alphacode. Science, 2022.
[23] Menon, A., Tamuz, O., Gulwani, S., Lampson, B., and Kalai, A. A machine learning framework for programming by example. In International Conference on Machine Learning. PMLR, 2013.
[24] Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang. L.., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W. et al. Webgpt: Browser-assister question-answerinn with human feedback. arXiv preprint arXiv:2 12.09332, 2021.
[25] Nijkamp, E., Pang, B., Hayashi, H., Tu, I.., Wang, H., Zhou, Y., Savarese, S., and Xiong, C. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203. / 13474, 2022.
[26] Nijkamp, E., Hayashi, H., Xiong, C., Savarese, S., and Zhou, Y. Codegen2: Lessons for training Ilms on programming and natural languages. arXiv preprint urXiv:2305.023009 2023.
[27] OpenAI. Gpt-4 technical report, 2023.
[28] OpenAl and hutps//opeeai.ccmm/bbgg//aaagg Chatgpt, 2022. URL httthtttt////enaaal com/ blog/ /cattpt.....
[29] Roziere, B., Lachaux. M.-A., Chanussot, L., and Lample, G.
Unsupervised translation of programming languages. Advances in neural information processing systems, 2020.
[30] Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint 2rXiv:21 2021.
[31] Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ili,, S., Hesslow, D., Castagné, R., Luccioni, A. S.,
Yvon, F., Gallé, M., al. Bloom: 176b-parameter open-access multilingual language model. arXiv preprint 10XXi:22/1.05/00 2022.
[32] Schick, T. and Schütze, H. Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001 (.V:2200..07666, 2020.
[33] Schick, T., Dwivedi- Yu, J., Dessi, R., Raileanu, R., Lomeli, M., Hambro, E.,, Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to use tools. Advances in Neurul Information Processing Systems, 2023.
[34] Shinn, N., Labash, B., and Gopinath, A. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303. 1166,, 2023.
[35] Shuster, K., Xu, J., Komeili, M., Ju, D., Smith, F. M., Roller, S., Ung, M., Chen, M., Arora, K., Lane, J., et Blenderbot a deployed conversational agent that continually learns to responsibly engage. arXiv preprint arXiv:2208.03188. 2022.
[36] Subramanian, S., Narasimhan, M., Khangaonkar, K., Yang. K., Nagrani, A., Schmid, C., Zeng. A., Darrell, T., and Klein, D. Modular visual question answering via code generation. arXiv preprint a30iv: 20...0099..33 2023.
12

---

[37] Surís, D., Menon, S., and Vondrick, C. Vipergpt: Visual inference via python execution for reasoning, In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
[38] Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, B. Stanford alpaca: An instructionffolloiing llama model, 2023.
[39] Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos,
T., Baker, L.., Du, Y., et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239 2022.
[40] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A. Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F,, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13977, 2023.
[41] Vemprala, S., Bonatti, R., Bucker, A., and Kapoor, A. Chatgpt for robotics: Design principles and model abilities. 2023, 2023.
[42] Wang. Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022.
[43] Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A., Stap, D., al. huperrmaturallinstuttion::: Generalization via declarative instructions on 1600+ nlp tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022.
[44] Wei, J., Wang. X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 2022.
[45] Xu, F. F,, Alon, U., Neubig, G., and Hellendoorn, J. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, 2022.
[46] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
[47] Zeng, A., Liu, X., Du Z., Wang. Z.. Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., Xia, X., et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02411 2022.
[48] Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint 2022.
arXiv:2205.0100
[49] Zhou, S.. Alon, U., Xu, F. F., Jiang, Z., and Neubig, G. Doeprompting: Generating code by retrieving the docs. The Eleventh International Conference on Learning Representations. 2022.
13

---

A Appendix
A.1 Dataset Details
Our dataset is multi-faceted, comprising three distinct domains: Torch Hub, Tensor Hub, and HuggingFace. Each entry within this dataset is rich in detail, carrying critical pieces of information that further illuminate the nature of the data. Delving deeper into the specifics of each domain, Torch Hub provides 95 APIs. The second domain, Tensor Hub, is more expansive with a total of 696 APls. Finally, the most extensive of them all, HuggingFace, comprises 925 APIs.
To enhance the value and utility of our dataset, we've undertaken an additional initiative. With each API, we have generated set of 10 unique instructions. These instructions, carefully crafted and meticulously tailored, serve as a guide for both training and evaluation. This initiative ensures that every API is not just represented in our dataset, but is also comprehensively understood and effectively utilizable.
In essence, our dataset is more than just collection of APIs across three domains. It is compre- hensive resource, carefully structured and enriched with added layers of guidance and evaluation parameters.
Domain Classification The unique domain names encompassed within our dataset are illustrated Fig. 7. The dataset consists of three sources with diverse range of domains: Torch Hub houses domains, Tensor Hub accommodates a much broader selection with 57 domains, while HuggingFace incorporates 37 domains. To exemplify the structure and nature of our dataset, we invite you refer to the domain names represented in Fig. 8.
API Call Task In this task, we test the model's capability to generate a single line of code, either in zero-shot fashion by leveraging API reference. Primarily designed for evaluation purposes, this task effectively gauges the model's proficiency in identifying and utilizing the appropriate API call.
API Provider Component This facet relates to the provision of the programming language. In this context, the API provider plays vital role as serves as a foundation upon which APIs are built and executed.
Explanation Element This component offers valuable insights into the rationale behind the usage of a particular API, detailing how it aligns with the prescribed requirements. Furthermore, when certain constraints imposed, this segment also incorporates those limitations. Thus, the explanation element serves a dual purpose, offering a deep understanding of API selection, as well as the constraints that might influence such selection. This balanced approach ensures comprehensive understanding of the API usage within the given context.
Code Example code for accomplishing the task. We de-prioritize this as we haven't 't tested the execution result of the code. We leave this for future works, but make this data available in-case others want to build on it.
A.2 Gorilla Details
We provide all the training details for Gorilla in this section. This includes how we divide up the training, evaluation dataset, training hyperparameters for Gorilla.
Data For HuggingFace, we devise the entire dataset into 90% training and 10% evaluation. For Torch Hub and Tensor Hub, we devise the data in to 80% training and 20% testing.
Training We train Gorillafor epochs with the 2e-5 learning rate with cosine decay. The details provide in Table 6. We finetune it on 8xA100 with 40G memory each.
14

---

Torch Hub domain names: Classification, Semantic Segmentation, Object Detection, Audio Separation, Video Classification, Text-to-Speech
Tensor Hub domain names: text-eequence-aligmeen, text-embedding, text-language- model, text-preprocessingg text-classification, text-generation, text-question-answering text- aarrival--uestiionanvverir text-segmentation, text-to-mel, image-classffication image- feature-vector, image-object-detection. image-segmentation, image-generator. image-pose- detection, image-rnn-agee image-augmentation, image-classifier, image-style-trannie umage-aesthetic-qualit,, image-depth-essimation image-supper-reolltioon image-deblurring.
mnage-extrapolatioo image-text wwwteeeeet-eeeiiioooe image-dehazing, image-deraining, image-
enhancemenmt, mmage-classffiction--gge logiis,,, image-frame- interpolltooo,,innn image-tex, rext-detection,
image-others, iideo-classification video-feature- ceature-extraction video- generation, video-audio-tex. video-text, audio- embedding, audio-event clssi-clasiiiiaation, audio- command-detection, setiio-prraaiigustssllasss synthesis, audio-synthesis, audio-pitcin audio-picch-extraction
seaiio-speech-t--teee audio-speech-
HuggingFace domain names: Multimodal Feature Extraction, Multimodal Text-to-Image, Multimodal Image-to- Text, Multimodal Text-to- Video. Multimodal Visual Question An- swering, Multimodal Document Question Answer, Multimodal Graph Machine Learning, Computer Vision Depth Estimation, Computer Vision Image Classification, Computer Vision Object Detection, Computer Vision Image Segmentation, Computer Vision Image-to-Iman Image, Computer Vision Unconditional Image Generation, Computer Vision Video Classification, Computer Vision Zero-Shor Image Classification, Natural Language Processing Text Classi- fication, Natural Language Processing Token Classification, Natural Language Processing Table Question Answering, Natural Language Processing Question Answering, Natural Lan- guage Processing Zero-Shot Classification, Natural Language Processing Translation, Natural Language Processing Summarization, Natural Language Processing Conversational, Natural Language Processing Text Generation. Natural Language Processing Fill-Mask. Natural Lan- guage Processing Text2Text Generation, Natural Language Processing Sentence Similarity, Audio Text-to-Speech, Audio Automatic Speech Recognition, Audio Audio-to-Audic Audio Audio Classification, Audio Voice Activity Detection, Tabular Tabular Classification, Tabu- lar Tabular Regression, Reinforcement Learning Reinforcement Learning, Reinforcement Learning Robotics
Figure 7: Domain names: Domain names with the three dataset. Tensor Hub the smallest dataset while the other two hubs contain many more models.
Table 6: Hyperparameters for training Gorilla
Hyperparameter Name
Value
learning rate
2e-5
batch size epochs warmup ratio weight decay
64
0.03
max seq length
2048
A.3 Performance Comparison
We provide full comparison of each model's performance in this section. In Fig 10 and Fig. 11, the full set of comparisons is provided. We see that especially in zero-shot case, Gorilla surpasses the GPT-4 and GPT-3.5 by large margin. The GPT-4 and GPT-3.5 gets around 40% accuracy in Torch Hub and Tensor Hub, which are two structured API calls. Compared to that, HuggingFace more flexible and diverse Hub, as result, the performance HuggingFace is not as competitive.
15

---

### User: Our customer is and to help them detect movement of different animals. Write a python program in to 2 lines to call API TensorFlowHub.
### Assistant:
<domain>: image-object--deeetion
<api_call>: detector hub.load( load( http:://hhbbberrooooooppppggensp
<api_providers: TensorFlow Hub,
<explanation>: This API call loads the pre-trained Single Shot Detector (SSD) MobileNet V2 model from TensorFlow Hub, which has been trained on the Open Images v4 dataset. This model is used for object detection in images. Once loaded, you can pass an image (and an optional score threshold for detection confidence) to the detector to get back the detected objects along with their bounding boxes, class labels, and associated confidence scores.
<code>: import tensorflow as tf; import tensorflow hub as hub; # Load the SSD MobileNet V2 model; detector hub.load(" luallooad"httpp https://tfhubbbc/ httpss//tthbbbe///gogg///oop/poe # Use the model to detect objects in an image (assuming 'image" is a TensorFlow tensor representing your image); result detector score_threshold-055)
wttp///oooggleopennnaeesss _4/ssdd mobilenet v2/1');
#### User: am an engineer at Uber and need to find API that can classify pedestrians, etc. from image of the scene. Write python program in to 2 lines to call API in TorchHub.
<Reference API>: /domain: Object Detection, framework: PyTorch, function- ality: Traffic Object Detection, Drivable Area Segmentation, Lane Detection, api api_name: HybridNets, api_call: forch.hub.loadrepo.. wat_ooorddrrdatte dddtauthnnhhyybrrdneett model='hybridnets' pretrained=True), api api_arguments: repo_or_dir, model, pretrained, python nathon_envirommenttt anviiroomeenteeegiireennnss: Python>=3.7, PyTorch==1110, example_code code:: 'import t datvuthanb/yybrddeess
torch; model torch. hub.load(
'hybridnets', pretrained=True); img torch.randn(I.3, 11,,,..,,3....3.33888 640,384); features, regression, classification, anchors, segmentation
model(igg). performance: dataset: [name: BDD 10K, accuracy: Traffic Object Detection: Recall (%): 92.8, m.P@0.5 (%): 77.3, Drivable Area Segmentation: Drivable mloU (%);
90.5, Lane Line Detection: Accuracy (%): 85.4 Lane Line IoU (%): 31.6], description: HybridNets an end2end perception network for multi-tasks. Our work focused on traffic object detection, drivable area segmentation and lane detection. HybridNets can run rall-imme on embedded systems, and obtains SOTA Object Detection, Lane Detection on BDD100K Dataset.)
### Assistant:
<domain>: Object Detection, <api_call>> model = www.h.hub.loadddaaaaaa........a. hat.uub..oocclaattttaaaay/ynrrrr "hybridnets', pretrained-True),
<api_provider>: PyTorch.
<explanation>: Load the pretrained Hybrid model for object detection from PyTorch Hub to classify pedestrians, cars, etc. in an image of the scene,
<code>: import torch; model = torch.hub.load ddatruthanhhhyybrddents' 'hybridnets', pre- trained=True)
Figure 8: Example of the Dataset: Two examples of the dataset, the above one is zero-shot (without information retrievers) and the bottom one is with information retriever.
16

---

generate_ video pipeline("text-toviieee mooee/iidddddddddd"oo"od"o"oooo model="your modeeemooelleccceee name")
vqa
"uquesttionaaswering...
molee=
tokenizer=" microosoft/clip--gaabaa
depth_estiman estimator =
"onell="itteliss/Miiaa"" tokenizer="intel.
isl/MiDaS")
Figure 9: Hallucination Examples: GPT-4 incurs serious hallucination errors in HuggingFace. We show couple of examples in the figure.
A.3.1 Evaluation
For ease of evaluation, manually cleaned up the dataset ensure each API domain only contains the valid call of form:
PPL_nameAAPLaar argi, API argg.... AP[_rgg argk)
Our framework allows the user to define any combination of the arguments to check. For Torch Hub, we check for the API name torch. hub. load with arguments repo _or __dir and model. For Tensor Hub, we check API name hub. KerasLaaer and hub. load with argument handle. For HuggingFace, since there are many API function names, we don't list all of them here. One specific note is that we require the pretrained_modee laaeedd_oddldmeme_____eee __nam___rr____ or_ththhhhhhhppppppppppp argument for all the calls except for pipeline. For pipel ine, we don't require the pretrained, podel_namee pptth argument since it automatically select a model for you once task is specified.
A.3.2 Hallucination
We found especially in HuggingFace, the GPT-4 model incurs serious hallucination problems. It would sometimes put a GitHub name that is not associated with the HuggingFace repository in the domain of pretrained, model_nnme or_orr_____ pptt.. Fig. demonstrates some examples and we also observe that GPT-4 sometimes assumes the user have local path the model like your, your_modell wwt_mmdd__mmmm. This is greatly reduced by Gorilla as we see the hallucination error comparison in Table 1.
A.3.3 AST Hallucination Metric
We evaluated the generated results on 100 LLM generations (randomly chosen from our eval set). The accuracy using AST subtree matching is 78%, consistent with human evaluation with 78% accuracy in calling the right API. All the generations that AST flagged incorrect, were the same ones that were manually also flagged as incorrect. Additionally, Gorilla generates supporting code to call API which includes installing dependencies (e.g., pip install transf environment variables, etc. When we manually attempted to execute end-to- code, 72% of all codes generated were executed successfully. It's worth noting that the 6% discrepancy were NOT semantic errors, but errors that arose due to factors external to the API in the supporting code we have included an example to illustrate this further. Considering the significant time and effort required for manual validation of each generation, our evaluation highlights the efficiency of using AST as robust offline metric.
[sentencepiece)
Here is a representative example, where we are able to load the correct model API. However, in the supporting code, after we have the output from the API, the zip() function tries to combine sentiments and scores together. However, since scores is float, it's not iterable. zip() expects both its arguments to be iterable, resulting in an 'float object is not iterable error.
17

---

Torch Hub 0-shot
Torch Hub with BM25-Bilrrieeer
Torch Hub with Retrreeereeer
Torch Hub with Gracie Retriever
66.12
60.21
60.21 61.82
Acorrio
40.30
38.7
38.17
18.81
16.51
Model
Model
Modk
Mode:
HuggingFace
HuggingFace BM25-Retrlever
HuggingFace with GPT-Retrlever
HeggingFace Retriever
91.26
71.68
Accurbyy
Accurnoy
Accuuboy
47.34
47.45
16.81
10.0
17.25
16.48
17.03
LLAMA GPT-3.5 6PS4
Claide Gerilla
LLAMA GPT-LS 699- Claude Gorill
LIAMA 6PT-L5 6PF- Claide Gerlis
LLAMA 6PF3S 6PT4 Claude Gorlss
Tensorflow Hub 0-shot
Tenserfiow Hub with BM25 Retriever
Tansorflow Hub with GPTRetriever
Tensorflow Huls with Oracle Retriever
$3.79
65.59
64.36
Accracy
Accuracc
41.25
41.89
34.01
18.2
12.55
LLAMA
Model
Model
Mode
Mode
Figure 10: Performance: We plot each model's performance different configurations. We that Gorilla performs extremely well the zero-shot setting. While even when the oracle answer is given, Gorilla still the best.
Table 7: Evaluating Gorilla (vs) DocPrompting Gorilla improves accuracy. while lowering the hallucination.
Accuracy f
Hallucination
DocPrompting
Gorilla
DocPrompting
Gorilla
61.72
71.68
17.36
10.95
A.3.4 Gorilla (VS) DocPrompting
We evaluate Gorilla and DocPrompting [49] on the HuggingFace Dataset from Table 1. For a 7B model, when trained on the same number of epochs, with and the same learning for both the models, Gorilla improves accuracy while reducing hallucination.
A.3.5 Sensitivity to pre-training
Gorilla's training recipe is robust the pre-training strategies and recipes of the underlying model. From Fig. 13 we demonstrate that all the three models can converge to within few percentage points in accuracy independent of the pre-trained base model.
18

---

Torch Hub
BMZz-Ratriwver
Torth Hub with GPT-Retriever
with Oraci Retriever
Accursey
Accuracy
2
Hollccnotton
Ha ucRaa.on on
Hall nnlucnnt
hollec nallon
HupgingFace @sshot
HuggingFace with BM25 Retriever
HuggingFace with GPTRetriover
Hugging with Oracle Retriever
Gzlte
x 6
Acquracy
Aenurery
Inllucintton
Ha Halluchation
Hallciistion
Rallccitionn
2. x 8
Acounsy
Grrrac
Accuracy
2 2
Calds
Iellccmatooo
Piallc mabion
Accuracy vs Hallucination: We plot each model's performance on different configurations. We found that in the zero-shot setting, Gorilla has the most accuracy gain while maintaining good factual capability. When prompting with different retrievers, Gorilla still capable avoid the hallucination errors.
Figure 11:
from transformers import pipeline
def foad_modell ():
classifier pipeline ''sentiment analyyss's
model='n1p "ooolllnlptoon/berr http:/brrttb-t---eeeeeeeeeeeeeee maae-muultiinggue waw.ggguaauuuueeeeeeeaeaa uncased sentiment ')
return classifier
def process_data datamennent classifier):
response classifier (comments)
sentiments = response (']lbaaell '] sppltt(
scores = response [0] [sscoee']]
result [['sentiment' sentiment, 'score': score)
for sentiment, score in zip(sentiments, scores)] return result
comments
Load the model classifier load_model ()
# Process the data
response process_ data (comments, classifier) print (response)
Figure 12: The API call by Gorilla model are accurate and bug-free, but the supporting zip () code has bug.
19

---

70
60
50
Accurecy
30
10
Gorill-LLaiMA
Gonilla MPT
Govilla Falcon
Pre-trained
Figure 13: For the train-eval datasct, our fine-tuning recipe, RAT, is robust the underlying base model.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper provides recipe teach LLMs use tools, and also presents
data-set for evaluating API calling, and metric for measuring hallucination. The paper studies them with rigorous evaluations.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [NA]
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the hyperparameters specified in the appendix, and the code and dataset is open-sourced at github. com/ wetshirraattggorrllll.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All code, data, and models are open-sourced under the Apache 2.0 license by the authors.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the hyperparameters are specified in the appendix, and all code, data, and models are open-sourced.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: All non-LLM experiments deterministic need error bars, and given the GPU costs involved, we perform LLM experiments once.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Provided in Appendix including the sample dataset.
21

---

9. Code Of Ethics
Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https://neur:::: cttpp///c////iiiicctceeeoeoooooe
Question:
Answer: [Yes]
Justification: Conform's with NeurlPS Code of Ethics
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Integrating language models with API calls significantly extends their utility, enabling wide range of applications from automating customer service to generating real- time content and facilitating data analysis. This integration can lead to more personalized and efficient user experiences across various platforms, language models process natural language inputs and interact with different APIs to fetch, interpret, and act on data in real time. For instance, in customer service, this mean providing instant, relevant responses to queries, reducing wait times, and improving overall satisfaction. In content generation, can enable dynamic creation of articles, reports, or summaries based on the latest data available from web services.
Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Unlike Images, etc where there are copyrights involved, APIs are meant to distributed. Hence, the incentives are very well aligned. For example, if Gorilla presents particular service's 's API, the service benefits from engagement.
Justification:
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All code, data, and models open-sourced under the Apache 2.0 license by the authors.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The open-source repository is actively
maintained at
gi thub... htmm/ShiiihiPPttiio/orillla
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
22


Reviews:
[{"review_id": "ZSfdG1r4c6", "reviewer": "Reviewer_MqVN", "summary": "This paper addresses a pipeline to call adequate APIs among massive pools to accomplish users’ instructions. For that, the authors construct and release the APIBench dataset that contains more than 1645 APIs, and propose the Gorilla model, which is a retrieval-aware finetuned Llama-7B model for API calls.", "questions": "- In section 4.1, “Note that for TorchHub and TensorHub, we evaluate all the models using AST tree accuracy score. *However, for HuggingFace, since the dataset is not exhaustive, for all the models except Gorilla, we only check if they can provide the correct domain names.*” is vague. The numbers of data for HF, TFH, and TH are 925, 801, and 95, respectively. Please clarify this.\n- Table 1 shows that TorchHub is hard to even with oracle API documents compared to HuggingFace and TensorFlowHub. Can you draw the reason?\n- (Typos) line 225. `his suggests`\n- (suggestions) Please elaborate more on Table 2 — e.g., the caption and highlights. Moreover, it’s hard to connect between the written explanation and the table, due to different numbers.\n- (suggestions) Absence of mentioning the full name of AST (Abstract Syntax Tree)", "limitations": "- Limitation section is absent, though the authors describe them in the check list. Broader impact and limitation could include implicit risks such as usage of unreliable APIs, license infringement, or unexpected results by incorrect function calls.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Constructing the APIBench with AST tree matching evaluation metrics as well as open-sourcing the trained Gorilla model will be largely benefit to the community.\n- The problem in this paper is timely in terms of LLM applications and eco-systems — i.e., automatizing API function calls.\n- The fine-tuned model, Gorilla, surpasses performance compared to current SOTA models.\n- The paper is well written and soundly provides experiment results and analysis.", "weaknesses": "- Lack of details about the training data for Gorilla, such as data stats and construction methods. It’s unclear how different the Gorilla training data and evaluation sets of APIBench are, which could lead to doubts about data contamination.\n- Experiment results analysis\n    - How robust are models to paraphrased user instructions corresponding to the same target model API?\n    - Providing performance for each domain could make it possible to analyze which domains are easy and hard.\n- Clarifications are needed:\n    - In the experimental result section, the authors report two metrics along with overall accuracy: **the error by hallucination and by selecting the wrong API call.** The metric equations or explanations for calculating the values need to be clarified.\n    - In section AST as a Hallucination Metric (line 251), how was experimented to attain the human evaluation of 78% accuracy?\n- It is required to mention the license of crawled APIs for usage.\n- A limitation section is absent, though the authors describe them in the checklist. Broader impact and limitation could include implicit risks such as usage of unreliable APIs, license infringement, or unexpected results by incorrect function calls.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and thoughtful comments. We are encouraged that you find our contributions of APIBench, and AST tree matching evaluation metric, along with open-sourcing of the LLM models to be valuable contribution to the community. \n\nWe clarify the questions below:\n\n**1. Details on Gorilla Training: Data and construction and Clarification on Sec 4.1**\n\nThe HuggingFace platform hosts and servers about 203,681 models. Post filtering for poor documentation, lack of dependencies, and those that have no information in their model card, etc, we pick the top 20 models from each domain. Post filtering, we arrive at a total of 925 models from HuggingFace. Note that some of the Model APIs are actually a family of APIs. For example, https://huggingface.co/docs/transformers/model_doc/resnet has `TFResNetModel` and `FlaxResNetModel` among others. In those scenarios, we decouple them to be independent APIs. Similarly, TensorFlow Hub is versioned into v1 and v2. The latest version (v2) has 801 models in total. After filtering, we are left with 626 models. Similarly, we extract all 95 models (exhaustive) from Torch Hub.\n\nPost extraction, we manually check every data-point for quality and correctness. For each API, we verify the dataset to ensure it is executable. Since our evaluation metric checks against the ground truth, having a correct answer using our metric is guaranteed to have high quality code. (Please also see our human eval, highlighting the relationship between the evaluation metric with the final execution accuracy.)\n\nThen, guided by the self-instruct paradigm Wang et al. (2022a), we employ gpt-4-0613 to generate synthetic instruction data. We provide three in-context examples, along with reference API documentation, and task the model with generating real-world use cases that call upon the API. We specifically instruct the model to refrain from using any API names or hints when creating instructions. We constructed 6 examples (Instruction-API pairs) for each of the 3 model hubs. These 18 examples were the only hand-generated data. For each of our 1,645 APIs, we generate 10 instruction-API pairs by sampling 3 of 6 corresponding instruction examples in each pair (illustrated in Figure 3). We then divide the data into a train, and test split (uniform) randomly.\nWe train Gorilla for 5 epochs with the 2e-5 learning rate with cosine decay, with batch size 64, warm-up ration 0.03, and 2048 max sequence length.\n\n**2. How robust are models to paraphrased user instructions corresponding to the same target model API?**\n\nThe models trained can respond to a diverse set of user instructions, given the diversity in our training data-set. As a yard-stick Gorilla models have been downloaded 10000+ times, and here are some examples randomly chosen:\n```\nExample 1 (q 638) “I am an illustrator, I want to create an appealing image based on a text description for commercial purposes.” \n\nExample 2 (q 704) “Assist me in finding the accurate information in a table related to the NYSE stock market”\n\nExample 3 (q 867)  “We want to communicate product information to online customers in France. 'Introducing the new eco-friendly water bottle made of high-quality stainless steel with double-wall insulation to keep your drinks cool for 24 hours or hot for 12 hours'.”\n```\n\n**3. Providing performance for each domain could make it possible to analyze which domains are easy and hard.**\n\nThis is a very valuable feedback and have already included this in the paper. We find the variance to be impacted by the number of models in the category. For, example, in HuggingFace, `Computer Vision Object Detection` has an accuracy of 0.77 compared to `NLP Text2Text Generation` which has an accuracy of 0.68. \n\n**4. The metric equations or explanations**\n\nThank you for raising this. We will clarify in the paper. Here’s a brief explanation: We first parse all the API calls in the API pool using Abstract Syntax Tree (AST). We then parse the LLMs output into the AST. We then check whether the LLMs output matches any AST (subtree match) in the pool. Note that we not only check the functional name but also the non-optional arguments. For example, in HuggingFace, one example could be image classification: pipeline('image-classification', model='fxmarty/resnet-tiny-beans'), we check the AST tree node function argument pipeline, value argument image-classification and fxmarty/resnet-tiny-beans. If they all matches the model’s output, we claim the model’s output matches this specific API. We then check the function description of model’s output and the matched API in our pool. If they are equivalent, we claim correct. Evaluation metrics: Accuracy is the #Correct / #Total. In order to be correct, the model’s output not only needs to match one API in the API pool, but the matched API’s function description needs to be the same as the ground truth. Hallucination is the #Not Matched / # Total. This simply means the model’s output doesn't match any API call, thus we cannot find the corresponding API in our dataset. \nAccuracy + Hallucination + Error (syntactic, etc) = 1\n\n\n**5. License of crawled APIs**\n\nThe licenses of crawled APIs are permissible under Apache 2.0. TensorFlow is licensed under the Creative Commons Attribution 4.0, and code samples are licensed under the Apache 2.0. Pytorch Hub follows the Linux Foundation Policies. The huggingface hubdocs are all on Apache 2.0.\n\n**6.  TorchHub is hard to even with oracle API documents**\n\nTorchHub APIs usually require pre-processing that is often not templated like in HuggingFace. This causes quite a bit of confusion for the model. For example, to load Tacotron 2 from Nvidia, the right API call is `tacotron2_model = torch.hub.load('NVIDIA/DeepLearningExamples:torchhub', 'nvidia_tacotron2', model_math='fp16')`. However, the model confuses this with the pre-processing step which is similar returning  `utils = torch.hub.load('NVIDIA/DeepLearningExamples:torchhub', 'nvidia_tts_utils')`.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LdEae17bWD", "reviewer": "Reviewer_Umrp", "summary": "This paper introduces Gorilla, a fine-tuned LLaMA model designed to improve large language models' ability to use APIs accurately. The authors created APIBench, a comprehensive dataset of ML APIs from HuggingFace, TorchHub, and TensorFlow Hub, and used self-instruct to generate instruction-API pairs for training. They fine-tuned LLaMA-7B with retrieval-aware training, incorporating tech like AST sub-tree matching for evaluating API call accuracy, retriever-aware training to adapt to API changes, and handling of constrained API calls. Results show that Gorilla outperforms existing LLMs (including GPT-4) on API call accuracy across multiple datasets, demonstrates ability to adapt to test-time changes in API documentation, and handles constrained API calls effectively. The paper presents a novel approach to improving LLMs' API usage capabilities, with promising results that outperform existing state-of-the-art models in this specific domain.", "questions": "* How does Gorilla's performance compare to other specialized code generation or API-focused models, not just general-purpose LLMs?\n* What is the performance impact of varying the size of the training dataset or the base model size (e.g. using LLaMA-13B instead of 7B)?\n* How does Gorilla perform on more complex multi-step API workflows rather than just single API calls?", "limitations": "NA - Appropriately discussed limitations.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "* Gorilla outperforms existing state-of-the-art language models, including GPT-4 and Claude, in API call accuracy across multiple datasets (TorchHub, HuggingFace, and TensorFlow Hub).\n* Gorilla significantly reduces API argument hallucination errors compared to other models, improving the reliability of API calls.\n* The retriever-aware training enables Gorilla to adapt to test-time changes in API documentation, allowing it to remain up-to-date with evolving APIs without requiring retraining.\n* Gorilla demonstrates the ability to understand and respect constraints (e.g., accuracy requirements) when selecting appropriate APIs, outperforming other models in constraint-aware API invocations.", "weaknesses": "A significant weakness of the Gorilla approach is that it relies heavily on knowledge augmentation, which is a common technique already used in various domains to improve language model performance on specific tasks. The use of retrieval-augmented generation to enhance API calling capabilities doesn't represent a novel improvement or implementation compared to similar approaches in other domains.\n\nEssentially, Gorilla applies existing knowledge augmentation techniques to the specific task of API invocation, rather than introducing a fundamentally new method for improving language model capabilities. While the results show improvements in API calling accuracy, the core approach of combining retrieval with language model fine-tuning is not innovative in itself. This limits the broader impact and generalizability of the work beyond the specific domain of API invocation.\n\nThe lack of significant methodological innovation suggests that similar performance improvements could potentially be achieved by applying existing retrieval-augmented generation techniques to the API calling task, without necessarily requiring the specific Gorilla architecture. This raises questions about the uniqueness and broader applicability of the Gorilla approach beyond the narrow domain explored in the paper.* While Gorilla performs well on the specific API datasets it was trained on, it's unclear how well it would generalize to entirely new APIs or domains not covered in the training data.\n\nOther Notes: \n* Although Gorilla shows some ability to handle constraints, its performance in this area is not significantly better than other models like GPT-3.5, suggesting room for improvement.\n* The paper doesn't compare Gorilla's performance with specialized API documentation tools or code completion systems, which might be more tailored for this specific task.\n* The paper doesn't provide an in-depth analysis of the cases where Gorilla fails, which could provide insights into its limitations and areas for improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and thoughtful comments. We are motivated that you found Gorilla can significantly reduce API argument hallucination errors compared to other models, the retriever-aware training to be a novel contribution that allows Gorilla to adapt to test-time changes in API documentation, allowing it to remain up-to-date with evolving APIs without requiring retraining, and that Gorilla demonstrates the ability to understand and respect constraints (e.g., accuracy requirements) when selecting appropriate APIs, outperforming other models in constraint-aware API invocations.\n\nWe clarify some of the questions below:\n\nWe agree with the reviewer that prior work including Vicuna, Orca [Mukherjee et al.], Textbooks Are All You Need [Gunasekar et al.] have demonstrated knowledge augmentation techniques. However, our study has been centered around connecting LLMs with APIs. We,\n\n1. Develop and manually curate a high-quality dataset that is valuable to the community. \n2. Introduce Retriever-Aware Training (RAT) which teaches the LLM to either use or ignore the retrieved context. This is a novel approach that helps Gorilla achieve superior performance. \n3. Propose evaluation metrics, including the first technique to define and measure hallucination for the domain of (functions and tools) APIs using Abstract Syntax Tree (AST). We also perform human evaluation to validate the competency of the metric. \n4. Study realistic API calls with constraints (e.g., “I would like to detect pedestrians and cars on the street. I want a model to run on a Raspberry PI with accuracy at least 65% on ImageNet”. Here, Gorilla needs to reason about the constraints - that the model needs to fit the memory budget while still meeting accuracy budgets.)\n5. Measure hallucination for popular models for APIs.\n\n**1. Generalization to out of domain**\n\nGorilla’s novel Retriever Aware Training (RAT) helps the model generalize very well to out of domain. For example, for the rebuttal we evaluated RAT trained Gorilla on 2000 APIs from a totally diverse set including hyperscalers (GCP, Azure, AWS), Postman, RapidApi etc,  and find that Gorilla (acc: 0.89) outperforms gpt-3.5-turbo-0125 (acc: 0.75). Some examples in this new set are: `stripe.Charge.create(amount={amount}, currency={currency}, source={source}, description={description})` to create a charge in Stripe, and `yfinance.Ticker({ticker}).dividends` to get dividend information of a stock from Yahoo Finance, which are completely out-of-domain to ML APIs demonstrating the ability to generalize well.\n\n** 2. Although Gorilla shows some ability to handle constraints, its performance in this area is not significantly better than other models like GPT-3.5, suggesting room for improvement.**\n\nWe agree with the reviewer, and highlight that GPT-3.5 is a closed source model about which little is known. With Gorilla, we propose techniques to fine-tune a 7B parameter model that can match if not beat performance of GPT-3.5. \n\n**3. How does Gorilla's performance compare to other specialized code generation or API-focused models, not just general-purpose LLMs?**\n\nThe strongest API-focused model when writing the paper is DocPrompting Zhou et al. (ICLR 2023) which looked at choosing the right subset of code including API along with a retriever. We demonstrate that when comparing Gorilla (vs) DocPrompting, Gorilla improves accuracy, while lowering the hallucination. We are happy to include any new baselines the reviewer suggests.\n\n|Accuracy ↑ | Accuracy ↑ | Hallucination ↓ | Hallucination ↓ |\n|---|---|---|---|\n|DocPrompting | **Gorilla** | DocPrompting | **Gorilla**|\n|61.72 |**71.68**| 17.36| **10.95**|\n\n**4. In-depth analysis of failure cases** \n\nThank you for your suggestion on providing an in-depth analysis of the cases where Gorilla fails, which could provide insights into its limitations and areas for improvement. We will include this in the paper. In the interest of space, we provide an example here: \nTorchHub APIs usually require pre-processing that is often not templated like in HuggingFace. This causes quite a bit of confusion for the model. For example, to load Tacotron 2 from Nvidia, the right API call is `tacotron2_model = torch.hub.load('NVIDIA/DeepLearningExamples:torchhub', 'nvidia_tacotron2', model_math='fp16')`. However, the model confuses this with the pre-processing step which looks very similar `utils = torch.hub.load('NVIDIA/DeepLearningExamples:torchhub', 'nvidia_tts_utils')`, returning the API for pre-processing instead of the model call.\n\n**5. What is the performance impact of varying model** \n\nWe have evaluated the performance of the model across three diverse base models : MPT-7B (0.70), Falcon-7B (0.66) and LLAMA-7B (0.71) on the HF dataset. The results are included in the paper (Appendix Figure 13) and demonstrate that all kept constant, our innovative Retrieval Aware Training (RAT) fine-tuning recipe is robust to the underlying base model. \n\n**6. How does Gorilla perform on more complex multi-step API workflows rather than just single API calls?**\n\nGorilla is focused at improving the LLM model to invoke single round APIs. Multi-step APIs is exciting future work and outside the scope of this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BqFCp3ItPd", "reviewer": "Reviewer_jdAv", "summary": "The authors present a new fine-tuned language model that is trained to map from user instructions to code snippets that invoke the appropriate APIs. The authors also introduce a new dataset which includes the information for roughly 1600 APIs from a variety of online sources, which is used to train the model. They conclude that their approach outperforms existing language models on the task of API call generation, as measured by both error rate and a novel “hallucination” metric.", "questions": "How do models which are fine-tuned compare to Gorilla? (If such a comparison is not appropriate, why?)\n\nWhat is the statistical significance of the results?\n\nWhat are the most common kinds of errors?\n\nDo the kinds of errors differ between the different categories of APIs?\n\nAs a final minor question: do the authors have an explanation for why GPT-3.5 appears to outperform GPT-4 across a few different problem settings? This seems like a surprising and somewhat notable result!", "limitations": "I feel that the authors should have a more robust discussion of the broader impact of their work.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "I feel that the problem statement is well-motivated and of general interest. Prior work has investigated allowing LLMs to make use of various tools, and improvements in that capability are likely to be well received. I appreciate that the authors have chosen a (relatively speaking) lightweight and open-source model, which increases the usability of their approach. I also appreciate the construction of a novel dataset, which could presumably be of interest to the larger community even in the absence of a specific model.", "weaknesses": "First, I wonder about baselines. The authors have done a good job of comparing their method against a variety of open-source and closed-source LLMs, but these systems are all “generalist” compared to the fine-tuned Gorilla model. It would have been useful to see a comparison to a fine-tuned version of another model (even one with less fine-tuning than is possible on the small Llama-7b) just to have a sense of how much improvement can be leveraged from the newly introduced dataset. I also wonder if it would have been possible to compare to specialized tool-based LLM models like Toolformer. If such a comparison is not appropriate or possible, I feel the authors should mention why.\n\nMy second question concerns statistical significance. The authors indicate the high cost of LLM experiments as the reason for omitting such analysis. I sympathize with this explanation, but feel that I would be remiss not to stress the importance of statistical testing as part of justifying claims that Gorilla’s “performance surpasses prompting the state-of-the-art LLMs in three massive datasets.” Without a measure of a variance, we cannot meaningfully conclude that apparent improvements achieved by Gorilla (which are often on the order of 1-3% in overall score) are the result of anything other than noise. To be sure, a 7B parameter model that achieves even comparable results to a massive LLM is a notable result (though see my earlier point about fine-tuned baselines), but I would temper claims about improvement in the absence of statistical justification.\n\nLastly, I feel that the paper would benefit greatly from a more robust error analysis. In the best-performing realistic setting (i.e. without access to an oracle), Gorilla achieves an overall accuracy of roughly 65%. This means that more than one third of the time it is returning an incorrect result (including errors, the authors point out, that could be silent to the end user). Obviously this is far from being ready for direct deployment without external validation. Given the prevalence of errors, it would be useful to better understand what kinds of errors are common: are they more likely to occur with particular kinds of APIs? Does the wording and / or complexity of the input prompt have a large effect? This is the kind of analysis that, in combination with the novel dataset, could help spur future work and improvement. In addition, given the potential for damaging errors, a more robust broader impact statement would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and their thoughtful comments. We are encouraged that you found our choice of model and dataset useful to the larger community. We will clarify some of the questions below:\n\n**1. Fine-tuned version of another model and comparison**\n\nWe agree that GPT and Claude models are generalist models. We have evaluated the performance of the model across three diverse base models : MPT-7B (0.70), Falcon-7B (0.66) and LLAMA-7B (0.71) on the HF dataset. The results are included in the paper (Appendix Figure 13) and demonstrate that for the same train-eval dataset, our innovate Retrieval Aware Training (RAT) fine-tuning recipe is robust to the underlying base model. \n\nToolformer, by it’s construction, requires the LLM to reason over the output which extends well for general purpose chat-style queries such as question answering, Wikipedia search engine, etc. Note that Gorilla supports any API call, and doesn’t rely on the output of the API to train the model - which is often not available. Further, Toolformer required a complete re-training (fine-tuning) when the API specification change. However, since Gorilla is trained with RAT, it is robust to changes in API documentation. \n\nThe strongest API-focused model when writing the paper is DocPrompting Zhou et al. (ICLR 2023) which looked at choosing the right subset of code including API along with a retriever. We demonstrate that when comparing Gorilla (vs) DocPrompting - which is a fine-tuned model, Gorilla improves accuracy, while lowering the hallucination. We are happy to include any new baselines the reviewer suggests.\n|Accuracy ↑ | Accuracy ↑ | Hallucination ↓ | Hallucination ↓ |\n|---|---|---|---|\n|DocPrompting | **Gorilla** | DocPrompting | **Gorilla**|\n|61.72 |**71.68**| 17.36| **10.95**|\n\n**2. What is the statistical significance of the results?**\n\nThank you for highlighting this. To demonstrate that Gorilla’s novel Retriever Aware Training (RAT) helps the model generalize very well to out of domain, for the rebuttal we evaluated RAT trained Gorilla on 2000 APIs from a totally diverse set including hyperscalers (GCP, Azure, AWS), Postman, RapidApi etc,  and find that Gorilla (acc: 0.89) outperforms gpt-3.5-turbo-0125 (acc: 0.75). Some examples in this new set are: `stripe.Charge.create(amount={amount}, currency={currency}, source={source}, description={description})` to create a charge in Stripe, and `yfinance.Ticker({ticker}).dividends` to get dividend information of a stock from Yahoo Finance, which are completely out-of-domain to ML APIs demonstrating the ability to generalize well.\n\n**3. Understanding the errors**\n\nThank you for your suggestion on including an analysis of errors. Given we have all the generations, this is an easy addition to the paper. In the interest of space, we present one example here, which also highlights errors that are possible even with an oracle retriever. TorchHub APIs usually require pre-processing that is often not templated like in HuggingFace. This causes quite a bit of confusion for the model. For example, to load Tacotron 2 from Nvidia, the right API call is tacotron2_model = torch.hub.load('NVIDIA/DeepLearningExamples:torchhub', 'nvidia_tacotron2', model_math='fp16'). However, the model confuses this with the pre-processing step which looks very similar utils = torch.hub.load('NVIDIA/DeepLearningExamples:torchhub', 'nvidia_tts_utils'), returning the pre-processing API instead of the model call. \n\nIn terms of categories, common types of errors include model-name hallucination `resnet-201` instead of `resnet-101`, hallucination in API call format from `torch.hub.load` to `torch.load`, chanes in directory structure e.g. `'facebookresearch/pytorch_GAN_zoo:hub` to `facebook/pytorch_GAN_zoo:hub`. The error categories vary in how often they occur as we go from 0-shot to BM-25 to oracle-retriever, but they remain steady as we vary datasets.  We will include a collection of examples for more qualitative analysis in the appendix of the paper which would be a rich playground for future work! \n\n**4. Why GPT-3.5 appears to outperform GPT-4 across a few different problem settings**\n\nThis is a great point of interest even for us. One potential explanation we got when we reached out to OpenAI is that GPT-4 is “lightly” RLHF’d when compared to GPT-3.5. Making 3.5 a stronger instruction follower. So, we can observe that as we move from 0-shot to oracle-retriever, 3.5 shines, since it is better at “following the instruction” of using the API to answer the user question, while GPT-4 on the other hand, tends to hallucinate even when provided with an API - oracle or not.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jpb5BTiTDf", "reviewer": "Reviewer_N9sF", "summary": "This paper proposes Gorilla, a fine-tuned LLaMA model, outperforms GPT-4 in crafting API calls and adapts well to document changes with a document retriever, reducing hallucination issues. It also provides APIBench, a new dataset for evaluation, includes APIs from HuggingFace, TorchHub, and TensorHub. Gorilla's integration with a retrieval system enhances tool usage accuracy and documentation updates, promising more reliable LLM outputs.", "questions": "1. In Figure 2, why GPT-3.5 is much better than GPT-4 in all settings?\n2. As most of the instructions are generated by GPT-4 referring to your hand-generated instruction seeds. How to ensure your seeds meets the real-world scenario and reach the real difficulties?\n3. As many APIs have overlapping functions, how to decide which API is better than the other similar one? How to maintain stable API calling to persist consistent results for the same request?\n4. Which Llama version is used as your foundation model? Llama2 or Llama3?\n5. In Table 1, why Gorilla always produces a large error in selecting wrong APIs? Do the results support your view?", "limitations": "The paper should provide a more comprehensive limitation part to point out potential overlooked issues, such as influence of synthetic training data and insufficient comparative experiments.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper trains a system which connects massive APIs and takes text instruction to get the corresponding API calls, along with a step-by-step explanation of the pipeline. It focuses on the details within an API call, which significantly mitigates the complexity involved in developing machine learning softwares.\n2. The paper also constructs APIBench by scraping a large corpus of ML APIs and developing an evaluation framework to check functional correctness. It uses AST subtree-matching metrics which helps measure hallucination and further contributes to the evaluation of ML API mastering techniques.\n3. The paper is well-writtened, presenting a detailed comparison with other related works, a clear structure on methods, and comprehensive experiments and analysis.", "weaknesses": "1. Although the paper overall does not present significant issues, I would appreciate seeing a performance comparison with more specific methods, such as other works that also utilize LLM for tool calling to build software pipelines, which may further demonstrate the contribution of techniques presented by Gorilla.\n2. The paper needs to better illustrate the contributions of this work and more clearly outline its potential impact on the community.\n3. Some writing detales need to be improved in the paper, such as \"the\" should be omitted in \"The model trained with the our method\" in Line 31, and \"AST\" should be replaced by \"Abstract Syntax Tree\" for the first time it appears in Line 36.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and thoughtful comments. We are encouraged you enjoyed the exposition of our pipeline and find the APIBench, and the AST subtree-matching metric to contribute to mastering techniques to evaluate APIs. \n\n**1. Highlighting the contributions and potential impact on community**\n\nGorilla is centered around connecting LLMs with APIs, and to this end, we:\n1. Develop and manually curate a high-quality dataset that is valuable to the community.\n2.RAT: We introduce Retriever-Aware Training (RAT) which teaches the LLM to either use or ignore the retrieved context. This is a novel approach that helps Gorilla achieve superior performance.\n3. Propose evaluation metrics, including the first technique to define and measure hallucination for the domain of (functions and tools) APIs. We also perform human evaluation to validate the competency of the metric.\n4. Study realistic API calls with constraints (e.g., “I would like to detect pedestrians and cars on the street. I want a model to run on a Raspberry PI with accuracy at least 65% on ImageNet”. Here, Gorilla needs to reason about the constraints - that the model needs to fit the memory budget while still meeting accuracy budgets.)\n5. Measure hallucination for popular models for APIs.\n\n**2. Comparison with other LLMs for tool calling**\n\nBeyond frontier labs whose models we have compared, within academic literature, the strongest baseline (API-focused model) when submitting the paper is DocPrompting Zhou et al. (ICLR 2023) which looked at choosing the right subset of code including API along with a retriever. We demonstrate that when comparing Gorilla (vs) DocPrompting, Gorilla improves accuracy, while lowering the hallucination. We are happy to include any new baselines you might suggest.\n\n|Accuracy ↑ | Accuracy ↑ | Hallucination ↓ | Hallucination ↓ |\n|---|---|---|---|\n|DocPrompting | **Gorilla** | DocPrompting | **Gorilla**|\n|61.72 |**71.68**| 17.36| **10.95**|\n\nWe also include a comparison with **prompting techniques {0-shot} and {3-shot}** in Table 6 of Appendix, and demonstrate Gorilla improves performance across both techniques for GPT-3.5 and GPT-4. \n\n**3. Writing Detales**\n\nThank you for flagging these. We agree that the writing mechanics of the paper can be improved and have already made editorial passes over the paper to improve the exposition of the paper. We will also include more comprehensive limitation part to point out potential issues, such as influence of synthetic training data and comparative experiments, and have already reflected the changes suggested. \n\n**4. Comparing GPT-3.5 vs GPT-4**\n\nWe would first like to highlight that with frontier models we can’t know for sure. Further, perhaps, this is orthogonal to this work. However, this intrigued us as well. One potential explanation we got when we reached out to OpenAI is that GPT-4 is “lightly” RLHF’d when compared to GPT-3.5. Making 3.5 a stronger instruction follower. So, we can observe that as we move from 0-shot to oracle-retriever, 3.5 shines, since it is better at “following the instruction” of using the API to answer the user question, while GPT-4 on the other hand, tends to hallucinate even when provided with an API - oracle or not.\n\n**5. Diverse real-world scenarios for instruction**\n\nFor the self-instruct generation, we provide three in-context examples, along with reference API documentation, and task the model with generating real-world use cases that call upon the API. We specifically instruct the model to refrain from using any API names or hints when creating instructions. We constructed 6 examples (Instruction-API pairs) for each of the 3 model hubs. Post this, we manually verify and audit each generation resulting in high quality data-set. Making this an important contribution to the community. \nFurther, as a yard-stick Gorilla models have been downloaded 10000+ times. \nHere are some examples randomly chosen to demonstrate the diversity and versatility: \n```\nExample 1 (q 638) “I am an illustrator, I want to create an appealing image based on a text description for commercial purposes.” \n\nExample 2 (q 704) “Assist me in finding the accurate information in a table related to the NYSE stock market”\n\nExample 3 (q 867)  “We want to communicate product information to online customers in France. 'Introducing the new eco-friendly water bottle made of high-quality stainless steel with double-wall insulation to keep your drinks cool for 24 hours or hot for 12 hours'.”\n```\n\n**6. How to decide which API is better than the other similar one**\n\nGiven a question, depending on the constraints we narrow down the set of right answers. For example, if the user requests an object detection model - any and all object detection APIs are the right answers. However, if a user specifies a constraint either explicitly or implicitly (e.g., they want the model to have a top-1 accuracy of xx on imagenet) then the set of APIs narrows down and a subset of them are “better” than other similar ones. Since, we create the data-set we can control for this. If two APIs are the right answer, then if either of them are called, we mark that as accurate. \n\n**7. In Table 1, why (does) Gorilla always produce a large error in selecting wrong APIs? Do the results support your view?**\n\nYes, this is quite intuitive. By teaching the model domain knowledge (with our novel Retriever Aware Training), the model tends to hallucinate less (not 0, but lesser than baselines). The error results from the model unable to identify the user intent and generating API for the wrong task.\n\n**8. Which Llama version is used as your foundation model?**\n\nLlama-2 model. We evaluate the performance across three diverse base models : MPT-7B (0.70), Falcon-7B (0.66) and LLAMA-7B (0.71). The results are included in the paper (Appendix Figure 13) and demonstrate that all kept constant, our innovative Retrieval Aware Training (RAT) fine-tuning recipe is robust to the underlying base model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are encouraged by the insightful reviews and appreciate the recognition of the key strengths of our work. The reviewers highlighted:
1. The timely relevance of our problem statement in the realm of Large Language Models (LLMs) and API ecosystems, emphasizing our novel approach in automatizing API function calls **[MqVN, jdAv]**. 
2. Our model, Gorilla, has been noted for its exceptional performance, surpassing current state-of-the-art (SOTA) models like GPT-4 and Claude in API call accuracy, and significantly reducing API argument hallucination errors **[Umrp, MqVN]**. 
3. The novel use of AST tree matching evaluation metrics for measuring LLM hallucination in calling APIs, and the construction of APIBench, as well as the open-sourcing of Gorilla, are particularly noted for their potential to benefit the community **[MqVN, N9sF]**. 
4. The paper’s clear, well-structured presentation, detailed comparative analysis, and comprehensive experimental results were also commended **[N9sF, MqVN]**. 
5. We acknowledge the positive feedback on the model’s ability to adapt to evolving APIs and its effectiveness in constraint-aware API invocations **[Umrp]**. 

We address individual concerns below, and all suggested revisions have been incorporated into the manuscript.

Meta-review: This work proposes Gorilla, a model that is able to achieve state of the art in API call generation. The authors will release the code and data (APIBench, which has AST matching metrics) to the community. The reviewers generally appreciate the work and impact that the model and dataset will have and comment on the clear presentation of the paper, as well as its actual performance compared to the baselines. They do raise some concerns, such as lack of training details, clarifications on metrics and results analysis, novelty, and evaluation (results, statistical significance). However, the authors responses are clear and seem convincing to the reviewers.
Decision: Accept (poster)