Title: Gorilla: Large Language Model Connected with Massive APIs
Venue: NeurIPS 2024
Accepted: yes
Keywords: LLM

Tool Use

APIs

Function Calling
Abstract: Large Language Models (LLMs) have seen an impressive wave of advances, with
models now excelling in a variety of tasks, such as mathematical reasoning and
program synthesis. However, their potential to effectively use tools via API calls
remains unfulfilled. This is a challenging task even for today’s state-of-the-art
LLMs such as GPT-4 largely due to their unawareness of what APIs are available
and how to use them in a frequently updated tool set. We develop Gorilla, a
finetuned LLaMA model that surpasses the performance of GPT-4 on writing API
calls. Trained with the novel Retriever Aware Training (RAT), when combined
with a document retriever, Gorilla demonstrates a strong capability to adapt to
test-time document changes, allowing flexible user updates or version changes.
It also substantially mitigates the issue of hallucination, commonly encountered
when prompting LLMs directly. To evaluate the model’s ability, we introduce
APIBench, a comprehensive dataset consisting of HuggingFace, TorchHub, and
TensorHub APIs. The successful integration of the retrieval system with Gorilla
demonstrates the potential for LLMs to use tools more accurately, keep up with
frequently updated documentation, and consequently increase the reliability and
applicability of their outputs. Gorilla’s code, model, data, and demo are available
at: https://gorilla.cs.berkeley.edu

Gorilla: Large Language Model Connected with Massive APIs

Abstract

Large Language Models (LLMs) have seen an impressive wave of advances, with models now excelling in a variety of tasks, such as mathematical reasoning and program synthesis. However, their potential to effectively use tools via API calls remains unfulfilled. This is a challenging task even for today's state-of-the-art LLMs such as GPT-4 largely due to their unawareness of what APIs are available and how to use them in a frequently updated tool set. We develop Gorilla, a finetuned LLaMA model that surpasses the performance of GPT-4 on writing API calls. Trained with the novel Retriever Aware Training (RAT), when combined with a document retriever, Gorilla demonstrates a strong capability to adapt to test-time document changes, allowing flexible user updates or version changes. It also substantially mitigates the issue of hallucination, commonly encountered when prompting LLMs directly. To evaluate the model's ability, we introduce APIBench, a comprehensive dataset consisting of HuggingFace, TorchHub, and TensorHub APIs. The successful integration of the retrieval system with Gorilla demonstrates the potential for LLMs to use tools more accurately, keep up with frequently updated documentation, and consequently increase the reliability and applicability of their outputs. Gorilla's code, model, data, and demo are available at: https://gorilla.cs.berkeley.edu

1 Introduction

The use of APIs and Large Language Models [10, 5, 31, 6, 27, 28] has changed what it means to program. Previously, building complex machine learning software and systems required extensive time and specialized skills. Now with tools like the HuggingFace API, an engineer can set up a deep learning pipeline with a few lines of code. Instead of searching through StackOverflow and documentation, developers can ask models like GPT for solutions and receive immediate, actionable code with docstrings. However, using off-the-shelf LLMs to generate API calls remains unsolved because there are millions of available APIs which are frequently updated.

We connect LLM's and massive API's with Gorilla, a system which takes an instruction, for example 'build me a classifier for medical images', and provides the corresponding API call and relevant packages, along with a step-by-step explanation of the pipeline. Gorilla uses self-instruct, fine-tuning, and retrieval to enable LLMs to accurately select from a large, overlapping, and changing set tools expressed using their APIs and API documentation. Further, our novel retriever-aware training (RAT) enables the model to adapt to test-time changes of APIs such as evolution in versions and arguments.

With the development of API generation methods comes a question of how to evaluate, as many APIs will have overlapping functionality with nuanced limitations and constraints. Thus, we construct

'Help me find an API to convert the spoken language in a recorded audio to text using Torch Hub.'

<domain>: Speech-to-Text

<api_provider>: TorchHub

<code>:

asr_model = torch.hub.load('snakers4/silero-models', 'silero_sst')

result = asr_model.transcribe(audio_path)

Correct

(Gorilla)

<domain>: Speech-to-Text

<api_provider>: TorchHub

<code>:

asr_model = torch.hub.load('snakers4/silero-models', 'asr', source='local')

result = asr_model.transcribe(audio_path)

API Argument

Hallucination

(GPT-4)

<domain>: Audio-Translation

<api_provider>: Pytorch

<code>:

import torchaudio

translation = torchaudio.pipelines.WAV2VEC2_ASR_PIPELINE("audio.wav")

Incorrect

API Call

(Claude)

Better

APIBench ( ∼ 1600 APIs) by scraping a large corpus of ML APIs and developing an evaluation framework that uses AST sub-tree matching to check functional correctness. Further, we draw a distinction between accuracy and hallucination, and propose an Abstract Syntax Tree (AST) based technique to measure hallucination.

Using APIBench, we finetune Gorilla, a LLaMA-7B-based model with document retrieval and show that it significantly outperforms both open-source and closed-source models like Claude and GPT-4 in terms of API functionality accuracy as well as a reduction in API argument hallucination errors. We show an example output in Fig. 1. Lastly, we highlight Gorilla's capability to comprehend and reason about user-defined constraints when choosing between APIs, an essential requirement for LLMs trained to accomplish tasks.

To summarize, this paper makes the following contributions:

We introduce Gorilla, the first system to enable large-scale API integration with LLMs, demonstrating state-of-the-art performance in generating accurate API calls across thousands of functions and libraries.

We develop Retriever-Aware Training (RAT), a novel technique that enables LLMs to effectively utilize retrieved API documentation at inference time, improving both accuracy and adaptation to API changes.

Wepresent APIBench, a comprehensive benchmark of ∼ 1600 machine learning APIs, along with new AST-based evaluation metrics that precisely measure both functional correctness and API hallucination.

2 Related Work

By empowering LLMs to use tools [33], we can grant LLMs access to vastly larger and changing knowledge bases and accomplish complex computational tasks. By providing access to search technologies and databases, [24, 39, 35] demonstrated that we can augment LLMs to address a significantly larger and more dynamic knowledge space. Similarly, by providing access to computational tools, [39, 1, 49, 36, 37] demonstrated that LLMs can accomplish complex computational tasks. Consequently, leading LLM providers [27], have started to integrate plugins to allow LLMs to invoke external tools through APIs.

Large Language Models Recent strides in the field of LLMs have renovated many downstream domains [10, 40, 48, 47], not only in traditional natural language processing tasks but also in program synthesis. Many of these advances are achieved by augmenting pre-trained LLMs by prompting [44, 14] and instruction fine-tuning [11, 30, 43, 15]. Recent open-sourced models like LLaMa [40], Alpaca [38], and Vicuna [9] have furthered the understanding of LLMs and facilitated their experimentation. While our approach, Gorilla, incorporates techniques akin to those mentioned, its primary emphasis is on enhancing the LLMs' ability to utilize millions of tools, as opposed to refining their conversational skills. Additionally, we pioneer the study of fine-tuning a base model by supplementing it with information retrieval - a first, to the best of our knowledge.

Tool Usage The discussion of tool usage within LLMs has seen an upsurge, with models like Toolformer taking the lead [33, 19, 20, 24]. Tools often incorporated include web-browsing [32], calculators [12, 39], translation systems [39], and Python interpreters [14]. While these efforts can be seen as preliminary explorations of marrying LLMs with tool usage, they generally focus on specific tools. Our paper, in contrast, aims to explore a vast array of tools (i.e., API calls) in an open-ended fashion, potentially covering a wide range of applications.

With the recent launch of Toolformer [33] highlights the exciting potential of using large language models (LLMs) for purposes beyond traditional chatbot applications. Moreover, the application of API calls in robotics has been explored to some extent [41, 4]. However, these works primarily aim at showcasing the potential of 'prompting' LLMs rather than establishing a systematic method for evaluation and training (including fine-tuning). Our work, on the other hand, concentrates on systematic evaluation and building a pipeline for future use.

LLMs for Program Synthesis Harnessing LLMs for program synthesis has historically been a challenging task [22, 7, 45, 16, 13, 29]. Researchers have proposed an array of strategies to prompt LLMs to perform better in coding tasks, including in-context learning [44, 18, 7], task decomposition [17, 46], and self-debugging [8, 34]. Besides prompting, there have also been efforts to pretrain language models specifically for code generation [25, 21, 26].

DocPrompting [49] looked at choosing the right subset of code including API along with a retriever. Gorilla presents distinct advancements over DocPrompting. First, the way the data-sets are constructed are different, leading to intersting downstream artifacts. Gorilla focuses on model usages where we also collect detailed information about parameters, performance, efficiency, etc. This helps our trained model understand and respond to finer constraints for each API. Docprompting focuses on generic API calls but not on the details within an API call. Second, Gorilla introduces and uses the AST subtreematching evaluation metric that helps measure hallucination which we find are more representative of code structure and API accuracy compared to traiditional NLP metrics. Finally, Gorilla focuses on instruction-tuning method and has "agency" to interact with users while DocPrompting focuses on building an NLP-to-Code generative model. On equal footing, we demonstrate that Gorilla performs better than DocPrompting in Appendix A.3.

3 Methodology

We first describe APIBench, a comprehensive benchmark constructed from TorchHub, TensorHub, and HuggingFace API Model Cards. We begin by outlining the process of collecting the API dataset and how we generated instruction-answer pairs. We then introduce Gorilla, a novel training paradigm with an information-retriever incorporated into the training and inference pipelines. Finally, we present our AST tree matching evaluation metric.

Dataset curation: 1,645 API calls. 94 from Torch

Hub (exhaustive), 626 from TensorFlow Hub v2

(exhaustive) and 925 from HuggingFace (Top 20

in each domain).

Self-instruct with in-context

examples to generate

16,450 {instruction,API} pairs

This is then used to

Cat

Cat

Cat

API: torch.hub.load(…)

'I want to see

some cats dancing

in celebration!'

Information

Retriever

Input:

###Task: Generate image

from text

###Reference API:

StableDiffusionPipeline.from_

pretrained (…)

API

:StableDiffusionPipelin

e.from_pretrained(stabilit

yai/stable-diffusion-2-1)

Zero-shot

Execution

Results!

GORILLA

GORILLA

HUB

JSON

3.1 Dataset Curation

To curate the dataset, we aggregate all model cards from HuggingFace's 'The Model Hub', PyTorch Hub, and TensorFlow Hub. Throughout the rest of the paper, we call these HuggingFace, Torch Hub, and TensorFlow Hub respectively for brevity.

API Documentation The HuggingFace platform hosts and servers about 203,681 models. However, many of them have poor documentation, lack dependencies, have no information in their model card, etc. To filter these out, we pick the top 20 models from each domain. We consider 7 domains in multimodal data, 8 in CV, 12 in NLP, 5 in Audio, 2 in tabular data, and 2 in reinforcement learning. Post filtering, we arrive at a total of 925 models from HuggingFace. TensorFlow Hub is versioned into v1 and v2. The latest version (v2) has 801 models in total, and we process all of them. After filtering out model cards with little to no information, we are left with 626 models. Similar to TensorFlow Hub, we extract 95 models (exhaustive) from Torch Hub. We then convert the model cards for each of these 1,645 API calls into a JSON object with the following fields: {domain, framework, functionality, api_name, api_call, api_arguments, environment_requirements, example_code, performance, description}. We provide more information in Appendix A.1. These fields were chosen to generalize beyond API calls within the ML domain, to other domains, including RESTful, SQL, and other potential API calls.

Instruction Generation Guided by the self-instruct paradigm [42], we employ GPT-4 to generate synthetic instruction data. We provide three in-context examples, along with reference API documentation, and task the model with generating real-world use cases that call upon the API. We specifically instruct the model to refrain from using any API names or hints when creating instructions. We constructed 6 examples (Instruction-API pairs) for each of the 3 model hubs. These 18 examples were the only hand-generated or modified data. For each of our 1,645 API datapoints, we generate 10 instruction-API pairs by sampling 3 of 6 corresponding instruction examples in each pair (Fig. 3).

API Call with Constraints API calls often come with inherent constraints. These constraints necessitate that the LLM not only comprehend the functionality of the API call but also categorize the calls according to different constraint parameters. Specifically, for machine learning API calls, two common sets of constraints are parameter size and a lower bound on accuracy. Consider, for instance, the following prompt: 'Invoke an image classification model that uses less than 10M parameters, but maintains an ImageNet accuracy of at least 70%.' Such a prompt presents a substantial challenge for the LLM to accurately interpret and respond to. Not only must the LLM understand the user's functional description, but it also needs to reason about the various constraints embedded within the request. This challenge underlines the intricate demands placed on LLMs in real-world API calls. It is not sufficient for the model to merely comprehend the basic functionality of an API call; it must

also be capable of navigating the complex landscape of constraints that accompany such calls. We also incorporate these instructions in our training dataset.

3.2 Gorilla

Our model, called Gorilla, is a retriever-aware finetuned LLaMA-7B model, specifically for API calls. As shown in Fig. 3, we employ self-instruct to generate {instruction, API} pairs. To fine-tune LLaMA, we convert this to a user-agent chat-style conversation, where each datapoint is a conversation with one round each for the user and the agent. We then perform standard instruction finetuning on the base LLaMA-7B model. For our experiments, we train Gorilla with and without the retriever. We would like to highlight that though we used the LLaMA model, our fine-tuning is robust to the underlying pre-trained model (see Appendinx A.3.5).

Retriever-Aware training (RAT) In retriever-aware training, the instruction-tuned dataset also appends to the user prompt, the relevant retrieved documentation with 'Use this API documentation for reference: <retrieved_API_doc_JSON>' . This is critical, because the retrieved documentation is not necessarily accurate - retrievers have imperfect re-call. By augmenting the prompt with potentially incorrect documentation, but the accurate ground-truth in the LLM response, we are in-effect teaching the LLM to 'judge' the retriever at inference time. During inference, if the LLM reasons that the retriever presented a relevant API document, it can use the API documentation to respond to the user's question, filling in additional details from the user's prompt. However, if after looking at the prompt, the LLM reasons that the retrieved API document is not relevant to the user's prompt, RAT trains the model to not get distracted by irrelevant context. The LLM then relies on the domain-specific knowledge baked-in during RAT training, to provide the user with the relevant API. Through RAT, we aim to teach the LLM to parse the second half of the question (API documentation) to answer the first half (user's query). We demonstrate that this (1) makes the LLM adapt to test-time changes in API documentation, (2) improves performance from in-context learning, and (3) reduces hallucination error.

Surprisingly, we find that augmenting a LLM with retrieval, does not always lead to improved performance, and can at-times hurt performance. We share more insights along with details in Sec 4.

Gorilla Inference During inference, the user provides the prompt in natural language (Fig. 3). This can be for a simple task (e.g., 'I would like to identify the objects in an image' ), or they can specify a vague goal, (e.g., 'I am going to the zoo, and would like to track animals' ). Gorilla, similar to training, can be used for inference in two modes: zero-shot and with retrieval. In the zero-shot setting, this prompt (with no additional prompt tuning) is fed to the Gorilla LLM model, which then returns the API call needed to accomplish the task or goal. In retrieval mode, the retriever (either of BM25 or GPT-Index) first retrieves the most up-to-date API documentation stored in the API Database. Before being sent to Gorilla, the API documentation is concatenated to the user prompt along with the message 'Use this API documentation for reference. ' The output of Gorilla is an API to be invoked. Besides the concatenation as described, we do no further prompt tuning in our system. While we also implemented a system to execute these APIs, to help the user accomplish the goal, that is not a focus of this paper.

3.3 Verifying APIs

Inductive program synthesis, where a program is synthesized to satisfy test cases, has found success in several avenues [3, 23]. However, test cases fall short when evaluating API calls, as it is often hard to verify the semantic correctness of the code. For example, consider the task of classifying an image. There are over 40 different models that can be used for the task. Even if we were to narrow down to a single family of densenet , there are four different configurations possible. Hence, there exist multiple correct answers and it is hard to tell if the API being used is functionally equivalent to the reference API by unit tests. Thus, to evaluate the performance of our model, we compare their functional equivalence using the dataset we collected. To trace which API in the dataset is the LLM calling, we adopt the AST tree-matching strategy. Since we only consider one API call in this paper, checking if the AST of the candidate API call is a sub-tree of the reference API call reveals which API is being used in the dataset.

hub

model_zoo

torch.hub.load('pytorch/vision:v0.10.0',

'densenet121', pretrained=True)

torch

hub

load

pytorch/vision

densenet121

pretrained = True

utils

Tensor

repo: huggingface/

pytorch-transformers

model:

densenet201

load

pretrained: False

pretrained: True

model:

densenet161

repo:

pytorch/vision

model:

densenet121

torch

optional

→

Identifying and even defining hallucinations can be challenging. We use the AST matching process to directly identify the hallucinations. We define a hallucination as an API call that is not a sub-tree of any API in the database - invoking an entirely imagined tool. This form of hallucination is distinct from invoking an API incorrectly which we instead define as an error. So, in our evaluations, error , hallucination , and accuracy add up to one.

AST Sub-Tree Matching We perform AST sub-tree matching to identify which API in our dataset is the LLM calling. Since each API call can have many arguments, we need to match on each of these arguments. Further, since, Python allows for default arguments, for each API, we define which arguments to match in our database. For example, we check repo_or_dir and model arguments in our function call. In this way, we can easily check if the argument matches the reference API or not. Fig. 4 illustrates an example subtree check for a torch API call. We first build the tree, and verify that it matches a subtree in our dataset along nodes torch.hub.load , pytorch/vision , and densenet121 . We do not check for match along leaf node pretrained=True since that is an optional argument.

4 Evaluation

When evaluating Gorilla, finetuned on APIBench ( train set ), we aim to answer the following questions: How does Gorilla compare to other LLMs on API Bench ( test set )? ( 4.1). How well does Gorilla adapt to test-time changes in API documentation? ( 4.2). How well can Gorilla handle questions with constraints? (4.3)

We demonstrate that Gorilla outperforms both open-source and close-source models for in-domain function calling. Further, trained with our novel retriever-aware training (RAT) technique, the Gorilla model generalizes to APIs that are outside of its training data (out-of-domain). In addition, we assess Gorilla's ability to reason about API calls under constraints. Lastly, we examined how integrating different retrieval methods during training influences the model's final performance.

Baselines We primarily compare Gorilla with state-of-the-art language models in a zero-shot setting and with 3-shot in-context learning. The models under consideration include: GPT-4 by OpenAI with the gpt-4-0314 checkpoint; GPT-3.5-turbo with the gpt-3.5-turbo-0301 checkpoint, both of which are RLHF-tuned models specifically designed for conversation; Claude with the claude-v1 checkpoint, a language model by Anthropic, renowned for its lengthy context capabilities; and LLaMA-7B, a state-of-the-art open-source large language model by Meta.

Retrievers The term zero-shot (abbreviated as 0-shot in tables) refers to scenarios where no retriever is used. The sole input to the model is the user's natural language prompt. For BM25, we consider

GPT-3.5

Gorilla

0

20

40

60

80

100

Accuracy (%)

60.21

59.13

60.21

14.51

61.82

Torch Hub with GPT-Retriever

GPT-3.5

Gorilla

47.34

44.57

41.37

10.17

47.45

HuggingFace with GPT-Retriever

GPT-3.5

Gorilla

65.59

43.94

55.62

15.62

64.96

Tensorflow Hub with GPT-Retriever

each API as a separate document. During retrieval, we use the user's query to fetch the most relevant (top-1) API. This API is concatenated with the user's prompt to query the LLMs. Similarly, GPTIndex refers to the state-of-the-art embedding model, text-embedding-ada-002-v2 from OpenAI, where each embedding is 1,536 dimensional. Like BM25, each API call is indexed as an individual document, and the most relevant document, given a user query, is retrieved and appended to the user prompt. Lastly, we include an Oracle retriever, which serves two purposes: first, to identify the potential for performance improvement through more efficient retrievers, and second, to assist users who know which API to use but may need to help invoking it. In all cases, when a retriever is used, it is appended to the user's prompt as follows: <user_prompt> Use this API documentation for reference: <retrieved_API_doc_JSON> . The dataset for these evaluations is detailed in Section 3. We emphasize that we have maintained a holdout test set on which we report our findings. The holdout test set was created by dividing the self-instruct dataset's instruction, API pairs into training and testing sets.

4.1 AST Accuracy on API call

We test each model for different retriever settings defined above (Table 1). We report the overall accuracy, the error by hallucination and the error by selecting wrong API call. Note that for TorchHub and TensorHub, we evaluate all the models using AST tree accuracy score. However, for HuggingFace, since the dataset cannot be exhaustive given the sheer number of models hosted, for all the models except Gorilla, we only check if they can provide the correct domain names. So this problem reduces to picking one of multiple choices. Across 0-shot and few-shot prompting strategies, Gorilla outperforms close-sourced and open-sourced models (Table 5).

Finetuning without Retrieval In Table 1 we show that lightly fine-tuned Gorilla is able to match, and often surpass performance in the zero-shot setting compared to closed-source, and open-source models - 20.43% better than GPT-4 and 10.75% better than GPT-3.5 (ChatGPT). When compared to other open-source models LLAMA, the improvement is as big as 83%. This suggests quantitatively, that as a technique to augment information and enforce adherence to syntax, fine-tuning is better than naive retrieval, at-least within the scope of invoking APIs.

Finetuning with Retrieval We now discuss how incorporating retrieval (RAT) during LLM finetuning enhances model performance. In this experiment, the base LLAMA model is finetuned with a prompt (instruction-generated), a reference API document (from a golden-truth oracle), and an example output generated by an LLM (GPT-4 in this case). As shown in Table 2, incorporating a ground-truth retriever in the finetuning pipeline yields notably improved results - 12.37% higher accuracy than training without retrieval in Torch Hub and 23.46% better in HuggingFace. However, at evaluation time, current retrievers show a significant performance gap compared to the ground-truth retriever: using GPT-Index at evaluation results in 29.20% accuracy degradation and using BM25 results in a 52.27% accuracy degradation. Despite this, considering the trends across models and retrievers, our findings indicate that finetuning an LLM with effective retrieval integration is preferable to zero-shot finetuning.

Hallucination with LLM One phenomenon we observe is that zero-shot prompting with LLMs (GPT-4/GPT-3.5) to call APIs results in dire hallucination errors. These errors,

while diverse, commonly manifest in erroneous behavior such as the model invoking the AutoModel.from_pretrained(dir_name) command with arbitrary GitHub repository names. Surprisingly, we also found that in TorchHub, HuggingFace and TensorFlow Hub, GPT-3.5 has less hallucination errors than GPT-4. This finding is also consistent for the settings when various retrieving methods are provided: 0-shot, BM25, GPT-Index and the oracle. This might suggest that RLHF plays a central role in turning the model to be truthful. Additional discussion in Appendix A.3.

AST as a Hallucination Metric We manually execute Gorilla's API generations to evaluate how well AST works as an evaluation metric. Executing every code generated is impractical within academic setting-for example, executing the HuggingFace model needs the required library dependencies (e.g., transformers, sentencepiece, accelerate), correct coupling of software kernels (e.g., torch vision, torch, cuda, cudnn versions), and required hardware support (e.g., A100 40G gpus). Hence, to make it tractable, we sampled 100 random Gorilla generations from our evalualtion set. The accuracy from our AST subtree matching is 78%, consistent with human evaluation of 78% accuracy in calling the right API. All the generations that AST flagged as incorrect, were the same ones that were manually flagged as incorrect. Additionally, Gorilla also generates supporting code to call the API which includes installing dependencies e.g., pip install transformers[sentencepiece]) , setting environment variables, etc. When we manually attempt to execute the code, 72% of all code generated executed successfully. It's worth noting that the 6% discrepancy are not semantic errors, but errors that arose due to factors external to the API, and in the supporting code. We have included the full example to illustrate this further in A.3.3. Considering the significant time and effort required for manual validation of each generation, the strong correlation between human evaluation and the AST evaluation further reinforces our belief in using the proposed AST as a robust offline metric.

USER: I want to automatically remove

the background from an input image

<Retrieval: ..fcn_resnet101..>

GORILLA:

api_call:

'torch.hub.load('pytorch/vision',

'fcn_resnet101', pretrained=True)'

USER: I want to automatically remove the

background from an input image. <Retrieval:

..NVIDIA/DeepLearningExamples:torchhub..>

GORILLA:

api_call:

'torch.hub.load('NVIDIA /

DeepLearningExamples:torchhub ', 'fcn_resnet50',

pretrained=True)'

USER: I want to automatically remove

the background from an input image.

<Retrieval:  ..fcn_resnet50..>

GORILLA:

api_call:

'torch.hub.load('pytorch/vision',

'fcn_resnet50', pretrained=True)'

Default Response

Update the model

Update the model repository

4.2 Test-Time Documentation Change

The rapidly evolving nature of API documentation presents a significant challenge for the application of LLMs in this field. These documents are often updated at a frequency that outpaces the retraining or fine-tuning schedule of LLMs, making these models particularly brittle to changes in the information they are designed to process. This mismatch in update frequency can lead to a decline in the utility and reliability of LLMs over time.

With the introduction of Gorilla's retriever-aware training, the RAT trained LLM readily adapts to changes in API documentation. This novel approach allows the model to remain relevant, even as the API documentation it relies on undergoes modifications. This is a pivotal advancement in the field, as it ensures that the LLM maintains its efficacy and accuracy over time, providing reliable outputs irrespective of changes in the underlying documentation.

For instance, consider the scenario illustrated in Fig. 6, where the training of Gorilla has allowed it to react effectively to changes in APIs. This includes alterations such as upgrading the FCN's ResNet-50 backbone to ResNet-101, as demonstrated in the second column of the figure. Since the model has encountered ResNet-101 as a backbone with other architectures, it interprets an FCN with a ResNet-101 backbone (unseen during training) as a relevant document at test time. Conversely, if the retriever suggests an FCN with a ResNet-60 backbone, the model-unfamiliar with ResNet-60 from RAT-assigns low confidence to this document and defaults back to FCN with ResNet-50. The third column in Fig. 6 further illustrates Gorilla's flexibility in adapting to shifts in model registries, such as from pytorch/vision to NVIDIA/DeepLearningExamples:torchhub , highlighting its ability to accommodate changes in preferred API sources as they evolve over time.

In summary, Gorilla's ability to adapt to test-time changes in API documentation offers numerous benefits. It maintains its accuracy and relevance over time, adapts to the rapid pace of updates in API documentation, and adjusts to modifications in underlying models and systems. This makes it a robust and reliable tool for API calls, significantly enhancing its practical utility.

4.3 API Call with Constraints

We now focus on the language model's capability of understanding constraints. For any given task, which API call to invoke is typically a tradeoff between a multitude of factors. In the case of RESTFul APIs, it could be the cost of each invocation ($) or the latency of response (ms), among many others. Similarly, within the scope of ML APIs, it is desirable for Gorilla to respect constraints such as accuracy, number of learnable parameters in the model, the size on disk, peak memory consumption, FLOPS, etc. In this section, we present a study evaluating the ability of different models in zero-shot and in the presence of retrievers to respect a given accuracy constraint. : if a user requests an image classification model that achieves at least 80% top-1 accuracy on the ImageNet dataset, then among the classification models hosted by Torch Hub, ResNeXt-101 32x16d , with a top-1 accuracy of 84.2%, would be the appropriate model to call, rather than MobileNetV2 , which has a top-1 accuracy of 71.88%.

For Table 4, we filtered a subset of the Torch Hub component of APIBench, retaining those models that had an accuracy metric defined for at least one-dataset the model was evaluated on, in its model card. We were left with 65.26% of TorchHub dataset from Table 1. We notice that with constraints, understandably, the accuracy drops across all models, with and without a retriever. Even in this challenging scenario, Gorilla is able to match the performance of the best-performing model GPT-3.5 when using retrievals (BM25, GPT-Index), and has the highest accuracy in the zero-shot setting. This highlights Gorilla's ability to navigate APIs while considering the trade-offs between constraints.

4.4 Finetuning (vs) Prompting: Gorilla 0-shot (vs) GPT 3-shot

To assess whether finetuning is truly necessary for APIs or if prompting alone is sufficient, we compare Gorilla in a zero-shot setting with three-shot in-context prompting for GPT-3.5 and GPT-4 models. In Table 5, "3-incont" denotes evaluation using three in-context examples, while "HF," "TH," and "TF" represent the HuggingFace, TorchHub, and TensorFlow Hub subsets of APIBench, respectively. Higher accuracy (Acc) and lower hallucination (Hall) rates are preferred. From Table 5, three-shot in-context learning improves the GPT models' ability to generate syntactically correct function calls, even matching accuracy on one subset (TorchHub). However, Gorilla 0-shot still outperforms the 3-shot GPT models on average.

5 Conclusion

LLMs are swiftly gaining popularity across diverse domains. APIs, serving as a universal language, are essential for enabling LLMs to communicate and operate effectively across diverse systems. In this paper, we introduced Gorilla, a state-of-the-art model for API invocation. Our Retriever Aware Training (RAT) approach empowers Gorilla with two essential capabilities: adapting dynamically to API changes at test time and reasoning through user-defined constraints when selecting suitable APIs. We also present APIBench, a comprehensive benchmark for assessing LLMs' function-calling abilities, and propose AST-based hallucination metrics for robust evaluation. Looking forward, we believe this work represents a first step towards transitioning LLMs from knowledge-bound models into flexible interfaces that interact with the digital world.

References

Andor, D., He, L., Lee, K., and Pitler, E. Giving bert a calculator: Finding operations and arguments with reading comprehension. arXiv preprint arXiv:1909.00109 , 2019.

Anthropic. Claude, 2022. URL https://www.anthropic.com/index/ introducing-claude .

Bavishi, R., Lemieux, C., Fox, R., Sen, K., and Stoica, I. Autopandas: neural-backed generators for program synthesis. Proceedings of the ACM on Programming Languages , (OOPSLA), 2019.

Brohan, A., Chebotar, Y., Finn, C., Hausman, K., Herzog, A., Ho, D., Ibarz, J., Irpan, A., Jang, E., Julian, R., et al. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on robot learning . PMLR, 2023.

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems , 2020.

Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y ., Lundberg, S., et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 , 2023.

Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021.

Chen, X., Lin, M., Schärli, N., and Zhou, D. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128 , 2023.

Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.

Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.

Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research , 2024.

Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021.

Devlin, J., Uesato, J., Bhupatiraju, S., Singh, R., Mohamed, A.-r., and Kohli, P. Robustfill: Neural program learning under noisy i/o. In International conference on machine learning . PMLR, 2017.

Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. Pal: Program-aided language models. In International Conference on Machine Learning . PMLR, 2023.

Iyer, S., Lin, X. V ., Pasunuru, R., Mihaylov, T., Simig, D., Yu, P., Shuster, K., Wang, T., Liu, Q., Koura, P. S., et al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212.12017 , 2022.

Jain, N., Vaidyanath, S., Iyer, A., Natarajan, N., Parthasarathy, S., Rajamani, S., and Sharma, R. Jigsaw: Large language models meet program synthesis. In Proceedings of the 44th International Conference on Software Engineering , 2022.

Kim, G., Baldi, P., and McAleer, S. Language models can solve computer tasks. arXiv preprint arXiv:2303.17491 , 2023.

Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Advances in neural information processing systems , 2022.

Komeili, M., Shuster, K., and Weston, J. Internet-augmented dialogue generation. arXiv preprint arXiv:2107.07566 , 2021.

Lazaridou, A., Gribovskaya, E., Stokowiec, W., and Grigorev, N. Internet-augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115 , 2022.

Li, R., Allal, L. B., Zi, Y ., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161 , 2023.

Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. Competition-level code generation with alphacode. Science , 2022.

Menon, A., Tamuz, O., Gulwani, S., Lampson, B., and Kalai, A. A machine learning framework for programming by example. In International Conference on Machine Learning . PMLR, 2013.

Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V ., Saunders, W., et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332 , 2021.

Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., and Xiong, C. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474 , 2022.

Nijkamp, E., Hayashi, H., Xiong, C., Savarese, S., and Zhou, Y. Codegen2: Lessons for training llms on programming and natural languages. arXiv preprint arXiv:2305.02309 , 2023.

OpenAI. Gpt-4 technical report, 2023.

OpenAI and https://openai.com/blog/chatgpt. Chatgpt, 2022. URL https://openai.com/ blog/chatgpt .

Roziere, B., Lachaux, M.-A., Chanussot, L., and Lample, G. Unsupervised translation of programming languages. Advances in neural information processing systems , 2020.

Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207 , 2021.

Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ili´ c, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 , 2022.

Schick, T. and Schütze, H. Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.07676 , 2020.

Schick, T., Dwivedi-Yu, J., Dessi, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems , 2023.

Shinn, N., Labash, B., and Gopinath, A. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366 , 2023.

Shuster, K., Xu, J., Komeili, M., Ju, D., Smith, E. M., Roller, S., Ung, M., Chen, M., Arora, K., Lane, J., et al. Blenderbot 3: a deployed conversational agent that continually learns to responsibly engage. arXiv preprint arXiv:2208.03188 , 2022.

Subramanian, S., Narasimhan, M., Khangaonkar, K., Yang, K., Nagrani, A., Schmid, C., Zeng, A., Darrell, T., and Klein, D. Modular visual question answering via code generation. arXiv preprint arXiv:2306.05392 , 2023.

Surís, D., Menon, S., and Vondrick, C. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023.

Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama model, 2023.

Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239 , 2022.

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Vemprala, S., Bonatti, R., Bucker, A., and Kapoor, A. Chatgpt for robotics: Design principles and model abilities. 2023 , 2023.

Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560 , 2022.

Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A., Stap, D., et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , 2022.

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 2022.

Xu, F. F., Alon, U., Neubig, G., and Hellendoorn, V. J. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming , 2022.

Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 , 2022.

Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., Xia, X., et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414 , 2022.

Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022.

Zhou, S., Alon, U., Xu, F. F., Jiang, Z., and Neubig, G. Docprompting: Generating code by retrieving the docs. In The Eleventh International Conference on Learning Representations , 2022.

A Appendix

A.1 Dataset Details

Our dataset is multi-faceted, comprising three distinct domains: Torch Hub, Tensor Hub, and HuggingFace. Each entry within this dataset is rich in detail, carrying critical pieces of information that further illuminate the nature of the data. Delving deeper into the specifics of each domain, Torch Hub provides 95 APIs. The second domain, Tensor Hub, is more expansive with a total of 696 APIs. Finally, the most extensive of them all, HuggingFace, comprises 925 APIs.

To enhance the value and utility of our dataset, we've undertaken an additional initiative. With each API, we have generated a set of 10 unique instructions. These instructions, carefully crafted and meticulously tailored, serve as a guide for both training and evaluation. This initiative ensures that every API is not just represented in our dataset, but is also comprehensively understood and effectively utilizable.

In essence, our dataset is more than just a collection of APIs across three domains. It is a comprehensive resource, carefully structured and enriched with added layers of guidance and evaluation parameters.

Domain Classification The unique domain names encompassed within our dataset are illustrated in Fig. 7. The dataset consists of three sources with a diverse range of domains: Torch Hub houses 6 domains, Tensor Hub accommodates a much broader selection with 57 domains, while HuggingFace incorporates 37 domains. To exemplify the structure and nature of our dataset, we invite you to refer to the domain names represented in Fig. 8.

API Call Task In this task, we test the model's capability to generate a single line of code, either in a zero-shot fashion or by leveraging an API reference. Primarily designed for evaluation purposes, this task effectively gauges the model's proficiency in identifying and utilizing the appropriate API call.

API Provider Component This facet relates to the provision of the programming language. In this context, the API provider plays a vital role as it serves as a foundation upon which APIs are built and executed.

Explanation Element This component offers valuable insights into the rationale behind the usage of a particular API, detailing how it aligns with the prescribed requirements. Furthermore, when certain constraints are imposed, this segment also incorporates those limitations. Thus, the explanation element serves a dual purpose, offering a deep understanding of API selection, as well as the constraints that might influence such a selection. This balanced approach ensures a comprehensive understanding of the API usage within the given context.

Code Example code for accomplishing the task. We de-prioritize this as we haven't tested the execution result of the code. We leave this for future works, but make this data available in-case others want to build on it.

A.2 Gorilla Details

We provide all the training details for Gorilla in this section. This includes how we divide up the training, evaluation dataset, training hyperparameters for Gorilla.

Data For HuggingFace, we devise the entire dataset into 90% training and 10% evaluation. For Torch Hub and Tensor Hub, we devise the data in to 80% training and 20% testing.

Training We train Gorillafor 5 epochs with the 2e-5 learning rate with cosine decay. The details are provide in Table 6. We finetune it on 8xA100 with 40G memory each.

Torch Hub domain names : Classification, Semantic Segmentation, Object Detection, Audio Separation, Video Classification, Text-to-Speech

Tensor Hub domain names : text-sequence-alignment, text-embedding, text-languagemodel, text-preprocessing, text-classification, text-generation, text-question-answering, textretrieval-question-answering, text-segmentation, text-to-mel, image-classification, imagefeature-vector, image-object-detection, image-segmentation, image-generator, image-posedetection, image-rnn-agent, image-augmentation, image-classifier, image-style-transfer, image-aesthetic-quality, image-depth-estimation, image-super-resolution, image-deblurring, image-extrapolation, image-text-recognition, image-dehazing, image-deraining, imageenhancemenmt, image-classification-logits, image-frame-interpolation, image-text-detection, image-denoising, image-others, video-classification, video-feature-extraction, videogeneration, video-audio-text, video-text, audio-embedding, audio-event-classification, audiocommand-detection, audio-paralinguists-classification, audio-speech-to-text, audio-speechsynthesis, audio-synthesis, audio-pitch-extraction

HuggingFace domain names : Multimodal Feature Extraction, Multimodal Text-to-Image, Multimodal Image-to-Text, Multimodal Text-to-Video, Multimodal Visual Question Answering, Multimodal Document Question Answer, Multimodal Graph Machine Learning, Computer Vision Depth Estimation, Computer Vision Image Classification, Computer Vision Object Detection, Computer Vision Image Segmentation, Computer Vision Image-to-Image, Computer Vision Unconditional Image Generation, Computer Vision Video Classification, Computer Vision Zero-Shor Image Classification, Natural Language Processing Text Classification, Natural Language Processing Token Classification, Natural Language Processing Table Question Answering, Natural Language Processing Question Answering, Natural Language Processing Zero-Shot Classification, Natural Language Processing Translation, Natural Language Processing Summarization, Natural Language Processing Conversational, Natural Language Processing Text Generation, Natural Language Processing Fill-Mask, Natural Language Processing Text2Text Generation, Natural Language Processing Sentence Similarity, Audio Text-to-Speech, Audio Automatic Speech Recognition, Audio Audio-to-Audio, Audio Audio Classification, Audio Voice Activity Detection, Tabular Tabular Classification, Tabular Tabular Regression, Reinforcement Learning Reinforcement Learning, Reinforcement Learning Robotics

Figure 7: Domain names : Domain names with the three dataset. Tensor Hub is the smallest dataset while the other two hubs contain many more models.

A.3 Performance Comparison

We provide a full comparison of each model's performance in this section. In Fig 10 and Fig. 11, the full set of comparisons is provided. We see that especially in zero-shot case, Gorilla surpasses the GPT-4 and GPT-3.5 by a large margin. The GPT-4 and GPT-3.5 gets around 40% accuracy in Torch Hub and Tensor Hub, which are two structured API calls. Compared to that, HuggingFace is a more flexible and diverse Hub, as a result, the performance on HuggingFace is not as competitive.

### User : Our customer is a zoo and we want to help them detect movement of different animals. Write a python program in 1 to 2 lines to call API in TensorFlowHub. ### Assistant :

<domain> : image-object-detection,

<api_provider> : TensorFlow Hub,

<explanation> : This API call loads the pre-trained Single Shot Detector (SSD) MobileNet V2 model from TensorFlow Hub, which has been trained on the Open Images v4 dataset. This model is used for object detection in images. Once loaded, you can pass an image (and an optional score threshold for detection confidence) to the detector to get back the detected objects along with their bounding boxes, class labels, and associated confidence scores. <code> : import tensorflow as tf; import tensorflow_hub as hub; # Load the SSD MobileNet V2 model; detector = hub.load('https://tfhub.dev/google/openimages_v4/ssd_mobilenet_v2/1'); # Use the model to detect objects in an image (assuming 'image' is a TensorFlow tensor representing your image); result = detector(image, score_threshold=0.5)

### User : I am an engineer at Uber and I need to find a API that can classify pedestrians, cars, etc. from an image of the scene. Write a python program in 1 to 2 lines to call API in TorchHub.

### Assistant :

<domain> : Object Detection, <api_call> : model = torch.hub.load('datvuthanh/hybridnets', 'hybridnets', pretrained=True),

<api_provider> : PyTorch,

<explanation> : Load the pretrained HybridNets model for object detection from PyTorch Hub to classify pedestrians, cars, etc. in an image of the scene,

<code> : import torch; model = torch.hub.load('datvuthanh/hybridnets', 'hybridnets', pretrained=True)

generate_video = pipeline("text-to-video", model="your_model_name")

vqa = pipeline("visual-question-answering", tokenizer="microsoft/clip-vqa-base")

model="microsoft/clip-vqa-base",

depth_estimator = pipeline("depth-estimation", model="intel-isl/MiDaS", tokenizer="intel- isl/MiDaS")

A.3.1 Evaluation

For ease of evaluation, we manually cleaned up the dataset to ensure each API domain only contains the valid call of form:

Our framework allows the user to define any combination of the arguments to check. For Torch Hub, we check for the API name torch.hub.load with arguments repo_or_dir and model . For Tensor Hub, we check API name hub.KerasLayer and hub.load with argument handle . For HuggingFace, since there are many API function names, we don't list all of them here. One specific note is that we require the pretrained_model_name_or_path argument for all the calls except for pipeline . For pipeline , we don't require the pretrained_model_name_or_path argument since it automatically select a model for you once task is specified.

A.3.2 Hallucination

Wefound especially in HuggingFace, the GPT-4 model incurs serious hallucination problems. It would sometimes put a GitHub name that is not associated with the HuggingFace repository in to the domain of pretrained_model_name_or_path . Fig. 9 demonstrates some examples and we also observe that GPT-4 sometimes assumes the user have a local path to the model like your_model_name . This is greatly reduced by Gorilla as we see the hallucination error comparison in Table 1.

A.3.3 AST as a Hallucination Metric

We evaluated the generated results on 100 LLM generations (randomly chosen from our eval set). The accuracy using AST subtree matching is 78%, consistent with human evaluation with 78% accuracy in calling the right API. All the generations that AST flagged as incorrect, were the same ones that were manually also flagged as incorrect. Additionally, Gorilla generates supporting code to call the API which includes installing dependencies (e.g., pip install transformers[sentencepiece] ), environment variables, etc. When we manually attempted to execute end-to-end code, 72% of all codes generated were executed successfully. It's worth noting that the 6% discrepancy were NOT semantic errors, but errors that arose due to factors external to the API in the supporting code - we have included an example to illustrate this further. Considering the significant time and effort required for manual validation of each generation, our evaluation highlights the efficiency of using AST as a robust offline metric.

Here is a representative example, where we are able to load the correct model API. However, in the supporting code, after we have the output from the API, the zip() function tries to combine sentiments and scores together. However, since scores is a float, it's not iterable. zip() expects both its arguments to be iterable, resulting in an 'float' object is not iterable error.

LLAMA GPT-3.5 GPT-4

Model

0

20

40

60

80

100

0

48.38

38.7

18.81

59.13

Torch Hub 0-shot

LLAMA GPT-3.5 GPT-4

Model

0

20

40

60

80

100

8.6

38.17

35.48

39.78 40.32

Torch Hub with BM25-Retriever

LLAMA GPT-3.5 GPT-4

Model

0

20

40

60

80

100

14.51

60.21

59.13

60.21

61.82

Torch Hub with GPT-Retriever

LLAMA GPT-3.5 GPT-4

Model

0

20

40

60

80

100

16.12

66.31 66.12

63.44

67.2

Torch Hub with Oracle Retriever

LLAMA GPT-3.5 GPT-4

Model

0

20

40

60

80

100

0

16.81

19.8

6.1

71.68

HuggingFace 0-shot

LLAMA GPT-3.5 GPT-4

Model

0

20

40

60

80

100

2.98

17.25 16.48

14.6

17.03

HuggingFace with BM25-Retriever

LLAMA GPT-3.5 GPT-4

Model

0

20

40

60

80

100

10.17

47.34

44.57

41.37

47.45

HuggingFace with GPT-Retriever

LLAMA GPT-3.5 GPT-4

Model

0

20

40

60

80

100

17.69

89.71

85.06

77.21

91.26

HuggingFace with Oracle Retriever

LLAMA GPT-3.5 GPT-4

Model

0

20

40

60

80

100

0

41.75

18.2

9.19

83.79

Tensorflow Hub 0-shot

LLAMA GPT-3.5 GPT-4

Model

0

20

40

60

80

100

8.9

54.16

34.01

35.18

41.89

Tensorflow Hub with BM25-Retriever

LLAMA GPT-3.5 GPT-4

Model

0

20

40

60

80

100

15.62

65.59

43.94

55.62

64.96

Tensorflow Hub with GPT-Retriever

LLAMA GPT-3.5 GPT-4

Model

0

20

40

60

80

100

12.55

95.03

55.91

74.74

94.16

Tensorflow Hub with Oracle Retriever

A.3.4 Gorilla (VS) DocPrompting

We evaluate Gorilla and DocPrompting [49] on the HuggingFace Dataset from Table 1. For a 7B model, when trained on the same number of epochs, with and the same learning rate for both the models, Gorilla improves accuracy while reducing hallucination.

A.3.5 Sensitivity to pre-training

Gorilla's training recipe is robust to the pre-training strategies and recipes of the underlying model. From Fig. 13 we demonstrate that all the three models can converge to within a few percentage points in accuracy independent of the pre-trained base model.

0

10

20

30

40

50

60

70

80

90

100

Hallucination

0

10

20

30

40

50

60

70

80

90

100

LLAMA

GPT-3.5

Gorilla

Torch Hub 0-shot

0

10

20

30

40

50

60

70

80

90

100

Hallucination

0

10

20

30

40

50

60

70

80

90

100

LLAMA

GPT-3.5

Gorilla

Torch Hub with BM25-Retriever

0

10

20

30

40

50

60

70

80

90

100

Hallucination

0

10

20

30

40

50

60

70

80

90

100

LLAMA

GPT-3.5

Gorilla

Torch Hub with GPT-Retriever

0

10

20

30

40

50

60

70

80

90

100

Hallucination

0

10

20

30

40

50

60

70

80

90

100

LLAMA

GPT-3.5

Gorilla

Torch Hub with Oracle Retriever

0

10

20

30

40

50

60

70

80

90

100

Hallucination

0

10

20

30

40

50

60

70

80

90

100

LLAMA

GPT-3.5

Gorilla

HuggingFace 0-shot

0

10

20

30

40

50

60

70

80

90

100

Hallucination

0

10

20

30

40

50

60

70

80

90

100

LLAMA

GPT-3.5

Gorilla

HuggingFace with BM25-Retriever

0

10

20

30

40

50

60

70

80

90

100

Hallucination

0

10

20

30

40

50

60

70

80

90

100

LLAMA

GPT-3.5

Gorilla

HuggingFace with GPT-Retriever

0

10

20

30

40

50

60

70

80

90

100

Hallucination

0

10

20

30

40

50

60

70

80

90

100

LLAMA

GPT-3.5

Gorilla

HuggingFace with Oracle Retriever

0

10

20

30

40

50

60

70

80

90

100

Hallucination

0

10

20

30

40

50

60

70

80

90

100

LLAMA

GPT-3.5

Gorilla

Tensorflow Hub 0-shot

0

10

20

30

40

50

60

70

80

90

100

Hallucination

0

10

20

30

40

50

60

70

80

90

100

LLAMA

GPT-3.5

Gorilla

Tensorflow Hub with BM25-Retriever

0

10

20

30

40

50

60

70

80

90

100

Hallucination

0

10

20

30

40

50

60

70

80

90

100

LLAMA

GPT-3.5

Gorilla

Tensorflow Hub with GPT-Retriever

0

10

20

30

40

50

60

70

80

90

100

Hallucination

0

10

20

30

40

50

60

70

80

90

100

LLAMA

GPT-3.5

Gorilla

Tensorflow Hub with Oracle Retriever

Gorilla-LLaMA

Gorilla-MPT

Gorilla-Falcon

Pre-trained models

0

10

20

30

40

50

60

70

Gorilla 0-shot on HF

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The paper provides a recipe to teach LLMs to use tools, and also presents a data-set for evaluating API calling, and a metric for measuring hallucination. The paper studies them with rigorous evaluations.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All the hyperparameters are specified in the appendix, and the code and dataset is open-sourced at github.com/ShishirPatil/gorilla .

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All code, data, and models are open-sourced under the Apache 2.0 license by the authors.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the hyperparameters are specified in the appendix, and all code, data, and models are open-sourced.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: All non-LLM experiments are deterministic so need no error bars, and given the GPU costs involved, we perform LLM experiments once.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Provided in Appendix including the sample dataset.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Conform's with NeurIPS Code of Ethics

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Integrating language models with API calls significantly extends their utility, enabling a wide range of applications from automating customer service to generating realtime content and facilitating data analysis. This integration can lead to more personalized and efficient user experiences across various platforms, as language models can process natural language inputs and interact with different APIs to fetch, interpret, and act on data in real time. For instance, in customer service, this can mean providing instant, relevant responses to queries, reducing wait times, and improving overall satisfaction. In content generation, it can enable dynamic creation of articles, reports, or summaries based on the latest data available from web services.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Unlike Images, etc where there are copyrights involved, APIs are meant to distributed. Hence, the incentives are very well aligned. For example, if Gorilla presents a particular service's API, the service benefits from engagement.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All code, data, and models are open-sourced under the Apache 2.0 license by the authors.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The open-source repository is actively maintained at github.com/ShishirPatil/gorilla

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Gorilla, a fine-tuned LLaMA model, outperforms GPT-4 in crafting API calls and adapts well to document changes with a document retriever, reducing hallucination issues. It also provides APIBench, a new dataset for evaluation, includes APIs from HuggingFace, TorchHub, and TensorHub. Gorilla's integration with a retrieval system enhances tool usage accuracy and documentation updates, promising more reliable LLM outputs.\n\nSTRENGTHS:\n1. The paper trains a system which connects massive APIs and takes text instruction to get the corresponding API calls, along with a step-by-step explanation of the pipeline. It focuses on the details within an API call, which significantly mitigates the complexity involved in developing machine learning softwares.\n2. The paper also constructs APIBench by scraping a large corpus of ML APIs and developing an evaluation framework to check functional correctness. It uses AST subtree-matching metrics which helps measure hallucination and further contributes to the evaluation of ML API mastering techniques.\n3. The paper is well-writtened, presenting a detailed comparison with other related works, a clear structure on methods, and comprehensive experiments and analysis.\n\nWEAKNESSES:\n1. Although the paper overall does not present significant issues, I would appreciate seeing a performance comparison with more specific methods, such as other works that also utilize LLM for tool calling to build software pipelines, which may further demonstrate the contribution of techniques presented by Gorilla.\n2. The paper needs to better illustrate the contributions of this work and more clearly outline its potential impact on the community.\n3. Some writing detales need to be improved in the paper, such as \"the\" should be omitted in \"The model trained with the our method\" in Line 31, and \"AST\" should be replaced by \"Abstract Syntax Tree\" for the first time it appears in Line 36.\n\nQUESTIONS:\n1. In Figure 2, why GPT-3.5 is much better than GPT-4 in all settings?\n2. As most of the instructions are generated by GPT-4 referring to your hand-generated instruction seeds. How to ensure your seeds meets the real-world scenario and reach the real difficulties?\n3. As many APIs have overlapping functions, how to decide which API is better than the other similar one? How to maintain stable API calling to persist consistent results for the same request?\n4. Which Llama version is used as your foundation model? Llama2 or Llama3?\n5. In Table 1, why Gorilla always produces a large error in selecting wrong APIs? Do the results support your view?\n\nLIMITATIONS:\nThe paper should provide a more comprehensive limitation part to point out potential overlooked issues, such as influence of synthetic training data and insufficient comparative experiments.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a new fine-tuned language model that is trained to map from user instructions to code snippets that invoke the appropriate APIs. The authors also introduce a new dataset which includes the information for roughly 1600 APIs from a variety of online sources, which is used to train the model. They conclude that their approach outperforms existing language models on the task of API call generation, as measured by both error rate and a novel “hallucination” metric.\n\nSTRENGTHS:\nI feel that the problem statement is well-motivated and of general interest. Prior work has investigated allowing LLMs to make use of various tools, and improvements in that capability are likely to be well received. I appreciate that the authors have chosen a (relatively speaking) lightweight and open-source model, which increases the usability of their approach. I also appreciate the construction of a novel dataset, which could presumably be of interest to the larger community even in the absence of a specific model.\n\nWEAKNESSES:\nFirst, I wonder about baselines. The authors have done a good job of comparing their method against a variety of open-source and closed-source LLMs, but these systems are all “generalist” compared to the fine-tuned Gorilla model. It would have been useful to see a comparison to a fine-tuned version of another model (even one with less fine-tuning than is possible on the small Llama-7b) just to have a sense of how much improvement can be leveraged from the newly introduced dataset. I also wonder if it would have been possible to compare to specialized tool-based LLM models like Toolformer. If such a comparison is not appropriate or possible, I feel the authors should mention why.\n\nMy second question concerns statistical significance. The authors indicate the high cost of LLM experiments as the reason for omitting such analysis. I sympathize with this explanation, but feel that I would be remiss not to stress the importance of statistical testing as part of justifying claims that Gorilla’s “performance surpasses prompting the state-of-the-art LLMs in three massive datasets.” Without a measure of a variance, we cannot meaningfully conclude that apparent improvements achieved by Gorilla (which are often on the order of 1-3% in overall score) are the result of anything other than noise. To be sure, a 7B parameter model that achieves even comparable results to a massive LLM is a notable result (though see my earlier point about fine-tuned baselines), but I would temper claims about improvement in the absence of statistical justification.\n\nLastly, I feel that the paper would benefit greatly from a more robust error analysis. In the best-performing realistic setting (i.e. without access to an oracle), Gorilla achieves an overall accuracy of roughly 65%. This means that more than one third of the time it is returning an incorrect result (including errors, the authors point out, that could be silent to the end user). Obviously this is far from being ready for direct deployment without external validation. Given the prevalence of errors, it would be useful to better understand what kinds of errors are common: are they more likely to occur with particular kinds of APIs? Does the wording and / or complexity of the input prompt have a large effect? This is the kind of analysis that, in combination with the novel dataset, could help spur future work and improvement. In addition, given the potential for damaging errors, a more robust broader impact statement would be beneficial.\n\nQUESTIONS:\nHow do models which are fine-tuned compare to Gorilla? (If such a comparison is not appropriate, why?)\n\nWhat is the statistical significance of the results?\n\nWhat are the most common kinds of errors?\n\nDo the kinds of errors differ between the different categories of APIs?\n\nAs a final minor question: do the authors have an explanation for why GPT-3.5 appears to outperform GPT-4 across a few different problem settings? This seems like a surprising and somewhat notable result!\n\nLIMITATIONS:\nI feel that the authors should have a more robust discussion of the broader impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Gorilla, a fine-tuned LLaMA model designed to improve large language models' ability to use APIs accurately. The authors created APIBench, a comprehensive dataset of ML APIs from HuggingFace, TorchHub, and TensorFlow Hub, and used self-instruct to generate instruction-API pairs for training. They fine-tuned LLaMA-7B with retrieval-aware training, incorporating tech like AST sub-tree matching for evaluating API call accuracy, retriever-aware training to adapt to API changes, and handling of constrained API calls. Results show that Gorilla outperforms existing LLMs (including GPT-4) on API call accuracy across multiple datasets, demonstrates ability to adapt to test-time changes in API documentation, and handles constrained API calls effectively. The paper presents a novel approach to improving LLMs' API usage capabilities, with promising results that outperform existing state-of-the-art models in this specific domain.\n\nSTRENGTHS:\n* Gorilla outperforms existing state-of-the-art language models, including GPT-4 and Claude, in API call accuracy across multiple datasets (TorchHub, HuggingFace, and TensorFlow Hub).\n* Gorilla significantly reduces API argument hallucination errors compared to other models, improving the reliability of API calls.\n* The retriever-aware training enables Gorilla to adapt to test-time changes in API documentation, allowing it to remain up-to-date with evolving APIs without requiring retraining.\n* Gorilla demonstrates the ability to understand and respect constraints (e.g., accuracy requirements) when selecting appropriate APIs, outperforming other models in constraint-aware API invocations.\n\nWEAKNESSES:\nA significant weakness of the Gorilla approach is that it relies heavily on knowledge augmentation, which is a common technique already used in various domains to improve language model performance on specific tasks. The use of retrieval-augmented generation to enhance API calling capabilities doesn't represent a novel improvement or implementation compared to similar approaches in other domains.\n\nEssentially, Gorilla applies existing knowledge augmentation techniques to the specific task of API invocation, rather than introducing a fundamentally new method for improving language model capabilities. While the results show improvements in API calling accuracy, the core approach of combining retrieval with language model fine-tuning is not innovative in itself. This limits the broader impact and generalizability of the work beyond the specific domain of API invocation.\n\nThe lack of significant methodological innovation suggests that similar performance improvements could potentially be achieved by applying existing retrieval-augmented generation techniques to the API calling task, without necessarily requiring the specific Gorilla architecture. This raises questions about the uniqueness and broader applicability of the Gorilla approach beyond the narrow domain explored in the paper.* While Gorilla performs well on the specific API datasets it was trained on, it's unclear how well it would generalize to entirely new APIs or domains not covered in the training data.\n\nOther Notes: \n* Although Gorilla shows some ability to handle constraints, its performance in this area is not significantly better than other models like GPT-3.5, suggesting room for improvement.\n* The paper doesn't compare Gorilla's performance with specialized API documentation tools or code completion systems, which might be more tailored for this specific task.\n* The paper doesn't provide an in-depth analysis of the cases where Gorilla fails, which could provide insights into its limitations and areas for improvement.\n\nQUESTIONS:\n* How does Gorilla's performance compare to other specialized code generation or API-focused models, not just general-purpose LLMs?\n* What is the performance impact of varying the size of the training dataset or the base model size (e.g. using LLaMA-13B instead of 7B)?\n* How does Gorilla perform on more complex multi-step API workflows rather than just single API calls?\n\nLIMITATIONS:\nNA - Appropriately discussed limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a pipeline to call adequate APIs among massive pools to accomplish users’ instructions. For that, the authors construct and release the APIBench dataset that contains more than 1645 APIs, and propose the Gorilla model, which is a retrieval-aware finetuned Llama-7B model for API calls.\n\nSTRENGTHS:\n- Constructing the APIBench with AST tree matching evaluation metrics as well as open-sourcing the trained Gorilla model will be largely benefit to the community.\n- The problem in this paper is timely in terms of LLM applications and eco-systems — i.e., automatizing API function calls.\n- The fine-tuned model, Gorilla, surpasses performance compared to current SOTA models.\n- The paper is well written and soundly provides experiment results and analysis.\n\nWEAKNESSES:\n- Lack of details about the training data for Gorilla, such as data stats and construction methods. It’s unclear how different the Gorilla training data and evaluation sets of APIBench are, which could lead to doubts about data contamination.\n- Experiment results analysis\n    - How robust are models to paraphrased user instructions corresponding to the same target model API?\n    - Providing performance for each domain could make it possible to analyze which domains are easy and hard.\n- Clarifications are needed:\n    - In the experimental result section, the authors report two metrics along with overall accuracy: **the error by hallucination and by selecting the wrong API call.** The metric equations or explanations for calculating the values need to be clarified.\n    - In section AST as a Hallucination Metric (line 251), how was experimented to attain the human evaluation of 78% accuracy?\n- It is required to mention the license of crawled APIs for usage.\n- A limitation section is absent, though the authors describe them in the checklist. Broader impact and limitation could include implicit risks such as usage of unreliable APIs, license infringement, or unexpected results by incorrect function calls.\n\nQUESTIONS:\n- In section 4.1, “Note that for TorchHub and TensorHub, we evaluate all the models using AST tree accuracy score. *However, for HuggingFace, since the dataset is not exhaustive, for all the models except Gorilla, we only check if they can provide the correct domain names.*” is vague. The numbers of data for HF, TFH, and TH are 925, 801, and 95, respectively. Please clarify this.\n- Table 1 shows that TorchHub is hard to even with oracle API documents compared to HuggingFace and TensorFlowHub. Can you draw the reason?\n- (Typos) line 225. `his suggests`\n- (suggestions) Please elaborate more on Table 2 — e.g., the caption and highlights. Moreover, it’s hard to connect between the written explanation and the table, due to different numbers.\n- (suggestions) Absence of mentioning the full name of AST (Abstract Syntax Tree)\n\nLIMITATIONS:\n- Limitation section is absent, though the authors describe them in the check list. Broader impact and limitation could include implicit risks such as usage of unreliable APIs, license infringement, or unexpected results by incorrect function calls.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}