Gorilla: Large Language Model Connected with Massive APIs

Authors: shishir g patil
tianjun  zhang
xin  wang
joseph e gonzalez
u c berkeley
microsoft  research
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 21:48:13

Abstract: Large Language Models (LLMs) have seen an impressive wave of advances, with
models now excelling in a variety of tasks, such as mathematical reasoning and
program synthesis. However, their potential to effectively use tools via API calls
remains unfulfilled. This is a challenging task even for today’s state-of-the-art
LLMs such as GPT-4 largely due to their unawareness of what APIs are available
and how to use them in a frequently updated tool set. We develop Gorilla, a
finetuned LLaMA model that surpasses the performance of GPT-4 on writing API
calls. Trained with the novel Retriever Aware Training (RAT), when combined
with a document retriever, Gorilla demonstrates a strong capability to adapt to
test-time document changes, allowing flexible user updates or version changes.
It also substantially mitigates the issue of hallucination, commonly encountered
when prompting LLMs directly. To evaluate the model’s ability, we introduce
APIBench, a comprehensive dataset consisting of HuggingFace, TorchHub, and
TensorHub APIs. The successful integration of the retrieval system with Gorilla
demonstrates the potential for LLMs to use tools more accurately, keep up with
frequently updated documentation, and consequently increase the reliability and
applicability of their outputs. Gorilla’s code, model, data, and demo are available
at: https://gorilla.cs.berkeley.edu

Main paper: 
 introduction: 
 The use of APIs and Large Language Models [10, 5, 31, 6, 27, 28] has changed what it means to program. Previously, building complex machine learning software and systems required extensive time and specialized skills. Now with tools like the HuggingFace API, an engineer can set up a deep learning pipeline with a few lines of code. Instead of searching through StackOverflow and documentation, developers can ask models like GPT for solutions and receive immediate, actionable code with docstrings. However, using off-the-shelf LLMs to generate API calls remains unsolved because there are millions of available APIs which are frequently updated.We connect LLM's and massive API's with Gorilla, a system which takes an instruction, for example "build me a classifier for medical images", and provides the corresponding API call and relevant packages, along with a step-by-step explanation of the pipeline. Gorilla uses self-instruct, fine-tuning, and retrieval to enable LLMs to accurately select from a large, overlapping, and changing set tools expressed using their APIs and API documentation. Further, our novel retriever-aware training (RAT) enables the model to adapt to test-time changes of APIs such as evolution in versions and arguments.With the development of API generation methods comes a question of how to evaluate, as many APIs will have overlapping functionality with nuanced limitations and constraints. Thus, we construct Incorrect API Call (Claude)Figure 1 : Examples of API calls. Example API calls generated by GPT-4 [27] , Claude [2] , and Gorilla for the given prompt. In this example, GPT-4 presents a model that doesn't exist, and Claude picks an incorrect library. In contrast, our Gorilla model can identify the task correctly and suggest a fully-qualified API call.Figure 2 : Accuracy (vs) hallucination in four settings, that is, zero-shot (i.e., without any retriever), and with retrievers. Commonly used BM25 and GPT retrievers, and the oracle -returns relevant documents with perfect recall, indicating an upper bound. Higher in the graph (higher accuracy) and to the left (lower hallucination) is better. Across settings, our model, Gorilla, improves accuracy while reducing hallucination.APIBench (∼ 1600 APIs) by scraping a large corpus of ML APIs and developing an evaluation framework that uses AST sub-tree matching to check functional correctness. Further, we draw a distinction between accuracy and hallucination, and propose an Abstract Syntax Tree (AST) based technique to measure hallucination.Using APIBench, we finetune Gorilla, a LLaMA-7B-based model with document retrieval and show that it significantly outperforms both open-source and closed-source models like Claude and GPT-4 in terms of API functionality accuracy as well as a reduction in API argument hallucination errors. We show an example output in Fig. 1 . Lastly, we highlight Gorilla's capability to comprehend and reason about user-defined constraints when choosing between APIs, an essential requirement for LLMs trained to accomplish tasks.To summarize, this paper makes the following contributions:1. We introduce Gorilla, the first system to enable large-scale API integration with LLMs, demonstrating state-of-the-art performance in generating accurate API calls across thousands of functions and libraries.2. We develop Retriever-Aware Training (RAT), a novel technique that enables LLMs to effectively utilize retrieved API documentation at inference time, improving both accuracy and adaptation to API changes. 
related work: 
 By empowering LLMs to use tools [33] , we can grant LLMs access to vastly larger and changing knowledge bases and accomplish complex computational tasks. By providing access to search technologies and databases, [24, 39, 35 ] demonstrated that we can augment LLMs to address a significantly larger and more dynamic knowledge space. Similarly, by providing access to computational tools, [39, 1, 49, 36, 37] demonstrated that LLMs can accomplish complex computational tasks. Consequently, leading LLM providers [27] , have started to integrate plugins to allow LLMs to invoke external tools through APIs.Large Language Models Recent strides in the field of LLMs have renovated many downstream domains [10, 40, 48, 47] , not only in traditional natural language processing tasks but also in program synthesis. Many of these advances are achieved by augmenting pre-trained LLMs by prompting [44, 14] and instruction fine-tuning [11, 30, 43, 15] . Recent open-sourced models like LLaMa [40] , Alpaca [38] , and Vicuna [9] have furthered the understanding of LLMs and facilitated their experimentation. While our approach, Gorilla, incorporates techniques akin to those mentioned, its primary emphasis is on enhancing the LLMs' ability to utilize millions of tools, as opposed to refining their conversational skills. Additionally, we pioneer the study of fine-tuning a base model by supplementing it with information retrieval -a first, to the best of our knowledge.Tool Usage The discussion of tool usage within LLMs has seen an upsurge, with models like Toolformer taking the lead [33, 19, 20, 24] . Tools often incorporated include web-browsing [32] , calculators [12, 39] , translation systems [39] , and Python interpreters [14] . While these efforts can be seen as preliminary explorations of marrying LLMs with tool usage, they generally focus on specific tools. Our paper, in contrast, aims to explore a vast array of tools (i.e., API calls) in an open-ended fashion, potentially covering a wide range of applications.With the recent launch of Toolformer [33] highlights the exciting potential of using large language models (LLMs) for purposes beyond traditional chatbot applications. Moreover, the application of API calls in robotics has been explored to some extent [41, 4] . However, these works primarily aim at showcasing the potential of "prompting" LLMs rather than establishing a systematic method for evaluation and training (including fine-tuning). Our work, on the other hand, concentrates on systematic evaluation and building a pipeline for future use.Harnessing LLMs for program synthesis has historically been a challenging task [22, 7, 45, 16, 13, 29] . Researchers have proposed an array of strategies to prompt LLMs to perform better in coding tasks, including in-context learning [44, 18, 7] , task decomposition [17, 46] , and self-debugging [8, 34] . Besides prompting, there have also been efforts to pretrain language models specifically for code generation [25, 21, 26] .DocPrompting [49] looked at choosing the right subset of code including API along with a retriever. Gorilla presents distinct advancements over DocPrompting. First, the way the data-sets are constructed are different, leading to intersting downstream artifacts. Gorilla focuses on model usages where we also collect detailed information about parameters, performance, efficiency, etc. This helps our trained model understand and respond to finer constraints for each API. Docprompting focuses on generic API calls but not on the details within an API call. Second, Gorilla introduces and uses the AST subtreematching evaluation metric that helps measure hallucination which we find are more representative of code structure and API accuracy compared to traiditional NLP metrics. Finally, Gorilla focuses on instruction-tuning method and has "agency" to interact with users while DocPrompting focuses on building an NLP-to-Code generative model. On equal footing, we demonstrate that Gorilla performs better than DocPrompting in Appendix A.3. 
methodology: 
 We first describe APIBench, a comprehensive benchmark constructed from TorchHub, TensorHub, and HuggingFace API Model Cards. We begin by outlining the process of collecting the API dataset and how we generated instruction-answer pairs. We then introduce Gorilla, a novel training paradigm with an information-retriever incorporated into the training and inference pipelines. Finally, we present our AST tree matching evaluation metric. During inference (lower half), Gorilla supports two modes -with retrieval, and zero-shot. In this example, it is able to suggest the right API call for generating the image from the user's natural language query.To curate the dataset, we aggregate all model cards from HuggingFace's "The Model Hub", PyTorch Hub, and TensorFlow Hub. Throughout the rest of the paper, we call these HuggingFace, Torch Hub, and TensorFlow Hub respectively for brevity.The HuggingFace platform hosts and servers about 203,681 models. However, many of them have poor documentation, lack dependencies, have no information in their model card, etc. To filter these out, we pick the top 20 models from each domain. We consider 7 domains in multimodal data, 8 in CV, 12 in NLP, 5 in Audio, 2 in tabular data, and 2 in reinforcement learning. Post filtering, we arrive at a total of 925 models from HuggingFace. TensorFlow Hub is versioned into v1 and v2. The latest version (v2) has 801 models in total, and we process all of them. After filtering out model cards with little to no information, we are left with 626 models. Similar to TensorFlow Hub, we extract 95 models (exhaustive) from Torch Hub. We then convert the model cards for each of these 1,645 API calls into a JSON object with the following fields: {domain, framework, functionality, api_name, api_call, api_arguments, environment_requirements, example_code, performance, description}. We provide more information in Appendix A.1. These fields were chosen to generalize beyond API calls within the ML domain, to other domains, including RESTful, SQL, and other potential API calls.Instruction Generation Guided by the self-instruct paradigm [42] , we employ GPT-4 to generate synthetic instruction data. We provide three in-context examples, along with reference API documentation, and task the model with generating real-world use cases that call upon the API. We specifically instruct the model to refrain from using any API names or hints when creating instructions. We constructed 6 examples (Instruction-API pairs) for each of the 3 model hubs. These 18 examples were the only hand-generated or modified data. For each of our 1,645 API datapoints, we generate 10 instruction-API pairs by sampling 3 of 6 corresponding instruction examples in each pair (Fig. 3 ).API Call with Constraints API calls often come with inherent constraints. These constraints necessitate that the LLM not only comprehend the functionality of the API call but also categorize the calls according to different constraint parameters. Specifically, for machine learning API calls, two common sets of constraints are parameter size and a lower bound on accuracy. Consider, for instance, the following prompt: "Invoke an image classification model that uses less than 10M parameters, but maintains an ImageNet accuracy of at least 70%." Such a prompt presents a substantial challenge for the LLM to accurately interpret and respond to. Not only must the LLM understand the user's functional description, but it also needs to reason about the various constraints embedded within the request. This challenge underlines the intricate demands placed on LLMs in real-world API calls. It is not sufficient for the model to merely comprehend the basic functionality of an API call; it must also be capable of navigating the complex landscape of constraints that accompany such calls. We also incorporate these instructions in our training dataset.Our model, called Gorilla, is a retriever-aware finetuned LLaMA-7B model, specifically for API calls.As shown in Fig. 3 , we employ self-instruct to generate {instruction, API} pairs. To fine-tune LLaMA, we convert this to a user-agent chat-style conversation, where each datapoint is a conversation with one round each for the user and the agent. We then perform standard instruction finetuning on the base LLaMA-7B model. For our experiments, we train Gorilla with and without the retriever. We would like to highlight that though we used the LLaMA model, our fine-tuning is robust to the underlying pre-trained model (see Appendinx A.3.5).In retriever-aware training, the instruction-tuned dataset also appends to the user prompt, the relevant retrieved documentation with "Use this API documentation for reference: <retrieved_API_doc_JSON>". This is critical, because the retrieved documentation is not necessarily accurate -retrievers have imperfect re-call. By augmenting the prompt with potentially incorrect documentation, but the accurate ground-truth in the LLM response, we are in-effect teaching the LLM to 'judge' the retriever at inference time. During inference, if the LLM reasons that the retriever presented a relevant API document, it can use the API documentation to respond to the user's question, filling in additional details from the user's prompt. However, if after looking at the prompt, the LLM reasons that the retrieved API document is not relevant to the user's prompt, RAT trains the model to not get distracted by irrelevant context. The LLM then relies on the domain-specific knowledge baked-in during RAT training, to provide the user with the relevant API. Through RAT, we aim to teach the LLM to parse the second half of the question (API documentation) to answer the first half (user's query). We demonstrate that this (1) makes the LLM adapt to test-time changes in API documentation, (2) improves performance from in-context learning, and (3) reduces hallucination error.Surprisingly, we find that augmenting a LLM with retrieval, does not always lead to improved performance, and can at-times hurt performance. We share more insights along with details in Sec 4.Gorilla Inference During inference, the user provides the prompt in natural language (Fig. 3 ). This can be for a simple task (e.g., "I would like to identify the objects in an image"), or they can specify a vague goal, (e.g., "I am going to the zoo, and would like to track animals"). Gorilla, similar to training, can be used for inference in two modes: zero-shot and with retrieval. In the zero-shot setting, this prompt (with no additional prompt tuning) is fed to the Gorilla LLM model, which then returns the API call needed to accomplish the task or goal. In retrieval mode, the retriever (either of BM25 or GPT-Index) first retrieves the most up-to-date API documentation stored in the API Database. Before being sent to Gorilla, the API documentation is concatenated to the user prompt along with the message "Use this API documentation for reference." The output of Gorilla is an API to be invoked. Besides the concatenation as described, we do no further prompt tuning in our system. While we also implemented a system to execute these APIs, to help the user accomplish the goal, that is not a focus of this paper.Inductive program synthesis, where a program is synthesized to satisfy test cases, has found success in several avenues [3, 23] . However, test cases fall short when evaluating API calls, as it is often hard to verify the semantic correctness of the code. For example, consider the task of classifying an image. There are over 40 different models that can be used for the task. Even if we were to narrow down to a single family of densenet, there are four different configurations possible. Hence, there exist multiple correct answers and it is hard to tell if the API being used is functionally equivalent to the reference API by unit tests. Thus, to evaluate the performance of our model, we compare their functional equivalence using the dataset we collected. To trace which API in the dataset is the LLM calling, we adopt the AST tree-matching strategy. Since we only consider one API call in this paper, checking if the AST of the candidate API call is a sub-tree of the reference API call reveals which API is being used in the dataset. On the left is an API call returned by Gorilla. We first build the associated API tree. We then compare this to our dataset, to see if the API dataset has a subtree match. In the above example, the matching subtree is highlighted in green, signifying that the API call is indeed correct. Pretrained=True is an optional argument.Identifying and even defining hallucinations can be challenging. We use the AST matching process to directly identify the hallucinations. We define a hallucination as an API call that is not a sub-tree of any API in the database -invoking an entirely imagined tool. This form of hallucination is distinct from invoking an API incorrectly which we instead define as an error. So, in our evaluations, error, hallucination, and accuracy add up to one.AST Sub-Tree Matching We perform AST sub-tree matching to identify which API in our dataset is the LLM calling. Since each API call can have many arguments, we need to match on each of these arguments. Further, since, Python allows for default arguments, for each API, we define which arguments to match in our database. For example, we check repo_or_dir and model arguments in our function call. In this way, we can easily check if the argument matches the reference API or not. Fig. 4 illustrates an example subtree check for a torch API call. We first build the tree, and verify that it matches a subtree in our dataset along nodes torch.hub.load, pytorch/vision, and densenet121. We do not check for match along leaf node pretrained=True since that is an optional argument. 
evaluation: 
 When evaluating Gorilla, finetuned on APIBench (train set), we aim to answer the following questions: How does Gorilla compare to other LLMs on API Bench (test set)? ( 4.1). How well does Gorilla adapt to test-time changes in API documentation? ( 4.2). How well can Gorilla handle questions with constraints? (4.3) We demonstrate that Gorilla outperforms both open-source and close-source models for in-domain function calling. Further, trained with our novel retriever-aware training (RAT) technique, the Gorilla model generalizes to APIs that are outside of its training data (out-of-domain). In addition, we assess Gorilla's ability to reason about API calls under constraints. Lastly, we examined how integrating different retrieval methods during training influences the model's final performance.Baselines We primarily compare Gorilla with state-of-the-art language models in a zero-shot setting and with 3-shot in-context learning. The models under consideration include: GPT-4 by OpenAI with the gpt-4-0314 checkpoint; GPT-3.5-turbo with the gpt-3.5-turbo-0301 checkpoint, both of which are RLHF-tuned models specifically designed for conversation; Claude with the claude-v1 checkpoint, a language model by Anthropic, renowned for its lengthy context capabilities; and LLaMA-7B, a state-of-the-art open-source large language model by Meta.Retrievers The term zero-shot (abbreviated as 0-shot in tables) refers to scenarios where no retriever is used. The sole input to the model is the user's natural language prompt. For BM25, we consider GPT-3. 5 each API as a separate document. During retrieval, we use the user's query to fetch the most relevant (top-1) API. This API is concatenated with the user's prompt to query the LLMs. Similarly, GPT-Index refers to the state-of-the-art embedding model, text-embedding-ada-002-v2 from OpenAI, where each embedding is 1,536 dimensional. Like BM25, each API call is indexed as an individual document, and the most relevant document, given a user query, is retrieved and appended to the user prompt. Lastly, we include an Oracle retriever, which serves two purposes: first, to identify the potential for performance improvement through more efficient retrievers, and second, to assist users who know which API to use but may need to help invoking it. In all cases, when a retriever is used, it is appended to the user's prompt as follows: <user_prompt> Use this API documentation for reference: <retrieved_API_doc_JSON>. The dataset for these evaluations is detailed in Section 3. We emphasize that we have maintained a holdout test set on which we report our findings. The holdout test set was created by dividing the self-instruct dataset's instruction, API pairs into training and testing sets.We test each model for different retriever settings defined above (Table 1 ). We report the overall accuracy, the error by hallucination and the error by selecting wrong API call. Note that for TorchHub and TensorHub, we evaluate all the models using AST tree accuracy score. However, for HuggingFace, since the dataset cannot be exhaustive given the sheer number of models hosted, for all the models except Gorilla, we only check if they can provide the correct domain names. So this problem reduces to picking one of multiple choices. Across 0-shot and few-shot prompting strategies, Gorilla outperforms close-sourced and open-sourced models (Table 5 ).In Table 1 we show that lightly fine-tuned Gorilla is able to match, and often surpass performance in the zero-shot setting compared to closed-source, and open-source models -20.43% better than GPT-4 and 10.75% better than GPT-3.5 (ChatGPT). When compared to other open-source models LLAMA, the improvement is as big as 83%. This suggests quantitatively, that as a technique to augment information and enforce adherence to syntax, fine-tuning is better than naive retrieval, at-least within the scope of invoking APIs.Finetuning with Retrieval We now discuss how incorporating retrieval (RAT) during LLM finetuning enhances model performance. In this experiment, the base LLAMA model is finetuned with a prompt (instruction-generated), a reference API document (from a golden-truth oracle), and an example output generated by an LLM (GPT-4 in this case). As shown in Table 2 , incorporating a ground-truth retriever in the finetuning pipeline yields notably improved results -12.37% higher accuracy than training without retrieval in Torch Hub and 23.46% better in HuggingFace. However, at evaluation time, current retrievers show a significant performance gap compared to the ground-truth retriever: using GPT-Index at evaluation results in 29.20% accuracy degradation and using BM25 results in a 52.27% accuracy degradation. Despite this, considering the trends across models and retrievers, our findings indicate that finetuning an LLM with effective retrieval integration is preferable to zero-shot finetuning.Hallucination with LLM One phenomenon we observe is that zero-shot prompting with LLMs (GPT-4/GPT-3.5) to call APIs results in dire hallucination errors. These errors, while diverse, commonly manifest in erroneous behavior such as the model invoking the AutoModel.from_pretrained(dir_name) command with arbitrary GitHub repository names. Surprisingly, we also found that in TorchHub, HuggingFace and TensorFlow Hub, GPT-3.5 has less hallucination errors than GPT-4. This finding is also consistent for the settings when various retrieving methods are provided: 0-shot, BM25, GPT-Index and the oracle. This might suggest that RLHF plays a central role in turning the model to be truthful. Additional discussion in Appendix A.3.We manually execute Gorilla's API generations to evaluate how well AST works as an evaluation metric. Executing every code generated is impractical within academic setting-for example, executing the HuggingFace model needs the required library dependencies (e.g., transformers, sentencepiece, accelerate), correct coupling of software kernels (e.g., torch vision, torch, cuda, cudnn versions), and required hardware support (e.g., A100 40G gpus).Hence, to make it tractable, we sampled 100 random Gorilla generations from our evalualtion set. The accuracy from our AST subtree matching is 78%, consistent with human evaluation of 78% accuracy in calling the right API. All the generations that AST flagged as incorrect, were the same ones that were manually flagged as incorrect. Additionally, Gorilla also generates supporting code to call the API which includes installing dependencies e.g., pip install transformers[sentencepiece]), setting environment variables, etc. When we manually attempt to execute the code, 72% of all code generated executed successfully. It's worth noting that the 6% discrepancy are not semantic errors, but errors that arose due to factors external to the API, and in the supporting code. We have included the full example to illustrate this further in A.3.3. Considering the significant time and effort required for manual validation of each generation, the strong correlation between human evaluation and the AST evaluation further reinforces our belief in using the proposed AST as a robust offline metric.USER: I want to automatically remove the background from an input image <Retrieval: ..fcn_resnet101..> GORILLA: api_call: "torch.hub.load('pytorch/vision', 'fcn_resnet101', pretrained=True)" USER: I want to automatically remove the background from an input image. <Retrieval: ..NVIDIA/DeepLearningExamples:torchhub..> GORILLA:api_call: "torch.hub.load('NVIDIA / DeepLearningExamples:torchhub ', 'fcn_resnet50', pretrained=True)" USER: I want to automatically remove the background from an input image. <Retrieval: ..fcn_resnet50..> GORILLA:api_call: "torch.hub.load('pytorch/vision', 'fcn_resnet50', pretrained=True)"Update the model Update the model repository The rapidly evolving nature of API documentation presents a significant challenge for the application of LLMs in this field. These documents are often updated at a frequency that outpaces the retraining or fine-tuning schedule of LLMs, making these models particularly brittle to changes in the information they are designed to process. This mismatch in update frequency can lead to a decline in the utility and reliability of LLMs over time.With the introduction of Gorilla's retriever-aware training, the RAT trained LLM readily adapts to changes in API documentation. This novel approach allows the model to remain relevant, even as the API documentation it relies on undergoes modifications. This is a pivotal advancement in the field, as it ensures that the LLM maintains its efficacy and accuracy over time, providing reliable outputs irrespective of changes in the underlying documentation.For instance, consider the scenario illustrated in Fig. 6 , where the training of Gorilla has allowed it to react effectively to changes in APIs. This includes alterations such as upgrading the FCN's ResNet-50 backbone to ResNet-101, as demonstrated in the second column of the figure. Since the model has encountered ResNet-101 as a backbone with other architectures, it interprets an FCN with a ResNet-101 backbone (unseen during training) as a relevant document at test time. Conversely, if the retriever suggests an FCN with a ResNet-60 backbone, the model-unfamiliar with ResNet-60 from RAT-assigns low confidence to this document and defaults back to FCN with ResNet-50. The third column in Fig. 6 further illustrates Gorilla's flexibility in adapting to shifts in model registries, such as from pytorch/vision to NVIDIA/DeepLearningExamples:torchhub, highlighting its ability to accommodate changes in preferred API sources as they evolve over time. In summary, Gorilla's ability to adapt to test-time changes in API documentation offers numerous benefits. It maintains its accuracy and relevance over time, adapts to the rapid pace of updates in API documentation, and adjusts to modifications in underlying models and systems. This makes it a robust and reliable tool for API calls, significantly enhancing its practical utility.We now focus on the language model's capability of understanding constraints. For any given task, which API call to invoke is typically a tradeoff between a multitude of factors. In the case of RESTFul APIs, it could be the cost of each invocation ($) or the latency of response (ms), among many others.Similarly, within the scope of ML APIs, it is desirable for Gorilla to respect constraints such as accuracy, number of learnable parameters in the model, the size on disk, peak memory consumption, FLOPS, etc. In this section, we present a study evaluating the ability of different models in zero-shot and in the presence of retrievers to respect a given accuracy constraint. : if a user requests an image classification model that achieves at least 80% top-1 accuracy on the ImageNet dataset, then among the classification models hosted by Torch Hub, ResNeXt-101 32x16d, with a top-1 accuracy of 84.2%, would be the appropriate model to call, rather than MobileNetV2, which has a top-1 accuracy of 71.88%.For Table 4 , we filtered a subset of the Torch Hub component of APIBench, retaining those models that had an accuracy metric defined for at least one-dataset the model was evaluated on, in its model card. We were left with 65.26% of TorchHub dataset from Table 1 . We notice that with constraints, understandably, the accuracy drops across all models, with and without a retriever. Even in this challenging scenario, Gorilla is able to match the performance of the best-performing model GPT-3.5 when using retrievals (BM25, GPT-Index), and has the highest accuracy in the zero-shot setting. This highlights Gorilla's ability to navigate APIs while considering the trade-offs between constraints.To assess whether finetuning is truly necessary for APIs or if prompting alone is sufficient, we compare Gorilla in a zero-shot setting with three-shot in-context prompting for GPT-3.5 and GPT-4 models. In Table 5 , "3-incont" denotes evaluation using three in-context examples, while "HF," "TH," and "TF" represent the HuggingFace, TorchHub, and TensorFlow Hub subsets of APIBench, respectively. Higher accuracy (Acc) and lower hallucination (Hall) rates are preferred. From Table 5 , three-shot in-context learning improves the GPT models' ability to generate syntactically correct function calls, even matching accuracy on one subset (TorchHub). However, Gorilla 0-shot still outperforms the 3-shot GPT models on average. 
conclusion: 
 LLMs are swiftly gaining popularity across diverse domains. APIs, serving as a universal language, are essential for enabling LLMs to communicate and operate effectively across diverse systems. In this paper, we introduced Gorilla, a state-of-the-art model for API invocation. Our Retriever Aware Training (RAT) approach empowers Gorilla with two essential capabilities: adapting dynamically to API changes at test time and reasoning through user-defined constraints when selecting suitable APIs. We also present APIBench, a comprehensive benchmark for assessing LLMs' function-calling abilities, and propose AST-based hallucination metrics for robust evaluation. Looking forward, we believe this work represents a first step towards transitioning LLMs from knowledge-bound models into flexible interfaces that interact with the digital world. 
 
 Appendix: Our dataset is multi-faceted, comprising three distinct domains: Torch Hub, Tensor Hub, and HuggingFace. Each entry within this dataset is rich in detail, carrying critical pieces of information that further illuminate the nature of the data. Delving deeper into the specifics of each domain, Torch Hub provides 95 APIs. The second domain, Tensor Hub, is more expansive with a total of 696 APIs. Finally, the most extensive of them all, HuggingFace, comprises 925 APIs.To enhance the value and utility of our dataset, we've undertaken an additional With each API, we have generated a set of 10 unique instructions. These instructions, carefully crafted and meticulously tailored, serve as a guide for both training and evaluation. This initiative ensures that every API is not just represented in our dataset, but is also comprehensively understood and effectively utilizable.In essence, our dataset is more than just a collection of APIs across three domains. It is a comprehensive resource, carefully structured and enriched with added layers of guidance and evaluation parameters.The unique domain names encompassed within our dataset are illustrated in Fig. 7 . The dataset consists of three sources with a diverse range of domains: Torch Hub houses 6 domains, Tensor Hub accommodates a much broader selection with 57 domains, while HuggingFace incorporates 37 domains. To exemplify the structure and nature of our dataset, we invite you to refer to the domain names represented in Fig. 8 .In this task, we test the model's capability to generate a single line of code, either in a zero-shot fashion or by leveraging an API reference. Primarily designed for evaluation purposes, this task effectively gauges the model's proficiency in identifying and utilizing the appropriate API call.This facet relates to the provision of the programming language. In this context, the API provider plays a vital role as it serves as a foundation upon which APIs are built and executed.Explanation Element This component offers valuable insights into the rationale behind the usage of a particular API, detailing how it aligns with the prescribed requirements. Furthermore, when certain constraints are imposed, this segment also incorporates those limitations. Thus, the explanation element serves a dual purpose, offering a deep understanding of API selection, as well as the constraints that might influence such a selection. This balanced approach ensures a comprehensive understanding of the API usage within the given context.Code Example code for accomplishing the task. We de-prioritize this as we haven't tested the execution result of the code. We leave this for future works, but make this data available in-case others want to build on it.We provide all the training details for Gorilla in this section. This includes how we divide up the training, evaluation dataset, training hyperparameters for Gorilla.Data For HuggingFace, we devise the entire dataset into 90% training and 10% evaluation. For Torch Hub and Tensor Hub, we devise the data in to 80% training and 20% testing.Training We train Gorillafor 5 epochs with the 2e-5 learning rate with cosine decay. The details are provide in Table 6 . We finetune it on 8xA100 with 40G memory each. Figure 8 : Example of the Dataset: Two examples of the dataset, the above one is zero-shot (without information retrievers) and the bottom one is with information retriever. generate_video = pipeline("text-to-video", model="your_model_name") vqa = pipeline("visual-question-answering", model="microsoft/clip-vqa-base", tokenizer="microsoft/clip-vqa-base") depth_estimator = pipeline("depth-estimation", model="intel-isl/MiDaS", tokenizer="intelisl/MiDaS") For ease of evaluation, we manually cleaned up the dataset to ensure each API domain only contains the valid call of form: API_name(API_arg 1 , API_arg 2 , ..., API_arg k ) Our framework allows the user to define any combination of the arguments to check. For Torch Hub, we check for the API name torch.hub.load with arguments repo_or_dir and model. For Tensor Hub, we check API name hub.KerasLayer and hub.load with argument handle. For HuggingFace, since there are many API function names, we don't list all of them here. One specific note is that we require the pretrained_model_name_or_path argument for all the calls except for pipeline. For pipeline, we don't require the pretrained_model_name_or_path argument since it automatically select a model for you once task is specified.We found especially in HuggingFace, the GPT-4 model incurs serious hallucination problems. It would sometimes put a GitHub name that is not associated with the HuggingFace repository in to the domain of pretrained_model_name_or_path. Fig. 9 demonstrates some examples and we also observe that GPT-4 sometimes assumes the user have a local path to the model like your_model_name. This is greatly reduced by Gorilla as we see the hallucination error comparison in Table 1 .We evaluated the generated results on 100 LLM generations (randomly chosen from our eval set). The accuracy using AST subtree matching is 78%, consistent with human evaluation with 78% accuracy in calling the right API. All the generations that AST flagged as incorrect, were the same ones that were manually also flagged as incorrect. Additionally, Gorilla generates supporting code to call the API which includes installing dependencies (e.g., pip install transformers[sentencepiece]), environment variables, etc. When we manually attempted to execute end-to-end code, 72% of all codes generated were executed successfully. It's worth noting that the 6% discrepancy were NOT semantic errors, but errors that arose due to factors external to the API in the supporting code -we have included an example to illustrate this further. Considering the significant time and effort required for manual validation of each generation, our evaluation highlights the efficiency of using AST as a robust offline metric.Here is a representative example, where we are able to load the correct model API. However, in the supporting code, after we have the output from the API, the zip() function tries to combine sentiments and scores together. However, since scores is a float, it's not iterable. zip() expects both its arguments to be iterable, resulting in an 'float' object is not iterable error. We present APIBench, a comprehensive benchmark of ∼ 1600 machine learning APIs, along with new AST-based evaluation metrics that precisely measure both functional correctness and API hallucination. 


Decision: This work proposes Gorilla, a model that is able to achieve state of the art in API call generation. The authors will release the code and data (APIBench, which has AST matching metrics) to the community. The reviewers generally appreciate the work and impact that the model and dataset will have and comment on the clear presentation of the paper, as well as its actual performance compared to the baselines. They do raise some concerns, such as lack of training details, clarifications on metrics and results analysis, novelty, and evaluation (results, statistical significance). However, the authors responses are clear and seem convincing to the reviewers.

Reviews: review: paper_summary: paper_summary: This paper addresses a pipeline to call adequate APIs among massive pools to accomplish users’ instructions. For that, the authors construct and release the APIBench dataset that contains more than 1645 APIs, and propose the Gorilla model, which is a retrieval-aware finetuned Llama-7B model for API calls.
strength_weakness: strength_weakness: - Lack of details about the training data for Gorilla, such as data stats and construction methods. It’s unclear how different the Gorilla training data and evaluation sets of APIBench are, which could lead to doubts about data contamination.
- Experiment results analysis
    - How robust are models to paraphrased user instructions corresponding to the same target model API?
    - Providing performance for each domain could make it possible to analyze which domains are easy and hard.
- Clarifications are needed:
    - In the experimental result section, the authors report two metrics along with overall accuracy: **the error by hallucination and by selecting the wrong API call.** The metric equations or explanations for calculating the values need to be clarified.
    - In section AST as a Hallucination Metric (line 251), how was experimented to attain the human evaluation of 78% accuracy?
- It is required to mention the license of crawled APIs for usage.
- A limitation section is absent, though the authors describe them in the checklist. Broader impact and limitation could include implicit risks such as usage of unreliable APIs, license infringement, or unexpected results by incorrect function calls.
questions: questions: - In section 4.1, “Note that for TorchHub and TensorHub, we evaluate all the models using AST tree accuracy score. *However, for HuggingFace, since the dataset is not exhaustive, for all the models except Gorilla, we only check if they can provide the correct domain names.*” is vague. The numbers of data for HF, TFH, and TH are 925, 801, and 95, respectively. Please clarify this.
- Table 1 shows that TorchHub is hard to even with oracle API documents compared to HuggingFace and TensorFlowHub. Can you draw the reason?
- (Typos) line 225. `his suggests`
- (suggestions) Please elaborate more on Table 2 — e.g., the caption and highlights. Moreover, it’s hard to connect between the written explanation and the table, due to different numbers.
- (suggestions) Absence of mentioning the full name of AST (Abstract Syntax Tree)
limitations: limitations: - Limitation section is absent, though the authors describe them in the check list. Broader impact and limitation could include implicit risks such as usage of unreliable APIs, license infringement, or unexpected results by incorrect function calls.
paper_summary: paper_summary: This paper introduces Gorilla, a fine-tuned LLaMA model designed to improve large language models' ability to use APIs accurately. The authors created APIBench, a comprehensive dataset of ML APIs from HuggingFace, TorchHub, and TensorFlow Hub, and used self-instruct to generate instruction-API pairs for training. They fine-tuned LLaMA-7B with retrieval-aware training, incorporating tech like AST sub-tree matching for evaluating API call accuracy, retriever-aware training to adapt to API changes, and handling of constrained API calls. Results show that Gorilla outperforms existing LLMs (including GPT-4) on API call accuracy across multiple datasets, demonstrates ability to adapt to test-time changes in API documentation, and handles constrained API calls effectively. The paper presents a novel approach to improving LLMs' API usage capabilities, with promising results that outperform existing state-of-the-art models in this specific domain.
strength_weakness: strength_weakness: A significant weakness of the Gorilla approach is that it relies heavily on knowledge augmentation, which is a common technique already used in various domains to improve language model performance on specific tasks. The use of retrieval-augmented generation to enhance API calling capabilities doesn't represent a novel improvement or implementation compared to similar approaches in other domains.

Essentially, Gorilla applies existing knowledge augmentation techniques to the specific task of API invocation, rather than introducing a fundamentally new method for improving language model capabilities. While the results show improvements in API calling accuracy, the core approach of combining retrieval with language model fine-tuning is not innovative in itself. This limits the broader impact and generalizability of the work beyond the specific domain of API invocation.

The lack of significant methodological innovation suggests that similar performance improvements could potentially be achieved by applying existing retrieval-augmented generation techniques to the API calling task, without necessarily requiring the specific Gorilla architecture. This raises questions about the uniqueness and broader applicability of the Gorilla approach beyond the narrow domain explored in the paper.* While Gorilla performs well on the specific API datasets it was trained on, it's unclear how well it would generalize to entirely new APIs or domains not covered in the training data.

Other Notes: 
* Although Gorilla shows some ability to handle constraints, its performance in this area is not significantly better than other models like GPT-3.5, suggesting room for improvement.
* The paper doesn't compare Gorilla's performance with specialized API documentation tools or code completion systems, which might be more tailored for this specific task.
* The paper doesn't provide an in-depth analysis of the cases where Gorilla fails, which could provide insights into its limitations and areas for improvement.
questions: questions: * How does Gorilla's performance compare to other specialized code generation or API-focused models, not just general-purpose LLMs?
* What is the performance impact of varying the size of the training dataset or the base model size (e.g. using LLaMA-13B instead of 7B)?
* How does Gorilla perform on more complex multi-step API workflows rather than just single API calls?
limitations: limitations: NA - Appropriately discussed limitations.
paper_summary: paper_summary: The authors present a new fine-tuned language model that is trained to map from user instructions to code snippets that invoke the appropriate APIs. The authors also introduce a new dataset which includes the information for roughly 1600 APIs from a variety of online sources, which is used to train the model. They conclude that their approach outperforms existing language models on the task of API call generation, as measured by both error rate and a novel “hallucination” metric.
strength_weakness: strength_weakness: First, I wonder about baselines. The authors have done a good job of comparing their method against a variety of open-source and closed-source LLMs, but these systems are all “generalist” compared to the fine-tuned Gorilla model. It would have been useful to see a comparison to a fine-tuned version of another model (even one with less fine-tuning than is possible on the small Llama-7b) just to have a sense of how much improvement can be leveraged from the newly introduced dataset. I also wonder if it would have been possible to compare to specialized tool-based LLM models like Toolformer. If such a comparison is not appropriate or possible, I feel the authors should mention why.

My second question concerns statistical significance. The authors indicate the high cost of LLM experiments as the reason for omitting such analysis. I sympathize with this explanation, but feel that I would be remiss not to stress the importance of statistical testing as part of justifying claims that Gorilla’s “performance surpasses prompting the state-of-the-art LLMs in three massive datasets.” Without a measure of a variance, we cannot meaningfully conclude that apparent improvements achieved by Gorilla (which are often on the order of 1-3% in overall score) are the result of anything other than noise. To be sure, a 7B parameter model that achieves even comparable results to a massive LLM is a notable result (though see my earlier point about fine-tuned baselines), but I would temper claims about improvement in the absence of statistical justification.

Lastly, I feel that the paper would benefit greatly from a more robust error analysis. In the best-performing realistic setting (i.e. without access to an oracle), Gorilla achieves an overall accuracy of roughly 65%. This means that more than one third of the time it is returning an incorrect result (including errors, the authors point out, that could be silent to the end user). Obviously this is far from being ready for direct deployment without external validation. Given the prevalence of errors, it would be useful to better understand what kinds of errors are common: are they more likely to occur with particular kinds of APIs? Does the wording and / or complexity of the input prompt have a large effect? This is the kind of analysis that, in combination with the novel dataset, could help spur future work and improvement. In addition, given the potential for damaging errors, a more robust broader impact statement would be beneficial.
questions: questions: How do models which are fine-tuned compare to Gorilla? (If such a comparison is not appropriate, why?)

What is the statistical significance of the results?

What are the most common kinds of errors?

Do the kinds of errors differ between the different categories of APIs?

As a final minor question: do the authors have an explanation for why GPT-3.5 appears to outperform GPT-4 across a few different problem settings? This seems like a surprising and somewhat notable result!
limitations: limitations: I feel that the authors should have a more robust discussion of the broader impact of their work.
paper_summary: paper_summary: This paper proposes Gorilla, a fine-tuned LLaMA model, outperforms GPT-4 in crafting API calls and adapts well to document changes with a document retriever, reducing hallucination issues. It also provides APIBench, a new dataset for evaluation, includes APIs from HuggingFace, TorchHub, and TensorHub. Gorilla's integration with a retrieval system enhances tool usage accuracy and documentation updates, promising more reliable LLM outputs.
strength_weakness: strength_weakness: 1. Although the paper overall does not present significant issues, I would appreciate seeing a performance comparison with more specific methods, such as other works that also utilize LLM for tool calling to build software pipelines, which may further demonstrate the contribution of techniques presented by Gorilla.
2. The paper needs to better illustrate the contributions of this work and more clearly outline its potential impact on the community.
3. Some writing detales need to be improved in the paper, such as "the" should be omitted in "The model trained with the our method" in Line 31, and "AST" should be replaced by "Abstract Syntax Tree" for the first time it appears in Line 36.
questions: questions: 1. In Figure 2, why GPT-3.5 is much better than GPT-4 in all settings?
2. As most of the instructions are generated by GPT-4 referring to your hand-generated instruction seeds. How to ensure your seeds meets the real-world scenario and reach the real difficulties?
3. As many APIs have overlapping functions, how to decide which API is better than the other similar one? How to maintain stable API calling to persist consistent results for the same request?
4. Which Llama version is used as your foundation model? Llama2 or Llama3?
5. In Table 1, why Gorilla always produces a large error in selecting wrong APIs? Do the results support your view?
limitations: limitations: The paper should provide a more comprehensive limitation part to point out potential overlooked issues, such as influence of synthetic training data and insufficient comparative experiments.

Comments: title: Reviewer please respond
Response to the author
How was the experiment conducted to attain the human evaluation of 78% accuracy?
comment: Dear reviewer,

Thank you for your efforts in reviewing this paper. Now that the authors have provided their response, do you have any further comments?

Thank you,
AC
I have updated the score, based on the rebuttal presented by the author.
Thank you for the response. I appreciate the clarification about why Toolformer would not be an appropriate direct comparison. A few questions and concerns remain, however.

First, I'm not sure I was totally clear about question (2) -- I am asking for some kind of a t-test to compare the performance of different models. Are the improvements from Gorilla statistically significant (i.e. P < 0.05)? This is not a question of out-of-domain generalization, but of the robustness of the apparent improvement.

For question (3) -- while I understand that constraints on space make including additional examples difficult, I would recommend at least a supplemental table in which the counts of different error types (e.g. "model name hallucination" or "formatting hallucination" as you present) are presented.
Thank you for your reply. Your explanation has mostly addressed my concerns. 

I hope you can incorporate some of the above content into the revised paper, such as the contributions and impact to the community, comparisons with other methods based on LLMs for tool calling, and some grammar issues. However, since there are still some experimental phenomena whose causes have not been fully explained, I suggest the author conduct a more in-depth analysis and investigation. Therefore, I have decided to keep my scores.
To keep the authors updated: 

Thank you for providing an in-depth response to my queries. I appreciate your clear rebuttals to my concerns, and tangible commitments to revise the work based on the feedback. 

I will read the rebuttal again, and respond with follow-up queries (if applicable). I intend to change my scores based on my improved understanding.
**7. human evaluation of 78% accuracy?**

This is from human evaluation by directly executing the code. We manually evaluated the generated results on 100 LLM generations (randomly chosen from our eval set). The accuracy using AST subtree matching is 78%. We observe that this is consistent with human evaluation that revealed a 78% accuracy in calling the right API. All the generations that AST flagged as incorrect, were the same ones that were manually also flagged as incorrect. Additionally, Gorilla also generates supporting code to call the API which includes installing dependencies (e.g., `pip install transformers[sentencepiece]`), environment variables, etc. When we manually attempted to execute these codes, 72% of all codes generated were executed successfully. It's worth noting that the 6% discrepancy are NOT semantic errors, but errors that arose due to factors external to the API in the supporting code - we have included an example to illustrate this further. Considering the significant time and effort required for manual validation of each generation, our data further reinforces our belief in the efficiency of using AST as a robust offline metric.
Here is a representative example, where we are able to load the correct model API. However, in the supporting code, after we have the output from the API, the `zip()` function tries to combine sentiments and scores together. However, since scores is a `float`, it's not iterable. `zip()` expects both its arguments to be iterable, resulting in an `'float' object is not iterable` error.

```
from transformers import pipeline
def load_model():
   classifier = pipeline('sentiment-analysis', model='nlptown/bert-base-multilingual-uncased-sentiment')
   return classifier

def process_data(comments, classifier):
   response = classifier(comments)
   sentiments = response[0]['label'].split()
   scores = response[0]['score']
   result = [{'sentiment': sentiment, 'score': score} for sentiment, score in zip(sentiments, scores)]
   return result

comments = "These comments are about our news website."
# Load the model
classifier = load_model()
# Process the data
response = process_data(comments, classifier)
print(response)
```