ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings
Authors: Shibo Hao
Tianyang Liu
Zhen Wang
Zhiting Hu
Venue: NeurIPS 2023 oral (2023)
Keywords: large language model
tool learning
TL;DR: We propose to use tool embeddings to augment large language models with tools

Abstract: Integrating large language models (LLMs) with various tools has led to increased attention in the field. Existing approaches either involve fine-tuning the LLM, which is both computationally costly and limited to a fixed set of tools, or prompting LLMs by in-context tool demonstrations. Although the latter method offers adaptability to new tools, it struggles with the inherent context length constraint of LLMs when many new tools are presented, and mastering a new set of tools with few-shot examples remains challenging, resulting in suboptimal performance. To address these limitations, we propose a novel solution, named **ToolkenGPT**, wherein LLMs effectively learn to master tools as predicting tokens through **tool embeddings** for solving complex tasks. In this framework, each tool is transformed into vector embeddings and plugged into the language model head. Once the function is triggered during text generation, the LLM enters a special function mode to execute the tool calls. Our experiments show that function embeddings effectively help LLMs understand tool use and improve on several tasks, including numerical reasoning, knowledge-based question answering and embodied decision-making.

ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings
Shibo Hao', Tianyang Liu', Zhen Wangh. Zhiting Hu' UC San Diego, "Mohamed bin Zayed University of Artificial Intelligence {s5hao, ti1040, zhw085, zhh019)@ucsd.edd
Abstract
Augmenting large language models (LLMs) with external tools has emerged as a promising approach to solving complex problems. However, traditional methods, which fine-tune LLMs with tool demonstration data, can be both costly and re- stricted a predefined set of tools. Recent in in-context learning paradigm alleviates these issues, but the limited context length only allows for a few shots of demon- strations, leading to suboptimal understandings of the tools. Moreover, when there are numerous tools to choose from, in-context learning could completely fail to work, this paper, we propose an alternative approach, ToolkenGPT, which combines the benefits of both sides. Our approach represents each tool as a token ("toolken") and learns an embedding for it, enabling tool calls in the same way as generating regular word token. Once toolken is triggered, the LLM is prompted to complete arguments for the tool to execute. TPT offers the flexibility plug in arbitrary number of tools by expanding the set of toolkens the fly. In addition, it improves tool use by allowing extensive demonstration data for learning the toolken embeddings. In diverse domains, including numerical reasoning, knowledge-bee basee question answering, and embodied plan generation, our approach effectively augments L.MMs with tools and substantially outperforms various latest baselines. ToolkenGPT demonstrates the promising ability to use relevant tools from large tool set in complex scenarios
Introduction
have established themselves powerful tools for diverse real-world applications, ranging from writing assistance to automated customer support [2] 6.] As these models continue evolve, there is a growing interest in their potential to interact with the real world and enhance their functionality through integration with other tools, such as the
Large Language Models (LLMs) 5 9
62 47
The capability of these models master and control wide array of tools not only serves as an indicator of their intelligence, but also signals a promising of their fundamental weaknesses. These include updating the latest world
calculator, databases, etc [50 61
56 53).
path to overcome knowledge [46), reducing their hallucinations (55) and executing symbolic operations 17 etc. However, the rapid emergence of new tools, such as advanced software libraries, novel APIs, or domain utilities [39] 36 229] introduces additional richness and complexity to the task of tool learning for LLMs. This continuous evolution accentuates the importance of empowering LLMs with the ability to adapt and master massive new tools swiftly.
Recent advancements in LLMSs have witnessed (Table T]. The first paradigm involves fine-tuning LLMS to 50 For example, there are enormous efforts to integrate the retrieval tool into
primary lines of research approaches for tool
integration with L.MSs 45
68 53
leam specific tools
Cooee is available at httpss///ggtttt..co////////////o
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Table 1:
Comparison of different tool learning paradigms. Plug-& means the LLMs can be equipped and unequipped with a tool flexibly. Note that it doesn't 't indicate zero-shot tool learning.
Frozen LMs
Massive Tools
Ability to Use
Tool Learning Paradigms
Plug-&-Play
Extensive Data
Fine-tuning [e.g.. 56 50
x
In-context learning ge.g.. 69
53
/
X
/
ToolkenGPT (Ours)
/
and the recent Toolformer [56] fine-tuned GPT-J to learn five tools. While this method could yield promising results, it is computationally expensive and lacks the adaptability to new tools. The second approach relies on in-context learning (6 49 53 where L.LMs learn how to use the tool through in-context demonstrations provided in the prompt. This method allows LLMs to handle newly introduced tools and drives successful applications like LangChain 7 and ChatGPT Plugin P However, in-context learning comes with its own unique limitations. Specifically, it struggles with the inherent limitation of context length, making it impossible to demonstrate massive tools in the context. Also, mastering new tools simply via few-shot examples could be challenging. For example, even the latest models like GPT-4 face difficulties when handling unusual tools [6].
LLMs 18 35| 58
In this paper, we introduce ToolkenGPT, an alternative solution that enables LLMs to master massive tools without the need for any L.MM fine-tuning, while still allowing for quick adaptation to new tools. The key idea of ToolkenGPT is to represent each tool as a new token ("toolken") ") to augment the vocabulary. Specifically, each tool associated with an embedding inserted into the L.LM head like a regular word token embedding. During generation, once toolken is predicted, the LLM temporarily switches into a special mode (through prompting) to produce input arguments for the tool to execute, and inject the outputs back into the generation (see Figure DT This approach offers an efficient way for LLMs to master tools by only learning the lightweight toolken embeddings. Consequently, ToolkenGPT combines the strengths of both fine-tuning and in-context learning paradigms while avoiding their limitations (Table T): Compared to in-context learning that can only accommodate a small number of tools and few-shot demonstrations, ToolkenGPT allows massive tools (by simply inserting respective toolkens in the vocabulary) and can use extensive demonstration data for learning toolken embeddings; In contrast LLM fine-tuning, the tool embeddings not only requires minimal training cost, but also provide a convenient means for plugging in arbitrary new tools on the fly by expanding the toolken vocabulary.
We demonstrate the flexibility and effectiveness of our ToolkenGPT in leveraging numerous external tools for solving diverse set of problems, spanning from numerical reasoning to knowledge-based question answering and embodied plan generation. In complex numerical reasoning problems that involve number of mathematical tools (numerical operations such as finding greatest common divisor), we show that ToolkenGPT can effectively utilize these tools during the reasoning process, which outperforms some of latest popular approaches, such as Chain-of-Thouug 66). For knowledge-baad based question answering. ToolkenGPT accommodates a substantial number of relation APIs (over 200) from the knowledge base, thereby facilitating factual predictions. Further- more, we apply our framework to task planning for embodied agents, where an agent interacts with an environment using tools, namely the actions and objects. The findings illustrate that our method offers better grounding by learning toolken embeddings for 58 grounded actions and objects than previous in-context learning and specialized decoding methods.
(65) and ReAct
2 Related Works
Fine-tuning LLMs to use tools. Early research relied heavily on fine-tuning to augment LMs with tools. In these works, LMS were mostly fine-tuned to use one or a few tools in a specific domain. For example, the retriever has been crucial tool for augmenting LLMs with external knowledge sources [7]. The prominent works in this line include REALM 18. RAG (35,, and RETRO [3. More recently, WebGPT (46) fine-tuned GPT-3 human web search behaviors to learn how to the web browser. With the advancements in LLMs, there has also been growing interest in tuning

---

these models on collection of general tools, including the QA model, calculator, translator, etc. Example works include TALM [50] and Toolformer [56]. However, LLM fine-tuning is costly and these tuned L.MS struggle to generalize to emergent updated tools. ToolkenGPT learns lightweight toolken embeddings for new tools, without any gradient calculation for the parameters of L.L.s. This enables efficient adaption to new tools and maintains a minimal GPU memory overhead for training toolken embeddings, at a cost similar to LLM inference.
In-context learning for tools. LLMs exhibit a strong in-context learning ability [5]. which becomes a prevalent method to use tools by showing tool descriptions and demonstrations in context 45
53]
Building on this idea, reasoning chains can be incorporated to tackle more complex problems This paradigm has given rise to popular industry products such as ChatGPT plugins and Langchain 77]. along with many successful applications in important research topics. For instance, a code interpreter can effectively address the L.LM's shortcomings in symbolic operations 40). Furthermore, by calling "tools" that have an effect on the virtual physical world, the LLM capable of guiding embodied agents to accomplish various household tasks Recent attempts to utilize LLMs controller to coordinate multiple neural models also achieve promising progress in multimodal reasoning tasks [57 43). Nevertheless, all methods based on in-context learning suffer from inferior performance in complex scenarios, where the tools are unfamiliar or numerous. One concurrent work, Li et al. (36] propose to retrieve the tools based on the text embedding of their documents, which may mitigate that issue. However, ToolkenGPT is fundamentally different from their method, in that the toolken embeddings can encode the implicit semantics of tools from extensive demonstrations, which can never be inferred from the surface text (A concrete example is shown in Figure 3. Also, note that ToolkenGPT is compatible with the recent advanced prompting techniques, e.g., Chain-of-Thhougle (CoT) 65 to improve the LLMs performance further.
669 32 49
R, 17 21 44
67
25
4
26
59
27
Efficient tuning of large language models. Adapting pre-trained frozen LLMs efficiently to new tasks is an active research area, leading to a surge of interest in parameter-ffficient fine-tuning (PEFT) methods [31 38
42 41 The idea is to only fine-tune small subset of parameters of the LLM while freezing most of its parameters, which bears similarity to our toolken embedding method. Which part of parameters to tune is the key to PEFT methods; for instance, Adapters (22) insert trainable layers. BitFit (7T) tunes the bias parameters, prompt tuning [3 [4]] appends parameters to the input embedding layer, and LoRA (33 learns low-rank matrices within specific dense layers, etc. However, existing PEFT methods have not proven suitable for efficient tool learning, and utilizing these methods on tool demonstrations may not efficiently capture the desired tool knowledge as ToolkenGPT does. To the best of our knowledge, we are the first to explore efficient tuning methods for predicting tools as tokens for tool learning of massive tools.
3 ToolkenGPT for Mastering Massive Tools
In this section, we present ToolkenGPT, which enables LLMs to learn and use massive tools for complex problem-solving without the need for heavily fine-tuning the LLM. We begin by introducing the background and notations of language modeling for tool use. Typically, LLMS model the
" probability of a sequence of word tokens = (41, I2...., (n)) as P(s)= E P(t ttss), where each word token comes from the vocabulary of the LLM, i.e. f, E V and tee denotes the partial word token sequence before i-th step. In practice, the user often sets the prefix of a sequence (referred to as the prompt) to steer LLMs to generate desired contents, e.g., answering question. Taking step deeper, the distribution of the next token is predicted as (()) = softmax(W, -1.1). where h4-1 E Rd is the last hidden state of the current context and W E glVxd is the embedding matrix for word tokens (also known language model head).
Given a set of useful tools T {T1, T...... J. our goal is to enable LLMs call a subset of these tools for solving the complex problem. Our flexible formulation allows tools to play a role by either returning some results that can help LLMs with text generation (e.g. calculation) or affecting the real-world environment (e.g. robot action). To call tool during generation, the LLM first needs to select a tool and then input the arguments. In the running examples shown in Figure]] during the answer generation process ("reasoning mode"), math operator square is selected as the tool, and an operand 16 is generated as the argument in the "tool mode". Once the external tool receives the call, it executes the tool and returns the result 256, back to the "reasoning mode".
3

---

Embeddings
Token Dist.
Reasoning mode
Liverpool
Question: John rectangular garden, of which length 64
meters and the width is 48 meters. He wants to divide the garden into Identical square sections, each with the largeet poseible area. What's the area each section?
Word
Tokens
find
CCD (64, 48)
Anawer: The maximal side length of each section 16 meters. Therefore, area
KB winner_o
256
Toolkens
father of
3 fill in text
mode switch
Math Toolkens
Tool mode
GCO
square
deeonsttrttoos
Execution
Toolkens
Robot
grob
walk
16 meters,
side lengih
<square>(16)
square
(16)
256
Argument completion
the tool
Figure I: Overview of ToolkenGPT framework. Toolken embeddings are appended to the language model head like regular word tokens. In the "reasoning mode" for solving the problem, the LLM generates text as usual, except that any plugged-in toolkens are also considered for the next token generation. Once toolken is predicted, (1) the LLM switch to the "tool mode", which provides few demonstrations of the same tool to complete the arguments. Then, (2) the tool call is executed, and (3) the result is sent back to the text to continue the reasoning mode until the final answer is generated.
3.1 Framework Overview
The core idea of ToolkenGPT is explicitly formulating tools as tokens (called "toolkens"), Each toolken is parameterized as a toolken embedding vector, and we denote a set of toolken embeddings as a matrix, i.e. W, E IR Tixd Assuming we have trained toolken embeddings (to be described in Section 3.2) first give overview of our framework by introducing how it works in inference. As shown in Figure [ the LLM is in the reasoning mode by default, generating the next token. Our framework allows the LLM consider word tokens and toolkens uniformly. Specifically, the tool embedding matrix is concatenated with We. Therefore, the LLM predicts the next token with the probability as follows:
((4 = softmax((W... [+] h1-1)
where the next token can be either word token or a toolken, i.c. t, E VUT, and [:] is the concatenation operation. As we can see, our formulation of tools toolken embeddings naturally allows for the fast adaption of new tools by expanding the toolken embedding matrix easily.
To execute tool, the LLM switches into the "tool mode" once its toolken is predicted as the next token (as shown in the "mode switch" in Figure [ which aims to generate the arguments for the tool. Specifically, the LLM pauses the generation and appends the current generated context to another prompt. The prompt in tool mode consists of in-context demonstrations for the predicted tool, showing how to generate the tool arguments by quoting the tool calls a special syntax of [tool] (arguments). Then the LLM can follow the pattern in demonstrations to complete the arguments of the current tool call. Contrasting previous methods [69 (53] that fully rely on in-context learning for tool learning, our framework only leaves the easy work of completing arguments to in-context learning. Besides, there would be abundant context space for extensive demonstrations of a single specified tool. This design shares similarities with the classic divide-ant and- conquer methods [33] 32 [3]. Finally, the arguments are sent to the specified tool for execution, and the returned value is sent back to the text in the reasoning mode.
3.2 Learning Toolken Embeddings
Our framework keeps the original LLM parameters frozen and introduces minimal additional training overhead with the toolken embeddings, W,. This embedding matrix contains the only parameters optimize, but unlike other efficient LLM tuning methods, e.g., prompt tuning [34 64 or prefix tuning [88, it does not require the gradients flowing through the major body of L.LM parameters, leading to much stable and efficient training. Therefore, the tuning of toolken embeddings

---

maintains nearly the same GPU memory as LLM inference. Whenever a new tool is added, the toolken embedding can be conveniently expanded and then, subsequent training on tool demonstration data involving the new tool gradually refines its embedding. Moreover, unlike in-context learning methods that only digest a few examples as training signals, ToolkenGPT capable of tuning toolken embeddings from massive demonstrations.
Drawing parallels to how infants learn a new tool through demonstrations from adults [5], in this pa- per, we primarily focus on learning toolken embeddings with tool demonstrations, which can be either in-domain training data or synthetic data generated by LLMs (see Section 4.1 and Section 4.2). We first describe the format of training data and the training objective and we use the same example from Figure too showcase how it can be used for training. Specifically, "the area is 256 square ...". can be tokenized into a word token sequence = ("the", "area", "is", "2". "5", "6", "square". "feet"....). To indicate when to predict the toolkens, we need a parallel sequence mixed with word tokens and toolkens, i.e. s' ("the", "aarea, "is", "[square]", "[N/A]", " [N/A]" "square", "feet"....) The subsequence of """ "6") in s where the returned tool results should fill in, and we choose the corresponding first token in s' as the toolken for the tool call with the following tokens are filled with [N/A], indicating neglect in loss calculation. Thus, given a dataset composed of paired sequences D = [(sss s')}. the training objective of ToolkenGPT is:
N
C(W-)-
Σ Σ
log P(t)
(o,0)) i=1
where P(G[(<2) is defined in Eq. and Di/ is the indicator function signaling we ignore the [N/A] tokens during the training. Thus, our training process is largely consistent with the inference in the reasoning mode. That is, to call a tool, the only job for the LLM to predict a toolken the beginning, and then the returned value will be filled back to the text. Here, [N/A] is introduced to skip the generation of the returned value of tool call.
There are two primary ways to get the paired data. First, some datasets provide ground truth tool calls along with natural language sequences, e.g. the facts in KB supporting the answer a question (Secion 4.2), or the calculation trace for solving a math problem (Section 4.1). To use the data for supervised learning, we preprocess them get the paired data required for training as described in the above paragraph. Second, we explore synthesizing tool demonstrations with LLMs. sharing a similar idea to self-instruct [63]. An intuitive interpretation of this process is to distill the knowledge inside LLM to the new toolken embeddings. Specifically, we can prompt LLMs with the tool document and a few demonstrations with special syntax indicating tool calling, e-g.. The capital of U.S. is <capital> ("U.S. ")= "Washington D.C. Conditioned on that, the LLMS can generate some new use cases that utilizes the given tool and quote the tool call with the same syntax. We can then easily locate the tool calls and process the data into the paired data for training.
4 Experiments
In this section, we apply ToolkenGPT to three distinct applications characterized by meaningful tool-use scenarios: arithmetic tools for numerical reasoning, database APIs for knowledge-basee question answering, and robot actions for embodied plan generation. We focus how methods can accurately call the tools and how successfully they can solve the tasks. Our experiments show that ToP can efficiently master massive tools while leveraging them to solve complex problems with improved performance, consistently better than advanced prompting techniques.
4.1 Numerical Reasoning
LLMs often struggle with mathematical tasks since the models are inherently designed for proba- bilistic estimation rather than symbolic operations. In this section, we aim to assess the tool-learning capabilities of ToolkenGPT, compared with in-context tool learning (e.g., ReAct (69 ). We first demonstrate that ToolkenGPT consistently matches or outperforms the performance of in-context learning with the availability of four basic arithmetic functions (+, -. x, +). Moreover, to benchmark the han-hinn capability in more complex math problems, we include more available tools, i.c., an expanded (13) set of functions, and create set of synthetic data. The results show that ToolkenGPT significantly outperforms baselines by training only on the synthetic data. Note that our focus is not to reach a state-of-the-an accuracy; Rather, the experiment is designed to evaluate the tool learning ability in the setting where certain tools are available.
5

---

Table 2: Results on the GSM8K-XL and FuncQA datasets. The numbers in parentheses indicate how many available tools are available. For GSM8K-XL and FuncQAune dataset, accuracy is evaluated based on an exact match (float numbers rounded to two decimals). In FuncQAmuli. we allow a margin of error of 0.1% to account for potential errors at each step of multi-hop reasoning.
FuncQA (13)
Method
GSMSK-XIL XL (4)
One-Hop
Multi-Hops
O-shot ChatGPT
0.17
0.55
0.09
ReAct 0.18
CoT 65
0.20
0.03
0.32
0.57
0.06
ToolkenGPT (Ours)
0.33
0.73
0.15
Datasets. To evaluate the tool-learning proficiency in numerical reasoning comprehensively, curate two new test datasets: (1) GSM8K-XI, an enhanced version of the existing GSM8K [IO] dataset. GSM8K is a dataset of linguistically diverse grade school math word problems, involving performing a sequence of calculations using 4 basic arithmetic operations (+, x, +) to reach the final answer. In the original GSM8K dataset, the numbers for calculations are typically small, which might be less challenging for the recent powerful LLMs 1O0 6.. So in the test set, we magnify the numbers to increase the computational difficulty for LLMs, which results in the GSMSK-XL dataset, featuring 568 test cases with much larger numbers. (2) FuncQA is synthetic dataset we created increase the complexity of math problems involving more arithmetic tools, which serves as much more challenging benchmark to test the model's tool-learning capabilities. Specifically, This dataset requires at least 13 operators (e.g.. power, sqrt, 1cm to solve, and it is challenging for both humans and LLMs to solve without an external calculator. Furthermore, FuncQA is categorized into two subsets: 68 one-hop questions (FuncQAane) solvable with just one operation, and 60 multi-hop questions (AuniQAAmtt) requiring few reasoning steps.
To train the toolken embeddings used in GSM8K-XL, we preprocess the original training set of GSM8K which has the calculation annotation as described in Section We get 6,054 examples, of which 1. 1,000 were allocated for validation, and 5,054 for the training data. For the FuncQA dataset, we prompt ChatGPT to generate some one-hop QA patterns for each operator, and then randomly assign values the patterns. This process yields 47 training data points and validation data points for each operator, resulting in a total of 611 samples for training and 39 samples for validation.
Comparison methods. We train toolken embeddings for each available math operator as described in Section 3.2 During inference, we prompt the LLM with 4-shot Chain-of-Thougle (65] examples enhance the reasoning ability of LLMs. The following baselines are evaluated for comparison: (1) 0-shot CharGPT is the straightforward method asking LLMs to answer a question. No examples will be provided in the context and tools are not available. We use ChatGPT as the base LLM in our experiment. This baseline measures the ability of the LLM to answer complex numerical reasoning problems with its own reasoning and calculation ability. (2) Chain-of-thouge (CoT) [65] more advanced prompting techniques. In this approach, a series of interconnected prompts are carefully crafted to guide the LLMs through step-by-step reasoning process. The example reasoning chains are the same as the ones we used for ToolkenGPT, but no functions are available. (3) ReAcr [69] combines reasoning and tools by prompting the LLMs to generate verbal reasoning traces and tool calls in an interleaved manner. Concretely, instead of just providing reasoning chains such as "... The cost 50*3.2= 160", ReAct incorporates special syntax to call operators, e.g."... The cost is (...=:cmlippyyy550..3...
Once the syntax is detected during inference, the tool would be called to calculate the result. We use the same reasoning chain examples as in both CoT and ToolkenGPT, with only slight differences in the tool calling syntax. LLaMA-33B [62 is used as the LLM for all settings other than zero-shot prompting. More experiment details are described in Appendix A
Result analysis. Table 2 shows the performance of all the methods on the GSM8K-XL and FuncQA datasets. On the GSM8K-XL dataset, 0-shot ChatGPT and few-shot learning with CoT struggle to calculate large numbers without the help of tools, while ReAct and ToolkenGPT manage to increase accuracy consistently by large margin. Generally, both methods can call the correct tools when necessary, as the toolset is comprised of only the four basic operators. However, for both FuncQA and datasets, learning to call applicable tools becomes challenging to ReAct as the

---

number of tools increases. In ReAct, though all the tools are listed at the beginning of the prompt, it infeasible include demonstrations of every tool in the limited context (In our experiment, we provide 4 examples including 5 tool demonstrations). As result, ReAct is susceptible to missing tool calls, making wrong tool calls, and predicting wrong arguments, especially for the tools not demonstrated in context. ToolkenGPPT outperforms all the baselines across both one-hop and multi- hop scenarios, showing superior tool learning ability when there are numerous tools. It is important to note that even though toolken embeddings are trained solely using one-hop synthetic data, and without any CoT examples, they still manage to enhance performance in multi-hop problem contexts and can be integrated effectively with CoT prompting. This implies degree of generalization of toolken embeddings, which is a very desired property that lowers the requirements of in-domain training data.
4.2 Knowledge-based Question Answering
LLMs are known often make factual errors and hallucinate 28 73 72 1 because of their limited knowledge [20]. Equipping them with access to knowledge bases (KBs) has been a promising research direction to reduce their hallucinations [58]. We formulate the access to the KB APIs querying the database [60] [6]. Thus, each relational query can be treated as a tool to which the input argument is subject entity, and the output the corresponding tail entity. An example tool call is "P1346(2005-06 FA CUP) - LIVERPOOL. F.C." "P6" is a relation identifier in Wikidata, representing the winner of a competition or similar event (referred to winner_oo of below for ease of reading). In this section, we show that ToolkenGPT can accurately query large knowledge base of up 234 tools (relations). We further show that even only with synthetic data (as described in Section 3.2 and explained below), we can train strong toolken embeddings that outperform popular tool-leanning methods.
Dataset. KAMEL [30] is a guestion-answeriigg dataset built with the facts in Wikidata. In line with ToolFormer (56). which uses its earlier version [51] benchmark to evaluate the tool use, we adopt KAMEL to evaluate the use of KB query tools. KAMEL contains knowledge about 243 relations from Wikidata, each of which is associated with question template (e.g. winner_oo of: "Who the winner of [S]?") to turn a fact in Wikidata into question. We have 234 tools in total for this dataset. In order to analyze the performance provided with different numbers of tools, we create four subsets by sampling from the original test set. Each subset consists of questions related to different numbers of relations, corresponding to 30, 60, 100, and 234, respectively. The size of each subset is 500.
Comparison methods. We set up two differ- variants of our framework. (1) ToolkenGPT (sup): We sample 200 examples per relation from the training set of KAMEL and train the toolken embeddings via supervised learn- ing. This setting represents real-world scenarios where sufficient in-domain training data is avail- able. (2) ToolkenGPT (syn): In a more challeng- ing setting where we assume in-domain training data is not available, we use the text descrip- tion of each relation to synthesize training data with ChatGPT, e.g. "The Nobel Peace Prize in 2020 was awarded to the United Nations World Food Programme for its efforts..." ", where the un- derlying tool call is winner of (NoBEL PFACE PRIZE IN (O2T)ED NATIONS WORLD FOOD PROGRAMME. On average, 40 cxamples are used to train each toolken embedding.
1.0
0.8
ToolkenGPT (sup)
Accuracy
0.6
ToolkenGPT (syn)
KL
ICL (desc)
0.4
Prompting
0.2
30
60
100
234
Number of relations (used tools) test
Figure 2: Performance of ToolkenGPT and base- lines on 4 testsets involving different numbers of tools (relations) from KAMEL. ICL is short for In- context Learning 53 Due to the context length limit of 2048 tokens, we list the descriptions and demonstrations of up to 30 relations for ICL and up to 60 relation descriptions for ICL (desc).
We introduce the following baselines for com- parisons: (1) Prompting (30) is a straightfor- ward method that answers the questions with the LLM's internal knowledge. We frame each question within the prompt "Question: [QUES- ThONAaThe answer is" and ask the L.LM to continue the sentence. (2) In-context Learning (ICI.) [53 is standard method to augment LLMs with tools as introduced in Section 2 Before asking the
7

---

question, we list the tool demonstrations and descriptions of all available tools. The demonstrations are shown in a specific syntax so that the LLM can generate similar style be parsed. An exam- ple demonstration for winner of is "Question: Who is the winner of 2005-06 FA Eup.nnArswwr:: FA Cup)=láverpool F.C." In a recent survey (53), this setting is referred to as "few-shot". (3) In-context Learning (desc) [53 is another common practice to augment LLMs with tools. The descriptions of all available tools will be provided in context, but their demonstrations are not directly shown. Instead, we show 8 demonstrations of the tools not included in the test subset to inform LLMs about the tool call format. This setting is referred to as "zero-shot"
The answer is <winner priinner_ofx(200554400
in Qin et al. [53]. The base model for all methods is LLaMA-13B 662 More experiment details described in Appendix B
Result analysis. We show the experiment results on 4 testsets involving different numbers of relations in Figure 2 Note that the number of involved relations is the number of tools we can use. For all testsets, the accuracy of Prompting is about 20%, which indicates L.Mss still struggle to store accurate facts in their parameters and it's necessary to augment them with a knowledge base. ToolkenGPT (sup) achieves the highest results with a large margin, showing that learning toolken embeddings is an effective method when there is massive in-domain training data. On the contrary, even though In-context learning also sees in-domain training data in the context, it still gets confused about which tools to call. Furthermore, the context length limit leads to drastic performance drops when there are more than 30 tools to use. The failure in the many-tools scene reveals the fundamental limitation of the in-context learning paradigm. ToolkenGPT (syn) also outperforms all other baselines in all subsets, without seeing any in-domain training data. The synthetic training data, often in very different expression styles from the dataset, still helps the LLM understand these relations.
This success reflects the flexibility of our framework which can be applied even if there is no in- domain training data available. In-context learning (desc) generally fails in this task, because the LLM has difficulties memorizing text descriptions shown in contexts and mapping them to relation identifiers. The results provide more evidence to the previous discovery that LLMs have trouble using unfamiliar tools [6. Based on this observation, it is reasonable to speculate that LLMs mostly recall the tools from their identifier instead of really learning to use tools from their descriptions.
4.3 Embodied Plan Generation
Recently, there have been many research attempts to utilize LLMs as the controller of embodied agents (25 59 27 (6)) Despite the preliminary success of prompting LLMs, teaching L.LMs about an environment and enabling them to make grounded predictions remain challenging. As discussed in Mialon et al. 45 tools that gather additional information (e.g. math or KB tools) and tools that have an effect the physical world (e.g. actions taken by embodied agents) can be called in similar styles by the LLM. In this section, we demonstrate how our framework can also be applied to plan generation for embodied agents. Compared previous methods that prompt LLMs, our ToolkenGPT can understand the environment better by learning toolken embeddings for agent action and object.
Dataset. VirtualHome [52 is simulation platform for typical houschold activities, and ActivityPro- grams knowledge base [22 consists of many tasks with plans executable in VirtualHome. We derive subset of 297 tasks from Activity,rograms.
Specifically, for each task, the model is given a high-level goal (e.g. "Read book"), a detailed instruction (e.g. "I would go lie down in my bed and open the book and start reuding.", ", and description of the environment, which includes the initial state of the agent, and the object list of the environment (e.g. "I am in ['home_op "hoom_office")) J. The objects I can munipulate are [ I'mail', 'freezer', "television' 'novel'J". The model is expected to output an executable plan, which is an ordered list of verb-object instructions (c.g. "[FIND] <novel>"). Each task comes with an initial and final state graph, enabling the verification of the generated plans with the simulator and the comparison of the resulting final state with ground truth. We split the dataset into a training set of 247 tasks and test set of 50 tasks, with a total of 25 verbs and 32 objects used in the dataset.
Comparison methods. We consider all the actions and objects in VirtualHome as tools. With an additional [END] function indicating the end of a plan, we have 58 toolkens in total. For this dataset, we do not need the argument generation process described in Figure because the tools do not take arguments. During inference, ToolkenGPT alternatively generates action toolkens and object toolkens, and ends with the [END] toolken. The toolken embeddings are trained on the training set.
8

---

Table 3: Results on VirtualHome. Grounding means the proportion of scripts in which all the actions and objects can be grounded to the environment. Executable means the proportion of scripts that can be executed in VirtualHome without violating any rules. Success means the proportion of scripts that leads to the correct final state. Success (R) a relaxed variant meaning the proportion scripts that have reached the correct final state, but not necessarily ending with it.
Method
Grounding
Executable
Success
Success (R)
In-context Learning
0.74
0.42
0.20
0.30
Translation 25
1.00
0.52
0.24
0.32
+ Grounded Decoding 27
1.00
0.66
0.38
0.42
ToolkenGPT (Ours)
1.00
0.82
0.68
0.70
We compare our method to the following baselines: (1) In-context Learning prompts the LLM and parses its outputs as the plan. The LLM is shown with the action list, 3 demonstration plans, and new task with its goal, detailed description, and environment description. This method is the base of most recent methods [25] 4 [27 that apply L.LMs to embodied AL. (2) Translation [25]: To avoid
plans that include unavailable actions or objects, Huang et al. [25] proposes to use a translation model to translate the LLM's generation to admissible instructions. Following Huang et al. (25), use SentenceRooERTa--aaa
(54] and translate the actions or objects to available ones with the highest cosine similarities. (3) Grounded Decoding [27] is a recent decoding-stage grouding method. The next token is predicted considering both LLM logits and "grounded functions". Specifically, we apply the affordance grounding function (27). encouraging LLMs to generate valid actions and objects. We do not consider other previous methods that heavily fine-tune the whole language model (37] The hase model of all methods is 1-3MB 662 More experiment details are described in Appendix C
Result analysis. We list results in Table B Though all valid actions and objects are ex-
Work
plicitly listed the context for the LLM us-
Go to office, sit at desk, turn on computer, enter
ing In-context Learning, it sometimes fails to ground its prediction to admissible instructions. Even though the actions and objects valid, they often violate the physical rule in Virtual- Home, resulting in low success rate. We notice
password, open application and begin work
[WALK] chome chome_office>
[WALK] chome_ <home_office>
[WALK] <desk>
[WALK] <chair>
[FIND] <desk>
[FIND] <chair>
[SIT] <desk>
[SIT] <chair>
that while most of the plans generated with In- context Learning appear reasonable humans, they are not grounded to the specific environ- ment of VirtualHome. Translation (25) helps solve some shallow grounding problems, e.g. [tv] [television], while Grounded Decod- ing 27 further improves executable and success rate by considering grounding carlier in the de- coding stage. Although these methods ensure all plans are grounded, neither significantly improves the LLM's understanding of actions and objects. leading to unsatisfactory executable and success rates. ToolkenGPT not only predict valid actions and objects naturally by its design, but also achieves the highest success rate by learning toolken embed- dings from more training tasks. A concrete example is shown in Figure to illustrate the difference: All the baselines predict [SIT] <desk>, presumably guided by the description "sit at desk", but in VirtualHome [SIT] refers to "sit on", and a desk is regarded as not sittable. ToolkenGPT is the only one to successfully learn this rule from demonstrations and instead predict [SIT] <chair>.
error desk not sitable!
Translation
ToolkenGPT
Figure 3: Case study on Home ToolkenGPT predicts a successful script while other baselines fail produce an executable one due to their misunderstanding of the SIT action.
4.4 Analysis
Computational Cost. We conduct experiments to compare ToolkenGPT with fine-tuning specifically using LoRA (23), in terms of computation efficiency and performance. Due to the cost of fine-tuning LLMs, we implement both methods on LLaMA-7B. The results listed in Table
Fine-tuning L.MSs results in slightly better performance than ToolkenGPT on FuncQA. Even though apply LoRA, which is known for efficiency, the time consumption for fine-tuning exceeds

---

Table 4: Comparison between ToolkenGPT and fine-tuning (LoRA) in terms of training cost and performance on FuncQA dataset. Both methods are based on Llama-7B.
Method
One-hop
Multi-hop
Computing Resource Training Time
ReAct
0.40
0.03
Prompting
0.10
0.00
Fine-tune w/ LoRA 23
0.62
0.07
A100 (80G)
40 min
ToolkenGPT 0.55
0.06
1 x RTX3000 (24G) 2 min
Table 5: Ablation study on FuncQA dataset with LLaMA-30B.
Table 6: ToolkenGPT with different con- figurations of training data on KAMEL.
Method
One-hop
Multi-hop
# Examples
Synthetic
Supervised
ReAct 0.57
0.06
10
0.36
0.56
Tool mode 0.60
0.07
20
0.46
0.90
ToolkenGPT 0.73
0.15
40
0.52
0.95
significantly when compared to training toolken embeddings. It is also worth noting that ToolkenGPT enjoys additional benefits other than efficiency (Table [ especially the blug-and-play of massive tools, thanks to the decoupled parameters for different tools.
Ablation Study. The design of ToolkenGPT benefits both tool selection and argument completion (tool mode in Figure D] To understand their respective contributions to the performance, we further implement baseline combining ReAct-style prompting and the sub-routine of argument completion (tool mode). In the tool mode, the LLM is prompted with demonstrations using only the selected tool, which will provide more relevant knowledge than ReAct prompt for argument completion. As shown in Table adding tool mode could indeed improve the vanilla ReAct prompting method by enhancing the accuracy of argument completion. However, ToolkenGPT still outperforms this improved baseline by large margin, indicating that toolken embeddings effectively help LLMs to decide when and which tool to call.
Training Data. In this section, we explore the effects of training data on learning the toolken embeddings. We choose extend our experiments on KAMEL (Section because there are two different sources of training data and it is easy to process synthesize more data. Specifically, we sample 10/20/40 training examples of each tool for both ToolkenGPT (sup) and ToolkenGPT (syn), and report the accuracy on the test set involving 30 tools.
The results are summarized in Table Under the same budget of data size, training with supervised data leads to better performance. Though we do not observe obvious mistakes in most of synthetic data instances, the distribution gap between synthetic data and test set may prevent toolken embedding from performing well. A larger training set benefits the performance for both data sources.
5 Conclusion
We presented ToolkenGPT, a new approach for augmenting frozen LLMs with massive external tools without expensive fine-tuning. Our method introduces the concept of toolken embedding for each tool, enabling LLMs to call and use different tools as easily as generating word tokens. Our approach overcomes the limitations of current fine-tuning and in-context learning paradigms, enabling LLMs to accommodate a much larger set tools and use extensive demonstration data for learning toolken embeddings. On series of tasks, ranging from numerical reasoning, knowledge-bsatd based question answering, to embodied plan generation, we observed a significant enhancement in LLM performance with the help of toolken embeddings. More importantly, ToolkenGPT is able rapidly adapt and leverage new tools, demonstrating its capacity to keep pace with the constantly evolving landscape massive tools. We expect future research learn robust toolken embeddings not only from demonstration data, but also other rich forms of experience (24), such as tool descriptions and input-output records. We are also interested in exploring the integration of toolken embeddings to recent advanced planning techniques [9], with the goal of developing an autonomous agent to solve complex wealwwrldd problems.
10

---

Acknowledgements
This project is partially supported by DARPA ECOLE HR001 12390063.
References
[1] Razvan Azamfirei, Sapna Kudchadkar, and James Fackler. Large language models and the perils of their hallucinations. Critical Care, 27(1):1-2, 2023.
[2] Michael Bommarito II and Daniel Martin Katz. Gpt takes the bar exam. arXiv preprint arXiv:2212.14402,, 2022.
[3] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. International conference machine learning, pages 2206-2240 PMLR, 2022.
[4] Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang. Ryan Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. In Conference Robot Learning, pages 287-318. PMLR. 2023.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 13:18771190,, 2020.
[6] Sébastien Bubeck, Varun Chandrasckaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv preprint arXiv:2303 12712, 2023.
[7] Harrison Chase. LangChain, 10 2022. URL http:://gitthbbb http://tt/hwwhcace wwwm///chaast////lgghhhaaa
[8] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211. 12588 2022.
[9] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint urXiv:2204.02311 2022.
[10] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthia Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
[11] Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. arXiv preprint arXiv:2203.06904, 2022.
[12] Iddo Drori, Sarah Zhang, Reece Shuttleworth, Leonard Tang, Albert Lu, Elizabeth Ke, Kevin Liu, Linda Chen, Sunny Tran, Newman Cheng, et al. A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level.
Proceedings of the National Academy of Sciences, (1932))e212343311 2022.
[13] Dheeru Dua, Shivanshu Gupta, Sameer Singh, and Matt Gardner. Successive prompting for decomposing complex questions. arXiv preprint 22Xi::2222.4009.. 2022.
[14] Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. Gpts are gpts: An early look at the labor market impact potential of large language models. arXiv preprint arXiv:2303.101 10130, 2023.
[15] Jacqueline Fagard, Lauriane Rat-Fischer, Rana Esseily, Eszter Somogyi, and JK O' O'Regan. What does it take for an infant to learn how to use tool by observation? Frontiers in psychology, 267, 2016.
[16] Bin Fu, Yungi Qiu, Chengguang Tang, Yang Li, Haiyang Yu, and Jian Sun. A survey complex question answering over knowledge base: Recent advances and challenges. arXiv preprint arXiv:2007.13069 2020.

---

[17] Luyu Gao, Aman Madaan, Shuyan Zhou, Uni Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Graham Neubig. Pal: Program-aite -ided language models. arXiv preprint arXiv:2211.10433 2022.
[18] Kelvin Guu, Kenton Lee. Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning. pages 3929-3938. PMLR, 2020.
[19) Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong. Zhen Wang. Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
[20] Shibo Hao, Bowen Tan, Kaiwen Tang, Bin Ni, Xiyan Shao, Hengzhe Zhang, Eric Xing, and Zhit- ing Hu. BertNet: Harvesting knowledge graphs with arbitrary relations from pretrained language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5000- 5015, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/ v 1120033.indings-acl..com
findings-acl.
[21] Joy He- Yueya, Gabriel Poesia, Rose Wang, and Noah D Goodman. Solving math word prob- lems by combining language models with symbolic solvers arXiv preprint 2:4102230..09102, 2023.
[22] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficien transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799. PMLR, 2019.
[23] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
[24] Zhiting Hu and Eric Xing. Toward a 'standard model' of machine learning. Harvard Data Science Review, 2022.
[25] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners:
Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 9118-9147. PMLR, 2022.
[26] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022.
[27] Wenlong Huang. Fei Xia, Dhruv Shah, Danny Driess, Andy Zeng, Yao Lu, Pete Florence, Igor Mordatch, Sergey Levine, Karol Hausman, al. Grounded decoding: Guiding text generation with grounded models for robot control. arXiv preprint arXiv:2303.008S. 2023.
[28] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
[29] Qiao Jin, Yifan Yang. Qingyu Chen, and Zhiyong Lu. Genegpt: Augmenting large language models with domain tools for improved access to biomedical information. ArXiv, 2023.
[30]
Jan-Christoph Kalo and Leandra Fichtel. Kamel: Knowledge analysis with multitoken enti- ties in language models. In Proceedings of the Conference on Automated Knowledge Base Construction, 2022.
[31] Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34:1022-1035, 2021.
[32] Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:22 2022.
[33] Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27.
Open Review, 62, 2022.
[34] Brian Lester, Rami A1-Rfou, and Noah Constant. The power of scale for efffmeter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, 2021.
12

---

[35] Patrick Lewis, Ethan Perez, Aleksandra Piktus. Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih. Tim Rocktäschel, et al. Retrieval-augmentt generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459 -9449, 2020.
[36] Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: benchmark for tool-augmented Ilms, arXiv preprint arXiv:2304.08244, 2023,
[37] Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang. Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, et al. Pre-trained language models for interactive decision decision-making. Advances in Neural Information Processing Systems, 35:31 199-31222, 2022.
[38] Xiang Lisa Li and Percy Liang, Prefix-tuning Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 4582 4597, 2021.
[39] Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, et al. Taskmatrix. ai: Completing tasks by connecting foundation models with millions of apis. arXiv preprint arXiv:2303. 16434, 2023.
[40] Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Lim+ p: Empowering large language models with optimal planning proficiency, arXiv preprint 214447230..11477, 2023.
[41] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhenghao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023.
[42] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable fine-tuning universally across scales and tasks. arXiv preprini arXiv: 10.00602. 2021.
[43] Pan Lu, Baolin Peng, Hao Cheng. Michel Galley, Kai- Wei Chang, Ying Nian Wu, Song- Chun Zhu, and Jianfeng Gao. Chameleon: Sug-and-plit compositional reasoning with large language models. arXiv preprint arXiv:2304.09842 2023.
[44] Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apid- ianaki, and Chris Callison-Burch. Faithful chain-of-though reasoning. arXiv preprint WxXiv:2301.13379 2023.
[45] Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, et al. Augmented language models: survey. arXiv preprint arXiv:2302.078.22 2023.
[46] Reichhrro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser- question-answeringg with human feedback. arXiv preprint arXiv:2112.09332. 2021.
[47] OpenAI. Gpt-4 technical report, 2023.
[48] Batu Ozturkler, Nikolay Malkin, Zhen Wang, and Nebojsa Jojic. Thinksum: Probabilistic reasoning over sets using large language models. arXiv preprini arXiv:2210.0129. 2022.
[49] Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. Art: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.0900 2023.
[50] Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. arXiv preprint arXiv:2205. 12255, 2022.
[51] Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint 1rXiv:190..01066, 2019.
[52] Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8494-8502 2018.
13

---

[53] Yujia Qin. Shengding Hu, Yankai Lin, Weize Chen, Ning Ding. Ganqu Cui, Zheni Zeng. Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shi Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bo Li, Ziwei Tang, Jing Yi, Yu Zhu, Zhenning Dai, Lan Yan, Xin Cong. Ya-Ting Lu, Weilin Zhao, Yuxiang Huang. Jun-Han Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang. Cheng Yang. Tongshuang Wu, Heng Ji, Zhiyuan Liu, and Maosong Sun. Tool learning with foundation models. ArXiv, abs/2304.0833 2023.
[54] Nils Reimers and Iryna Curevych. Sentence-bert: Sentence embeddings using siamese bert- networks. arXiv preprint arXiv: 10084, 2019.
[55] Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, et al. Recipes for building open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 300-325, 2021.
[56] Timo Schick, Jane Dwivedi-Yo Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023.
[57] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yucting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303. 17580, 2023.
[58] Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmenta- tion reduces hallucination in conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3784-3803, 2021.
[59] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209. 11302, 2022.
[60] Alon Talmor and Jonathan Berant. The web as a knowledge-brrd base for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 641-651, 2018.
[61] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.00 008239,, 2022.
[62] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 2023.
[63] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instruc- tions. arXiv preprini urXiv:2212. 1.. 10560, 2022.
[64] Zhen Wang, Rameswar Panda, Lconid Karlinsky, Rogerio Feris, Huan Sun, and Yoon Kim. Mul- titask prompt tuning enables parameter-efficient transfer learning. In The Eleventh International Conference on Learning Representations, 2023.
[65] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. / /903, 2022.
[66] Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang, Zichao Yang, and Zhiting Hu. Language Models Meet World Models: Embodied Experiences Enhance Language Models. NeurlPS, 2023.
[67] Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. Translating natural language planning goals with langaala models. arXiv preprint arXiv:2302.05128. 2023.
[68] Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans. Foun- dation models for decision making: Problems, methods, and opportunities. arXiv preprint 20X3v:2303.04129 2023.
14

---

[69] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting language models. arXiv preprint arXiv:2210.03629, 2022.
[70] Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and Meng Jiang. survey of knowledge-enhanced text generation. ACM Compuling Surveys, 54(1 1s):1-38, 2022.
[71] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efficiie fine-tuning for transformer-bes based masked language- model.. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1-9, 2022.
[72] Yuheng Zha, Yichi Yang. Ruichen Li, and Zhiting Hu. AlignScore: Evaluating factual consistency with a unified aligmment function. Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11328-11348, Toronto, Canada, July 2023. Association for Computational Linguistics. URL https://ccaathhllogg org 2023 acl- acl-long. 634
[73] Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. Text alignment is an efficient unified model for massive nlp tasks. NewrIPS, 2023.
15

---

A Details of Numerical Reasoning
In this section, we describe the data processing workflow (including building the testing datasets and synthesizing training data), list the prompts used for different methods, and describe the training setting. For a fair comparison, we use the same prompts for CoT and the reasoning mode of ToolkenGPT. With the same example questions, we label the calculation process in place get the prompts for R For the tool mode of ToolkenGPT, we randomly sample examples of the specified tool from the training set, and transform them into ReAct-style prompts. Because large number of tools are used, we show the prompts in the supplementary file instead of listing them here.
A.1
GSM8K-XI
A.1.1
Data Synthesis
To build an enhanced version GSM8K-XL -XL by magnifying numbers in GSM8K, we take the following steps:
1. We prompt ChatGPT(gpt-3. 5-turbo) with examples replace the numbers with appropriate placeholders, The prompt is presented below.
2. In order to validate the correctness of the number replacements, we develop a verification function for the GSMSk dataset, which calculates the formulas embedded in the solutions (See the example prompt below. Formulas are quoted with and w.) to get answer. By substituting the original numbers back into the rewritten question-answer pairs and comparing the execution results with the original answer, can identify any erroneous rewrites and exclude them from our dataset. This ensures the reliability and accuracy of the number replacements.
3. Once have confirmed the accuracy of the rewriting process, we proceed with magnifying the numbers. We replace the original numbers with their respective cubic values, exponentially amplifying their magnitudes. However, it's worth noting this strategy doesn't make sense in every scenario. For instance, when dealing with percentage values (e.g., 10%), direct cubic magnification to 1000% would be inappropriate. In such cases, maintain the original values to preserve their intended meaning.
4. With the magnified numbers in place, we seamlessly substitute the placeholders within the rewritten question-ansee answer pairs. By employing the solving function developed earlier, we can process the magnified formulas and obtain accurate solutions, resulting in the GSMSK-XL dataset.
"role": "system",
"content": "Your task is rewrite the given question and answer in more general form using variables instead of specific numbers."
"role": "user"
"content": "Question: Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much dollars does she make every day at the farmers' market <<16-3-4-9>29 duck eggs day. \nShe makes every day at the farmer's market. \n#н## 18"
Janet sells 16 3 4 $c<9-2-18>>18
8,
"role": "assistant",
"As Yao et al. 669 doesn't evaluate ReAct on numerical reasoning, we don't follow the exact format (i.e. trajectories marked by [THINK] and ), but design a more natural way to combine CoT reasoning and tool calling (as shown later sections). We refer to this format "ReAct-style" prompts.
16

---

"content"; "Question: Janet's ducks lay {v_1} egg per day. She eats (v_2) for breakfast every morning and bakes muffins for her friends every day with [v_3}. She sells the remainder at the farmers' market daily for $(v_4) per fresh duck egg. How much dollars does she make every day at the farmers'
Janet
sells {v_1} {v_2} {v_3} <<(v (el1iivv22--__- (1-___3)=(__1>>2>>033 1ab__>11>><________ 1} duck
egge day. AnShe makes {r_1} * {v_4} $<<{r_ ()=(v__4=(r_2>>>(x___
every day at the farmer's market. \n# {r_2)"
"role": "user",
"content": "Question: wooden bridge can carry no more than 5000 pounds. delivery truck filled with identical boxes, each weighing 15 pounds, will pass over the bridge. The combined weight of the driver and the empty truck is 3755 pounds. What is the maximum number of boxes which can be loaded onto the truck while not exceeding the bridge's weight limit?\nAnswer. The boxes can weigh to 5000 pounds 3755 pounds = <5000-3766-1246>>222 pounds in total. AnThere can be 1245 15 <<1245/15-83>>83 boxes loaded onto the truck without exceeding the bridge's weight limit. \n#*## 83"
"role": "assistant",
"content"; "Question: A wooden bridge can carry no more than {v_1} pounds. delivery truck filled with identical boxes, each weighing {v_2} pounds, will pass over the bridge. The combined weight of the driver and the empty truck is {v_3} pounds. What the maximum number of boxes which can be loaded onto the truck while not exceeding the bridge's weight limit?/nAnswer: The boxes can weigh up to [v_1} pounds {v_3} pounds <<tu1D)-(v_3/><<<1)>>>> pounds in total. \nThere can be {r_1} {v_2}
<<(r At/[v_2]=(C (r>><)))))(((((rr..>.>>)))>>)>>) 2} boxes loaded onto the truck without exceeding the bridge's weight limit. \n#### (r_2}"
"role": "user" "content": [INPUT]
A.1.2 Training Details
As mentioned in Section 4.1 the toolken embeddings are trained with subset of 5,063 examples. An additional 1,000 examples are reserved for validation. The embeddings were trained with learning rate of 5e-4, performing early stopping based on the development set, with maximum of 10 epochs.
A.1.3 Prompt for GSM8K-XL XL Dataset
Prompt for Direct Prompting with ChatGPT:
Solve the following math problem step by step, and then provide the final answer in the format: 'So, the answer xxx,'
[QUESTION]
Prompt for Chain of Thought (CoT) and ToolkenGPT reasoning mode:
Answer the following questions step by step.
Question: Mark has 3 tanks for pregnant fish. Each tank has 4 pregnant fish and each fish gives birth young. How many young fish does have the end?
17

---

Answer: He has 4+3=12 pregnant fish They give birth to 12+20-240 fish #### 240
contest are divided into three rounds: easy, average, and hard. There are corresponding points given for each round. That is 2, 3, and points for every correct answer in the easy, average, and hard rounds, respectively. Suppose Kim got 6 correct answers in the easy; correct answers in the average; and correct answers in the difficult round, what are her total points in the contest?
Question:
The math questions
:nswer: Kim got points/round x 2 round 12 points in the easy round. She got 2 points/round
rounds 6 points in the average round. She got points/round x 5 rounds = 20 points in the difficult round. So her total points is 12 points + 6 points 20 points 38 points. #### 38
Question: clothing store sells 20 shirts and 10 pairs of jeans. A shirt costs $10 each and pair of jeans costs twice as much. How much will the clothing store earn if all shirts and jeans are sold?
Answer: Twenty shirts amount to $10 x 20 $200. The cost of each pair of $20. So pairs of jeans amount to $20 x 10 $200.
jeans $10
Therefore, the store will earn $200 + $200 = $400 if all shirts and jeans are sold. #### 400
Question: Arnold's collagen powder has 18 grams of protein for every scoops. His protein powder has 21 grams of protein per scoop. And his steak has 56 grams of protein.
has scoop of collagen powder, scoop of protein powder and his steak, how many grams of protein will he consume?
Answer: 2 scoops of collagen powder have 18 grams of protein and he only has 1 scoop so he consumes 18/2 grams of protein He has 9 grams collagen powder, 21 grams of protein powder and 56 grams in his steak for a total of 9+21+56 86 grams of protein #### 86
Question:
[QUESTION]
Answer:
Prompt for ReAct:
Answer the following questions with <add>, <subtract>, <multiply>, <divide> operators
Question: Mark has tanks for pregnant fish. Each tank has 4 pregnant fish and each fish gives birth to 20 young. How many young fish does he have the end?
Answer: He has <mllipyy) 3)=12 pregnant fish They give birth to 12=20=<multiply>> (12, 20)=240 fish #### 240
Question: The math questions in contest are divided into three rounds: easy, average, and hard. There are corresponding points given for each round. That 2, 3, and points for every correct answer in the easy, average, and hard rounds, respectively. Suppose Kim got 6 correct answers the easy; correct answers the average; and correct answers in the difficult round, what are her total points in the contest?
Answer: Kim got points/round 2 round <multiply> 2)=12 points the easy round. She got 2 points/round x rounds <multiply>>2, 3) points in the average round. She got 4 points/round x rounds Kmultiply> (4, 5)=20 points in the difficult round. So her total points is 12 points points 20 points <add> (12, 6, 20)=38 points. #### 38
18

---

Question: A clothing store sells 20 shirts and 10 pairs of jeans. A shirt costs $10 each and pair of jeans costs twice much. much will the clothing store earn if all shirts and jeans are sold?
Answer: Twenty shirts amount to $10 x 20 $<mltiply>>10, 20)=200. The cost of each pair of jeans is $10 x 2 $<mltiply> 2)=20. So 10 pairs of jeans amount to $20 x 10 $<ultiply>> (20, =0)=200. Therefore, the store will earn $200 + $200 $<add> (200, 200)=400 if all shirts and jeans
are sold. #### 400
Question: Arnold's collagen powder has 18 grams of protein for every 2 scoops. His protein powder has 21 grams of protein per scoop. And his steak has 56 grams of protein. If he has scoop of collagen powder, 1 scoop of protein powder and his steak, how many grams of protein will he consume?
Answer: 2 scoops of collagen powder have 18 grans of protein and he only has 1 scoop so he consumes 18/2 <divide> (18, 29 grams of protein He has 9 grams collagen powder, 21 grams of protein powder and grams in his steak for a total of 9+21+56 = <add> (9, 21, 56)=86 grams of protein ####
Question:
[QUESTION]
Answer:
A.2 FuncQA
A.2.1
Training Details
As mentioned in Section 4.1 Toolken embeddings are trained using subset of 611 examples, with additional 39 examples reserved for validation purposes. The learning perform early stopping based on the development set, with the maximal training epochs to be 20.
is 1e-4, and
A.2.2 Prompt for Synthetic Training Data
In Section 4.1
we discussed the utilization of ChatGPT for synthesizing the training set. To create the training data, we begin by manually crafting two examples that adhere the desired format, and then use the following specific prompt to generate more examples. However, important to acknowledge that the prompt does not guarantee the generation of examples that strictly conform to the required format. So, we apply filtering process to remove any non-conforming instances. Furthermore, the generation process often produces duplicate examples, necessitating subsequent de-duplication step.
You are a math question generator for teachers, and your task is to generate some questions and answers using function [FUNC] to solve and can be solved within one single step. You not need give specific numbers, so that the teachers can fill any numbers they want. Here are two examples that use the function [FUNC].
[EXAMPLE_1]
[EXAMPLE_2]
[FUNC] is a function to [DESCRIPTION] Now, let's mimic the format of examples to generate various real world QA pairs using the function [FUNC] that can be solved within one step. The numbers should be replaced by [ARG] and [ANSWER] as the examples given above.
A.2.3 Prompt for FuncQA One-Hop
Prompt for Shoo with ChatGPT:
19

---

Solve the following math problem, and then provide the final answer in the format: 'So, the answer xxx."
[QUESTION]
Prompt for Chain of Thought (CoT) and ToolkenGPT:
Q: If Amy's income increases by 4% annually, how many times will it multiply in 11 years?
A: In 11 years, Amy's s income will increase by 1.04-11=1.8 times. So, the answer is 1.54.
Q: If store sells 147 bananas today and 354 more bananas tomorrow, how many bananas does the store sell in total?
A: The store sells 147 bananas today 354 more bananas tomorrow, so the total number of bananas sold is 147+354=501, So, the answer is 501.
Q: A man had 789.4 4 dollars in his wallet. He spent 11.99 dollars on movie ticket. How much money does have left now?
A: The man had 789.4 4 dollars in his wallet and spent 11.99 dollars on movie ticket, so he 777.41 dollars.
4-11.991.91....... 99=799 dollars left. So, the answer
Q: If cake weighs 3.77 pounds and is divided into 13 equal pieces, how much does each piece weight?
A: Each piece of the cake weighs 3. 3.77/13-0.27 .99 pounds. So, the answer is
29 pounds.
Q: [QUESTION]
A:
Prompt for ReAct:
Answer the following question with <add>, <subtract>, <multiply>, <divide>, <power>, <sqrt>, <log>, <lem>, <ged>, <ln>, <choose>, <remainder>, <permutate>:
Q: If Amy's income increases by 4% annually, how many times will it multiply in 11 years?
A: In years, Amy's income will increase by 1.04-11 <power> (1.04 11)=1111156 =1.54 times. So, the answer is 1.54.
Q: If store sells 147 bananas today and 354 more bananas tomorrow, how many bananas does the store sell in total?
A: The store sells 147 bananas today and 354 bananas tomorrow, the total number of bananas sold is 147+354=<add> (147, 354) =501. So, the answer is 501.
Q: had 789.4 dollars in his wallet. He spent 11.99 dollars on movie ticket. How much money does he have left now?
A: The man 789.4 4 dollars in his wallet and spent 11.99 dollars on movie ticket, so he has 789, 1-194.11111 99=<eubtract> (789. 1...:,,,.1..999118 11. dollars left. So, the answer 777.41
Q: If cake weighs 3.77 pounds and is divided into 13 equal pieces, how much does each piece weight?
A: Each piece the cake weighs 3.77/13=<dividee (3.77, 13)=0.30.22 29 pounds.
So, the answer is 0.29.
Q: [QUESTION]
20

---

A:
A.2.4 Prompt for FuncQA Multi-Hop
Prompt for Zero-Shot with ChatGPT:
Solve the following math problem step by step, and then provide the final answer in the format: 'So, the answer xxx.
[QUESTION]
Prompt for Chain of Thought (CoT) and ToolkenGPT:
Answer the following questions step by step:
Question: coin is tossed 8 times, what is the probability of getting exactly 7 heads ?
Answer: The total number of possible outcomes to toss a coin times is
2-8=256. The number of ways of getting exactly 7 heads is 8C7-8. The probability of getting exactly heads is 8/256-0. 03125. #### 0. 03125
Question: If paint costs $3.2 per quart, and a quart covers 12 square feet, how much will it cost paint the outside of cube 10 feet each edge?
Answer: The total surface area of the ft cube is 6+10-2=6+10=6600
square feet. The number of quarts needed is 600/12=50. The cost is 50+3.2=160. #### 160
Question: 1x)(x)=2, 1og(y)=0.1, what Answer: =2, so x=10-2=100; 00.,, so y=10-0. 10000.111.00 1=1.26;; x-y=100-1.26-98.777 74, so log(x-y)=1og(98.777)) (9)))))1.1)) 74) #### 1.99
value 1xg(x-y) ?
Question: How many degrees does the hand travel when the clock goes 246 minutes?
Answer: The hour hand travels 360 degrees in 12 hours, so every hour it travels 360/12=30 degrees. 246 minutes is 246/60=4.1 1 hours. The hour hand travels 4+30=112 degrees. #### 123
Question: [QUESTION]
Answer:
Prompt for ReAct:
Answer the following questions with <add>, <subtract>, <multiply>, <divide>, <power>, <sqrt>, <log>, <lem>, <ged>, <ln>, <choose>, <remainder>, and <permutate>:
Question: coin is tossed 8 times, what is the probability of getting exactly 7 heads?
coin times is 2-8=<power> (2, 8)6256. The number of ways of getting exactly 7 heads is 88 The probability getting exactly heads
Answer:
The total number of possible outcomes
8/256=<divide> (8, =0. 03125. #### 0. 03125
Question: If paint costs $3.2 2 per quart, and quart covers 12 square feet, how much will it cost paint outside of cube 10 feet on each edge?
Answer: The total surface area the 10 cube is 6+10~2=6*<pover>>> (10)))=10000<<mmiiippp>>> (6,10)))6600 160)=600 square feet. The number of quarts needed is 600/12=<aivide><6000 (2) The cost
#### 160
21

---

Question: 1xg(x)=2, 1og(y)=0.1, what value log(x-y) ?
Answer: 1xg x=107>><ppwer> 1og(y)=0.1, so 2x 26) =98. -9, so 11 ((9.))1199 #*## 1.99
Question: How many degrees does the hour hand travel when the clock goes 246 minutes?
Answer: The hour hand travels 360 degrees in 12 hours, every hour it travels 360/12><divide> (360, (2)6012))=00 =3 degrees. 246 minutes is 246/60=<divide> (24e:)iddd2(2660))))) 60) =4.1 hours. The hour hand travels degrees. #### 123
4. 1.11300<<mlliippyy4441,.,..7
Question: [QUESTION]
Answer:
B Details of Knowledge-based QA
In this section, show how transform each Wikidata relation identifier (e.g. P1346) into natural language description and then describe the method to synthesize data and the training settings.
B.1 Getting Text Description
KAMEL provides question template for each relation. We randomly sample 3 facts from the dataset and instantiate them into question-anse answer pair, and use the following prompt to generate a description for them with ChatGPT:
Given question template and some example answer, you need to def ine API that can help you answer the question.
Q 1.1: What is the original language of The Wonderful Galaxy of Oz
1.1: Japanese
1.2: What is the original language of Wild Field?
A 1.2: Russian
1.3: What is the original language of Nadigan?
1.3: Tamil
API 1: Bigginal_laguuage (title): gets the original language of art work
Q 2.1: What languages does Judah Maccabee speak?
2.1: Hebrew
Q 2.2: What languages does Ronelda Kamfer speak?
2.2: Afrikaans
Q 2.3: What languages does Leibush Lehrer speak?
2.3: Yiddish
API 2: @ooken_languagee (name) gets the spoken languages of person
3.1: [Q1]
A 3.1: [A1]
3.2: [Q2]
A 3.2: [A2]
3.3: [Q3]
A 3.3: [A3]
API 3:
B.2
Synthetic Data
We prompts to synthesize diverse training data, and aggregate the samples from each prompt.
*Note that the natural language descriptions are not necessary for ToolkenGPT which is directly trained with demonstrations, but they are crucial for the in on-context learning baselines, especially for ICL (desc), which can only understand tools through language descriptions.
22

---

Here are some examples of using functions for text generation (after the function call, the sentence should continue with the returned value of the function call):
1. star_ _eaarrrttinnee (product)) gets the star rating the product on scale from 0 to 5.
Example 1.1: The new iPhone 12 Pro Max is already generating a lot of buzz, thanks to its <f>star_rating ("iPhone 12 Pro 4ax")="4.7" 7"</f>4.7 star rating.
2. Literary_genre(book)) gets the literary genre of book Example 2.1: Literature is often categorized by genre, such as drama, romance, science fiction. The Harry Potter series is popular example of the <f>literary -eereeeeerrrreeeeee ("Harry Potter")""lattter"""">aaaaa features imaginary worlds and magical elements.
genre, which
3. current_ current_location (oscatiionuusr__liii
gets the current location of a user.
Example 3.1: If you're trying to coordinate plans with friend, it's helpful to know their current location. You can ask the question "Where are you right now?" and the function <tcurrent_location "1234")="Ner York"</f>New York as an example response.
4. number_ fimber_of_moviee htt.ooooooooiiiiiieeeeeee (director)) gets the number movies directed by a specific director.
Example 4.1: Martin Scorsese is one of the most celebrated movie directors of all time. He has directed total of <f>nuberr_od movies ("Martin
"corsese")==7""<<("7"" movies throughout 5. word_ frrd_definiitinn (word): (word))) gets the definition of particular word Example 5.1: Writers and English language learners can enhance their vocabulary by knowing the definition of unfamiliar words. The definition of the word "eccentric" is and slightly straage"<<"Fumccoventti ("eeintttom"""cecerrrr""""""" and slightly strange.
career.
6. number_oft wtw.sptffffoloooeeeeeeeet (artooo (artist): gets the number of Spotify followers for the artist.
Example 6.1: Taylor Swift's latest album is a hit and her fan base is growing rapidly. In fact, her number of Spotify followers of today is <f>number_of of Biaimbee_o__sottff_feee. (oal ("Taylor "wift")="49,en 879, 220"</f>49,,.,// 9,, 879, 220. 6. [DESCRIPTION]
Please continue to generate 10 examples using the function [NAM]], starting with 6.1 601
Here are some examples of using functions for text generation (after the function call, the sentence should continue with the returned value of the function call):
1. current_weatee (city) gets the current weather of city. Example 1.1: What's the weather in Beijing now? The weather is prcuurent__eathee (""el""""""""""""""""""""""""""
now. Example 1.2: Do you know what's the weather in San Diego now? The weather is Prccurrent_weahheee ("San
now.
2. salculatoorforrmlla gets the calculation result of formula. Example 2.1: What's sum 213 and 5032? The answer is <f>calculator( ("22"""""""""""""""""""""""""11
Example 2.2: What's difference between 2015 and 33? The answer is ("alculltorr2201--3333 "19"""")))"99""//1111""
3. [DESCRIPTION]
Please continue to generate 10 examples using the function [NAM]] starting with 3.1 to 3.10.
B.3 Training Details
Toolken embeddings are trained with learning rate of 1e-4, performing early stopping based on the development set, and trained for maximum of 5 epochs.
23

---

C Details of Embodied Plan Generation
In this section, we describe the preprocessing of VirtualHome, and list all the prompts and training details.
C.1 Preprocessing
We collect all scripts from Activity [25] and filter the dataset with the following steps: (1) filter out all the scripts that not executable, don't cause any state changes in VirtualHome (2) deduplicate the scripts with the same goal and instruction. (3) discard the script that involves two different objects of the same name (4) find the verbs and objects that occur more than 10 times in the data, and keep the scripts composed of only these verbs and objects.
Note that our preprocessing is different from Huang et al. 25 where they regard high-level goal as a task. We treat two scripts with the same goal but different instructions as distinet tasks because different instructions often indicate different action sequences, which may Iead to different final state graphs, e.g., for a high-level goal "Reading", some the instructions mention "Turn on desk lamp" while others don't. Huang al. 25 relies on human annotation to evaluate the correctness of the generated script, which actually lowers the difficulties learning the environment, because humans may assign correct label long the plan looks "reasonable". On the contrary, use the Evolving Graph f to strictly match the resulting state and ground truth state. This serves as an automatic and more objective evaluation.
C.2 Prompts
We show the prompts for LLMs to generate plans below. Note that all methods use the same prompts this experiment.
I household robot and I can take actions from [[IND] [[IT]' [SWITCHON] ', [TURNTO] [LOOKAT]], [[TYP]], [WALK]', '[LIE] [GRAB] [READ]', [[ATCH]] , [POINTAT]', [TOUCH] [[WITTCOOF]] [OPEN] [PUSH] [[PUTOBJBAC]', [CLOSE] [[RIIN]] "[RUN], [DROP] [PULL]'
Task 1:
am in ['bathroom]]. The objects can manipulate are ['faucet', 'keyboar', 'television', 'coffe_maker" 'chair', 'button', 'pillow', iphone,, 'cup', 'couch', 'freezer', 'desk', 'oven', 'light', 'table', "bedroom', Sdining_room's 'cupboard', 'computer', 'sink', 'mail', 'bed', 'mouse', 'home_ 'home_office" office'].
Goal:
Write an email
Hint:
i went near the computer and turned it on. then sent the mail
Plan:
[WALK]
<home_office>
[WAKK] <table>
[FNND] <table>
[WALK] <table>
[FIND] <computer>
[TURNTO] <computer>
[LOOKAT] <computer>
[TURNTO] <computer>
[SWITCHON] <computer>
[FIND] <mail>
[TURNTO] <mail>
Task 2:
24

---

I am in @rome_office'. The objects I can manipulate are ['faucet', 'noeel', 'keyboard', 'television', 'newspaper', 'chair', 'coffe _aatfe_ 'pillow', 'phone', 'check', 'couch', 'freezer', 'desk', 'toothbrush', Poven', 'light', food' 'table', 'bookmark', 'bedroom', Saainggrromm" 'computer', 'sink', 'mail', 'bed', 'cat', 'mouse', 'home_office",, 'pot']. Goal:
Work
Hint:
Find the computer. Turn it on by pressing the on button. Wait for it to load. Use the mouse and keyboard to perform your tasks on screen.
Plan:
[FIND] <computer>
[SWITCHON] <computer>
[FIND] <mouse>
[TOUCH] <mouse>
[FIND] <keyboard>
[TOUCH] <keyboard>
Task 3:
am ['bathroom]], The objects can manipulate are ['dishwashe'', 'faucet', 'keyboard', 'television', 'newspaper', 'chair', 'coffe_ rooffemaker', maker', Ppillow', 'phone', 'cup', 'check', 'couch', 'freezer', 'desk', 'oven', "light', food_foodd' 'plate', 'table', 'bookmark', 'bedroom', "dining_room', 'cupboard', 'computer', 'sink', 'bed', 'cat', 'mouse', "home_office,, 'pot'].
Goal:
Pick up phone
Hint:
first when i hear the ringing sound i will my living room and picks
up and i will say hello
Plan:
[RUN] chome_offiie
[WALK] <chair>
[FIND] <chair>
[SIT] <chair>
[FIND] <phone>
[GRAB] <phone>
Task
[QUESTION]
C.3 Training Details
Toolken embeddings are trained with learning rate of Ie-4, performing early stopping based on the development set, with a maximum of 10 epochs.
D
Computational Resources
In terms of computational resources, we train and test ToolkenGPT based on LLaMA-13B and LLaMA-33B using 2 and Nvidia RTX 3090 GPUs, respectively.
E
Safeguard Statement
In this paper, we primarily focus on the applications of mathematical, knowledge-based, and embodied planning problems, posing no significant ethical or harmful concerns. We recognize that future research on border applications of tool learning may pose a risk of misuse, and we recommend careful consideration of all aspects of safety before it's applied in the real world.
25


Reviews:
[{"review_id": "hakthc5beD", "reviewer": "Reviewer_C7TV", "summary": "This study introduces an approach to teach Large Language Models (LLMs) to utilize numerous tools through learning embeddings for these tools. The embeddings are incorporated into the LLMs' output layer, invoking the use of tools at relevant steps. The inference process alternates between reasoning and tool-use modes to tackle novel tasks. Empirical results validate the efficacy of ToolkenGPT in utilizing a massive number of tools across various tasks, including numerical reasoning, knowledge-based question answering, and embodied plan generation.", "questions": "- Is it possible to perform instruction tuning using the same training data?\n- Can ToolkenGPT facilitate the zero-shot plug-and-play utilization of new tools?", "limitations": "The authors have discussed the limitations thoroughly.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The proposed method is innovative and scales well to a massive number of tools with limited examples.\n- The method is effective across a variety of tasks and demonstrates advanced multi-turn planning capability.\n- The paper is well-structured and clearly written.", "weaknesses": "- While the authors assert the training efficiency of ToolkenGPT, empirical results reflecting computation costs and performance comparisons are missing.\n- Given the current feasibility of instruction tuning (e.g., Alpaca, Vicuna), it would be beneficial to compare the proposed method with the fine-tuning of the entire model using the same training data. This comparison could clarify if there's a tradeoff between performance and computation cost.\n- Although the authors claim a plug-and-play capability for ToolkenGPT, this feature hasn't been evaluated in the empirical studies. Additionally, if I understand correctly, we still need to train the embeddings for new tools. The authors should clarify this plug-and-play capability in more detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer C7TV for acknowledging that our method is innovative, effective and scales well to massive tools with limited examples. Below we address the reviewer's comments and questions point by point.\n\n(a) Computation costs and performance. Thanks for the great comment! We understand the importance of demonstrating computational efficiency and we present additional results on computation resources and performance in global response (a) and we will include such results in the revised version.\n\n(b) Comparing with full-model fine-tuning. Thanks so much for the great suggestion! Please refer to the global response (a) to see a direct comparison with full-model fine-tuning.\n\n\\(c\\) Clarifying Plug-and-play capability. Thanks for this great clarification question! We made an explanation of the term plug-and-play in the global response, and we will update the paper in the final version to make it clearer.\n\n(d) Feasibility of instruction tuning. We appreciate this great suggestion! Based on our understanding, instruction tuning for tools shares a great overlapping with full-model fine-tuning using the same training data, as explained in the previous response (b). Some minor revisions might include mixing multi-tool data together and changing the prompt format a little bit. Therefore, we believe instruction tuning could be feasible with the same training data, although the computation cost may vary depending on the training methods for the instruction tuning. Further investigation into this direction of tuning the full model on a collection of API instructions is a very promising direction for the future research of tool-augmented LLMs.\n\n(e) Zero-shot plug-and-play for new tools. Thanks for the great suggestion! Please refer to our general response (b) for discussing the \"plug-and-play\" capacity of ToolkenGPT.\n"}, {"review_id": "28peNPY6Y1", "reviewer": "Reviewer_xBHJ", "summary": "The paper presents ToolkenGPT, a framework for extending LMs with tool use. For each new tool, a new token is added to the output vocabulary of the LM and the embedding for that token is trained with annotated or synthetic examples. When the LM generates that token, the model is switched to a different mode and prompted with examples for that particular tool in order to generate the necessary arguments for that tool call. ", "questions": "On line 152, \"and does requires the gradients of LLM parameters\" should presumably be \"and does not require the gradients of LLM parameters\"?\n\nWhen generating the training data, do you do any filtering to make sure that the tool can actually correctly handle the given query? Seems like that would help.\n\nFor most of the tasks it is unclear how many synthetic data examples you generate. Please clarify.\n\nWith in-context learning and 234 tools to describe, are you actually able to fit all the tool examples or descriptions into the context? Or does some of this information simply get truncated? Please clarify.", "limitations": "The potential risks of letting LMs make calls to external APIs, with automatically generated arguments, should also be addressed.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The method seems novel and useful, pushing forward the area of research on extending LM capabilities by teaching them to call external tools.", "weaknesses": "Evaluation on three different types of tasks is great. However, the choice of baselines is the main weakness of the paper. The method seems to use substantial amounts of data for fine-tuning the token embeddings, whereas all the chosen baselines are only restricted to zero-shot or few-shot prompting. Comparison with Toolformer, which fine-tunes the whole model using similar data, would be more fair.\n\nThere are a number of writing errors. Please make sure to proof-read the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer xBHJ for acknowledging that our method is novel and useful, and will push forward the field of tool-augmented LLMs. We appreciate the reviewer's great suggestions and questions. We will further proofread the paper and fix the typos in the final version. Herein, we address the reviewer's questions as follows:\n\n(a) **Choices of baselines**. Thanks for the thoughtful comment! We thank the reviewer for recognizing the coverage of our evaluation and please refer to our global response (a) for following the reviewer's suggestion to compare with Toolformer and fine-tuning a model on similar data. \n\n(b) **Training data generation and filtering**. Thanks for the great clarification question and we apologize if the process of generating training data is not very clear in the experiment section. As shown in the appendix, for math datasets, we prompt ChatGPT to produce questions and their corresponding answers. Instead of giving specific numbers, we instruct the model to use placeholders that follow our particular format. We retain only those cases that are parsable and align with our format requirements. Afterward, we programmatically assign random numbers and compute the results.\n\nFor example, \n```\nQuestion: If the price of a stock increases by 10% every day, by how many times will its initial value increase after [ARG_0] days?\nAnswer: After [ARG_0] days, the value of the stock will increase by a factor of 1.1^[ARG_0]=<power>(1.1, [ARG_0])=[ANSWER] \nArgs: ['int']\n```\n(c) **Number of synthetic data examples**. Regarding the training samples we synthesized, for the math dataset FuncQA, we synthesize 50 examples for each operation; for KAMEL, we synthesize at most 50 examples per relation (some examples are not successfully parsed and removed from the dataset, leading to 40 examples on average per relation).\n\n(d) **In-context learning on 234 tools**. Thanks for the question! It would be infeasible to fit all tool examples into the context. As indicated in Section 4.2 and Figure 2, we tried our best to fit as many tools into the context, reaching the 2048-tokens limit of LLaMA, which is about . The detailed mechanism of this baseline will be clarified in the revised version.\n\n(e) **Discussing potential risks of connecting LLMs with external APIs**. We appreciate this great comment on the potential risks of tool-augmented LLMs. We believe addressing such a potential risk requires community-wide efforts. We will delve deeper into this issue in the future work section of our revised paper.\n"}, {"review_id": "eEiZID33be", "reviewer": "Reviewer_Mpoh", "summary": "This paper presents ToolkenGPT, a novel approach to generalizing LLMs to massive external tools. The key is to represent each tool as a \"token\", such that when LLMs generate the tool token(toolken), the associated tool will be invoked and executed. The learning of toolken is easy and efficient since there is no need to update the whole LLMs or backward the gradients through LLMs. Toolkens are appended at the last layer of the LLMs, making the cost similar to just doing inference. The authors have conducted extensive experiments and compared to proper baselines. The results are convincing.", "questions": "NA", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The proposed method could make a good contribution to tool learning. In particular, the use of toolken is a novel yet elegant way to solve the massive tools adaption and generation problem.\nThe proposed method is efficient, yet to learn and extend. \nThe paper is clearly written and easy to follow.\nThe evaluation is comprehensive and convincing.", "weaknesses": "It would make the evaluation stronger if the authors can include some of the following experiments: \n\n1. Lacking comparison with fine-tuning-based methods, e.g., toolformer. In Numerical Reasoning tasks, I believe these methods could work quite well.\n\n2. The target is the massive tool-learning scenario,  it would be great to try some tasks that involve multiple quite different tool calls at the same time, e.g., qa and calculator. Although the experiments on VirturalHome are great, these 58 tools are mostly similar.\n\n3. Testing whether the training and inclusion of toolken will affect the language generation capacity, i.e., adding some NLU tasks to see if there is any performance drop.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer Mpoh for acknowledging that our proposed method is novel and elegant to learn massive tools efficiently, and makes a good contribution to the field of tool learning. We address the reviewer's comments on the evaluation as follows:\n\n(a) Comparing with full-model fine-tuning. Thanks for the great comment! For the comparison with Toolformer, please see our global response (a).\n\n(b) **Consider different tools simultaneously**. Thanks for the great suggestion and we appreciate the interesting setting pointed out by the reviewer. We initially focused on demonstrating the potential of ToolkenGPT in a scenario where a set of task-specific tools are used to solve one single task. As the reviewer mentioned, we've tried our best to increase the number and the diversity of such tools, i.e., 13 tools in math, 234 tools in KBQA, and 58 tools in VirtualHome. Since these tools are created for similar goals in specific tasks, they somehow are related to each other due to the nature of the evaluated tasks. Considering distinct tools, like QA and calculator, would require creating more challenging scenarios and tasks. We believe that such a set of experiments involving multiple, diverse tool calls could further highlight the flexibility and capability of ToolkenGPT, which serves as a realistic, multi-tool scenario for future works. \n\n(c) **Influence of Toolken on LLM generation**. Thanks for such an interesting comment! The potential influence on language generation is an interesting angle to discuss further. First of all, the training of toolken won't affect the generation capability of LLMs, as it doesn't change any pre-trained parameters for the generation. Second, for some NLG tasks like the summarization task that don't require external tools, the pure generation capacity from LLMs is enough, and thus, we can easily disable the learned tools by masking their toolkens in the output vocabulary, keeping the original generation capacity intact. This serves as a demonstration of the plug-and-play capacity of ToolkenGPT, as explained in the global response (a). \n\nLast but not least, in the most generic setting where we just keep the learned toolken embeddings and simply prompt the LLM for a generation task, we conducted an additional experiment to test the influence of toolken embeddings on NLG tasks. Specifically, we leverage the trained toolkens from math datasets and apply ToolkenGPT to two data-to-text generation datasets, WebNLG [1] and E2E [2]. We performed 10-shot learning (_randomly picked_) on the dev set of the WebNLG dataset (872) and the test set of the E2E dataset (630). We report the traditional NLG metrics, ROUGE-2 score from Huggingface evaluate and use greedy decoding for the generation.\n\nRTable 3: Performance of ToolkenGPT on NLG tasks.\n\n| Dataset      | Llama-30B | Llama-30B w/ toolkens | \\#Func Calls |\n|--------------|----------|-----------------------|------------|\n| WebNLG      | 0.56     | 0.56                  | 0          |\n| E2E       | 0.46     | 0.46                  | 0          |\n\nRTable 3 shows that the inclusion of toolkens does not have any influence on the generation capabilities of LLMs on these two NLG tasks. As we can see from RTable 3, toolkens are not activated during the generation with the number of function calls as zero. However, in rare cases where the generation tasks are very similar to the toolken training text, the LLM may mistakenly call external tools, which is an interesting problem to investigate in the future. Nonetheless, we hope these additional results could further clarify the impact of toolken embeddings on NLG tasks and we will include them in the revised version.\n\n**References**:\n\n[1] Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. The WebNLG challenge: Generating text from RDF data. ICNL, 2017.\n\n[2] Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. The E2E dataset: New challenges for end-to-end generation. arXiv preprint arXiv:1706.09254, 2017.\n"}, {"review_id": "vhlcTnHz7Y", "reviewer": "Reviewer_5YmU", "summary": "This paper proposes a new method to help LLM utilize tools. Essentially, the method trains an additional output embedding token (toolken) for each tool. During inference, if LLM outputs a toolken, a special routine of using the corresponding tool is triggered and the returned result of the tool is then replaced with the toolken for continual inference. As a result, the proposed method can accommodate more tools and unfamiliar tools, compared to in-context prompting approaches like ReAct. The author has shown the effectiveness and advantage of the proposed method across multiple domains including math reasoning, knowledge based QA and embodied AI.", "questions": "1. The paper did not provide empirical comparison between ToolkenGPT with finetuning approach like Toolformer and TALM. Though you mentioned that ToolkenGPT requires much less GPU memory than those methods, is it possible to provide some empirical comparison to those methods?", "limitations": "The author has adequately discussed the limitations.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Compared to full model finetuning like Toolformer or TALM, the proposed method is much more efficient as it doesn't require gradient of LLM parameters and only tunes an additional output embedding matrix. \n2. Empirical results on multiple domains verified the advantage of ToolkenGPT over ReAct, as it can better accommodate more tools.\n3. ToolkenGPT is easy to implement and is able to be applied for various domains, which can potentially foster more research on the direction of enhancing LLMs with tools.", "weaknesses": "1. several typos are present: a) line 2, missing period after \"problems\"; b) line 92, missing space \"whichhelps\"; c) line 204, versatile *and* adaptive; d) line 234, 503.2 --> 50*3.2\n2. One limitation is ToolkenGPT requires access to model activations thus not capable of being applied on closed-sourced LLMs like ChatGPT, while pure prompting method like ReAct doesn't have such limitation. As such, it's desirable to compare with a baseline of ReAct w/ ChatGPT. However, in the experiments, the author only compare with ChatGPT with 0-shot prompting (Table 2)\n3. There are two parts of the proposed methods: toolken embedding and sub-routine of tool call. It's desirable to show some ablation of the latter part. For example, show a baseline of ReAct + sub-routine of tool call with demonstrations. In this way, readers can better assess the benefit of each component of proposed method", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 5YmU for acknowledging that our proposed method is much more efficient, easy to implement, and can be applied to various domains to foster more research in the field of tool-augmented LLMs. We also thank the reviewer for spotting several typos and we will fix them all in the revised version. Below we resolve the reviewer's concerns and questions.\n\n(a) **Baseline of ReAct w/ ChatGPT**. We understand the reviewer's concern regarding the potential limitation of ToolkenGPT in its inability to be applied on closed-sourced LLMs like ChatGPT. From the perspective of the open-sourced community, we agree that a comparison with a baseline of ReAct w/ ChatGPT would provide a broader context and help further validate the efficacy of our method. As shown in Table 2, we already compare our method with ReAct but based on the same based model (LLaMA-30B), mainly for a fair comparison. As ChatGPT is a much stronger base model than LLaMA-30B, applying ReAct to ChatGPT is expected to boost the performance. However, our ToolkenGPT should also be benefited from stronger open-sourced LLMs, like the recent Llama 2 models, which we believe should be an interesting future work to explore.\n\n\n(b) **Ablation study**. Thanks for the great suggestion! To better assess the benefit of each component of ToolkenGPT, we conduct an ablation study for the two main components of our method - toolken embedding and sub-routine of tool call (tool mode) as follows. \n\nRTable 4: Ablation study on tool embeddings and tool mode.\n\n| Method | FuncQA (one-hop) | FuncQA (multi-hop) |\n| -------- | -------- | -------- |\n| ReAct     | 0.57     | 0.06     |\n| ReAct + tool mode     | 0.60     | 0.07     |\n| ToolkenGPT | 0.73 | 0.15 |\n\nRTable 4 is closely related to Table 2 from the paper, where we use a hard math dataset, FuncQA with 13 math tools. We further implement a baseline of combing ReAct prompt and the sub-routine of tool call (tool mode) from our ToolkenGPT. As shown in this table, the tool mode used in ToolkenGPT could help improve the correctness of tool calling and marginally improve the vanilla ReAct prompting method. We also see that ToolkenGPT still outperforms this improved baseline by a large margin, indicating that our toolken embeddings still outperform ReAct greatly in terms of deciding when and which tool to call. We hope these additional results could help provide clear evidence of the contribution of each component to the performance of ToolkenGPT.\n\n\n(c\\) Comparing with fine-tuning approaches. Please refer to the global response (b) for addressing the comparison with full-model fine-tuning methods, like Toolformer."}, {"review_id": "ZJd46ts4Tx", "reviewer": "Reviewer_QLCo", "summary": "The authors propose \"ToolkenGPT.\" This method allows language models to take advantage of external tools without full model fine-tuning, and with the ability to show more tools and with greater detail than in-context learning typically allows. They show that use of \"toolkens\" (tool tokens) improves performance over baselines on arithmetic, knowledge-base QA, and embodied plan generation.", "questions": "See weaknesses", "limitations": "The authors have a nice discussion of limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In terms of originality, the paper introduces a new and innovative approach to external tool access for language models. It seems like a hybrid of fine-tuning and few-shot learning, but uses both in a novel and interesting way. Though there's not much direct comparison to prior work, it is at least acknowledged.\n\nIn terms of quality, the experimental setup is sound. It seems evident that ToolkenGPT performs better than simple (but well designed) baselines.\n\nIn terms of clarity, the paper is well-written and easy to follow. I haven't read the appendix in detail, but it seems to provide enough information for reproducing the experiments.\n\nIn terms of significance, this work brings up new ideas that seem very useful.\n", "weaknesses": "* The biggest weakness of this paper is in evaluation.\n    * ToolkenGPT is not directly compared to prior work. It is only compared to things along the lines of zero-shot or few-shot baselines. At the end of the paper I'm left wondering how ToolkenGPT would stack up against e.g., a full model fine-tuning method like Toolformer. I can see that ToolkenGPT allows for demonstrating more tools with more detail than few-shot. It also seems like it's probably cheaper to use ToolkenGPT than to fine-tune. But I'm not really sure of how accuracy for the two tuning methods compare.\n    * In evaluation there are sometimes confounding variables that render results less useful. For example, in Section 4.2 5x more data is used for training ToolkenGPT (sup) than ToolkenGPT (syn). Is the improved performance due to more training samples or higher data quality?\n    * Evaluation sometimes lacks specificity. For example, in Section 4.2 it's unclear whether the 30 relations ICL has access to in the e.g., 234 relation case are required to include the correct relation. That seems like a very important detail.\n    * Overall this paper covers an impressive breadth of evaluation, but I wish there was one in-depth evaluation where # of tools, model size, and number of training data points were carefully considered for strong few-shot and fine-tuning (e.g., fine-tuning on all the exact data used to train the toolkens) baselines along with ToolkenGPT while carefully controlling for other variables.\n* I think the comparison between fine-tuning, few-shot, and ToolkenGPT could cleaner\n    * I'm not sure of the difference between \"plug-and-play\" and \"frozen LM\" in Table 1. I wouldn't say ToolkenGPT is \"plug-and-play\" given the model-specific training required for embeddings.\n    * I'm not sure what is really meant by \"massive tools.\" What is stopping a fine-tuning method from learning such tools?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviwer QLCo for acknowledging that our approach is new, innovative and interesting, brining up new and useful ideas to the field. Below we address all the reviewer's concerns and questions.\n\n(a) Comparing with full-model fine-tuning. We appreciate that the reviewer acknowledge that our proposed ToolkenGPT allows for more tools with more details than few-shot and is much cheaper than fine-tuning. We agree that a comaprison with full-model fine-tuning would bolster our evaluation and help clarify the unique advantages of our method. We explicitly address such an important comparison in the global response (b). \n\n(b) **Training data for ToolkenGPT (sup) and ToolkenGPT (syn)**. We appreciate the great observation from the reviewer and provide further experiment results to investigate the influence on training sample quantity and quality. Specifically, we sample 10/20/40 training examples of each tool for both ToolkenGPT (sup) and ToolkenGPT (syn), and report the accuracy on the 30-tools testset.\n\nRTable 5: Influence of training quantity and quality on ToolkenGPT.\n\n| # Examples per tool | ToolkenGPT (syn) | ToolkenGPT (sup) |\n| -------- | -------- | -------- |\n| 10     | 0.364     | 0.564     |\n| 20     | 0.464     | 0.896     |\n| 40     | 0.524     | 0.948     |\n\nRTable 5 shows the performance of ToolkenGPT (sup) and ToolkenGPT (syn) under the same amount of training data. As we can see, both the size and source of training data matter here, and the supervised data clearly contribute to the higher performance in the Figure 2 of the paper. We will include such ablation study into the revised version and we hope this could clarify the question from the reviewer. \n\n(c\\) **Tool selection for ICL baseline**. We appreciate the reviewer for mentioning this very important detail. To illustrate this process clearly, let's index all the relations from 1 to 234. As explained in Line 273 of the paper, we have 4 datasets that involve the relations 1-30 (30-relation dataset), 1-60 (60-relation dataset), 1-100 (100-relation dataset), 1-234 (234-relation dataset), respectively.\n\n- The prompt for `ICL` includes the descriptions and the demonstrations of relations 1-30.\n- The prompt for `ICL (desc)` inclues the descriptions of relations 1-30 (for the first dataset) or 1-60 (for other datasets), and demonstrations of 8 random relations. \n\nIt's worth noting that, even in the cases where `ICL` / `ICL (desc)` has access to every helpful tools for the dataset, i.e., for the 30-relations dataset, the performance is still inferior to ToolkenGPT (see Figure 2, datapoints in the first column). This indicates that even if the context length limit is resolved, in-context learning methods could still struggle to choose from massive tools. We hope this will clarify the question from the reviewer and we will clarify this detail clearly in the revised version.\n\n(d) **Comparison between fine-tuning, few-shot and ToolkenGPT**. We thank the reviewer for brining up this clarification question. First, we refer the reviewer to the global response (a) for a comprehensive comparison with fine-tuning. We also address the comment on the \"plug-and-play\" capaciy of ToolkenGPT in the global response (b). To sum up, we understand the reviewer's concern on the overlapping between \"frozen LMs\" and \"plug-and-play\". The aspect of \"frozen LMs\" in our paper focuses on the training efficiency, while \"plug-and-play\" is more challenging and requires more elegant design of parameters/prompts to reuse previous tools and plug them into new tasks. \n\nRegarding the definition of \"Massive tools\", we mean a large collection of tools that our model can learn and utilize. We agree that fine-tuning methods can learn such tools, but the innovation in our work is the ability to learn them with quick adaption and without exhaustive fine-tuning. When massive tools are presented, fine-tuning requires more training data (because we have more parameters to optimize) and more computing resources, which is costly to obtain, while our ToolkenGPT can \"scales well to a massive number of tools with limited examples\" mentioned by R-C7TV. More importantly, an implicit characteristic of \"massive tools\" is the dynamics of tools such that new tools are kept invented and updated every day. Thus, the growing massive tools make fine-tuning prohibitive to update model parameters frequently. We hope such a more detailed explanation could resolve the reviewer's concern and we will definitely clarify such a comparison more clearly in the revised version."}]

Author rebuttal:
We are grateful to all reviewers for their detailed and constructive feedback! We are encouraged to see that reviewers find:
- Our proposed ToolkenGPT is an "innovative, novel and useful", "a novel yet elegant way" (R-C7TV, R-xBHJ, R-QLCo, R-Mpoh), which "brings up new ideas", "fosters more research" and "pushes forward the area of research" on tool-augmented LLMs (R-xBHJ, R-5YmU, R-QLCo);
- ToolkenGPT is "efficient, yet to learn and extend" (R-Mpoh), offers a "much more efficient" method (R-5YmU) that "scales well to a massive number of tools with limited examples" (R-C7TV);
- Our "experimental setup is sound" with "well-designed baselines" (R-QLCo), our "empirical results on multiple domains" verify its advantage (R-5YmU), "the evaluation is comprehensive and convincing" (R-Mpoh)
- ToolkenGPT "is easy to implement and is able to be applied for various domains" (R-5YmU), and we "provide enough information for reproducing the experiments" (R-QLCo)

We have addressed all the questions raised by reviewers with additional experiments and thorough clarifications via separate responses to each reviewer. There are two common concerns we would like to address in the global response.

(a) **Comparison with full-model fine-tuning**. As shown in Table 1 from the paper, there are many advantages of ToolkenGPT in comparison with full-model fine-tuning, including training efficiency, quick adaptation to new tools, scaling to massive tools with limited examples, etc. While most reviewers agree on the above advantages, it is also interesting to compare with the full-modal fine-tuning, in terms of computation efficiency and performance. We implement different methods with Llama-7B on FuncQA, and show the comparison in RTable 1.

RTable 1: Comparison between ToolkenGPT and finetuning (using LoRA) in terms of training cost and performance on FuncQA dataset (using Llama-7B).

| Model           | One-hop  | Multi-hop | Computing Resource  | Training Time (all) |
|-----------------|---------|----------| ---|---|
| Fine-tune w/ LoRA |   0.62 |     0.074 | 1 \* A100 (80G) | ~40 min |
| ToolkenGPT |   0.55  |     0.058 |  1 \* RTX3090 (24G) | ~ 2 min |
| ReAct | 0.40   |  0.030  | - | -  |
| Baseline | 0.10  |  0.00    | - | - |

Though finetuning performs better than ToolkenGPT, it's significantly more expensive. Even though we use LoRA (parameter-efficient finetuning) and more computing resource (e.g., GPU memories, to increase the batch size), finetuning still costs about 20× more time than training toolken embeddings.

We also compare ToolkenGPT and Toolformer [1], a recent full-model fine-tuning method, in terms of efficiency on the learning of mathematic tools. Note that it's infeasible for us to compare performance, as Toolformer is not open-sourced and is based on GPT-J, a different base LM than ours (Llama). It's also computationally infeasible for us to reproduce Toolformer, given the extreme computational cost as shown in the table below.

RTable 2: Comparison between ToolkenGPT and Toolformer (math).

| Method | Computing Resource | Number of Training Examples |Time (per epoch)|
|---|---|---|---|
| ToolkenGPT (30B) | 4 \* RTX3090 (24GB) | 5k | ~16min |
| ToolkenGPT (13B) | 2 \* RTX3090 (24GB)| 5k | ~7min |
| ToolkenGPT (7B) | 1 \* RTX3090 (24GB)| 5k | ~5min |
| ToolFormer | 8 \* A100 (40GB) | 25k (only for math) | Unknown |

RTable 2 presents the comparison of computing resources and time for training our ToolkenGPT and Toolformer. It's clearly evident that our ToolkenGPT can be trained with much less computing resources, compared with the full-model fine-tuning method, Toolformer. Note that Toolformer's training time is not reported in the original paper.

(b) **Plug-and-play capacity of ToolkenGPT**. As mentioned by R-C7TV and R-QLCo, the definition of "plug-and-play" in Table 1 of the paper could be further clarified. By “plug-and-play”, we mean that new toolken embeddings are disentangled with other parameters, and can be enabled/disabled or directly added to an existing tool set effortlessly. The "plug-and-play" defined in the paper doesn't mean there is no need to train the embedding, but refers to that we are able to reuse the trained toolken embeddings and plug them into any downstream settings flexibly when needed. We will clarify this term in the updated paper.

Besides, it's also possible to train toolken embeddings without preparing training data manually. As demonstrated in Section 4.1 (FuncQA) and 4.2 (ToolkenGPT-syn), we can synthesize the training data for new tools. We can observe that ToolkenGPT (syn)---no in-domain training data is used---outperforms strong CoT baselines.

**References**

[1] Schick, Timo, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. "Toolformer: Language models can teach themselves to use tools." arXiv preprint arXiv:2302.04761 (2023).

[2] Miao, Shen-Yun, Chao-Chun Liang, and Keh-Yih Su. "A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers." In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 975-984. 2020.

Meta-review: The authors propose ToolkenGPT, an approach to teach language models to use tools that requires no modifications of the underlying parameters of the LLM. Essentially, separate embeddings (tool tokens -> toolkens) are learned in the output layer of  the LLM. These separate embeddings, when used in generation, invoke tool use. The reviewers have a consensus that this paper is innovative, scales well for use with many tools, and they like the efficiency and modularity of the approach. There were some concerns about how the approach fares against fine-tuning, which was addressed by the authors. Essentially, the model outperforms zero-shot and few-shot baselines handily, but is not quite as performant as fine-tuning. This makes sense, given that the underlying LLM parameters are frozen. However, the benefits of ToolkenGPT are that it can be easily extended to new tools, is much cheaper to adapt (since underlying LLM parameters can remain fixed, although their gradients do need to be estimated to update the too lembeddings), and moreover, does not harm the generation quality of the LLM through catastrophic forgetting. All reviewers agree that the paper makes a solid contribution to the field, and we hope that the reviews and discussions during the rebuttal paper make it into the final version of the paper (i.e. more detailed discussion of fine-tuning, terminology change of “massive tools” -> “massive amounts of tools”, clarifications about what is meant by “plug-and-play”, etc.).
Decision: Accept (oral)