SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Comparison methods. We train toolken embeddings for each available math operator as described in Section 3.2. During inference, we prompt the LLM with 4-shot Chain-of-Thought [65] examples to enhance the reasoning ability of LLMs. The following baselines are evaluated for comparison: (1)  $0$ -shot CharGPT is the straightforward method asking LLMs to answer a question. No examples will be provided in the context and tools are not available. We use ChatGPT as the base LLM in our experiment. This baseline measures the ability of the LLM to answer complex numerical reasoning problems with its own reasoning and calculation ability. (2) Chain-of-thoughts (CoT) [65] is a more advanced prompting techniques. In this approach, a series of interconnected prompts are carefully crafted to guide the LLMs through a step-by-step reasoning process. The example reasoning chains are the same as the ones we used for ToolkenGPT, but no functions are available. (3) ReAct [69] combines reasoning and tools by prompting the LLMs to generate verbal reasoning traces and tool calls in an interleaved manner. Concretely, instead of just providing reasoning chains such as "... The cost is  $50*3.2 = 160$ ", ReAct incorporates special syntax to call operators, e.g. "... The cost is  $50*3.2 = <\text{multiply}>(50,3.2) = 160$ ". Once the syntax is detected during inference, the tool would be called to calculate the result. We use the same reasoning chain examples as in both CoT and ToolkenGPT, with only slight differences in the tool calling syntax. LLaMA-33B [62] is used as the LLM for all settings other than zero-shot prompting. More experiment details are described in Appendix A.
Result analysis. Table 2 shows the performance of all the methods on the GSM8K-XL and FuncQA datasets. On the GSM8K-XL dataset, 0-shot ChatGPT and few-shot learning with CoT struggle to calculate large numbers without the help of tools, while ReAct and ToolkenGPT manage to increase accuracy consistently by a large margin. Generally, both methods can call the correct tools when necessary, as the toolset is comprised of only the four basic operators. However, for both FuncQA one and FuncQA multi datasets, learning to call applicable tools becomes challenging to ReAct as the
number of tools increases. In ReAct, though all the tools are listed at the beginning of the prompt, it is infeasible to include demonstrations of every tool in the limited context (In our experiment, we provide 4 examples including 5 tool demonstrations). As a result, ReAct is susceptible to missing tool calls, making wrong tool calls, and predicting wrong arguments, especially for the tools not demonstrated in context. ToolkenGPT outperforms all the baselines across both one-hop and multihop scenarios, showing superior tool learning ability when there are numerous tools. It is important to note that even though toolken embeddings are trained solely using one-hop synthetic data, and without any CoT examples, they still manage to enhance performance in multi-hop problem contexts and can be integrated effectively with CoT prompting. This implies a degree of generalization of toolken embeddings, which is a very desired property that lowers the requirements of in-domain training data.
4.2 Knowledge-based Question Answering
LLMs are known to often make factual errors and hallucinate [28, 73, 72, 1] because of their limited knowledge [20]. Equipping them with access to knowledge bases (KBs) has been a promising research direction to reduce their hallucinations [58]. We formulate the access to the KB as APIs querying the database [60, 16]. Thus, each relational query can be treated as a tool to which the input argument is a subject entity, and the output is the corresponding tail entity. An example tool call is "P1346(2005-06 FA CUP)  $\rightarrow$  LIVERPOOL F.C." "P1346" is a relation identifier in Wikidata, representing the winner of a competition or similar event (referred to winner_of below for ease of reading). In this section, we show that ToolkenGPT can accurately query a large knowledge base of up to 234 tools (relations). We further show that even only with synthetic data (as described in Section 3.2 and explained below), we can train strong toolken embeddings that outperform popular tool-learning methods.
Dataset. KAMEL [30] is a question-answering dataset built with the facts in Wikidata. In line with ToolFormer [56], which uses its earlier version [51] as a benchmark to evaluate the tool use, we adopt KAMEL to evaluate the use of KB query tools. KAMEL contains knowledge about 243 relations from Wikidata, each of which is associated with a question template (e.g. winner_of: "Who is the winner of [S]?" to turn a fact in Wikidata into a question. We have 234 tools in total for this dataset. In order to analyze the performance provided with different numbers of tools, we create four subsets by sampling from the original test set. Each subset consists of questions related to different numbers of relations, corresponding to 30, 60, 100, and 234, respectively. The size of each subset is 500.
Comparison methods. We set up two different variants of our framework. (1) ToolkenGPT (sup): We sample 200 examples per relation from the training set of KAMEL and train the toolken embeddings via supervised learning. This setting represents real-world scenarios where sufficient in-domain training data is available. (2) ToolkenGPT (syn): In a more challenging setting where we assume in-domain training data is not available, we use the text description of each relation to synthesize training data with ChatGPT, e.g. "The Nobel Peace Prize in 2020 was awarded to the United Nations World Food Programme for its efforts...", where the underlying tool call is winner_of(NOBEL Peace PRIZE IN 2020)→UNITED NATIONS WORLD FOOD PROGRAMME. On average, 40 examples are used to train each toolken embedding.
We introduce the following baselines for comparisons: (1) Prompting [30] is a straightforward method that answers the questions with the LLM's internal knowledge. We frame each question within the prompt "Question: [QUES-
Figure 2: Performance of ToolkenGPT and baselines on 4 testsets involving different numbers of tools (relations) from KAMEL. ICL is short for Incontext Learning [53]. Due to the context length limit of 2048 tokens, we list the descriptions and demonstrations of up to 30 relations for ICL and up to 60 relation descriptions for ICL (desc).
TION\nThe answer is" and ask the LLM to continue the sentence. (2) In-context Learning (ICL) [53] is a standard method to augment LLMs with tools as introduced in Section 2. Before asking the
question, we list the tool demonstrations and descriptions of all available tools. The demonstrations are shown in a specific syntax so that the LLM can generate in a similar style to be parsed. An example demonstration for winner_of is "Question: Who is the winner of 2005-06 FA Cup?nAnswer: The answer is <winner_of>(2005-06 FA Cup)=Liverpool F.C." In a recent survey [53], this setting is referred to as "few-shot". (3) In-context Learning (desc) [53] is another common practice to augment LLMs with tools. The descriptions of all available tools will be provided in context, but their demonstrations are not directly shown. Instead, we show 8 demonstrations of the tools not included in the test subset to inform LLMs about the tool call format. This setting is referred to as "zero-shot" in Qin et al. [53]. The base model for all methods is LLaMA-13B [62]. More experiment details are described in Appendix B.
Result analysis. We show the experiment results on 4 testsets involving different numbers of relations in Figure 2. Note that the number of involved relations is the number of tools we can use. For all testsets, the accuracy of Prompting is about  $20\%$ , which indicates LLMs still struggle to store accurate facts in their parameters and it's necessary to augment them with a knowledge base. ToolkenGPT (sup) achieves the highest results with a large margin, showing that learning toolken embeddings is an effective method when there is massive in-domain training data. On the contrary, even though In-context learning also sees in-domain training data in the context, it still gets confused about which tools to call. Furthermore, the context length limit leads to drastic performance drops when there are more than 30 tools to use. The failure in the many-tools scene reveals the fundamental limitation of the in-context learning paradigm. ToolkenGPT (syn) also outperforms all other baselines in all subsets, without seeing any in-domain training data. The synthetic training data, often in very different expression styles from the dataset, still helps the LLM understand these relations.
This success reflects the flexibility of our framework which can be applied even if there is no indomain training data available. In-context learning (desc) generally fails in this task, because the LLM has difficulties memorizing text descriptions shown in contexts and mapping them to relation identifiers. The results provide more evidence to the previous discovery that LLMs have trouble using unfamiliar tools [6]. Based on this observation, it is reasonable to speculate that LLMs mostly recall the tools from their identifier instead of really learning to use tools from their descriptions.
4.3 Embodied Plan Generation
Recently, there have been many research attempts to utilize LLMs as the controller of embodied agents [25, 59, 4, 27, 66]. Despite the preliminary success of prompting LLMs, teaching LLMs about an environment and enabling them to make grounded predictions remain challenging. As discussed in Mialon et al. [45], tools that gather additional information (e.g. math or KB tools) and tools that have an effect on the physical world (e.g. actions taken by embodied agents) can be called in similar styles by the LLM. In this section, we demonstrate how our framework can also be applied to plan generation for embodied agents. Compared to previous methods that prompt LLMs, our ToolkenGPT can understand the environment better by learning toolken embeddings for agent action and object.
Dataset. VirtualHome [52] is a simulation platform for typical household activities, and ActivityPrograms knowledge base [52] consists of many tasks with plans executable in VirtualHome. We derive a subset of 297 tasks from ActivityPrograms.
Specifically, for each task, the model is given a high-level goal (e.g. "Read book"), a detailed instruction (e.g. "I would go lie down in my bed and open the book and start reading.", and a description of the environment, which includes the initial state of the agent, and the object list of the environment (e.g. "I am in ['home_office']". The objects I can manipulate are ['mail', 'freezer', 'television', ..., 'novel']". The model is expected to output an executable plan, which is an ordered list of verb-object instructions (e.g. "[FIND] <novel>"). Each task comes with an initial and final state graph, enabling the verification of the generated plans with the simulator and the comparison of the resulting final state with ground truth. We split the dataset into a training set of 247 tasks and a test set of 50 tasks, with a total of 25 verbs and 32 objects used in the dataset.
Comparison methods. We consider all the actions and objects in VirtualHome as tools. With an additional [END] function indicating the end of a plan, we have 58 toolkens in total. For this dataset, we do not need the argument generation process described in Figure 1 because the tools do not take arguments. During inference, ToolkenGPT alternatively generates action toolkens and object toolkens, and ends with the [END] toolken. The toolken embeddings are trained on the training set.
Table 3: Results on VirtualHome. Grounding means the proportion of scripts in which all the actions and objects can be grounded to the environment. Executable means the proportion of scripts that can be executed in VirtualHome without violating any rules. Success means the proportion of scripts that leads to the correct final state. Success (R) is a relaxed variant meaning the proportion of scripts that have reached the correct final state, but not necessarily ending with it.
Method	Grounding	Executable	Success	Success (R)
In-context Learning	0.74	0.42	0.20	0.30
+ Translation [25]	1.00	0.52	0.24	0.32
+ Grounded Decoding [27]	1.00	0.66	0.38	0.42
ToolkenGPT (Ours)	1.00	0.82	0.68	0.70
We compare our method to the following baselines: (1) In-context Learning prompts the LLM and parses its outputs as the plan. The LLM is shown with the action list, 3 demonstration plans, and a new task with its goal, detailed description, and environment description. This method is the base of most recent methods [25, 4, 27] that apply LLMs to embodied AI. (2) Translation [25]: To avoid plans that include unavailable actions or objects, Huang et al. [25] proposes to use a translation model to translate the LLM's generation to admissible instructions. Following Huang et al. [25], we use SentenceRoBERTa-large [54] and translate the actions or objects to available ones with the highest cosine similarities. (3) Grounded Decoding [27] is a recent decoding-stage grounding method. The next token is predicted considering both LLM logits and "grounded functions". Specifically, we apply the affordance grounding function [27], encouraging LLMs to generate valid actions and objects. We do not consider other previous methods that heavily fine-tune the whole language model [37]. The base model of all methods is LLaMA-13B [62]. More experiment details are described in Appendix C.
Result analysis. We list results in Table 3. Though all valid actions and objects are explicitly listed in the context for the LLM using In-context Learning, it sometimes fails to ground its prediction to admissible instructions. Even though the actions and objects are valid, they often violate the physical rule in VirtualHome, resulting in a low success rate. We notice that while most of the plans generated with Incontext Learning appear reasonable to humans, they are not grounded to the specific environment of VirtualHome. Translation [25] helps solve some shallow grounding problems, e.g.  $[\mathrm{tv}]\rightarrow [\mathrm{television}]$ , while Grounded Decoding [27] further improves executable and success rate by considering grounding earlier in the decoding stage. Although these methods ensure
all plans are grounded, neither significantly improves the LLM's understanding of actions and objects, leading to unsatisfactory executable and success rates. ToolkenGPT not only predict valid actions and objects naturally by its design, but also achieves the highest success rate by learning toolken embeddings from more training tasks. A concrete example is shown in Figure 3 to illustrate the difference: All the baselines predict [SIT] <desk>, presumably guided by the description "sit at desk", but in VirtualHome [SIT] refers to "sit on", and a desk is regarded as not sittable. ToolkenGPT is the only one to successfully learn this rule from demonstrations and instead predict [SIT] <chair>.
Computational Cost. We conduct experiments to compare ToolkenGPT with fine-tuning, specifically using LoRA [23], in terms of computation efficiency and performance. Due to the cost of fine-tuning LLMs, we implement both methods on LLaMA-7B. The results are listed in Table 4.
Fine-tuning LLMs results in slightly better performance than ToolkenGPT on FuncQA. Even though we apply LoRA, which is known for efficiency, the time consumption for fine-tuning exceeds
Go to office, sit at desk, turn on computer, enter password, open application and begin work
Figure 3: Case study on VirtualHome. ToolkenGPT predicts a successful script while other baselines fail to produce an executable one due to their misunderstanding of the SIT action.
Table 4: Comparison between ToolkenGPT and fine-tuning (LoRA) in terms of training cost and performance on FuncQA dataset. Both methods are based on Llama-7B.
Method	One-hop	Multi-hop	Computing Resource	Training Time
ReAct	0.40	0.03	-	-
Prompting	0.10	0.00	-	-
Fine-tune w/ LoRA [23]	0.62	0.07	8 × A100 (80G)	40 min
ToolkenGPT	0.55	0.06	1 × RTX3090 (24G)	2 min
Table 5: Ablation study on FuncQA dataset with LLaMA-30B.
Method	One-hop	Multi-hop
ReAct	0.57	0.06
+ Tool mode	0.60	0.07
ToolkenGPT	0.73	0.15
Table 6: ToolkenGPT with different configurations of training data on KAMEL.
# Examples	Synthetic	Supervised
10	0.36	0.56
20	0.46	0.90
40	0.52	0.95
significantly when compared to training toolken embeddings. It is also worth noting that ToolkenGPT enjoys additional benefits other than efficiency (Table 1), especially the plug-and-play of massive tools, thanks to the decoupled parameters for different tools.
Ablation Study. The design of ToolkenGPT benefits both tool selection and argument completion (tool mode in Figure 1). To understand their respective contributions to the performance, we further implement a baseline combining ReAct-style prompting and the sub-routine of argument completion (tool mode). In the tool mode, the LLM is prompted with demonstrations using only the selected tool, which will provide more relevant knowledge than ReAct prompt for argument completion. As shown in Table 5, adding a tool mode could indeed improve the vanilla ReAct prompting method by enhancing the accuracy of argument completion. However, ToolkenGPT still outperforms this improved baseline by a large margin, indicating that toolken embeddings effectively help LLMs to decide when and which tool to call.
Training Data. In this section, we explore the effects of training data on learning the toolken embeddings. We choose to extend our experiments on KAMEL (Section 4.2), because there are two different sources of training data and it is easy to process or synthesize more data. Specifically, we sample 10/20/40 training examples of each tool for both ToolkenGPT (sup) and ToolkenGPT (syn), and report the accuracy on the test set involving 30 tools.
The results are summarized in Table 6. Under the same budget of data size, training with supervised data leads to better performance. Though we do not observe obvious mistakes in most of synthetic data instances, the distribution gap between synthetic data and test set may prevent toolken embedding from performing well. A larger training set benefits the performance for both data sources.
We presented ToolkenGPT, a new approach for augmenting frozen LLMs with massive external tools without expensive fine-tuning. Our method introduces the concept of toolken embedding for each tool, enabling LLMs to call and use different tools as easily as generating word tokens. Our approach overcomes the limitations of current fine-tuning and in-context learning paradigms, enabling LLMs to accommodate a much larger set of tools and use extensive demonstration data for learning toolken embeddings. On a series of tasks, ranging from numerical reasoning, knowledge-based question answering, to embodied plan generation, we observed a significant enhancement in LLM performance with the help of toolken embeddings. More importantly, ToolkenGPT is able to rapidly adapt and leverage new tools, demonstrating its capacity to keep pace with the constantly evolving landscape of massive tools. We expect future research to learn robust toolken embeddings not only from demonstration data, but also other rich forms of experience [24], such as tool descriptions and input-output records. We are also interested in exploring the integration of toolken embeddings to recent advanced planning techniques [19], with the goal of developing an autonomous agent to solve complex real-world problems.
This project is partially supported by DARPA ECOLE HR00112390063.
[69] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
[70] Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and Meng Jiang. A survey of knowledge-enhanced text generation. ACM Computing Surveys, 54(11s):1-38, 2022.
[71] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1-9, 2022.
[72] Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. AlignScore: Evaluating factual consistency with a unified alignment function. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11328-11348, Toronto, Canada, July 2023. Association for Computational Linguistics. URL https://aclanthology.org/2023.acl-long.634.
[73] Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. Text alignment is an efficient unified model for massive nlp tasks. NeurIPS, 2023.
A Details of Numerical Reasoning
In this section, we describe the data processing workflow (including building the testing datasets and synthesizing training data), list the prompts used for different methods, and describe the training setting. For a fair comparison, we use the same prompts for CoT and the reasoning mode of ToolkenGPT. With the same example questions, we label the calculation process in place to get the prompts for ReAct<sup>3</sup>. For the tool mode of ToolkenGPT, we randomly sample 4 examples of the specified tool from the training set, and transform them into ReAct-style prompts. Because a large number of tools are used, we show the prompts in the supplementary file instead of listing them here.
A.1 GSM8K-XL
A.1.1 Data Synthesis
To build an enhanced version GSM8K-XL by magnifying numbers in GSM8K, we take the following steps:
1. We prompt ChatGPT(gpt-3.5-turbo) with two examples to replace the numbers with appropriate placeholders. The prompt is presented below.
2. In order to validate the correctness of the number replacements, we develop a verification function for the GSM8k dataset, which calculates the formulas embedded in the solutions (See the example prompt below. Formulas are quoted with « and ».) to get an answer. By substituting the original numbers back into the rewritten question-answer pairs and comparing the execution results with the original answer, we can identify any erroneous rewrites and exclude them from our dataset. This ensures the reliability and accuracy of the number replacements.
3. Once we have confirmed the accuracy of the rewriting process, we can proceed with magnifying the numbers. We replace the original numbers with their respective cubic values, exponentially amplifying their magnitudes. However, it's worth noting this strategy doesn't make sense in every scenario. For instance, when dealing with percentage values (e.g.,  $10\%$ ), a direct cubic magnification to  $1000\%$  would be inappropriate. In such cases, we maintain the original values to preserve their intended meaning.
4. With the magnified numbers in place, we seamlessly substitute the placeholders within the rewritten question-answer pairs. By employing the solving function developed earlier, we can process the magnified formulas and obtain accurate solutions, resulting in the GSM8K-XL dataset.
{ "role": "system", "content": "Your task is to rewrite the given question and answer in a more general form using variables instead of specific numbers." }, { "role": "user", "content": "Question: Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market\nAnswer: Janet sells 16 - 3 - 4 = <<16-3-4=9>>9 duck eggs a day.\nShe makes 9 * 2 = $<<9*2=18>>18 every day at the farmer's market.\n##### 18" } { "role": "assistant",
<sup>3</sup>As Yao et al. [69] doesn't evaluate ReAct on numerical reasoning, we don't follow the exact format (i.e. trajectories marked by [THINK] and [ACT]), but design a more natural way to combine CoT reasoning and tool calling (as shown in later sections). We refer to this format as "ReAct-style" prompts.