Toolformer: Language Models Can Teach Themselves to Use Tools
Authors: Timo Schick
Jane Dwivedi-Yu
Roberto Dessi
Roberta Raileanu
Maria Lomeli
Eric Hambro
Luke Zettlemoyer
Nicola Cancedda
Thomas Scialom
Venue: NeurIPS 2023 oral (2023)
Keywords: Language Models
Zero-Shot Learning
Tool Use
APIs
TL;DR: We introduce Toolformer, a language model trained in a self-supervised way to know when and how to use external tools, achieving substantially improved zero-shot performance across a variety of downstream tasks.

Abstract: Language models (LMs) exhibit remarkable abilities to solve new tasks from just a few examples or textual instructions, especially at scale. They also, paradoxically, struggle with basic functionality, such as arithmetic or factual lookup, where much simpler and smaller specialized models excel. In this paper, we show that LMs can teach themselves to *use external tools* via simple APIs and achieve the best of both worlds. We introduce *Toolformer*, a model trained to decide which APIs to call, when to call them, what arguments to pass, and how to best incorporate the results into future token prediction. This is done in a self-supervised way, requiring nothing more than a handful of demonstrations for each API. We incorporate a range of tools, including a calculator, a Q&A system, a search engine, a translation system, and a calendar. Toolformer achieves substantially improved zero-shot performance across a variety of downstream tasks, often competitive with much larger models, without sacrificing its core language modeling abilities.

Toolformer: Language Models Can Teach Themselves to Use Tools
Timo Schick
Jane Dwivedi-Yu
Roberto Dessit
Roberta Raileanu
Maria Lomeli
Eric Hambro
Luke Zettlemoyer Nicola Cancedda Thomas Scialom
FAIR, Meta
'nniversitat Pompeu Fabra
Abstract
Language models (LMs) exhibit remarkable abilities to solve new tasks from just a few examples or textual instructions, especially at scale. They also, paradoxically, struggle with basic functionality, such as arithmetic factual lookup, where much simpler and smaller specialized models excel. In this paper, we show that LMs can teach themselves to use external tools via simple APIs and achieve the best of both worlds. We introduce Toolformer, a model trained to decide which APIs to call, when to call them, what arguments pass, and how to best incorporate the results into future token prediction. This is done a sulf-super way, requiring nothing more than handful of demonstrations for each API. We incorporate range of tools, including calculator, Q&A system, a search engine, a translation system, and calendar. Toolformer achieves substantially improved zero-shot performance across a variety of downstream tasks, often competitive with much larger models, without sacrificing its core language modeling abilities.
1
Introduction
Large language models achieve impressive zero and few-shot results on a variety of natural language processing tasks (Brown et al., 2020 Chowdhery et al. 2022] i.a.). However, these models have several inherent limitations that can at best be partially addressed by further scaling. These limitations include inability access up-to-date information
recent events (Komeili et al., 2022) and the related tendency to hallucinate facts (Maynez et al., 2020 Ji et al. 2022). difficulties understanding low-resource languages (Lin et al. 2021). a lack of mathematical skills to perform precise calculations
(Patel et al. 2021) and an unawareness of the progression of time (Dhingra et al. (022).
A simple way overcome the limitations of today's 's language models is to give them the ability to use external tools such as search engines, calculators, or calendars. However, existing approaches either rely on large amounts of human annotations (Komeili et al. |022 Thoppilan al. (022) limit tool use to task- task-specific settings only (c.g.. Gao et al. 2022; Parisi et al., 2022). hindering a
more widespread adoption of tool use in LMs. Therefore, propose Toolformer, model that learns to use tools in a novel way, which fulfills the following desiderata:
Tool use should be Iearned in self-supervised way without large amounts of human annotations. This is important not only because of the costs associated with such annotations, but also because what humans find useful may be different from what a model finds useful.
The I.M should not lose any of its generality and should be able to decide for itself when and how to use which tool. In contrast to existing approaches, this enables a much more comprehensive use of tools that not tied to specific tasks.
Our approach for achieving these goals is based on the recent idea of using large LMs with in-coneext learning (Brown al. 2020) generate entire datasets from scratch (Schick and Schütze, 2021b
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

The New England Journal of Medicine registered trademark of [QA("Who is the publisher of The New England Journal Medicine?") Massachusetts Medical Society) the MMS.
Out 1400 participants, 400 (or [Calculator(400/ / 1400) 0.29] 29%) passed the test.
The name derives from tortuge", the Spanish word for [MT("tortuga") -uurtee turtle.
The Brown Act is Califomii's low [WikiSearch( ("Boown Act') - The Ralph M. Brown Act an act of the California State Legislature that guarantees the public's right attend and participate meetings of local legislative bodies.J that requires legisi agiiiative bodies, like city councils, to hold their meetings open to the public.
Figure I:
Exemplary predictions of Toolformer. The model autonomously decides to call different APIs (from top to bottom: a question answering system, calculator, a machine translation system, and Wikipedia search engine) to obtain information that is useful for completing a piece of text.
LM Dataset with API Calls
LM Cataser
Sample Calls
Execute Calls
Filter API Calls
x Pittsburgh
6 What other Pittsburgh known by?
! Steel City
Lic Steel City) min(L L((e)
Fittsburgh
also known
also known [QA(What
Xnn the Steel City
= Which country is
= United States
L United States) min(L,(c) 6), L,(e))
Steel City))
Pittsburgh
the Steee City.
Figure 2: Key steps our approach, illustrated for a question answering tool: Given an input text x, we first sample position i and corresponding API call candidates ...c...... We then execute these API calls and filter out all calls which do reduce the loss L, over the next tokens. All remaining API calls are interleaved with the original text, resulting in new x".
Honovich et al. 2022 Wang al. 2022 Given just handful of human-written examples of how API can be used, we let LM annotate a huge language modeling dataset with potential API calls. We then use self-supervised loss determine which these API calls actually help the model in predicting future tokens. Finally, we finetune the LM itself on the API calls that considers useful. As illustrated in Figure T through this simple approach, LMs can learn control a variety of tools, and to choose for themselves which tool to use when and how.
As approach is agnostic of the dataset being used, that was used to pretrain a model in the first place. This ensures that the model does not lose any of its generality and language modeling abilities. We conduct experiments downstream tasks, demonstrating that after learning to use tools, Toolformer, which based on pretrained GPT-J model (Wang and Komatsuzaki 2021) with 6.7B parameters, achieves much stronger zero-shot results, clearly outperforming much larger GPT-3 model (Brown et al.) 2020) and several other baselines on various tasks.
apply it the exact same dataset
variety of different
2 Approach
Our aim is to equip language model M with the ability to use different tools through API calls. We represent API calls tuples e (ac: ie) where a, is the name of the API and i, is the corresponding input. Given an API call e with corresponding result we denote the linearized sequences of the API call not including and including its result, respectively, as:
e(c) = <API> a (t.) </API>
e(c. r) = <API> a (i.) -> </API>
where "<API>", "</API>" and "_"" are special tokens Some examples of linearized API calls inserted into text sequences are shown in Figure
In practice, we use the token sequences" [", "J" and "->" > to represent "<API>", "</API>" and "-*", respectively. This enables our approach to work without modifying the existing LM's vocabulary. For reasons of readability, still refer to them as "<API>", "</API>" and "_"" throughout this section.

---

Your tesk is add calls to a Question Answering API to piece text. The questions should help you get informetion reguired complete the text. You can call the API by writing [QAfquestion]] where "question" the question you want to ask, Here some examples of AFI calls:
Input:
Joe Biden was born in Scranton, Pennsylvania.
Output:
Joe Biden was born (Qeee was Joe Biden bom?')) Scranton, [QA("In which state is Scranton?")
Pennsylvania.
Input: Coca- CaccaCCola,oo Coke, carbonated soft drink manufactured by the Coca-Cola Company.
Output: Coca-Cola, or [QA("What other name is Coca-Cola known by?")] Coke, carbonated soft drink manufactures by [QA("Who manufactures Coca "ola"" the Coca Cola. Company.
Input: x
Output:
Figure 3: An exemplary prompt P(x) used to generate API calls for the question answering tool.
Given dataset C fx',..., augmented with API calls. This is done in three steps, illustrated in Figure 2 First, we exploit the in-context learning ability of M to sample large number of API calls. We then execute them and finally check whether the obtained responses are helpful for predicting future tokens; this is used as a filtering criterion. After filtering, dataset C", and finetune M itself on this dataset. Each step is described in more detail below.
...., x x of plain texts, first convert this dataset into dataset C*
merge API calls for different tools, resulting in the augmented
Sampling API Calls
For each API, we write a prompt P(x) that encourages the LM to annotate an example x = ......... Tn with API calls. An example such a prompt for question answering tool is shown in Figure B Let Par
2i,..., 2n) be the probability that M assigns to token 2n++1 as a continuation for the sequence ø1, ....... 2n- We first sample up to & candidate positions for doing API calls by computing, for each i {1, ....... n2. the probability p. Par Par (<API> P(x), 21s-1) that M assigns to starting an API call at position i. Given a sampling threshold тN. we keep all positions I = {i P; ] if there are more than ht such positions, we only keep the top k. For each position i E I, we then obtain up to m API calls ct ....., Th by sampling from M given the sequence (P(x), 11, .. 11 <API>| prefix and </API>
end-of-sequence token.
Executing API Calls As a next step, we execute API calls generated by M. How this is done depends entirely on the API itself - for example, it can involve calling another neural network, executing a Python script or using a retrieval system to perform search over a large corpus. The response for each API call e, needs to be single text sequence T..
Filtering API Calls Let i be the position of the API call c; in the sequence x 21.... Fna and let be the response from the API. Further, given sequence (w; i E N) of weights, let
(z(z) = Σ w log (M) z, 2,11:-11
be the weighted cross entropy loss for M over the tokens T,,... Fn if the model is prefixed with some text sequence z. We compare two different instantiations of this loss:
L L,(e(c,)r:))
L min (Li(e), L((((,,)))
where denotes an empty sequence. The former is the weighted loss over all tokens Xi, - if the API call and its result are given to M as a prefix the latter is the minimum of the losses obtained from (i) doing API call all and (ii) doing an API call, but not providing the response. Intuitively, an API call is helpful to M if providing it with both the input and the output of this call makes it easier for the model predict future tokens, compared to not receiving the API call at all, or receiving only its input. Given filtering threshold Ty. we thus only keep API calls for which L; - L 2 holds, i.e., adding the API call and its result reduces the loss by at least Ty, compared to not doing any API call or obtaining no result from it.
We provide e(c,, rx) as a prefix instead of inserting it at position because M is not yet finetuned on any examples containing API calls. so inserting it in the middle of x would interrupt the flow and not align with patterns in the pretraining corpus, thus hurting perplexity.
3

---

Table 1: Examples of inputs and outputs for all APIs used.
API Name
Example Input
Example Output
Question Answering
Where was the Knights of Columbus founded? Fishing Reel Types
New Haven, Connecticut
Wikipedia Search
Spin fishing Spin fishing is distinguished between fly fishing and hait cast fishing by the type of rod and reel used. There are two types of reels used when spin fishing, the open faced reel and the closed faced reel.
Calculator
2 4
35
Calendar
Today Monday, January 30, 2023.
Machine Translation
súreté nucléaire
nuclear safety
Model Finetuning After sampling and filtering calls for all APIs, we finally merge the remaining API calls and interleave them with the original inputs. That is, for an input text x = 2 In with a corresponding API call and result (c), position
we construct the new sequence x" = -1 c(2, ri), Tin' we proceed analogously for texts with multiple API calls. Doing this for all C results the new dataset C* augmented with API calls. We use C to finetune M, using a standard language modeling objective. Crucially, apart from inserted API calls, C contains the exact same texts as C, the original dataset. As consequence, finetuning M on C exposes it to the same content as finetuning on C. Moreover, as API calls are inserted in exactly those positions and with exactly those inputs that help M predict future tokens, finetuning on C enables the language model to decide when and how to use which tool, based purely on its own feedback.
When generating text with M after finetuning with our approach, we perform regular decoding until M produces the "-"" token, indicating that it next expects the response for an API call. At this point, interrupt the decoding process, call the appropriate API to get response, and continue the decoding process after inserting both the response and the </API> token.
Inference
Tools
We explore various tools to address different shortcomings of LMs. The only constraints impose are that (i) their inputs and outputs can be represented as texts, and (ii) we can obtain a few demon- explore question answering system, a Wikipedia
strations of their intended use. Concretely, search engine, a calculator, a calendar, and a machine translation system. Examples for the APIs associated with each of these tools shown in Table T We briefly discuss all tools below.
Question Answering Our first tool is a question answering system based on another LM that can answer simple factoid questions. Specifically, we use Atlas (Izacard et al. 2022). retrieval- augmented LM finetuned on Natural Questions (kwiatkowski et al. 2019).
Calculator As a second tool, we use a calculator that can perform simple numeric calculations; we only support the four basic arithmetic operations. Results are always rounded two decimal places.
Wikipedia Search Our third tool is a search engine that, given a search term, returns short text snippets from Wikipedia. Compared to our question answering tool, this search enables model to get more comprehensive information on subject, but requires it to extract the relevant parts by itself. As our search engine, indexes the Wikipedia dump from KILT (Petroni et al. 2021).
BM25 retriever (Robertson al. 1995 Baeza- Yates et al. 1999) that
Machine Translation System Our fourth tool is machine translation system based LM that can translate a phrase from any language into English. More concretely, we use the 600M parameter NLLB (Costa-jussa, jussa et al. 2022) our multilingual machine translation model that works for 200 languages (including lew-resuurce ones). The source language is automatically detected using the fastText classifier (Joulin et al. 2016), while the target language is always set English.
Calendar Our final tool is calendar API that, when queried, returns the current date without taking any input. This provides temporal context for predictions that require some awareness of time.

---

Table 2: Number of examples with API calls in C* for different values of our filtering threshold Ty.
Number of Examples
API
7 = 0.5
T 1.0
T 2.0
Question Answering
51,987
18,526
5,135
Wikipedia Search
207,241
60,974
13,944
Calculator
3,680
994
Calendar
61,811
20,587
3,007
Machine Translation
3,156
1,034
229
4 Experiments
We investigate whether our approach enables a LM to use tools without any further supervision and to decide for itself when and how to call which tool. To test this, select a variety of downstream tasks where we assume at least one of the considered tools to be useful, and evaluate performance in zero-shot settings (Section (.2). Beyond that, also ensure that our approach does not hurt the model's core LM abilities; we verify this by looking at perplexity on two language modeling datasets (Section 4.3 Finally, investigate how tool affected by model size (Section 4.4).
4.1
Experimental Setup
We use a subset CCNet (Wenzek et al. 2020) as our dataset C' and GPT-J (Wang and Komatsuzaki 202I) as language model M. To reduce the computational cost of annotating C' with API calls, we define heuristics for some APIs to get a subset of C. for which API calls are more likely to be helpful than for an average text. For example, we only consider texts for the calculator tool if they contain at least three numbers. Details the heuristics used are given in Appendix A. For obtaining C from C, perform all steps described in Section 2 and additionally filter out all examples for which all API calls were eliminated the filtering step For the weighting function, we use
W
Wt's
with W, max 0.2
SEN W.
to make sure that API calls happen close to where the information provided by the API is actually helpful for the model. The thresholds т, and Ty are chosen individually for each tool ensure sufficient number of examples; see Appendix A for details. Table [Shows final dataset augmented with API calls. We finetune M on C* using a batch size of 128 and learning rate of 1 10 with linear warmup for the first 10% of training. Finetuning details are given in Appendix B. In our experiments, we mainly compare GPT-J and the following models:
shows relevant statistics of our
GPT-J CC: GPT-J finetuned on C, our subset of CCNet without any API calls.
Toolformer: GPT-J finetuned on C*, subset of CCNet augmented with API calls.
Toolformer (disabled): The same model as Toolformer, but API calls are disabled during decoding. This is achieved by manually setting the probability the <API> token 0.
We additionally compare to OPT (66B) (Zhang et al. 2022) and the original davinci variant of GPT-3 (175B) (Brown et al. 2020), two models that are about 10 and 25 times larger than GPT-J.
4.2 Downstream Tasks
We evaluate on various downstream tasks considering a prompted zero-shot setup: Models are instructed to solve each task in natural language (see Appendix C), but we provide no examples. This is in contrast to prior work on tool use (e.g.. Gao et al. 2022: Parisi et al. 2022). where models are provided with -atasct- examples of how tool can be used to solve a concrete task. We choose this more challenging setup as we are interested in seeing whether Toolformer works in precisely those cases where a user does not specify advance which tools should used in which way.
*While this tiltering alters the distribution of training examples, we assume that the remaining examples are close enough to the original distribution so that M 'ss language modeling abilities remain unaffected. This assumption is empirically validated in Section 4.3
5

---

Table 3: Results subsets of LAMA and various benchmarks requiring mathematical reasoning. For LAMA, Toolformer uses the question answering tool for most examples, clearly outperforming all baselines of the same size and achieving results competitive with GPT-3. For the math benchmarks, Toolformer makes extensive use of the calculator tool, clearly outperforming OPT and GPT-3. Best results with GPT-J based model shown bold, best results overall are underlined.
LAMA
Math Benchmarks
Model
SQuAD
Google-RE -RE
T-REx
ASDiv
SVAMP
MAWPS
GPT-J
17.8
4,9
31.9
7.5
5.2
9.9
GPT-J CC
19.2
5.6
33.2
9.6
5.0
9.3
Toolformer (disabled)
22.1
6.3
34,9
14.8
6.3
15.0
Toolformer
33,8
11.5
53,5
40.4
29.4
44.0
OPT (66B)
21.6
2.9
30.1
6.0
7.9
GPT-3 (175B)
26.8
7.0
39.8
14.0
10.0
19.8
We use greedy decoding, but with one modification for Toolformer: We let the model start an API call whenever <API> is one of the k most likely tokens. For k = 1, this corresponds to regular greedy decoding; we instead use & = 10 increase the disposition of our model to make use of APIs. At the same time, we allow at most one API call per input to make sure the model does not get stuck in loop where constantly calls APIs. The effect these modifications explored Appendix
LAMA We evaluate our models on the SQuAD, Google RE and T-REx subsets of the LAMA benchmark (Petroni et al. 2019). For each of these subsets, the task is to complete short statement with missing fact (e.g., a date or a place). As LAMA was originally designed to evaluate masked LMs Devlin et al. 2019), we filter out examples where the mask token is not the final token, so that all examples can be processed and added complexity from not informing the model that a single word is required, for all models we use a slightly more lenient evaluation criterion than exact match and simply check whether the correct word is within the first five words predicted by the model. As LAMA is based on statements obtained directly from Wikipedia, we prevent Toolformer from using the Wikipedia Search API to avoid giving it unfair advantage. As shown in Table 3 (left), all GPT-J models without tool use achieve similar performance. Crucially, Toolformer clearly outperforms these baseline models, improving upon the best baseline by 11.7, 5.2 and 18.6 points, respectively. It also clearly outperforms OPT (66B) and GPT-3 (175B), despite both models being much larger. This is achieved because the model independently decides ask the question answering tool for the required information in almost all cases (98.1 %);; for only very few examples, it uses a different tool (0.7%) or no tool at all (1.2%).
left-to-right fashion. To account for different tokenizations
Math Benchmarks We test mathematical abilities on ASDiv (Mao et al. 2020). SVAMP Pattl et al. 2021) and the MAWPS benchmark (Koncel-Kedziorse, et al. 2016). We again account for the fact that we test all models in a zero-shot setup by using a more lenient evaluation criterion: As the required output is always number, we simply check for the first number predicted by the model Results are shown Table 3 (right). While GPT-J and GPT-J + CC perform about the same, Toolformer achieves stronger results even without API calls. We surmise that this is because the model is finetuned on many examples of API calls and their results, improving its own mathematical capabilities. Nonetheless, allowing the model to make API calls more than doubles performance for all tasks, and also clearly outperforms the much larger OPT and GPT-3. This is because across all benchmarks, for 97.9 77.9% of all examples the model decides to ask the calculator tool for help.
Question Answering We look Web Questions (Berant et 2013). Natural Questions (Kwiatkowski et al. 2019) and TriviaQA (Joshi et al. 2017). For evaluation, we check whether the first 20 words predicted by a model contain the correct answer instead of requiring an exact match. For Toolformer, we disable the question answering tool as this would make solving the tasks trivial. Results are shown Table (left). Once again, Toolformer clearly outperforms all other models based on GPT-J, relying on the Wikipedia search API (99.3%) to find relevant information. However,
An exception to this is if the model's prediction contains equation (e.g.. "The correct answer is 5+3=8"), in which case consider the first number after the "=" sign be its prediction.

---

Table 4: Results for various question answering datasets and temporal datasets. Using the Wikipedia search tool for most examples, Toolformer clearly outperforms baselines of the same size, but falls short of GPT-3 (175B) for question answering tasks. For temporal datasets, Toolformer outperforms all baselines, but does not make use of the calendar tool for TEMPLAMA.
LAMA
Temporal Datasets
Model
WebQS
NQ
TriviaQA
TEMPLAMA
DATESET
GPT-J
18.5
12.8
43.9
13.7
3,9
GPT-J CC
18.4
12.2
45.6
12.9
2.9
Toolformer (disabled)
18.9
12.6
46.7
12.7
5.9
Toolformer
26.3
17.7
48.8
16.3
27.3
OPT (66B)
18.6
11.4
45.7
14.5
1.3
GPT-3 (175B)
29.0
22.6
65.9
15.5
0.8
Table 5: Results on MLQA for Spanish (Es), German (De), Hindi (Hi). Vietnamese (Vi), Chinese (Zh) and Arabic (Ar). While using the MT tool to translate questions is helpful across all languages, further pretraining on CCNet deteriorates performance; thus, Toolformer does not consistently outperform GPT-J. The final rows correspond to models that given contexts and questions in English.
Model
Es
De
Hi
Vi
Zh
Ar
GPT-J
15.2
16.5
1.3
8.2
18.2
8.2
GPT-J CC
15.7
14.9
0.5
8.3
13.7
4.6
Toolformer (disabled)
19.8
11.9
1.2
10.1
15.0
3.1
Toolformer
20.6
13.5
1.4
10.6
16.8
3.7
OPT (66B)
0.3
GPT-3 (175B)
3.4
1.1
0.1
1.7
17.7
0.1
GPT-J (All En) GPT-3 (All En)
24.3
27.0
23.9
23.3
23.1
23.6
24.7
27.2
26.1
24.9
23.6
24.0
Toolformer still lags behind the much larger GPT-3 (175B) model. This is likely due to both the simplicity of our search engine (in many cases, it returns results that are clearly not good match for a given query) and the inability of Toolformer to interact with it, e.g., by reformulating its query results not helpful by browsing through multiple of the top results.
Multilingual QA We evaluate all models MLQA (Lewis al., 2019). multilingual QA benchmark. Context for each question is provided in English, while the question can be in Arabic, German, Spanish, Hindi, Vietnamese, Simplified Chinese. Our evaluation metric is the percentage of times the model's generation, capped at 10 words, contains the correct answer. Results are shown in Table 5 API calls improve Toolformer's performance for all languages, suggesting that it has learned to make use of the machine translation tool. Depending on the language, this tool is used for 63.8% 94.9% of all examples; the only exception is Hindi, for which it is used in only 7.3% of cases. However, Toolformer does not consistently outperform GPT-J as finetuning on CCNet deteriorates performance for some languages. OPT and GPT-3 perform surprisingly weak across all languages, mostly because they fail to provide an answer in English despite being instructed to do so. potential reason for GPT-J not suffering from this problem that it was trained on more multilingual data than both OPT and GPT-3, including EuroParl (Koehn 2005). As upper bound, we also evaluate GPT-J and GPT-3 on variant of MLQA where both the context and the question are provided in English. In this setup, GPT-3 performs better than all other models, supporting our hypothesis that its subpar performance on MIQA due to the task's multilingual aspect.
Temporal Datasets We evaluate all models on TEMPLAMA (Dhingra et al. 2022) and new dataset that we call DATESET. TEMPLAMA contains cloze queries about facts that change with time. DATESET, described Appendix D. generated through a series of templates, but populated using a combination of random dates/durations (e.g.. "What day of the week was it 30 days ago?"). For both tasks, we use the same evaluation as for the original LAMA dataset. Results shown in Table (right) illustrate that Toolformer outperforms all basclines for both TEMPLAMA and DATESET. However,

---

LAMA
Math Benchmarks
QA Benchmarks
40
35
25
25
30
20
25
15
Toolforner
10
Toolformer (disabled)
5
GPT3
0
0
2000
4000
6000
0
2000
4000
6000
0
2000
4000 6000
Model Parameters (M)
Model Parameters (M)
Model Parameters
Figure 4: Average performance on LAMA, our math benchmarks and our QA benchmarks for GPT-2 models of different sizes and GPT-J finetuned with our approach, both with and without API calls. While API calls are not helpful to the smallest models, larger models learn how to make good use of them. Even for bigger models, the gap between predictions with and without API calls remains high.
closer inspection shows that improvements TEMPLAMA can not be attributed to the calendar tool, which is only used for 0.2% of all examples, but mostly to the Wikipedia search and question answering tools. This makes sense given that entities in TEMPLAMA that even knowing the date alone would be of little help. The best course of action for this dataset - first querying the calendar API to get the current date, and then querying the QA system with this date - is not only prohibited by our restriction of using at most one API call, but also hard to learn for Toolformer given that all API calls in its training data are sampled independently. For DATESET, on the other hand, the considerable improvement Toolformer compared to other models can be fully accredited to the calendar tool, which it makes use of for 54.8% of all examples.
often specific and rare
4.3 Language Modeling
We want to ensure that language modeling performance of Toolformer does not degrade through finetuning with API calls. To this end, evaluate models on two language modeling datasets: WikiText (Merity et al. 2017) and subset of 10,000 randomly selected documents from CCNet (Wenzek et al.] (020) that were not used during training. Finetuning CCNet leads slightly improved performance on the CCNet evaluation subset (perplexity improves from 10.6 to 10.5), but slightly deteriorates performance on WikiText (9.9 to 10.3), presumably because the original pretraining data for GPT-J is more similar to WikiText than our subset of CCNet. Most importantly, however, training on C* does not lead an increase in perplexity compared training on when API calls are disabled at inference time, giving perplexities of 10.5 and 10.3, respectively
4.4 Scaling Laws
We investigate how the ability to ask external tools for help affects performance vary the size of our LM. To this end, we apply our approach not just to GPT-J, but also to four smaller models from the GPT-2 family (Radford al. 2019)), with 124M, 355M, 775M and 1.6B parameters, respectively. We do so using only subset of three tools: the question answering system, the calculator, and the Wikipedia search engine. Apart from this, follow the experimental setup described in Section 4.1 Ssows that the ability to leverage the provided tools only emerges at around 775M parameters: smaller models achieve similar performance both with and without tools. An exception to this the Wikipedia search engine used mostly for QA benchmarks; we hypothesize that this is because the API is comparably easy to use. While models become better at solving tasks without API calls as they grow in size, their ability to make good use of the provided API improves at the same time. Thus, there remains a large gap between predictions with and without API calls even for our biggest model.
'We do not evaluate the perplexity of Toolformer with API calls enabled as computing the probability PM (z: .i,..., 1) of token o given æi,..., 2 would require marginalizing over all potential API calls that the model could make at position t, which is intractable.
8

---

In Appendix G, we extend these investigations in scale to the LLaMA v1 7B model Touvron et al. 2023) to see how tool-use scales with model capability instead of size. We find that the utility of weaker tools such as the WikiSearch tool vanish for these stronger base-models, and we demonstrate the value of generating and scoring with a strong model, compared simply finetuning.
5 Related Work
Language Model Pretraining There are various approaches that augment LMs with some form of additional textual information during pretraining, including various forms of metadata (Keskar et al. 2019), HTML tags (Aghajanyan et al. 2021). Wikipedia markup (Schick et al. 2022), or related texts obtained from an information retrieval system (Guu et al. 2020; Borgeaud et al. 2022 Izacard et al.,
2022). For all of these approaches, additional information always provided, regardless of whether is helpful or not. In contrast, Toolformer learns for itself to explicitly asks for the right information.
Tool Use Several approaches aim to equip LMs with the ability to use external tools such as search
engines (Komeili et al. [2022] Thoppila et al., [022 Lazaridou et al. [2022] Shuster et al. 2022 Yao
al. 2022). web browsers (Nakano et [ calculators (Cobbe et al.] [021 [ Toppilan et al.
2022). translation systems Thoppilan et al. (022)) and Python interpreters (Gao et al. 2022). The
way these models learn to use tools can roughly be divided into two approaches: Either they rely
on large amounts of human supervision (Komeili et al. [222,] Nakano et al.
202 Thoppilan et al.)
2022) or they work by prompting the language model in a few-shot setup tailored towards a specific task where it is known priori which tools needs to be used (Gao et al., 2022 Lazaridou et al. 2022
Yao et al. 2022). In contrast, the self-super nature of Toolformer enables it to learn how and
when to use tools without requiring specific prompt that shows task task-specific examples of how a tool could be used. Perhaps most closely related to our work is TALM (Parisi et al. [0022), an approach that uses similar self-supervised objective for teaching model to calculator and search engine, but explores this only in settings where a model is finetuned for downstream tasks.
Bootstrapping The idea of using self-training and bootstrapping techniques to improve models has been investigated in various contexts, ranging from word sense disambiguation (Yarowsky) 1995). relation extraction (Brin, 1999 Agichtein and Gravano 2000), parsing (McClosky et al. 2006). sequence generation (He et al. (020). few-shot text classification (Schick and Schüitze| [22 201 la) and retrieval (Izacard and Grave 2021) to reasoning (Zelikman et al. (022). In a similar spirit, Toolformer is trained on its own predictions after applying a perplexity- based filtering step.
6
Limitations
While our approach enables LMs to learn how to use a variety of tools in a self-supervised way, there some clear limitations to what can be achieved with method in its current form. One such limitation is the inability of Toolformer to use tools in a chain (i.e., using the output of one tool as an input for another tool). This is due to the fact that API calls for each tool are generated independently; as a consequence, there are no examples of chained tool use in the finetuning dataset, since this would necessitate multiple API calls per example. Our current approach also does not allow the LM to use tool in an interactive way - especially for tools such as search engines, that could potentially return hundreds of different results, enabling a LM browse through these results or refine its search query in a similar spirit to Nakano et al. (202T) can be crucial for certain applications. Beyond this, we found models trained with Toolformer often be sensitive to the exact wording of their input when deciding whether or not to call an API; this is perhaps unsurprising given that LMs are known to be very sensitive the prompt they are provided with in both zero- and few-shot settings (Jiang et al. and Schütze 2021a). Depending on the tool, our method is also very sample-inefficien for example, processing more than a million documents results in only a few thousand examples of useful calls to the calculator API. A potential solution to this problem might be to iteratively apply our approach, similar to how this is done related bootstrapping approaches (Schick and Schütze 2021a Izacard and Grave 2021 Parisi et al.) 2022). Finally, when deciding whether or not to make an API call, Toolformer currently does not take into account the tooe-depen,ent, computational cost incurred from making an API call.

---

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving. Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. 2021. Improving 7 Conclusion
We have introduced Toolformer, a LM that learns a self-supervised way how to use different tools such search engines, caleulators, and translation systems via simple API calls. This is done by finetuning on sampled API calls that are filtered based on whether they reduce perplexity on future tokens. Toolformer considerably improves zero-shot performance of a 6.7B parameter GPT-J model, enabling it to even outperform a much larger GPT-3 model on a range of different downstream tasks.
References
Armen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, and Luke Zettlemoyer. 2021. Hilm: Hyper-text pre-training and prompting of language models
Eugene Agichtein and Luis Gravano. 2000. Snowwall: Extracting relations from large plain-texx Mexnttext collections) Proceedings of the Fifth ACM Conference on Digital Libraries, DL '00, page 85 94, New York, NY, USA. Association for Computing Machinery.
Ricardo Baeza- Yates, Berthier Ribeiro-Neto, et al. 1999. Modern information retrieval, volume 463. ACM press New York.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs, In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533-1544, Seattle, Washington, USA. Association for Computational Linguistics.
language models by retrieving from trillions of tokens
Sergey Brin. 1999. Extracting patterns and relations from the world wide web In The World Wide Web and Databases, pages 172- 183, Berlin, Heidelberg. Springer Berlin Heidelberg.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voos Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners) In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko. Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev. Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hycontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier- Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022, Palm: Scaling language modeling with pathways
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, et 2021, Training verifiers to solve math word problems. arXiv preprint arXiv:2110.1416.
10

---

Marta R Costa-jussä, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht. Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv: arXiv:220.004672.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-41 Minneapolis, Minnesota. Association for Computational Linguistics.
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022. Time-aware language models as temporal knowledge bases Transactions of the Association for Computational Linguistics, 10:257-273.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2022. Pal: Program-aided language models
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-We Wei Chang. 2020. Realm
Retrrieallauummented language model pre-training
Junxian He, Jiatao Gu, Jiajun Shen, and Marc' Aurelio Ranzato. 2020, Revisiting self-training for neural sequence generation In International Conference on Learning Representations.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2022. language models with (almost) no human labor
Unnatural instructions: Tuning
Gautier Izacard and Edouard Grave. 2021. Distilling knowledge from reader to retriever for question answering In International Conference on Learning Representations.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-YV Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. |Atlas: Few-shot learning with retrieval augmented language models)
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022. [Survey of hallucination in natural language generation) ACM Computing Surveys.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Pupers), pages 1601-1611, Vancouver, Canada. Association for Computational Linguisties.
Anmand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016. Fasttext. zip: Compressing text classification models. arXiv preprint arXiv:1612.03651.
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. Cirl: A conditional transformer language model for controllable generation
Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of machine translation summit x: papers, pages 79-86.
Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2022. Internet-augmente augmented dialogue generation In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 8460-8478. Dublin, Ireland. Association for Computational Linguistics.
Rik
Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. MAWPS: A math word problem repository] In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human I Language Technologies, pages 1157, San Diego, California. Association for Computational Linguistics.
11

---

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research Transactions of the Association for Computational Linguistics, 7:452-466.
Angeliki Lazaridou. Elena Gribovskaya, Wojeiech Stokowiee, and Nikolai Grigorev. 2022. Internet- augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05/ 15.
Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019. Mlqa: Evaluating cross lingull extractive question answering. arXiv preprint arXiv:1910.07477
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O' Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. 2021.
Few-shot learning with multilinuual language models
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and Factuality in abstractive summarization
David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 152-159, New York City, USA. Association for Computational Linguistics.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models In International Conference on Learning Representations.
Shen-yun Miao, Chao-Chun Liang, and Keh-Yil Su. 2020. A diverse corpus for evaluating and developing English math word problem solvers In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistccs, pages 975 975-984, Online. Association for Computational Linguisties.
Reicchiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang. Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021. WWbgppt: Browser-assisted question-answeriine answering with human feedback
Aaron Parisi, Yao Zhao, and Noah Fiedel. 2022. Talm: Tool augmented language models
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080-2094, Online. Association for Computational Linguistics.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: a benchmark for knowledge intensive language (asks) In Proceedings of the 2021 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2523-2544, Online. Association for Computational Linguistics.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP--JCNLPP) pages China. Association for Computational Linguistics.
Hong Kong,
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. Okapi at trec-3. Nist Special Publication Sp, 109:109.
12

---

Timo Schick, Jane Dwivedi- Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis. Gautier Izacard. Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel. 2022. Peer: A collaborative language model
Timo Schick and Hinrich Schütze. 2021a. Exploiting cloze-questions for few-shot text classification and natural language inference In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. pages for Computational Linguistics.
Online. Association
Timo Schick and Hinrich Schlitze. 2021b. Generating datasets with pretrained language models In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 6943-6951, Online and Punta Cana, Dominican Republic. Association for Computational Linguisties.
Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung. Moya Chen, Kushal Arora, Joshua Lane, Morteza Behrooz, William Ngan, Spencer Poff, Naman Goyal, Arthur Szlam, Y-Lan Boureau, Melanie Kambadur, and Jason Weston. 2022. Blenderbot 3
deployed conversational agent that continually learns to responsibly engage
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng. Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang. Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Mare Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022.
Lamda: Language models for dialog applications
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation
language models
Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A Billion Parameter Autoregressive Language Model. http://ggttttoooooo////g/gg///// transformer-
-rrerl/ jax
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions)
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data In Proceedings of the Twelfih Language Resources and Evaluation Confer- ence, pages 4003 4012, Marseille, France. European Language Resources Association.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting language models]
David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods In 33rd Anmual Meeting of the Association for Computational Linguistics, pages 189-196, Cambridge, Massachusetts, USA. Association for Computational Linguisties.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. 2022. Star: Bootstrapping reasoning with reasoning
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Vietoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-trained transformer language models)
13


Reviews:
[{"review_id": "IDlztE0D9v", "reviewer": "Reviewer_jLhs", "summary": "This paper proposes a method to finetune pretrained autoregressive language models such that they learn when and how to use external tools to achieve good performance in downstream tasks. Following an in-context learning scheme, humans provide a few examples of inserting API calls at appropriate location in the natural language sentence, which is ised by the trained language model to automatically generate an API-augmented variation of a natural language corpus. These API augmented sentences are filtered via a threshold over a criterion measuring whether adding the API calls and their result as a prefix improves the perplexity of the natural language sentence under the LM. This filtered API augmented dataset is used for further finetuning of the pretrained LM so that it learns to insert API calls at appropriate places. This approach called toolformer is compared against using the LM alone, finetuning the LM on the natural language corpus, using toolformer but aritificially suppressing its ability to call an API, and larger general purpose language models. 5 different APIs are considered in this setting. The model is evaluated on several downstream tasks for which access to the APIs might be beneficial.", "questions": "See above", "limitations": "See above", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-- The paper is very well motivated. This capability of querying external API while generating text is a natural solution to the pathologies like hallucination that the language models exhibit today. This approach is a step toward endowing a language model with such capabilities reliably.\n\n-- The experimental setup is well designed and the choices of APIs, baselines, and downstream tasks to evaluate on lead to informative analysis.\n\n-- This approach outperforms baselines convincingly on the downstream tasks while not drastically affecting the language modeling capabilities as measured by perplexity on the held-out set.", "weaknesses": "-- From the writeup, this approach doesn't seem to generate multiple API calls in a sentence and also doesn't perform well with nested API calls. More discussion on this would be useful.\n\n-- While it is discussed in the limitation section and Table 2, a more thorough analysis of sample efficiency of this approach would be helpful. How many sentences are enough to learn tool-use functionality? Is it possible to collect enough high-quality API augmented sentences easily?\n\n-- Ablation study: performance as a function of filtering threshold that controls the quality of the API-augmented sentences would give more insight into the learnability of tool-use and sensitivity to the \"correctness\" of the API-augmented dataset.\n\n-- A thorough error analysis of failure modes would improve the understanding of the limitations of the proposed approach more clearly.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> From the writeup, this approach doesn't seem to generate multiple API calls in a sentence and also doesn't perform well with nested API calls. More discussion on this would be useful.\n\nThis is currently touched upon in the Limitations section: “API calls for each tool are generated independently; as a consequence, there are no examples of chained tool use in the finetuning dataset.” We will make clearer that “chained tool use” encompasses multiple API calls. \n\n> While it is discussed in the limitation section and Table 2, a more thorough analysis of sample efficiency of this approach would be helpful. How many sentences are enough to learn tool-use functionality? Is it possible to collect enough high-quality API augmented sentences easily?\nAblation study: performance as a function of filtering threshold that controls the quality of the API-augmented sentences would give more insight into the learnability of tool-use and sensitivity to the \"correctness\" of the API-augmented dataset.\n\nWhile we agree with the reviewer that it would be valuable to quantify how many examples are necessary for tool-use, this undoubtedly varies according to the specific tool. For some tools like a search engine, this may not need that many samples, but the same is not necessarily true for more complex APIs like scheduling a meeting.  Furthermore, finding good opportunities to generate complex examples in a corpus like CCNet poses yet another challenge. The proposed ablation experiments would also require significant compute (at least ten model training ablations for the five tools) and the results may not generalize.\n\n> A thorough error analysis of failure modes would improve the understanding of the limitations of the proposed approach more clearly.\n\nWhile we did not conduct an extensive and quantitative error analysis, we do have evidence that suggests failure modes are prevalent with some combinations of task and tool. For example, for the DateSet dataset, the Calendar tool is necessary for every example, but we observe that the Toolformer tends to call it less frequently than needed (92.9% of the time). Note however that our evaluation datasets are skewed towards requiring immediate tool use, and that a quantitative error analysis on those would not necessarily shed light on failure patterns in less biased conditions.  Additionally, the task of attributing each incorrect answer across all evaluations to a specific failure mode (failing to call the tool, calling the wrong tool, calling the tool incorrectly, receiving a useless or wrong result from the tool, or failing to come to the wrong conclusion using the tool results) is a major challenge that requires expert human annotation and ideally a broader set of tools.  We leave a general study of failure modes across data sources for future work, and instead include in the camera-ready version a discussion of the different types of failure modes and relevant statistics that we are able to report. Please see the general response for more details."}, {"review_id": "g8Jh0MVJrU", "reviewer": "Reviewer_6zEw", "summary": "This paper proposes an innovative method for enabling Language Models (LMs) to utilize tools. The authors prompt the LM to generate API calls based on human demonstrations, which are then executed in tools. Any non-contributing API calls are filtered out. A dataset is then augmented with these API calls, and used to fine-tune the LM. The Toolformer surpasses larger models in many tasks, offering a significant contribution to the field.", "questions": "To summarize the inquiries raised in the Weaknesses section:\n\nDoes the use of square brackets as API tokens interfere with the standard usage of square brackets in the text?\n\nWhy does Toolformer underperform in comparison to GPT-J+CC in German and Arabic in the MLQA experiment?\n\nUnder what circumstances does Toolformer fail to utilize tools effectively?\n", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper outlines a remarkably simple yet effective strategy for curating a dataset that empowers LMs to utilize tools. The method is well-explained and detailed, boasting a universal applicability across multiple datasets and tools. The authors have carried out extensive, well-designed experiments that showcase the performance boost facilitated by their method. The comparison experiment involving Toolformer, a disabled Toolformer, and GPT-J+CC is particularly commendable, as it eliminates the potential of additional fine-tuning data contributing to performance improvement.\n\nThis research addresses a practical and intriguing topic that is likely to attract considerable interest from both the research and industrial communities. The potential to integrate more sophisticated tools and utilize larger LMs holds promise for advancing LM capabilities.\n", "weaknesses": "The proposed method has some limitations. First, there's a dependency on fine-tuning when adapting the LM to new tools, which could impede broad usage and necessitate additional work.\n\nSecondly, the use of square brackets for the \"<API>\" token, without any special escaping mechanism, might present issues when square brackets form part of the original text.\n\nLastly, the MLQA experiment raises a few questions. The performance of OPT(66B) and GPT-3(175B) suffers due to their inability to provide answers in English, suggesting a potentially inappropriate evaluation setting. Toolformer also underperforms GPT-J in certain languages, seemingly due to the impact of fine-tuning on CCNet. However, it's unclear why Toolformer lags behind GPT-J+CC in German and Arabic. The MLQA experiment fails to convincingly support the paper's main claims.\n\nAdditionally, the paper lacks an analysis of cases where the LM fails to use tools effectively during inference. For instance, the reasons behind the LM's failure when using a calculator during inference are not investigated. Is it due to the inability to generate the <API> token or the candidate? Or does it fail to provide the correct answers even when the API call is successful? An examination of these failed cases and the issues causing them would be enlightening.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The proposed method has some limitations. First, there's a dependency on fine-tuning when adapting the LM to new tools, which could impede broad usage and necessitate additional work.\n\nAt the time this work was conducted there was no evidence that effective tool use could be achieved purely through in-context learning. Indeed, in the absence of reproducible descriptions of how models such as ChatGPT and GPT-4, which do exhibit tool use capabilities, were trained, one has to entertain the possibility that they too were fine-tuned towards tool use in some way. Our work introduces a simple, effective, and reproducible way to endow a model that had no tool training with the capability to use tools, something that we believe is relevant to anyone wishing to train their own model. \n\n> Secondly, the use of square brackets for the \"<API>\" token, without any special escaping mechanism, might present issues when square brackets form part of the original text. Does the use of square brackets as API tokens interfere with the standard usage of square brackets in the text?\n\nThis is an understandable concern, however, an API call requires not only the square bracket but also that the specific tool name follows the square bracket (e.g., <API> Calendar() </API>). Consequently, it would be highly unlikely for parts of the original text to be confused with a true API call, but use cases where usage of the square bracket is common would have to consider this issue, which we have added to the Limitations section.\n\n> Why does Toolformer underperform in comparison to GPT-J+CC in German and Arabic in the MLQA experiment?\n\nOur hypothesis is that the GPT-J+CC model already has some knowledge of other languages so often the model doesn’t need to translate the question into English in order to answer it correctly. In fact, performing the API call can sometimes confuse the model due to the special characters, so the model’s answers may be worse in such cases. However, it is not clear why GPT-J+CC performs better for some languages but not others. \n\nFrom our analysis, the model calls the correct API (MT in this case) with appropriate arguments (either the entire or part of the question) most of the time. The relative poor performance on the MLQA benchmark comes from the inability of answering these questions, even after they’ve been translated into English. However, this is orthogonal to the main goal of the paper which is learning when and how to call a certain API, which we believe our model does to a reasonable extent.   \n\n> Under what circumstances does Toolformer fail to utilize tools effectively?\n\nPlease see the general response for more details."}, {"review_id": "geTabhPuGD", "reviewer": "Reviewer_dirG", "summary": "This paper proposes an approach to augment language models with the ability to call \"tools\" during decoding, such as a calculator, retrieval system, or machine translation system. This requires only a few human-written examples, and then uses the LM to generate a larger fine-tuning datasets constructed from raw text. When fine-tuned on this dataset, and augmented with the ability to execute external tools, performance of the LM is improved for a range of downstream tasks, across various model scales.", "questions": "* The use of a threshold based on the likelihood assigned by the LM with and without the tool use is clever, but I also wonder whether this could be misleading in some cases. For instance, the LM may have been trained (?) on some of the CCNet data, so this may lead to an overly optimistic likelihood without tool usage relative to the optimal tool usage at test time, especially for, .e.g., temporally-sensitive facts. Were any such limitations related to the filtering process observed in practice?", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* The paper proposes a relatively elegant way to integrate tools with language models in a way that requires only a limited amount of human-written examples of API calls per tool. The proposed method to synthetically construct the fine-tuning dataset appears to work well in practice. \n* By showcasing a variety of tools and their impact across a collection of tasks, this paper showcases the potential impact of integrating such tools and their ability to address some common limitations of LMs. The paper seems likely to influence future work.", "weaknesses": "* I did not find any significant weaknesses in the proposed approach, execution of the experiments, or technical descriptions in the paper.\n* My only gripe is in the wording of the title claim that LMs can \"teach themselves to use tools\". I can see what the authors mean, as a LM is used to generate the fine-tuning data, but I don't find this to be a helpful description of the method and I think the paper would read better without this bit of hype. Additionally, for new tools, the approach still requires a prompt, a handful of examples, and heuristics for selecting relevant subsets of a corpus. Anyways, this gripe shouldn't be blocking for publication, and I don't expect the authors to change their selected title.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The use of a threshold based on the likelihood assigned by the LM with and without the tool use is clever, but I also wonder whether this could be misleading in some cases. For instance, the LM may have been trained (?) on some of the CCNet data, so this may lead to an overly optimistic likelihood without tool usage relative to the optimal tool usage at test time, especially for, e.g., temporally-sensitive facts. Were any such limitations related to the filtering process observed in practice?\n\nWe agree that when the LM has been trained on some data, this may lead to an overly optimistic likelihood without tool usage. We actually don't see this necessarily as a limitation: for some data/knowledge already in the model's weights, it is positive to not call a tool but rely on its weights. Conversely, when the model \"knows it doesn't know\", which is the case with well calibrated probabilities, we want it to call the tool. This is quite analogous to humans that would rely on, for example, a calculator, for complex calculus. If the training data covers a sufficiently long time span, temporally-sensitive questions will tend to have more uncertain answers and would fall into this category. We agree that more quantitative analysis would be interesting to measure such behavior."}, {"review_id": "2n1Y4mhLr2", "reviewer": "Reviewer_Vx6Q", "summary": "The paper explores an interesting area to extend large language models (LLMs) with external tools. The authors show that LLMs can teach themselves to better utilize tools. They tested GPT-j on several tools (calculator, QA system, search engine, translator, and calendar). The experimental results well support the claim, and the model even surpasses GPT-3 despite owning far fewer parameters.", "questions": "NA", "limitations": "NA", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "+ The paper is well-written and easy to follow.\n\n+ The idea is novel and the supporting experimental results are extensive.\n\n+ The authors study a very interesting topic which I believe will be impactful in the LLM era.", "weaknesses": "The experiments are only conducted on GPT-j (a non-instruction tuned model), which I believe is not enough, considering the existence of more powerful open-source LLMs such as LLaMA and Vicuna. \n\nI actually tested that Vicuna, ChatGPT, and GPT-4 already have excellent capabilities in utilizing the tools mentioned in the paper (almost perfect). These models can skillfully manipulate tools based on very simple prompting, achieving far better performance than the reported number in this paper (I'm not sure why this paper only includes GPT-3 as the baseline, which apparently performs poorer than most of the current LLMs). Hence I doubt whether the proposed method could still benefit well-tuned SOTA LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The experiments are only conducted on GPT-j (a non-instruction tuned model), which I believe is not enough, considering the existence of more powerful open-source LLMs such as LLaMA and Vicuna.\n\nWe completely agree with the reviewer, but at the time during which this work was conducted, neither of the models were available. We are actively working on the experiments with the LLaMA family of models and intend to include those results.\n\n> I actually tested that Vicuna, ChatGPT, and GPT-4 already have excellent capabilities in utilizing the tools mentioned in the paper (almost perfect). These models can skillfully manipulate tools based on very simple prompting, achieving far better performance than the reported number in this paper (I'm not sure why this paper only includes GPT-3 as the baseline, which apparently performs poorer than most of the current LLMs). Hence I doubt whether the proposed method could still benefit well-tuned SOTA LLMs.\n\nIt is challenging to respond fully to this question without details about the specific experiment that the reviewer conducted, but it is plausible that models like GPT-4 are capable of utilizing tools when the specific tool description is included in the prompt (and given that the details of this model are not released, it is not implausible that ChatGPT and GPT-4 have already seen tool usage in the pre-training or alignment stage). However, the very act of mentioning the tool potentially hints to the model that it should use a tool, thereby inadvertently simplifying the problem of tool usage. Our approach aims to enable Toolformer to automatically know when to leverage these tools and how. We conjecture that simple prompting is not sufficient when more, possibly redundant, tools become available: verifying this conjecture is part of our future research plans.\n"}]

Author rebuttal:
We thank the reviewers for their time and efforts to review, discuss and improve the paper. We have written responses to each reviewer in turn.

Two of the reviewers asked for more details on the types of errors we have seen in the evaluation. While a detailed and quantitative classification of each failure for each task and tool would require expert human annotation, we do notice broad trends. Namely, we observed that failure can arise from the following: 

Failing to call the tool:
- Our evaluation and finetuning datasets are distributionally different since the former are all ultra-short form QA style tasks while the latter consists of CCNet paragraphs, with only a few tool calls per paragraph.  This difference in distribution likely leads to under-use of the tool in our desired setting.  We correct for this by triggering a tool-use when a start token is in the top 10 tokens for the Toolformer, but this is clearly an example of ‘failing to call the tool’. We note that tools are not called for every question, especially where the answer is strongly in-weights.

Calling the wrong tool:
- We find Toolformer very often calls an appropriate tool and can often judge the context correctly. In the maths section, we do observe occasional calls to Question Answering and WikiSearch likely because examples using these tools are much more frequent in the fine-tuning dataset than those using the calculator tool.

Calling the tool incorrectly. 
- Many of the tools we use cannot be called ‘incorrectly’, taking either no arguments or strings as inputs - all of which are valid as API calls.
In the case of the Calculator tool, at data-augmentation time, we see many incorrect/invalid generations (even with constrained decoding to arithmetic tokens), but since the final dataset contains useful, correct API calls, the fine-tuned model often generates valid calculations. However, these calls are often very simple (often two number +,-, /, *), and have low complexity.  

Receiving the wrong result from the tool or failing to come to the right conclusion
- In some cases we see that a tool response is either useless or incorrect - most often this can be seen with the WikiSearch tool which uses a naive BM25 information retrieval algorithm on Wikipedia. 
- The model’s response to this varies and sometimes ignores the result of the tool, while at other times incorporates it.  More investigation is needed to understand _when_ or _why_ a tool is ignored.  
Anecdotally, we observe that the natural continuation requires a highly specific answer, but the tool has not returned it - for instance  “Harry Styles was born in [WikiSearch(Harry styles) -> Harry Styles is an British singer and actor, who has starred in My Policeman] Worcestershire.”

Meta-review: This paper proposes augmenting LLMs with the ability to call "tools" (e.g., as a calculator), which is done by providing an initial (small) set of examples, which is then used to augment more examples by leveraging LLM to generate a larger fine-tuning datasets for "instruction tuning" for tool-use.

The proposed technique (cascade of steps) is straightforward, in an elegant way, which also enables the model to generalize showcased by the capability of using a wide range of tools.

All the reviewers agreed the value this paper adds and acknowledged the potential impact of the paper. Accept as a spotlight.
Decision: Accept (oral)