SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 5: Results on MLQA for Spanish (Es), German (De), Hindi (Hi), Vietnamese (Vi), Chinese (Zh) and Arabic (Ar). While using the MT tool to translate questions is helpful across all languages, further pretraining on CCNet deteriorates performance; thus, Toolformer does not consistently outperform GPT-J. The final rows correspond to models that are given contexts and questions in English.
Model	Es	De	Hi	Vi	Zh	Ar
GPT-J	15.2	16.5	1.3	8.2	18.2	8.2
GPT-J + CC	15.7	14.9	0.5	8.3	13.7	4.6
Toolformer (disabled)	19.8	11.9	1.2	10.1	15.0	3.1
Toolformer	20.6	13.5	1.4	10.6	16.8	3.7
OPT (66B)	0.3	0.1	1.1	0.2	0.7	0.1
GPT-3 (175B)	3.4	1.1	0.1	1.7	17.7	0.1
GPT-J (All En)	24.3	27.0	23.9	23.3	23.1	23.6
GPT-3 (All En)	24.7	27.2	26.1	24.9	23.6	24.0
Toolformer still lags behind the much larger GPT-3 (175B) model. This is likely due to both the simplicity of our search engine (in many cases, it returns results that are clearly not a good match for a given query) and the inability of Toolformer to interact with it, e.g., by reformulating its query if results are not helpful or by browsing through multiple of the top results.
Multilingual QA We evaluate all models on MLQA (Lewis et al., 2019), a multilingual QA benchmark. Context for each question is provided in English, while the question can be in Arabic, German, Spanish, Hindi, Vietnamese, or Simplified Chinese. Our evaluation metric is the percentage of times the model's generation, capped at 10 words, contains the correct answer. Results are shown in Table 5. API calls improve Toolformer's performance for all languages, suggesting that it has learned to make use of the machine translation tool. Depending on the language, this tool is used for  $63.8\%$  to  $94.9\%$  of all examples; the only exception is Hindi, for which it is used in only  $7.3\%$  of cases. However, Toolformer does not consistently outperform GPT-J as finetuning on CCNet deteriorates performance for some languages. OPT and GPT-3 perform surprisingly weak across all languages, mostly because they fail to provide an answer in English despite being instructed to do so. A potential reason for GPT-J not suffering from this problem is that it was trained on more multilingual data than both OPT and GPT-3, including EuroParl (Koehn, 2005). As an upper bound, we also evaluate GPT-J and GPT-3 on a variant of MLQA where both the context and the question are provided in English. In this setup, GPT-3 performs better than all other models, supporting our hypothesis that its subpar performance on MLQA is due to the task's multilingual aspect.
Temporal Datasets We evaluate all models on TEMPLAMA (Dhingra et al., 2022) and a new dataset that we call DATESET. TEMPLAMA contains cloze queries about facts that change with time. DATESET, described in Appendix D, is generated through a series of templates, but populated using a combination of random dates/durations (e.g., "What day of the week was it 30 days ago?"). For both tasks, we use the same evaluation as for the original LAMA dataset. Results shown in Table 4 (right) illustrate that Toolformer outperforms all baselines for both TEMPLAMA and DATESET. However,
Figure 4: Average performance on LAMA, our math benchmarks and our QA benchmarks for GPT-2 models of different sizes and GPT-J finetuned with our approach, both with and without API calls. While API calls are not helpful to the smallest models, larger models learn how to make good use of them. Even for bigger models, the gap between predictions with and without API calls remains high.
closer inspection shows that improvements on TEMPLAMA can not be attributed to the calendar tool, which is only used for  $0.2\%$  of all examples, but mostly to the Wikipedia search and question answering tools. This makes sense given that entities in TEMPLAMA are often so specific and rare that even knowing the date alone would be of little help. The best course of action for this dataset – first querying the calendar API to get the current date, and then querying the QA system with this date – is not only prohibited by our restriction of using at most one API call, but also hard to learn for Toolformer given that all API calls in its training data are sampled independently. For DATESET, on the other hand, the considerable improvement of Toolformer compared to other models can be fully accredited to the calendar tool, which it makes use of for  $54.8\%$  of all examples.
4.3 Language Modeling
We want to ensure that language modeling performance of Toolformer does not degrade through finetuning with API calls. To this end, we evaluate our models on two language modeling datasets: WikiText (Merit et al., 2017) and a subset of 10,000 randomly selected documents from CCNet (Wenzek et al., 2020) that were not used during training. Finetuning on CCNet leads to slightly improved performance on the CCNet evaluation subset (perplexity improves from 10.6 to 10.5), but slightly deteriorates performance on WikiText (9.9 to 10.3), presumably because the original pretraining data for GPT-J is more similar to WikiText than our subset of CCNet. Most importantly, however, training on  $\mathcal{C}^*$  does not lead to an increase in perplexity compared to training on  $\mathcal{C}$  when API calls are disabled at inference time, giving perplexities of 10.5 and 10.3, respectively.[5]
4.4 Scaling Laws
We investigate how the ability to ask external tools for help affects performance as we vary the size of our LM. To this end, we apply our approach not just to GPT-J, but also to four smaller models from the GPT-2 family (Radford et al., 2019), with 124M, 355M, 775M and 1.6B parameters, respectively. We do so using only a subset of three tools: the question answering system, the calculator, and the Wikipedia search engine. Apart from this, we follow the experimental setup described in Section 4.1. Figure 4 shows that the ability to leverage the provided tools only emerges at around 775M parameters: smaller models achieve similar performance both with and without tools. An exception to this is the Wikipedia search engine used mostly for QA benchmarks; we hypothesize that this is because the API is comparably easy to use. While models become better at solving tasks without API calls as they grow in size, their ability to make good use of the provided API improves at the same time. Thus, there remains a large gap between predictions with and without API calls even for our biggest model.
<sup>5</sup>We do not evaluate the perplexity of Toolformer with API calls enabled as computing the probability  $p_{M}(x_{t} \mid x_{1}, \ldots, x_{t-1})$  of token  $x_{t}$  given  $x_{1}, \ldots, x_{t-1}$  would require marginalizing over all potential API calls that the model could make at position  $t$ , which is intractable.
In Appendix G, we extend these investigations in scale to the LLaMA v1 7B model (Touvron et al., 2023) to see how tool-use scales with model capability instead of size. We find that the utility of weaker tools such as the WikiSearch tool vanish for these stronger base-models, and we demonstrate the value of generating and scoring with a strong model, compared to simply finetuning.
Language Model Pretraining There are various approaches that augment LMs with some form of additional textual information during pretraining, including various forms of metadata (Keskar et al., 2019), HTML tags (Aghajanyan et al., 2021), Wikipedia markup (Schick et al., 2022), or related texts obtained from an information retrieval system (Guu et al., 2020; Borgeaud et al., 2021; Izacard et al., 2022). For all of these approaches, additional information is always provided, regardless of whether it is helpful or not. In contrast, Toolformer learns for itself to explicitly asks for the right information.
Tool Use Several approaches aim to equip LMs with the ability to use external tools such as search engines (Komeili et al., 2022; Thoppilan et al., 2022; Lazaridou et al., 2022; Shuster et al., 2022; Yao et al., 2022), web browsers (Nakano et al., 2021), calculators (Cobbe et al., 2021; Thoppilan et al., 2022), translation systems (Thoppilan et al., 2022) and Python interpreters (Gao et al., 2022). The way these models learn to use tools can roughly be divided into two approaches: Either they rely on large amounts of human supervision (Komeili et al., 2022; Nakano et al., 2021; Thoppilan et al., 2022) or they work by prompting the language model in a few-shot setup tailored towards a specific task where it is known a priori which tools needs to be used (Gao et al., 2022; Lazaridou et al., 2022; Yao et al., 2022). In contrast, the self-supervised nature of Toolformer enables it to learn how and when to use tools without requiring a specific prompt that shows task-specific examples of how a tool could be used. Perhaps most closely related to our work is TALM (Parisi et al., 2022), an approach that uses a similar self-supervised objective for teaching a model to use a calculator and a search engine, but explores this only in settings where a model is finetuned for downstream tasks.
Bootstrapping The idea of using self-training and bootstrapping techniques to improve models has been investigated in various contexts, ranging from word sense disambiguation (Yarowsky, 1995), relation extraction (Brin, 1999; Agichtein and Gravano, 2000), parsing (McClosky et al., 2006), sequence generation (He et al., 2020), few-shot text classification (Schick and Schütze, 2021a) and retrieval (Izacard and Grave, 2021) to reasoning (Zelikman et al., 2022). In a similar spirit, Toolformer is trained on its own predictions after applying a perplexity-based filtering step.
While our approach enables LMs to learn how to use a variety of tools in a self-supervised way, there are some clear limitations to what can be achieved with our method in its current form. One such limitation is the inability of Toolformer to use tools in a chain (i.e., using the output of one tool as an input for another tool). This is due to the fact that API calls for each tool are generated independently; as a consequence, there are no examples of chained tool use in the finetuning dataset, since this would necessitate multiple API calls per example. Our current approach also does not allow the LM to use a tool in an interactive way – especially for tools such as search engines, that could potentially return hundreds of different results, enabling a LM to browse through these results or to refine its search query in a similar spirit to Nakano et al. (2021) can be crucial for certain applications. Beyond this, we found models trained with Toolformer to often be sensitive to the exact wording of their input when deciding whether or not to call an API; this is perhaps unsurprising given that LMs are known to be very sensitive to the prompt they are provided with in both zero- and few-shot settings (Jiang et al., 2020; Schick and Schütze, 2021a). Depending on the tool, our method is also very sample-inefficient; for example, processing more than a million documents results in only a few thousand examples of useful calls to the calculator API. A potential solution to this problem might be to iteratively apply our approach, similar to how this is done in related bootstrapping approaches (Schick and Schütze, 2021a; Izacard and Grave, 2021; Parisi et al., 2022). Finally, when deciding whether or not to make an API call, Toolformer currently does not take into account the tool-dependent, computational cost incurred from making an API call.
We have introduced Toolformer, a LM that learns in a self-supervised way how to use different tools such as search engines, calculators, and translation systems via simple API calls. This is done by finetuning on sampled API calls that are filtered based on whether they reduce perplexity on future tokens. Toolformer considerably improves zero-shot performance of a 6.7B parameter GPT-J model, enabling it to even outperform a much larger GPT-3 model on a range of different downstream tasks.