What Are Tools Anyway? A Survey from the Language Model Perspective
Authors: Zhiruo Wang
Zhoujun Cheng
Hao Zhu
Daniel Fried
Graham Neubig
Venue: COLM (2024)
Keywords: tool, agent, program
TL;DR: In this survey, we formally define what is a tool, when and how tools help, and existing issues with tool-related works.

Abstract: Language models (LMs) are powerful yet mostly for text generation tasks. Tools have substantially enhanced their performance for tasks that require complex skills. However, many works adopt the term “tool” in different ways, raising the question: What is a tool anyway? Subsequently, where and how do tools help LMs? In this survey, we provide a unified definition of tools as external programs used by LMs, and perform a systematic review of LM tooling scenarios and approaches. Grounded on this review, we empirically study the efficiency of various tooling methods by measuring their required compute and performance gains on various benchmarks, and highlight some challenges and potential future research in the field.

Published as a conference paper at COLM 2024
What Are Tools Anyway?
A Survey from the Language Model Perspective
Zora Zhiruo Wang"
Zhoujun Cheng"
Hao Zhu
Daniel Fried*
Graham Neubig
CCrneeiie Mellon University "Shanghai Jiao Tong University {zhiruow,
dffirud dfried, gneubig eecs.
cmu. cmu. edu
Abstract
Language models (LMs) are powerful yet mostly for text generation tasks. Tools have substantially enhanced their performance for tasks that require complex skills. However, many works adopt the term "tool" in different ways, raising the question: What is a tool anyway? Subsequently, where and how do tools help LMs? In this survey, we provide unified definition of tools as external programs used by LMs, and perform a systematic review of LM tooling scenarios and approaches. Grounded on this review, we empirically study the efficiency of various tooling methods by measuring their required compute and performance gains on various benchmarks, and highlight some challenges and potential future research in the field !
1 Introduction
Language Models (LMs) have become increasingly effective in solving text-generation tasks, by taking in natural language (NL) instructions from users and outputting NL responses, such answering the "What is the capital of the US?" with "Washington D.C.". However, LMS often struggle to perform tasks that require complex skills (e.g., math or complex rea- soning), and are fundamentally unable to solve other tasks that require access to information not in their training data or parametric knowledge (e.g., the current weather or date).
To solve this problem, researchers and practition- ers are turning to LMs enhanced with tools, which help facilitate the task-solving process of LMs, or extend LMs with new abilities that the LM does not possess otherwise (Qin al., 2023 Mialon al. 2023). For example, a calculator tool can facilitate math calculations, or a get.time time () tool can enable LMs to obtain time. Inspired by the tools used by humans (Shumaker et al. 2011), some works intro- duce application-specitis sof tware as tools, such as a search engine to obtain knowledge (Lazaridou et al. 2022 Komeili et al., 2022) or translator to
extend
real world
L
3 pm
get time()
(35.223 x /
basic
calculator
textgen
facilitate
① 4537
16 ① 22223 yrelds
then divida
the number by
2 square number
4 reault is 453.7
Figure 1: Illustration of tools extending and facilitating LM task-solving.
process unknown languages (Schick et al., 2023).
With the development of numerous application programming interfaces (APIs) on the web, many works collect APIS as tools to access world data in real-time (Balog et al. 2016 Xu et al.
2023: Qin et al. 2024) and via multiple modalities [Tang et al. 2023). Instead of black-box
APIIs other works use crafted functions to query over structured tables (Wang et al.
2024a
Cao et al. 2023 or images (Surís al. 2002),, where tools can be created by human
Gupta
& KKmbhhaii 2022) or model experts (Wagg et al. 2023a Cai et al. 2023 Wang et al.
2024b).
However, despite this broad and burgeoning area of tool use in LMs, existing surveys only cover certain tool categories such as software (Mialon et al. 2023) or APIs (Qin et al. (023).
In this paper, we provide a unified view of tool use across a broad range of scenarios.

---

Published as a conference paper at COLM 2024
We start with proposing unified definition of tools and explain why tools help task-solving ($2). We present the basic tool-use paradigm ($3) and study variety tool-using scenarios by enumerating which tools exist and to which tasks they apply (54). Next, we study advanced approaches for complex tool usage and even make new tools if they are unavailable for the task ($5) We then summarize existing testbeds and evaluation metrics across LM tooling works, and highlight several missing aspects with concrete metric suggestions ($6). Lastly, grounding on our empirical analysis about when tools are effective, we identify the most efficient tooling approaches and the tasks that benefit most from tools (57)
2 Background
2.1 What are tools?
Because LMs are products of the digital world, tools employed by LMs are often computer programs that are executable in corresponding environments. Referring back to animal-used tools Shumaker et al. (2011) defined as "the external employment of an unattached or manipulable attached environmental object." Similar to human-used physical tools, LM-used program tools should also be external to the employer (i.e., the LM) and are part of the environment. Also, instead of arbitrary program snippets, tool is function (e.g., _lus_one), meaning that can be applied to other objects (e.g., data) and yield an output (e.g. plus_ [1)s_one(() -0 2).
Existing definitions of LM-used tools touch on some of these aspects. Qin et al. (2023) appeal to the similarity to human tool use, but do not define what entails tool. Mialon et al. (2023
define tool as "an external module that is typically called using rule or a special token and whose output is included in the augmented LM's context." We argue for somewhat broader definition than this, which encompasses wide variety of more recent works on tool usage:
Definition 1. An LM-used tool is a function interface to computer program that runs externally to the LM, where the LM generates the function calls and input arguments in order use the tool.
2.2 Why are tools helpful?
Tools can help task-solving in mainly three ways, as reflected by their functions: perception, action, and computation. A tool may belong to one or more of these three categories.
Perception Perception tools provide or collect information from the environment. An example is using a get time () API to obtain the current time, which is not included in the LM's parametric knowledge learned from training.
Action Action tools can exert actions on the environment and change its state. For example, executing make bakkepost((( (website, post) can change the content on a website.
Computation Computation tools do not perceive or modify the external environment, but use programs to tackle complex computational tasks. For example, a calculator is computation tool for mathematical calculation. Note that the computation also includes more general acts of computing beyond numerical calculation. Therefore, a translator is also computation tool that can be used to translate between languages.
tool can fall into multiple categories. For instance, search tool can perform both computation and perception: it computes document similarity to find relevant ones, but also perceives the environment (the web) and fetches data (returned documents) from it. In similar spirit, SQL queries can be used as computation tools (e.g., SELECT SQRT(16) / 10 AS result), perception tools for viewing data (e-g., SELECT name FROM data), action tools to modify data (e.g., INSERT INTO data VALUES name), or all of the above (e.g., INSERT INTO counts (grp.id, grp_cnt) SELECT grp-id, COUNT (*) FROM data GROUP BY grp.id).
2.3 Tools and "Agents"
There has recently been a burgeoning of work on LM-powered agents (X et al. 2023 Sumers et al.) 2024 Russell & Norvig (2010) define agents as "anything that can be viewed as percetoing its environment through sensors and acting upon that environment through actuators." According to this definition, agents are programs that use perception tools to perceive the
2

---

Published as a conference paper at COLM 2024
situated environment, or action tools to interact with the environment. Models that only use computation tools and do not interact with their environments through perception or action tools arguably do not fall under the category of "agents" according to this definition.
3 The basic tool use paradigm
We show illustrative example of basic tool-use paradigm introduced by Schick et al. (2023), which many tool-related works adopt (Figure 2).
How is the weather today?
is check weather()
Assuming an LM communicates with users mainly in NI., upon receiving, user query such "How is the weather today?", the LM then proceeds to generate either text tool calls. In the exam- ple, the LM starts with generating few tokens of text "It is .... When the LM needs to seek external tools to complete the task, e.g,, get real- time weather information, it generates tokens of the tool name and corresponding input arguments check.weather () to construct a complete tool call- ing expression. This expression will trigger shift from text-generation mode to -xol-execution mode. The tool- tool-hosting server executes the expression and returns the execution output "sunny" to the LM. The returned output replaces the tool call in the LM-generated tokens (from "It is "oock_weather() to "It is sunny"). Accordingly, LM shifts back to the text generation mode and continues to finish the response by generating new text tokens, e.g,, adding "today." Finally, the LM returns the response to the user.
call
replace
is sunny today
weather
return API onipur
Figure 2: The basic tool use paradigm. LM calls check weather tool by gen- erating text tokens. This call triggers the server to execute the call and re- turn the output sunny using which the LM replaces the API call tokens in the response to the user.
In order for LMs to use this basic paradigm of using tools, current works mainly leverage inference-time prompting and training-ting time learning methods.
Inference-time prompting Leveraging the ability of LMs to learn in-context (Brown et al. 2020), many works provide tool information through prompt and expect LMs to acquire abilities to use these tools from input contexts. This is achieved by providing instructions
about the task, example pairs of queries and solutions that use tools (Gupta & Kembhavi
2022; Lu et al. 2023a: Paranjape et al., 2023
|023|Shen (hhnn et al. 2023a Yang et al. 2023 and/or or
documentation of the tools' functionality (Hsieh et al. 2023).
Learning by training Beyond learning tools from test-time contexts, LMs can learn from examples that use these tools during training. LMs can simply be trained to generate tool- using solutions, where the examples can be manually annotated by humans (Li et al./ 2023). synthesized by larger teacher LMs (Tang et al. 2023 Qin et al. 2024 Huang et al. 2024), or
bootstrapped by the test-time LM itself (Schick et al.. 2023).
4 Scenarios where tools are useful
While LMs may easily solve many tasks without tools, many other tasks greatly benefit from tool use. In this section, we study a broad range of scenarios where tools have been used to assist agents. We discuss tasks where human-created, task-specific tools can improve their performance ($4. as well as scenarios where tools may not be as useful ($4.2)
4.1 Utilizing existing tools for specific applications
While it is difficult to exhaustively enumerate every scenario where tools could be useful, summarize some major categories of tool use in Table and below.
B Knowledge access LMs store limited knowledge during training due to both limits in (i) the data that they are trained on and (ii) the ability of LMs to accurately memorize and utilize their training data. Knowledge-accessinn
tools can help alleviate this issue. SQL and SPARL executors can provide access to data in structured knowledge bases (Thoppilan)
et al. 2022 Hao et al. 2023) graphs (Zhuang et al. 2023). An search engine over the
3

---

Published as a conference paper at COLM 2024
Category
Example Tools
sql. (que:x: str) -> answer: any
2 Knowledge access
search.engiiee(ueryy. str) document: str
calculatoorfoormui. str) -> value: int float
Computation activities
petthnnnnnterrreteerrrrrggg str) -> result: any borkshheet.insert.rowwcc list, index: int) -> None
get www.weeeeeeeittyaaneeeee str) -> weather: str
Interaction w/ the world
get_location(ip: str) -> location: str
calendar www.cheeeeeetsddaaeeeeeeee str) -> events: list
cat_image.dellte((nnee,, deleeteimmage..d str) -> None
Non-textual modalities
spotify.play. cla.fy..lllymmmsiccnnee str) None
visual.qa(query: str, image: Image) -> answer: str
QA(question: str) -> answer: str
C%
Special-skilled LMs
translation(text: str, Language: str) >> text: str
Table 1: Exemplar tools for each scenario. A tool may fall into one or more categories.
web Yao et al. 2023 Schick et al., 2023 Paranjape et al., 2023) can enable LMs to access
more up-to-date information (Komeili et al. 2022 Lazaridou et al. 2022 More generally,
retrieval-augmented systems (Asai et al. 2023) can be seen as using a retriever tool.
ii Computation activities Complex computing activities such math calculations are known to be challenging for neural LMs (Schick et al., 2023). While even a calculator can enhance LMs' numeracy abilities (Parisi et al., 2022 Hao et al. 2023), more generic Python |
programs are also employed to aid reasoning tasks (Gao et al. 20232] Chen et al. 2023 Wang et al. 2023b). For more complex professional jobs, business tools are also apphed, such as using worksheet to manipulate Google Sheets (Xu et al. 2023), or even tools for financial, medical, education, advertising domains (Tang et al. (023) Huang et al. 2024
Interaction with the world LMs without tools are fundamentally unable to perceive and act in the world around them, necessitating tools that can interact with the world. For instance, tools can help LMs access real-time information such as weather (Xu et al. 2023
Tang et al. )023), positional knowledge such location (Qin et al. (024). Meanwhile tools can enable LMs to take actions on the world such as managing calendars (Schick et al.
2023) and emails QQn et al. 2024). Beyond web-based activities, tool-augmented LMs can engage in physical activities in embodied environments, such as fishing with rods or mining with axes in the Minecraft world (Wang et al.) 2023a); further propagate to the real-world tasks to perform cooking (Singh et al. 2022: Shridhar et al. 2020), plotting (Liang et al., 2023), and even conducting chemical research (Boiko al. 2023).
Non-textual modalities While many LMs only consume and generate texts, some
works bring in access to visual (Gupta & Kembhavi 2022 Suris al. 2023), audio (Yang
al. 2023 Gao et al. 2023a), or other modalities. For example, LMs can access images with
cat iage APIs (Xu et al. 2023 Tang et al. 2023) or songs (Huang et al. 2024 provided by
spotify, even answer questions about them (Gupta & Kembhavi 2022
Gao et al. (023a).
0 Special-skilled LMs using the main LM as a task planner to dispatch requests to other LMs. Schick et al. (2023)
Some works propose to use specialized LMs as tools, essentially
propose QA models to fill in factoid details in responses, Thoppilan et al. (2022) /Schick
al. (2023); : Paranjape et al. (2023) use machine translation models to assist multilingual tasks. Beyond specitic tasks, some works adopt multiple neural models from Hugginface or similar platforms (Patill et al. 2023 Shen et al. 2023a or further fine-tune them on various data /Viswanathan et al., 2023). ). Compared to the base LM, these tool models mainly vary in their specialized skills, and may may not have architectural differences to the base LM.
4.2 Where are tools not useful?
Despite that tools can be helpful under many scenarios discussed above, it is also important to note scenarios where tools are arguably not very helpful. Some example tasks where tools have not (yet) been used to great effect include machine translation, summarization,
4

---

Published as a conference paper at COLM 2024
and sentiment analysis (among others). These are tasks that are not easy to perform using non-ML methods (c.f. accessing databases can be done using SQL), and can be performed with high accuracy by a powerful LM alone. One intuitive reason is that the tools currently leveraged for these tasks are neural networks and have limited advantages over the base LMM
Advanced tool-use methods
Given this understanding of the basic tooling paradigm and the scenarios in which tools are useful, we now discuss more advanced approaches for tools. Concretely, we study multi-tool selection and complex usage ($5.1) tooling under programmatic contexts (55.2), and creation of tools when they are not available a-priori (53.)
5.1 Complex tool selection and usage
Depending on the number of tools available, the system may include an implicit or explicit tool selection module. If tools are already designated for the task (Lazaridou et al. 2022 Thoppilan et al. 2022), then no tool selection is needed. If small number (e.g., 5-10) of tools are available, metadata and use cases of these tools can be provided input contexts along with the user query (Schick et al., 2023 Paranjape et al., 2023), and LMs can directly select tools from contexts via standard generation process. If the toolbox size further grows (e.g,, to hundreds), fitting all tools into model inputs is not feasible. Thus an extra retrieval step is often incorporateds ted: retriever model short-lists the most relevant tools and feeds their metadata to the solution-generation LM. Specifically, Zhou et ( Qin et al. 20224) train retriever models that map NL to tool documentation. Yuan et al. (2023) ask LMs write hypothetical descriptions then find similar tools. More easily, some directly use off-the-shelf embeddings (Meng et al. 2024 OpenAl) or training-free retrievers (Robertson et al. 2009).
For complex queries that require multiple tools to solve, the common approach so far is to break down the task and sequentially tackle each step by using a single tool (Paran- jape et al. 2023). However, this sequential multi-step paradigm may not be reflective of more complex or realistic tool usage. For example, a user may prefer nested function calls (get_local tt ('Pittsburgh') to allow information hiding or encapsula- tion (Rogers 2001), parallel calls to reduce round trips with the API (Eleti et al. 2023), or iterative calls of buy teiceeeeeeeeeete until it returns True to indicate successtul transaction.
5.2 Tools in programmatic contexts
Unlike text-based tasks where tools are auxiliary modules to LMs, on programmatic tasks, where code LMs can solve the problem by generating programs, tools can be seen as compositions of basic functions. In this part, we discuss tools in programmatic tasks for domain-specific (55.2.1) and general-purpose problems (55.2.2).
Focus on varied tools Depending the tasks of interest, existing works focus on different types of tools under programmatic contexts. With the increasing complexity of these tools and presumably decreasing familiarity of LMs about them, there are works that adopt (i) built-in functions of a programming language (PL) to augment LMs in symbolic reasoning, (ii) external libraries in pre-designed packages to tackle complex open-domain coding queries (Wang et al. 2023c), and (iii) utility functions unseen at training time to solve specific tasks.
build-in functions tools Q: The bakers baked 200 loaves of bread....
external libraries - tools Q: The table shows how many... What the max number of vacation days across years?
utility functions - tools
How many loaves of bread did they have left?
Q. Who is wearing the dress?
The bakers started with 200 loaves. They 62.
Year Vacation days
2013
2014
2015
LM
sold 93 the moming.... The
23
18
11
loeves baked, loaves returned 2000 sold_moming sold afternantnnnnns 93, 39 answer'a loaves, baked loaves sold morning [ Joaves_ssld sold_ aftemoon lovees retumedd
import pandes as
drese_hox locate_ objects(inage,, "dresa")
pd DataFrame (2013, 2015,
dress_ragiion crop_rogionnmmmg dresa_box)
"Vacation days". (23, 18, 11])
answer = uasual.gal wew.alaalccreesseeoooo gragiin.sssreegiann
oaweesrreuueed
(ayx_days off Ve'"Vactiion days' imax(]
question- "Woo is wearing the dress?")
CodeLL
CodeLM
CodeLM
Figure 3: Tools involve built-in functions, external libraries, task-specific utility functions.
5

---

Published as a conference paper at COLM 2024
5.2.1 Domain-specific semantic parsing
NL-to-code generation systems have been studied for many years on special-domain tasks
such as querying databases (Zelle & Mooney 1996 Zettlemoyer & Collins, 2012) or knowl-
edge graphs (Berant et al. 2013). Code produced by these systems is often domain-specific logical forms (DSL) designed by experts, such as lambda expressions (Liang 2013 SQL
queries (Yu al. 2018), or the recent QDMR grammar (Wolfson et al. (020). Many agentic
tasks also adopt DSL to operate in task-specific environments, such as click or type in web
navigation (Liu et al. 2018 Yao et al. 2022 Zhou et al., 2024), place tem in the embodied Minecraft world (Wang et al. 2023a),
set.joint_targe for robots (Yu et al. (023). While DSL builf-in actions can be directly used, for complex queries, solution programs with basic DSLs alone can be hard to interpret or cumbersome to use.
5.2.2 General-purpose code generation
Recent code generation systems have expanded from using DSL to more general-purpose PLs such Python Java (Yin & Neubig 2017: Chen al. 2021). These languages enable more programming flexibility and readily apply to versatile scenarios. As we have introduced using built-in actions as tools in $4.1 1 we discuss more on two other common categories of tools for code LMs, namely external libraries and task-specific utility functions.
From the usage of PLs, built-in functions are internal to whichever environment, whereas third-party libraries lie externally and need to be imported to tackle specific contexts, such as Figure 3 (middle). Aligning with this conception, Zhang et al. use Python libraries such as matplotlib to plot figures and pandas to manage data.
External libraries
(2023)
Utility functions Expert-crafted utility functions, usually unseen training time, can also be used as tools. E.g., in Figure 3 (right), the highlighted poctte.objects (Gupta & Kembhavi 2022) loads neural models to detect object regions in images. Also, Cheng et al. (2023) GPT as a tool to query world facts external to table contents, Cao et al. 2023 further design macro APTs for advanced tabular operations. However, because human tool creation requires expertise and effort, some works explore using LMs to create new tools.
5.3
Tool creation and reuse
tools make new tools
While one can readily use tools for tasks equipped with pre-designed tools, for tasks that do not have readily-applicable, human-created tools, some works explore using LMs to make tools and use them.
find max(values)
what's the max value?
available tools
vqalq: stri
pick
what's the time now?
get_ time(
con use existing
Domain-specific library abstraction Works using DSLs often compose frequently-used--oggehe tions as shorteut tools. Ellis et al. (2023) learn func- tion abstractions such as length from lambda primitives (e.g., 0, +) for list or string tasks. Wong et al. [2021); [Bweers et al. (2023) build libraries bottom-up from corpus of DSL programs. Grand et al. 2023) use LLMS to abstract libraries with nuto-documentation. For agentic tasks, |Lu et al.
Figure 4: LM makes tools when no tools readily apply to the task.
ac-
(018) learn common workflows to guide web navigation, e-g., com- posing basic (click, like actions to form a higher-level login action click (iik(e'ooglll )).
General-purpose tool making On general-purpose PLs, the DSL-oriented methods above may explode the search space and have limited utility. Instead, recent works leverage LMs' guide search. To start, Wang al. (2023a) design an automatic curriculum in Minecraft to make and use Java program tools. LATM (Cai et al.) 2023 use
procedural knowledge
LMs to build, verify, and use Python tools on BigBench (bench authors 2023). CREATOR QQan et al. 2023) extend tool-making to math and table tasks, and improves task success by creating tools yet repetitively for every example, thus CRAFT (Yuan et al. 2023) add heuristic-based training to reduce tool repetition. Towards more efficient pipelines, ReGAL (Stengel-Liskin et al. 2024) learns from refactoring existing programs, while TroVE (Wang al. 2024b) utilize inference- -ime execution signal and induces reusable tools on-the-tly.

---

Published as a conference paper at COLM 2024
6 How to evaluate tool use?
In this section, we study existing LM-tooling benchmarks ($6.1) and their evaluation metrics 6.2 lastly, we discuss the missing yet important evaluation aspects of tools ($6.3)
6.1
Testbeds for evaluating tools
LM tool use can be evaluated on (i) repurposed existing datasets that can additionally benefit from tools 6.1.1 and (ii) newly crafted benchmarks that necessitate tool (...12).
6.1.1 Repurposed Existing Datasets
Many tasks solvable by using LMs, yet often with great difficulty inefficiency. There- fore, some works use tool-augmented LMs as an alternative approach to solve these tasks.
Most of these datasets require reasoning. Starting from NI. questions that demand com- plex reasoning with Big-bench (bench authors 2023), mathematical problems with MATH (Hendrycks et al. 2021), to answering world facts NaturalQuestions (Kwiatkowski et al., 2019) and IriviaQA (Joshi et al., 2017). Beyond free-form texts, datasets with structured data also benefit from tools, particularly table-based QA with tabular math world problems in TabMWP (Lu et al. 2023b), Wikipedia tables in WTQ (Pasupat & Liang/ 2015), and complex- structured tables in Hrlab (Cheng et al. 2022). Beyond text modality, datasets involve other modalities benefit from modality--xtendding tools, e.g,, answering questions about an image in GQA (Hudson & Manning, 2019), or image pairs NLVR2 (Suhr et al. 2019).
Because tool use is an alternative method to solve these datasets, evaluations of these tool- augmented systems follow the standard evaluation process of these datasets, i.e, answer exact match. Note that to obtain the answers, all tool calls need to be executed, since the execution outputs are integrated into the final answers, as introduced in $3
6.1.2 Aggregated API Benchmarks
Existing benchmarks can only benefit from a limited set of tools, yet there are far more tools we can utilize to perform versatile tasks, particularly the web APIs created by human developers. Table 2.
we list recent benchmarks for using APIs from various sources.
Benchmark
Tool Source
Example Curation
Domain ($4.
Executable
ToolBench
existing dataset
adopted, human annotated model synthesized
ToolBench2
RapidAPI
ToolQA
existing dataset
model synthesized
8
/
ToolAlpaca
PublicAPIs
model synthesized
8
X
API-Bank
PublicAPIs
human annotated model synthesized
Meta'oool
OpenAl Plugins
Gorilla
HE, Torch, TF
model synthesized
X
HuggingGIT
HF
human annotated
C
X
Task Bench
HE, PublicAPIs
model synthesized
C
X
Table 2: Benchmarks that use API tools to solve tasks. HF is short for HuggingFace.
Tool sources Tools are mainly aggregated from existing datasets or public APIs. Bench- marks aggregating existing datasets (Xu et al. 2023 Zhuang et al., 2023) are often limited in domains. Other works scrape APIs from online sources such as Public APIs (Tagg al. 2023), RESTful APIs [Tang et al., 2023),
the OpenAl plugin list (Huang al. 2024). Beyond human-written APIs (Li et al. 2023), neural models from ML platforms can be similarly formatted APIs (Patil et al. 2023 Shen et al. 2023a However, tools come from heterogeneous sources, it is hard to select the best from or unify all these benchmarks.
Example curation While most examples adopted from existing datasets human annotated (Xu et al., 2023),, only [i al. (2023) do so for scraped APIs, by surveying 500 people and creating 314 dialogues manually. Most other works prompt GPT models to
7

---

Published as a conference paper at COLM 2024
synthesize examples QQn et al. 2024 Tang et al. 2023 Shen et al. 2023b Zhuang et al. 2023 Huang et al. 2024), however, leading to issues of naturalness and executability.
First, LMs are used to create examples given a heuristically selected set of tools, leading to two issues: (i) users may not use the selected tools together in practice and (ii) the synthesized examples may not reflect the natural use cases of these tools. Second, 5 out of benchmarks in Table 2 do not support tool execution, often to alleviate the cost of hosting unstable APIs, which cause issues it evaluation. Instead of matching final responses using
lexical- Li et al. 2023) or neural-based metrics (Tang et al., 2023 Qin et al. 2024), works
resort to pseudo matching of API calling expressions with lexical Tang et 2023 Shen
et al. 2023a Huang et al. 2024 and syntactical (Patil et al. 2023 Shen et al. 2023b) means.
6.2 What metrics are measured now?
Task completion Tools are used to assist task solving. Most works that allow tool execu- tion evaluate the task completion score to quantify the effectiveness of utilizing tools.
Tool selection Another common metric the accuracy of selecting the correct tools. More concretely, it can serve as a measure of LM planning abilities - the process of breaking down a task into multiple steps and selecting tools to complete individual steps.
Tool reusability While tool reusability is often deemed important in took-making ap- proaches (Cai et al. 2023: Yuan et al. 2023), only WWag et al. (0024) evaluates it by the size of toolboxes. Reusable tools have generic functions, can be efficiently (re)used multiple times, and facilitate human verification in speed and accuracy aspects (Wang et al. 2024b).
6.3
What properties are missing?
Efficiency of tool integration The benefits of tools come with the cost of extra computation, particularly for teaching LMs to use tools via training or prompting. Besides performance gain, reporting the computation overhead can enable fairer comparisons of different works.
Beyond task accuracy, the performance of tools themselves is also important. Tool performance covers multiple aspects such returning results quickly, using less compute, and not failing unexpectedly. One way to measure these aspects is to conduct API Ehsan et al., 2022) on their runtime, memory usage, and success rate.
Quality of tools
testing (Yasar 2022
Reliability of unstable tools
Particularly for tools that involve neural models or randomized components, their output quality may be unstable and unpredictable. For example, the VQA tool Guppa & Kembhavi 2022) may answer some questions correctly but others incorrectly. essential be aware of and reduce this uncertainty contrast to stable, rule-based tools.
Reproducible testing Many world-interacting tools may return different results at dif- ferent times. E.g., check weather may return "sunny" today but "cloudy" tomorrow. This prreproducible behavior poses great challenges to create static evaluation benchmarks with reference answers. While some works alleviate this by evaluating API calls without execut- ing them, more rigorous method could be parallel lesting (Sharma et al. 2018) executing model-generated and reference programs in parallel and matching their final outputs.
Safe usage Many systems may only opt to use tools if they are trusted to be secure (Barbir et al. 2007). At the very least, users favor tools that can be easily understood and verified. Yet there may be more security issues such as authentication and data integrity (Ehsan et al., We encourage readers to peruse the referenced works above for thorough studies.
2022
7 Trade-offs in tool usage
Using tools often brings better performance, however, should we always use tools? More concretely, is the performance gain from using tools worthy of the computation cost spent for LMs to learn tools? In Table 3) we empirically study the performance gain and learning cost of various methods on their tested datasets, from which we discover more efficient (i.e,, achieve greater gains with less compute) methods and tasks that benefit more from tools.
8

---

Published as a conference paper at COLM 2024
For each work and each dataset they experimented with we measure the performance gain after LM learned or made tools to solve tasks, against baseline LMs with no prior exposure to tool-related information. We also quantify the computation cost of these methods in token-consuming training and inference processes. More computation details in g
# Tokens (M)
Type
Method
Task
A Perf.
# Params (B)
train
test
cloze math
14.7
6.7
642.1
269.0
+ 30.4
3864.2
421.0
ToolFormer
QA
+ 5.8
6.7
1101.2
189.0
tool use
multilingual temporal
0.2
6.7
606.0
274.0
+ 13.0
6.7
508.8
202.0
API-Bank
API
+ 24.4
190414.6
0.0
ToolAlpaca
API
+ 45.2
241889.3
0.0
science table
+ 2.6
0.0
88.3
Chameleon
1.9
0.0
325.9
LATM
BigBench + 29.1
28.5
4720.0
math
+ 4.5
0.0
5113.6
CREATOR
tool making
table
0.0
6827.6
math
13.2
4126.6
4098.5
CRAFT
table
+ 17.2
2750.6
5018.2
math
+ 21.0
0.0
1825.2
TroVE
table
+ 12.0
1358.8
Table 3:
Computation cost (number of tokens in M and parameters in B) of tooling methods and their performance gain on experimented datasets. To fairly compare costs on datasets with different sizes, we report the average number of tokens spent on testing example.
What tasks benefit the most from tools? In general, tasks that cover multiple domains experience the highest increase, such ToolAlpaca in tool-using and BigBench in tool- making scenarios. Substantial gains may be expected on API benchmarks, because all examples synthesized use cases for designated tools ($4.1), no-tool baselines are deprived of necessary components (i.e., tools) to solve the task, thus achieving much lower accuracy.
On existing datasets, the ToolFormer method in Figure 5) the most efficient on MATH problems, showing the biggest 30.4 increase with little computation. While other tasks im- prove less, multilingual tasks even degrade by -0.2 points, despite using a similar amount of compute. This variance across tasks aligns with our discussion in $4 tasks neces- sitating ML-based tools benefit less from them.
Performance Gain
30
math
20
clloee
temporal
10
question answering
multilingusl
What methods are efficient in tool-making? While it is hard to fairly compare works testing on different datasets, the three tool-making methods (Creator, CRAFT, TROVE), experiment on the same MATH and TabMWP datasets Figure 6) thus enable fair comparisons. TROVE is the most efficient method, costing only 1.2-1.4K tokens while increas- ing the accuracy by 12.0-21.0 points. In contrast, CREATOR and CRAFT less efficient, costing 3.8-6.0 times of compute, yet achieve only minimal (0.0-4.5%) or comparable (4.1-
50 100
150
Computation Cost (M B)
Figure 5: ToolFormer method on various tasks.
5.0%) accuracy increases.
Training-time vs inference-time cost Training time and inference-time costs may not be equally important many practitioners, since inference may be run many times but training often only needs to be done once. If we only consider inference-time cost in Table 3
We did not measure some works due to insufficient resources.

---

Published as a conference paper at COLM 2024
the efficiency ranking of tool-using methods changes: Tool- Former requires more compute than API-Bank and ToolAl- paca. If the user has sufficient budgets for training and higher demand on inference-time efficiency, the training approaches proposed by API-Bank and ToolAlpaca could be more suitable.
Performance Gain
30
Math
Table
20
TroVE
ORAFT
10
CREATOR
Final Remarks
5
10
Our survey presents a definition for LM-used tools, various scenarios to apply tools, systematic summaries of existing methods, and an empirical analysis guide when (on what tasks) and how (use what methods) one should use tools. We believe tools can greatly extend and facilitate LM abilities, and hope our work elicits more discussions and research developments in (i) curating benchmarks with natural use cases and executable tools, (ii) using multi-faceted evaluation metrics as in $6 and (ii) exploring more realistic and challenging scenarios for tool-using and tool-making techniques.
Computation Cost (M B)
Figure 6: Comparing different
tool-making methods.
Acknowledgments
We thank Saujas Vaduguru, Sherry Tongshuang Wu, Jiawei Liu, Shihao Liang, Pengfei Liu for the helpful discussions. Zora Zhiruo Wang is supported by the CMU Teng Family Presidential Fellowship. Hao Zhu is supported by NSF EÁGER Award #2141751.
References
Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen. Retrieval-based language models and applications. In Proceedings of the 61st Anmual Meeting of the Association for Computa- tional Linguistics (Volume 6: Tutorial Abstracts), Association for Computational Linguistics, 2023. URL /ttpis:://aaaattie /ttp///anoooooooorg/////////////
Matej Balog, Alexander L Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. Deepcoder: Learning to write programs. arXiv preprint 1XXim:661..0190 2016.
Abbie Barbir, Chris Hobbs, Elisa Bertino, Frederick Hirsch, and Lorenzo Martino. Challenges of testing web services and security in soa implementations. Test and Analysis of Web Services, pp. 395 440, 2007.
BIG bench authors. Beyond the imitation game:
Quantifying and extrapolating the ca- pabilities of language models. Transactions on Machine Learning Research, 2023. URL
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1533-1544, 2013.
Daniil Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes, Autonomous chemical research with large language models. Nature, 624(7992) 2023.
Matthew Bowers, Theo X. Olausson, Lionel Wong, Gabriel Grand, Joshua B. Tenenbaum, Kevin Ellis, and Armando Solar-Lezama. Top-down synthesis for library learning. Proc. ACM Program. Lang., 2023. URL https::///ooooogg////00/ 10tp://///0001155/11233353
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020. URL httppp:://ppooeddngss.eetttee cc/ e/paper files/ htts/paper/222100 hial
10

---

Published as a conference paper at COLM 2024
Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as tool makers. arXiv preprint arXio:2305.17126, 2023. httpstps: arxiv.org//ppp / 2305.17126
Yihan Cao, Shuyi Chen, Ryan Liu, Zhiruo Wang, and Daniel Fried. Api-assisted code generation for question answering on varied table structures. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, 2023. URL
//aclanthology.org/// http::/rr//2....mmmmmmmaa.....a mainp-
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained on code. Xiv preprint ar Akiv:2107.03374 2021.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2023, URL https:/ J http://oeeeiiieeeett////m forteetttoooddddo
YfZ4ZPt82d
Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian- Guang Lou, and Dongmei Zhang. Hitab: A hierarchical table dataset for question answering and natural language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), May 2022.
Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. Binding language models in symbolic languages. In The Eleventh International Conference on Learning Representations, 2023. URL htttp:///peeneeiiewnnt// wwwpet//dnoomddd-i-P22ddd
Adeel Ehsan, Mohammed Ahmad M. E. Abuhaliqa, Cagatay Catal, and Deepti Mishra. Restful api testing methodologies: Rationale, challenges, and solution directions. Applied Sciences, 2022. URL https:/ http://wwwwwmmmplcocooo/////////
Atty Eleti, Jeff Harris, and Logan Kilpatrick. Function calling and other api updates, 2023. URL https:///ppenai.com /blog/ /wwp:/ttoo-clllinnnnnddoeeee-i-e
Kevin Ellis, Lionel Wong, Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lore Anaya Pozo, Luke Hewitt, Armando Solar-Lezama, and Joshua B Tenenbaum. Dreamcoder: growing generalizable, interpretable knowledge with sleep bayesian program learning. Philosophical Transactions of the Royal Society A, 2023.
Difei Gao, Lei Ji, Luowei Zhou, Kevin Qinghong Lin, Joya Chen, Zihan Fan, and Mike Zheng Shou. Assistgpt: A general multi-modal assistant that can plan, execute, inspect, and learn. aiv preprint ar 206.24006008640 2023a.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pp. 10764-10799, PMLR, 2023b.
Gabriel Grand, Lionel Wong, Matthew Bowers, Theo X Olausson, Muxin Liu, Joshua Tenenbaum, and Jacob Andreas. Lilo: Learning interpretable libraries by compressing and documenting code. arv preprint ar XrXiv:2310.19791, 2023.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual
reasoning without training. arv preprint 1rXio:22f1.11559, 2022. URL httpss/ /arxiv/o//xxvv//////// pdf/2211.1155)
Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. In shirth- Conference on Neural Information Processing Systems, 2023. URL https:// http:///ppperreeeeeeefrruumm
BHXsb69bSx
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. Xiv preprint arXiv:2103.03884, 2021. URL
11

---

Published as a conference paper at COLM 2024
Cheng-Yu Hsieh, Si-An Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, and Tomas Pfister. Tool documentation enables zero-shot tool-usage with large language models. arviv preprint arXiv:2308.006777 2023.
Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun. Metatool benchmark for large language models: Deciding whether to use tools and which to use. Xiv preprint arXivv2310.03128, 2024.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2017. JRLhttpsst
accanthology.ongPP17--11
Mojtaba Komeili, Kurt Shuster, and Jason Weston. Inteenet-augmented dialogue generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022. URL /ttthttss:// http://ccaantoooggogg/////22.///
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszko- reit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answer- ing research. Transactions of the Association for Computational Linguistics, 2019. URL httpss//acaanhholoy.orgg Q09-1
Angeliki Lazaridou, Elena Gribovskaya, Wojeiech Stokowiec, and Nikolai Grigorev. Internet- augmented language models through few-shot prompting for open-domain question answering. arXiv preprint ar arXir:2203.05115. 2022. URL httpp::/
abwp:/ggaab///2000 2203.05115
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented lims. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL http:////caaaaatlllggg///////// htwp://oooooo.oo///222///m/////e 2023.emmll-mmin..mmmm 187
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEÉ International Conference on Robotics and Automation (ICRA), pp. 9493-9500. IEEE, 2023.
Percy Liang. Lambda dependency-based compositional semantics. ar arXio:1309.4408, 2013.
ar preprint
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, and Percy Liang. Reinforcement learning web interfaces using workflow-guided exploration. In International Conference on Learning Representations, 2018. /tL/sttps:// httppss//ppppeeeeeevveeee//rr/rr
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, SongSnn Song-Chun co
Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a. URL http:://geeenevvvvve////uounddop
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chur Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXio preprint ar XrXi::2209.1466 14610, 2023b. URL https://arxxivorg/ pdtp://9.//2221110000
Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfr- mubeedding-mistrleennii text retrieval with transfer learning. Salesforce Al Research Blog, 2024. URL
12

---

Published as a conference paper at COLM 2024
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pa- sunuru, Roberta Raileanu, Baptiste Roziere, Timo Schick, Jane Dwivedi- Yu, Asli Ce- likyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. Augmented language models: a survey. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL
OpenAL. New embeddings models and api updates. URL https://openai.com//
Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. Art: Automatic multi-step reasoning and tool-use for large language models. arviv preprint (rXi::2333.09014, 2023. http:sss:: http::///axxvvorgg////// 2330 2303 09014
Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. arv preprint 12XX::2200.122555 2022. URL https::/ /awxvvorr/ abs/ 2205.12255
Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Compulational Linguistics and the 7th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers). Association for Computational Linguistics, July 2015. URL https:/ htc://accaee Bilanthology.org/P55--II..
Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. arXit preprint ar arXi::2305.180 153334 2023.
Cheng Qian, Chi Han, Yi R. Fung, Yujia Qin, Zhiyuan Liu, and Heng Ji. Creator: Disentan- gling abstract and concrete reasonings of large language models through tool creation. Xre preprint arXto:2305.14318, 2023. /tttss/ss///axiii
Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang, Jing Yi, Yuzhang Zhu, Zhenning Dai, Lan Yan, Xin Cong, Yaxi Lu, Weilin Zhao, Yuxiang Huang, Junxi Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang, Cheng Yang, Tongshuang Wu, Heng Ji, Zhiyuan Liu, and Maosong Sun. Tool learning with foundation models. arv preprint ar Xfo:22304.08354, 2023.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toollim: Facilitating large language models to master 16000+ real-world apis. In The Twelfth International Conference on Learning Representations, 2024. URL https:/
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends@ Information Retrieval, 344)3333-38, 2009.
Wm. Paul Rogers. Encapsulation is not information hiding, JavaWorld, 2001. URL hhtpss//: /
Stuart J Russell and Peter Norvig. Artificial intelligence a modern approach. London, 2010.
Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. ar preprint ar arXit:2302.04761, 2023. URL https;
/ //arxiv.org/ attpivv.orr//ss///3303/ 20000///332.06611
Abhinav Sharma, M Revathi, et al. Automated api testing. In 2018 3rd International Conference on Inventive Computation Technologies (ICICT), pp. 788-791. IEEF, 2018
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. In Thirth- Conference on Neural Information Processing Systems, 2023a. URL https:/ /opentrview..nt/ forum?id=yHdTTs<Y6CC
13

---

Published as a conference paper at COLM 2024
Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yueting Zhuang. Taskbench: Benchmarking large language models for task automation. Xrv preprint ar drXio:2311.18660 2023b.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zett and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 10740-10749, 2020.
Robert W Shumaker, Kristina R Walkup, and Benjamin B Beck. Animal tool behavior: the use and manufacture of fools by animals. JHU Press, 2011.
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In Workshop on Language and Robotics at CoRL 2022, 2022. httpp:://ppeeeeeeeeeee//rroee
Elias Stengel-Eskin, Archiki Prasad, and Mohit Bansal. Regal: Refactoring programs to discover generalizable abstractions. Xio preprint WXio:2401.16467, 2024.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisfics. Association for Computational Linguistics, 2019. URL htttps//aclanthologg.orrg PP9--1444
Theodore Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas Griffiths. Cognitive architectures for language agents. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL httpt::/tt//peeeeee http:////eeeeeeeeeeeee////odmddd
Survey Certification.
Didac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. Xrviv preprint XXix:2303.08128, 2023.
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. arxiv preprint ar xrio:23366003011 2023.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas Claire Cui, Marian Croak, Ed Chi, and Quoc Le. Lamda: Language models for dialog applications. arXiv preprint Xiv:2201.000 200.0999,, 2022. https://s httpss/////xxx/g///////// abs/ 2201.08239
Vijay Viswanathan, Chenyang Zhao, Amanda Bertsch, Tongshuang Wu, and Graham Neubig. Prompt2model: Generating deployable models from natural n language instruc- tions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics, 2023, URL http:///acaattoooggro///222mm//m
Ben Wang and Aran Komatsuzaki. Gpt-i-6b: A 6 billion parameter autoregressive language wwwp:mmesssttttteeeeee transformer-jax
model, 2021. URL https:/ http:////tt////gttttuoo/////////
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023a. URL http:///ppenevvvvweee/oooumtttap
14

---

Published as a conference paper at COLM 2024
Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. Mint: Evaluating Iims in multi-turn interaction with tools and language feedback. arXiv preprint ar Xrix:2309.10691, 2023b.
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better Ilm agents. arxio preprint ar XrX0:22402.01030, 2024a.
Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig. Execution-based evaluation for open-domain code generation. In Findings of the Association for Compulational Linguistics: EMNLP 2023, pp. 1271 Association for Computational Linguistics, December 2023c. URL httphhhttt//aace httpht/s:://ccaatoooogggg//
2023. findings-emnll....
Zhiruo Wang, Daniel Fried, and Graham Neubig. Trove: Inducing verifiable and efficient toolboxes for solving programmatic tasks. ar Xrxiv preprint arXio:2401.12869 2024b. URL
https///aanii.ooggab///2//0/0/
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the Twelfth Language Resources and Evaluation Conference, 2020. URL https:/ httpss///aaaatholgg..ooooo hat.mnoooooogyoro//2200rre----. 2020.
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. Break it down: question understanding benchmark. Transactions of the Association for Computational Linguistics, 2020. URL lttps:// / /ww.org//oooo//////o 1ttp://11111/2//a..10333333 t00099
Catherine Wong, Kevin M Ellis, Joshua Tenenbaum, and Jacob Andreas. Leveraging lan- guage to learn program abstractions and search heuristics. In Proceedings of the 38th International Conference on Machine Learning, Proceedings of Machine Learning Research, 2021. URL http::///ppoeeediggmiiiigeee/eee wtr.:iggssm.....ss.. v139/ /
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. arviv preprint ar ato:2309.078644 2023.
Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. On the tool manipulation capability of open-source large language models. arXiv preprint arXiv:2305.16504, 2023.
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. Xiv preprint arXie:2303.1138/ 2023.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. shop: Towards scalable real-world web interaction with grounded language agents. In Advances in Neural Information Processing Systems. Curran Asso- ciates, Inc., 2022. URL htttps://prooeeeeeee http://pooeeeddisseerrpsssso /ppperappeepaeeee filpeaiillleeeeepppppppppaaaeale patess/apprr20000 / Compe///bbbcc-pppeeooCoeneeeeeee 22adleeeelDeeeeeeePPP/aaaabbeeee
Web-
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023. httttp::///oppeeeie..eettt
forum?id=WE..luyUU.
Kinza Yasar. Software testing, 2022. URL httpp://wwww.eeccaar...ooo////ts
software-testing
Pengcheng Yin and Graham Neubig. A syntactic neural model for general-purpose code
generation. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 440- 450, Vancouver, Canada, July 2017, Association for Computational Linguistics. doi: 10.18653/v1/P17-1100 URL https:/ huR.hhtps///aannoooooo/////////
15

---

Published as a conference paper at COLM 2024
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingming Yao, Shanelle Roman, et al. Spider: large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint ar WrXiv:189908880 2018.
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang- Huei Lee, Montse Gonza- lez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, al. Language to rewards for robotic skill synthesis. arXio preprint ar Xio:2306.08647, 2023.
Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi R. Fung, Hao Peng, and Heng Ji. Craft: Customizing llms by creating and retrieving from specialized toolsets. arXiv preprint (rXiv:2309.17428, 2023.
John M Zelle and Raymond J Mooney. Learning to parse database queries using inductive logic programming, In Proceedings of the national conference on artificial intelligence, pp. 1050-1055, 1996.
Luke S Zettlemoyer and Michael Collins. Learning map sentences logical form: Struc-
tured classification with probabilistic categorial grammars. ar arviv preprint ar arXio:1207.1420 2012.
Kechi Zhang, Huangzhao Zhang, Ge Li, Jia Li, Zhuo Li, and Zhi Jin. Toolcoder: Teach code generation models to use api search tools. arXiv preprint xrXi:2305.000332 2023. URL
hittps://arivv..o abs /bbb///33000032
Shuyan Zhou, Uri Alon, Frank F. Xu, Zhiruo Wang, Zhengbao Jiang, and Graham Neubig. Docprompting: Generating code by retrieving the docs. In The Eleventh International Conference on Learning Representations, 2023. URL http://oppeneeiee.eeefoooou
ZTCxT212Ru
Shuyan Zhou, Frank Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig, Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, 2024. URL httpss///ppeeeeiieenttt/////// formtm/df
oKn9c6ytLx
Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. Toolqa: A dataset for Ilm question answering with external tools, or Xrv preprint Xiv:23006 133044 2023.
16

---

Published as a conference paper at COLM 2024
A Detailed computation process for tooling trade-offs
For each method measured in we describe the detailed processes in estimating their computation cost and performance improvement. For open-source models, we estimate cost C = 6ND, where N is the number of tokens and D is the parameter size (Figure 7 left). Because the parameter size D of closed-source GPT is unknown, we only measure the number of extra tokens N per example (Figure 7| right).
APT
Cloze
Math
Multilingual
Math
Table
BigBench
Science
60
30
Performance Gain
ToolAlpaca
LATM
O Chameleon
LATM
TroVE
40
20
ToolFormer
API-Bank
20
CRAFT
ToolForeer
API-Bank
10
0
ToolAlpaca
Chameleon
Creator
0
-20
0
0
2
Computation Cost (M BB
Number of Tokens (K)
Figure 7: Computation cost of different approaches using open-source (left) and closed- source (right) models, and their performance gain on experimented datasets. We use different colors to represent tasks and different shapes to represent methods.
A.1 Methods using known-sized models
For methods using models whose parameter sizes are known, we estimate the computation cost by the FLOPs during any additional modules such training and inference with additional context. In general, the computation cost is majorly affected by (1) the number of tokens processed, and (2) the parameter size of models.
API-Bank (Li et al. 2023) This work trains the Lynx model that uses tools to solve problems in the proposed API-Bank dataset. The Lynx model is initialized by Alpaca 7B parameters, and trained on the API-Bank training set with 3 epochs. Therefore, we adopt the Alpaca 7B as the baseline and Lynx as the tool-using model, where the 3-epoch training is the additional computation cost introduced to enable tool use. We calculate the total number of tokens involved in the training process, including the example i/ and additional instructions. Because the baseline and proposed method use the same prompt at inference time, no additional computation is required. Regarding task performance, we adopt the total correctness across all evaluation systems, reported in Table 3. We report the difference between the fine-tuned Lynx-7B and the zero-shot Alpaca-7B.
AoolAlpaca (Tang et al., 2023) This work proposes the ToolAlpaca dataset and trains Vicuna models to use tools. The baseline models are Vicuna-7B and Vicuna-13B models. The trained tool-using models are called ToolAlpaca-7B and ToolAlpaca-13B models. All ToolAlpaca models are trained on the training split for epochs, so we estimate the cost during this training process for 7B and 13B models, respectively, We adopt the 'overall' results reported in Table 3, on examples with both simulated tools and real-world APIs, and report their average results. We measure the performance gain by the difference between the ToolAlpaca-7/13B and Vicuna-7/ 1otuna
Toolformer (Schick al. 2023 This work integrates five tools - question answering
system, calculator, Wikipedia search, machine translation system, and calendar - respec- tively for five tasks transformed from a subset of CCNet (Wenzek et al., 2020). Starting with GPTT- models (Wang & Komatsuzaki 2021 as the no-tool baseline, they train on 25k model-syyntessized examples for each tool and obtain the Toolformer models, causing total of 1M FLOPs for each task. At inference time, they add special instruction and in-context examples to prompt tool using, resulting in extra compute. Because each task contains multiple datasets, report the average results to represent the general task performance.
17

---

Published as a conference paper at COLM 2024
A.2 Models with unknown size
While many of the works use GPT-3355 or GPT-4 models that do not release their parameter size, we estimate the cost by using the number of tokens processed in extra modules.
Chameleon (Lu et al., 2023a) This work proposes to take a tool-augmented approach to improve on two existing datasets - ScienceQA and TabMWP. Because all experiments use ChatGPT and GPT-4 models, whose parameter sizes are unknown, we only examine results with (the better) GPT-4 model to fairly compare with other methods using GPT- 4. Specifically for the ScienceQA dataset, we adopt the Chain-of-Thought (CoT) baseline reported the paper, and report task accuracy as in the ALL column Table 3. We calculate the difference in number of tokens between the proposed Chameleon methods against the
CoT baseline. For the TabMWP dataset, we adopt the Program-ot oiraaam----Thouhttt (PoT) baseline
and similarly calculate the token number difference using the provided results
We adopt
numbers in the ALL column in Table 4 as the TabMWP accuracy.
LATM (Cai et al. 2023) This work proposes to LMs to make tools for individual tasks in BigBench. Compared to the thain-of-thought (CoT) baseline, the proposed LATM method integrates training, validation, and inference stages to make tools and solve questions. We estimate the compute cost by the additional number of tokens used for LATM than for CoT. We measure each method by averaging its accuracy across all six selected tasks.
CRAFT (Yuan et al. 2023) This work uses LMs to make tools for math, table, and image reasoning tasks. We calculate the number of tokens used during training and inference,
using its released code and data CRAFT similarly implements CoT as the baseline, and proposes further training, verification, and finally testing in the CRAFT method. We report its task accuracy on the representative datasets from each task MATH, TabMWP, and GQA- - to enable fairer comparison with other works having overlapping datasets.
CREATOR (Qian et al. 2023) As prior work for CRAFT, CREATOR similarly tests on MATH and table tasks, but designs its methods differently. addition to CoT, this work implements stronger program-oriented baseline called Program-of-Thought (PoT). We also adopt PoT as the main baseline without tool making or using. The CREATOR method operates at test time, with multiple steps through tool making, solution generation, verification, rectification, etc. We calculate the difference in number of tokens between the CREATOR approach and the baseline PoT setting. We adopt the task accuracy reported in Table 2 (MATH) and Table 3 (TabMWP) from the original paper.
TroVE (Wang et al. 2024b) TroVE also induces tools without training supervision. This work adopts the primitive baseline, a presumably stronger version of PoT yet without much textual explanation. The main implementation change in TroVE is the three-mode generation and multi- sampling. We calculate the additional tokens used in TroVE compared to the primitive baseline. The dataset reports task accuracy, solution complexity, and toolbox size, we only adopt the task accuracy to fairly compare with other works.
http:///gittbbbcmm//llaanaaaaaat
18


Reviews:
[{"review_id": "PODo4oXibb", "reviewer": "Reviewer_H5ew", "summary": "The paper surveys previous works about LMs with tool-use.\nSpecifically, it proposes a definition of tool-use as calling methods that are executed externally to the LM.\nIt then details about types of tool-use and tool types, and advanced tool use scenarios like tool selection and programmatic contexts.\nFinally, it presents an empirical investigation of the performance and computation cost of different tool-use approaches.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your recognition of the importance and usefulness of our survey paper!\n\n- **R1: survey lacks some works**\nThank you for pointing out these works! \nFirst, the three works you pointed out were published within three months of our publication date, so we didn’t get a chance to include them all, but we are happy to add them to our revised version. \nAlso, thanks for pointing out the newsletter source. We believe our paper has discussed most of the papers mentioned in it, but we are happy to include the earlier papers left in our revised version. (Specifically, the only tool-related earlier paper left out is *Neural Programmer-Interpreters (Reed and Freitas, 2015)*, and we will add it)\n\nIn general, we believe we have provided a comprehensive scope of tool-related topics, and thank you again for providing these additional papers to further enrich the content of our paper!\n\n- **R2: reviewer did not learn enough**\nThank you for your feedback! Our survey aims to serve as a useful handbook for LM-used tools. Compared to existing methodology or survey works, our paper *uniquely points out* the definition of tools (section 2), a clear taxonomy of tools (section 4), a systematic summary of testbeds (section 5), and most importantly, proposes five essential evaluation aspects (section 6, along with empirical study about them in section 7) that have not yet been evaluated in tool-related literature.\nWe particularly discuss “different tool-use methods” in section 3 and 5. We did not spend more content on it due to content length limitations and the importance of other aspects (as mentioned above). However, because tool-use methods to date are mostly homogeneous (e.g., train then test), this lack of diversity may also give the impression of a lack of pros-and-cons discussion.\nIn addition to these aspects, if you have other specific topics of interest, feel free to point them out and we are happy to try to incorporate them in our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Tool use is an important subject today when developing language models.\nAs there is a lot of work on the subject, it may be useful to have such surveys in hand.\nThe survey covers different aspects of tool use in a useful high-level perspective.", "reasons_to_reject": "I find the survey and empirical results a bit lacking - I don't think the survey is very comprehensive (e.g. some relevant recent work is missing, for example https://aclanthology.org/2023.findings-emnlp.926/, https://aclanthology.org/2024.eacl-short.10/, https://aclanthology.org/2024.eacl-long.7/). Some earlier works in this area were also missing, e.g. as mentioned in this blog post on the subject: https://newsletter.ruder.io/p/tool-augmented-llms\n\nIn general, I don't think I learned enough from the survey - it may be somewhat useful for someone completely new to the research area, but I think it can be improved further by discussing more works and diving a bit deeper into how different tool-use methods work and the pros and cons for each.", "questions_to_authors": "typo on page 3 - SPARL\n\nsection 2.1 - \"tools Shumaker et al. (2011)\" - missing comma\n\nintro - \"to obtain time\" --> \"to obtain the current time\"?\n\nabstract - \"powerful yet mostly\" --> \"powerful, yet mostly\"?", "extra_scores": {}}, {"review_id": "ZKSAmOKVvs", "reviewer": "Reviewer_m17M", "summary": "The authors aim to unify various understanding and definitions of \"tools\" in connection with language models. This aim is highly relevant and needed to guide interested (lay) people as well as researchers and developers and help them to communicate better. \n\nThe paper contains a nice overview on existing definitions and frameworks and evaluations metrics.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of the importance and quality of our work!\n\n- **Q1: Better caption for Figure 1**\nThanks for the feedback. Our figure1 intends to illustrate the text paragraphs around it, to emphasize how (calculator/api) tools come naturally to address two example limitations of current LMs (in math and in getting up-to-date information). We will cite figures1&4 and make their captions clearer in the revised version.\n\n- **Q2: Replace section sign with ‘section’**\nThank you, we will update this in the revised version!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written and highly significant. The authors will be able to address the issues raised and submit a revised version.", "reasons_to_reject": "There are no reasons to reject the paper", "questions_to_authors": "What exactly does figure 1 show? The caption does not help very much in understanding it. Please provide a better caption. Additoinally, this figure is not referenced in the text. Please do so and provide contextualizing information to the reader: what are they supposed to see in this figure? Figure 4 is also not referenced in the text.\n\nRefering to sections in the paper with the paragraph sign is rather odd. Please use \"section 2\" etc.", "extra_scores": {}}, {"review_id": "3SNeRD4ckt", "reviewer": "Reviewer_oLaS", "summary": "This study systematically surveys tools associated with language models (LMs), specifically external programs utilized in LM applications. Beginning with a unified definition of \"tools\" in the context of LMs, the paper outlines various categories applicable to text-based applications and more sophisticated scenarios. Additionally, the study explores evaluation methods and metrics pertinent to these external programs within the realm of LM.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of the various aspects covered by our survey!\n\n- **R1: RAG is tool**\nWe agree, and **we have explicitly mentioned** that “RAG is a tool” in the paper: we exemplify RAG as a “knowledge access” tool in Table 1, and we discuss RAG in Sec 4.1 the “knowledge access” part.\n\n- **R2: Merge into one category**\nWe would like to clarify that **not all these tools are to obtain knowledge**. E.g., make_post() tool from “interact. w/ world” takes actions to change the world instead of obtaining data. So merging these categories would not be appropriate.\nWe group tools in a fine-grained way to highlight their uniqueness and provide more insights. E.g., readers can learn that “skilled LM” tools may not always do tasks correctly, which is not true for get_time() from “interact. w/ the world” that always perfectly solves the task. But readers may not grasp this insight if we merge these tools. Nonetheless, readers are free to further interpret based on our discussions.\n\n- **R3: Lack bottleneck discussions**\nWe **discuss multiple bottlenecks** of tools across the paper: tasks that tools cannot easily solve in Sec 4.2 and 5.1; tasks that no existing tools can solve in Sec 5.3; limitations in collecting/evaluating tools in Sec 6.1. Lastly, we provide an empirical analysis in Sec 7 on accuracy-efficiency performance, responding to the reviewer's “how well they perform” question.\n\nIn general, we believe **our work has sufficient discussions** for the above points regarded as missing by the reviewer, and has clear contributions in definition, taxonomy, evaluation, and empirical study of tools. We would appreciate it if the reviewer could reconsider these aspects. We are happy to take any more feedback!\n\n- **Q1: why programmatic tools**\nSince programming is also a popular medium of tool use, we find it critical to clarify how program and tool relate to each other.\n\n- **Q2: Guidance to create tools**\nPlease refer to Sec 5.3 for details on various tool-making testbeds/methods and use the most suitable one. From our empirical analysis in Sec 7, we recommend TroVE when w/ data/compute limitations, and CRAFT otherwise.\n\n- **Q3: Benefits of tools**\nAs we stated in Sec1 and Figure1, tools can help LM task-solving (do math accurately) and extend LM abilities (know the current time), which the strongest LM, even with fine-tuning, is fundamentally unable to do. We would like to note that RAG is one form of tool-using, so RAG already benefits from using tools.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A unified definition of “tools” is proposed for LMs.\nA range of relevant works have been listed and categorized in application scenarios.\nThe evaluation of such “tools” has been discussed, particularly the metrics.", "reasons_to_reject": "The contribution of this paper may not be sufficient, considering the coverage and the analysis. For example, retrieval-augmented generation (RAG) can also be viewed as an external tool for LMs, particularly, large language models, to generate more accurate content based on given documents.\nAs for the categorization, it seems to me that “knowledge access”, “Interaction w/ the world”, and “Special-skilled LMs” can be classified as the same functional category, e.g., obtaining external knowledge or information.\nIn addition, readers may be more interested in identifying bottlenecks for such “tools” and how well they can perform on downstream tasks.", "questions_to_authors": "1. Why are the tools in programmatic contexts special and singled out as a section?\n2. Can you be more specific on tool creation? Can you give us some guidance on how to create a tool for a specific downstream task based on LMs?\n3. What are the benefits of using tools for LMs, compared to other methods like fine-tuning and RAG?", "extra_scores": {}}, {"review_id": "Cnjn8UaPY3", "reviewer": "Reviewer_XLAS", "summary": "This survey paper provides a unified definition of tools as external programs used by language models (LMs) to enhance their abilities. It systematically reviews various scenarios where LMs use tools, including knowledge access, computation, interaction with the real world, and processing non-textual data. The paper also discusses advanced tool usage methods, tool creation approaches, evaluation benchmarks and metrics, and empirically analyzes the trade-offs between performance gains and computation costs when using different tooling methods across tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of the clarity, comprehensiveness, and insightfulness of our work!\n\n- **R1: Is survey suitable for COLM**\nAs stated in the FAQ of the official COLM website (https://colmweb.org/faq.html): *“We don't have a special track for survey papers. If you wish, you can submit a survey paper and we will route it to the appropriate area chairs and reviewers and based on their judgment, you will get a decision.”*\nFrom this, we infer that  COLM accepts survey papers and will find the most helpful ACs and reviewers for them. Therefore, we believe our survey paper is appropriate for COLM.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Well-Written and Easy to Understand: The paper is well-organized, clearly written, and easy to follow, even for readers new to the topic of language model tooling. The authors provide clear definitions, illustrative examples, and a logical flow that aids comprehension.\n\n- Comprehensive Coverage: The survey comprehensively covers the landscape of language model tooling, including tools for knowledge access, computation, real-world interaction, and multi-modal processing. It examines basic paradigms, advanced methods, benchmarks, and evaluation practices.\n\n- Insightful Analysis and Future Directions: Through empirical analysis, the paper identifies efficient tooling approaches and tasks that benefit most from tools. It also critically examines missing evaluation aspects like tool reliability and safety, highlighting important future research directions.", "reasons_to_reject": "My only concern is that, as a survey paper, this work does not present any novel technical contributions or algorithms. Therefore, I am not sure whether it is suitable to publish this paper in COLM?", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a survey of tool usage in LLMs, which includes a unifying definition of tools and a categorization of tooling scenarios and methods while considering their computational cost and performance.

Tool usage in LLMs is a timely and growing research area. Surveys that aggregate many existing works from a high-level perspective could be useful in making this research area easily accessible to a wide audience and newcomers. 

It seems that researchers already working on tool usage in LLMs may not benefit substantially from this work. While it highlights general challenges and future directions for this research area, these could be made more comprehensive and actionable by considering additional works not covered in the current version of the paper, as well as a more in-depth comparison between existing methods.

[At least one review was discounted during the decision process due to quality]
Decision: Accept