CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets
Authors: Lifan Yuan
Yangyi Chen
Xingyao Wang
Yi Fung
Hao Peng
Heng Ji
Venue: ICLR 2024 poster (2024)
Keywords: Tool Creation
Tool Retrieval
LLM
Multimodality
Reasoning
Primary area: general machine learning (i.e., none of the above)
TL;DR: Automatically create a scalable and task-specific tool base using LLMs, and retrieve tools to tackle downstream samples at inference time.

Abstract: Large language models (LLMs) are often augmented with tools to solve complex tasks. By generating code snippets and executing them through task-specific Application Programming Interfaces (APIs), they can offload certain functions to dedicated external modules, such as image encoding and performing calculations. However, most existing approaches to augment LLMs with tools are constrained
by general-purpose APIs and lack the flexibility for tailoring them to specific tasks. In this work, we present CRAFT, a general tool creation and retrieval framework for LLMs. It creates toolsets specifically curated for the tasks and equips LLMs with a component that retrieves tools from these sets to enhance their capability to solve complex tasks. For each task, we collect specific code solutions by prompting
GPT-4 to solve the training examples. Following a validation step ensuring the correctness, these solutions are abstracted into code snippets to enhance reusability, and deduplicated for higher quality. At inference time, the language model retrieves snippets from the toolsets and then executes them or generates the output conditioning on the retrieved snippets. Our method is designed to be flexible and
offers a plug-and-play approach to adapt off-the-shelf LLMs to unseen domains and modalities, without any finetuning. Experiments on vision-language, tabular processing, and mathematical reasoning tasks show that our approach achieves substantial improvements compared to strong baselines. In addition, our in-depth analysis reveals that: (1) consistent performance improvement can be achieved by
scaling up the number of tools and the capability of the backbone models; (2) each component of our approach contributes to the performance gains; (3) the created tools are well-structured and reliable with low complexity  and atomicity.

Published as a conference paper at ICLR 2024
CRAFT: CUSTOMIZING LLMs BY CREATING AND RETRIEVING FROM SPECIALIZED TOOLSETS
Lifan Yuan, Yangyi X Wang, Yi R. Fung, Hao Peng, Heng Ji
University of Illinois Urbana-Chane Chama-aaa
[lievanyuan gema @maai 1
com
(yangyic3, xingyao6, yi gifu222, haopeng, hengja .ddu
ABSTRACT
Large language models (LLMs) are often augmented with tools to solve complex tasks. By generating code snippets and executing them through task-specific Ap- plication Programming Interfaces (APIs), they can offload certain functions to dedicated external modules, such as image encoding and performing calculations. However, most existing approaches to augment LLMs with tools are constrained by general-purpose APIS and lack the flexibility for tailoring them specific tasks, In this work, we present CRAFT, general tool creation and retrieval framework for LLMs. It creates toolsets specifically curated for the tasks and equips LLMs with a component that retrieves tools from these sets to enhance their capability to solve complex tasks. For each task, we collect specific code solutions by prompting GPT-4 to solve the training examples. Following a validation step ensuring the correctness, these solutions are abstracted into code snippets to enhance reusabil- ity, and deduplicated for higher quality. At inference time, the language model retrieves snippets from the toolsets and then executes them or generates the output conditioning on the retrieved snippets. Our method is designed to be flexible and offers a plug-and-play approach to adapt off-the-shelf LLMs to unseen domains and modalities, without any finetuning. Experiments on vision-language, tabular processing, and mathematical reasoning tasks show that our approach achieves substantial improvements compared strong baselines. In addition, our in-depth analysis reveals that: (1) consistent performance improvement can be achieved by scaling up the number of tools and the capability of the backbone models; (2) each component of our approach contributes to the performance gains; (3) the created tools are well-structured and reliable with low complexity and atomicity.
INTRODUCTION
Large language models (LLMs) have emerged transformative tools in AI, exhibiting capabilities in complex problem-solving, including reasoning, planning, and producing creative outputs (Brown et al., 2020; Touvron et al., 2023b;a; Yuan et al., 2023). Recent evidence has shown that LLMs can dynamically interact with the environment through external tools, which grants them access to information beyond their pretrained parameters (Qin al., 2023a; Mialon et al., 2023; Schick et al., 2023). For example, these models can generate code snippets and call APIs provided by visual tools like image encoding models, solve problems that involve images or videos (Wu al., 2023; Shen et al., 2023; Yang et al., 2024).
Success has been achieved by integrating LLMs with large-scale, general-purpose tool collec- tions (Qin et al., 2023b; Tang et al., 2023; Surís et al., 2023; Gao et al., 2023a; Chen et al., 2022a; Gao et al., 2023b; Patil et al., 2023). However, adapting LLMs to many domains and evolving applications involves working with more specialized APIs tailored to address specific challenges. which are often inadequately represented in general-pur purposs toolsets. In response, this work proposes to integrate LLMs with highly customizable toolsets that are curated for specific problems of interest.
"Equal contribution. The first author conducts this research during an internship at UTUC. 'The code is available https: http//gittttt wtt.tttub.cmm//m Saannyunn//e -yuun/CRAT1... CRAFT.

---

Published as a conference paper at ICLR 2024
Question: in what room of a house would you most likely find these types of chairs?
CRAFT (Ours)
query, heiplul convsponding decst can [...]
Buttow
Created Toolset
cer towrthyy acetce around, pndecttiud
Pe
Retrieved Tools
Socetring
L det ccantity abgecta, 10, locatfont....) ouf sesermirg, M2jaet Insutiontt..))
Eeteretive everatt atncoprere of e. gleee leeattes,
ViperPPT 2023)
LLM: [Direct problem- solviing]
LLM: [Use retrieved toal from created tool base for problem--olvinng
chart ppetchee trape_aatch. Finett chatn!?
faagn, partes Imsrrateenceaap
def execute, (leggn) >> stri
trate_prtch Iesge"atch typec idrorsfy. www.yyyyo.y......... (trar patch, "chadr") possiile_ "ltying race", "dtniin rese!, 'kitchen', 'bodroon'', "offficc'
suery 'Wha type of chate 14, thix??
chetr Eype shelt sesch, simple, (curr() "chatee
Automasically casermine the
(cratr_Sypa in
carrect lncation glven lat:
return 'xt is difficolt to detereing varlatle "cheir typu"
neturn chatr_type
("eliningg'
Exect match betweon
rooe: ceternfiee object. Locatfon:fnoge.. psttc, chstr, Sypes, possiste, return room Output: "dl ming race*
more floible
foult-to
Name: Acto's per move
The movie critic likd count the number actors each movie he saw. What is the smallest number actors?
Question:
CRAFT (Ours)
acters
LLM: solve the the heipful hoote ght be lepa ... and the consesonnding decsiii ing can be [...]
Created Toolset
Bssed Ds
quastion
calas
calca ata Trans, rattollec
apt
Frenerce
Retrieved Tools
Seckering
caf
coant, Teen, be lae, tiree s
find Lowest nusber, fn, sttstten Lood
Taies Frepuency caste and
coder ioem with (reasein
CREATOR (Clian 61 M., 2023)
LLM: [Direct problem-solvingg
LLM: (Use retrieved toal from created lool base problem-solvingg
find, emall leet www.teeaaaaaaaoo pttttrrrr.
cestlest of actorstd"))
return rind I twestt twwet.ontoe..tee www.ue.ttnn.no..oo....... www.:/nnnttteteceeee hnu.stee_eesfcrr "Steel", * "Latt")
rar (s
dil ([1))
Crrecily reuse correct Soars avard unacpected errors.
actoes apperd
of a pd. Du ad..Dtafrowe(("Sttr
D2y 8, 4, 5, 6, 71
prr__antltest,nnoo suserr
return wincecoora)
batail (fcttrracr)))
sequence
"Laat's : L2, 3, To
(2, 3, 51, 16, 5, 2r To, 91331
[.
6, 71
Cotput: Voluebrrort winth eng anply Secuenec
find_awaltest_nneeee (ctoors
Figure 1: Previous approaches directly solve the given problem by generating code solutions, which may contain errors. CRAFT first creates a toolset that contains diverse, reusable, and correct tools that are executable code snippets. During inference, CRAFT employs a multi-view matching approach, incorporating information about the target problem, API names, and docstrings, to identify and utilize relevant tools, enhancing its problem- capabilities.
Our approach, dubbed CRAFT, constructs toolset customized for a given task (see Figure 1). In contrast to previous approaches that only incorporate one single type of tool (Cai et al., 2023) or create unverified and non-reusable tools (Qian et al., 2023), our toolset contains diverse, reusable, and correct APIs that can tackle various problems. This is achieved through an automated process, by instructing LLMs to generate specific code solutions to solve training problems of the task related ones. The specific solutions are then abstracted into code snippets, which can later be instantiated to solve similar problems. Dedicated validation and deduplication steps ensure the correctness of the tools and reduce redundancy, thereby enhancing the quality of the toolset.
At inference time, precisely identifying and retrieving relevant tools for the given problems is challenging, especially given the constructed large toolset. Existing solutions typically rely on pre-selected tools (Parisi al., 2022), heuristic-based tool selection strategy (Shen et al., 2023), and simple similarity measure (Qin et al., 2023b), which may be unsuitable or insufficient to pinpoint the related tools from large toolset given the problems. CRAFT implements a retrieval component that takes into account the target problem, the names of the tools (a.k.a, APIs), and their docstrings through a multi-view matching function. The retrieved snippets are then added to the prompt of LLMs so that the retrieved tools can be invoked in the generated code solutions.
The empirical effectiveness of CRAFT is validated through experiments on visual question answering. tabular processing, and mathematical reasoning tasks. Compared to strong baselines, CRAFT achieves an average of 43.16% relative improvement in Fl score compared to the best baselines in vision-language tasks, where the LLMs are required to interact with various visual tools to encode the images. Through our carefully designed analysis, we find (I) the performance continually increases as the number of tools and the capability of the backbone models increase; (2) Each component
2

---

sentence representation learning method based on Published as a conference paper at ICLR 2024
Sampled (Problem, Answer) Pairs
(1) Tool Generation
(2) Tool Abstraction
Question Whtch
sommen halding the
L LLM: Spocific Solution
Abstract Tool
Answe
Question: Wiiil site
Specifie Solution
LIME Abstract Tool
lediyy bear"
Specific Solition
-
Ansaes Question: what lncaved newt to the cog?
1nTH GiMI Etr.
LLM: Abstract
Revarra TureSic decering
art canee sndeert and aggnnets
abur
Diaetty anjest toutee aloued to sprifi aljest
voribble
spnc
Gary (077 otigral pery
40 anteer
(4) Tool Deduplication
(3) Validation
ga 1o
def feed closart abdaetc...l
X
Toolsat For VQA
www. (aggettto
Figure 2: The toolset construction pipeline creates diverse, reusable, and correct tools that are executable code snippets, which can generalize L.MSs to specialized domains and tasks.
design incorporated in CRAFT contributes to the performance gains; (3) the created tools exhibit atomicity and possess low complexity, underscoring their robust structures and reliability.
The contribution of this work two-fold. First, we introduce CRAFT, broadly applicable framework to customize LLMs to various tasks and domains via tool creation and retrieval. Second, we release the created toolsets that include diverse, reusable, and correct tools, which are useful for various downstream tasks. Estimatedly, it costs around 2,500S in total for the toolsets construction.
2
CRAFT
We introduce CRAFT to address the challenges faced by prior research in the following two aspects: The establishment of an extensive toolset of diverse, reusable, and correct tools, in contrast to the reliance on limited examples (Cai et al., 2023; Qian et al., 2023); (2) Tool Retrieval: The effective retrieval of relevant tools from large toolset, tailored to the specific question, thereby de- parting from the conventional approach of simplistic similarity matching (Qin et al., 2023b; Patil et al., 2023). By instantiating the retrieved code and adding it to the prompt, L.LMs can then use the tools by calling the function perform complex operations rather than implement every detail from scratch.
(I) Tool Creation:
2.1 TOOL CREATION
Based on a source dataset, namely general instruction dataset or a training dataset that contains problem-answer pairs, CRAFT constructs the toolset through four steps: Generation, Abstraction, Verification, and Deduplication, which are illustratied in Figure and will be described as follows.
Generation. To create toolset containing diverse tools that be adopted address various problems, we apply an iterative approach to sample problem-answer pairs from the source dataset. At a high level, the generation step involves iteratively sampling problems from the source dataset, generating code solutions, and filtering out incorrect ones. We use Q to denote the set of sampled problems and R, to denote the set of remaining problems after the i-th iteration. Q is initialized with random samples from the entire source dataset and R; is initialized as the rest. At each iteration, we use the highest similarity between each 4. E R, and any O. E Q as the similarity between each gr and set Q. To enhance the diversity of the toolset, Q is updated by adding & problems that are least similar to Q, where & represents the desired number of samples for each iteration. This min-max sampling strategy is: Q 4 Q U argToop, TrgTopK min returns the indices of the top & elements with the smallest values from a set, which is set to 100 in implementation, and sim (-) denotes the cosine similarity of the representation vectors computed by SimCSE, a state-of-the- contrastive learning (Gao et al., 2021).
(max,,eq sim((r. qs) gr E R.). Function
3

---

Published as a conference paper at ICLR 2024
For each problem 9, Q. we instruct GPT-4 (OpenAI, 2023) to generate specific solution in Python that can be executed by an interpreter to get the answer. The prompts are shown in Appendix C. We keep those code solutions that are bug-free and can produce correct outputs, and discard everything else to ensure the correctness of the created tools.
Abstraction. The generated code solutions tailored for the given problems, keeping them from being useful for others. The abstraction step aims to promote the reusability of the toolset, ensuring that each tool can be adopted to tackle broader range of similar problems. This abstraction step is achieved by instructing GPT-4 to replace all specific variable names with general ones (e.g.. desk and wrap textual inputs of internal function calls as arguments of the tool (e.g., date = df [ col umm name is passed in by tool users) within the code piece, substituting them with more generic counterparts to adapt to similar problems (see Figure 2). In addition, we instruct GPT-4 to assign a suitable and general function name and compose corresponding docstring to elucidate the functionality of created tools. The prompt is described in Appendix C.
-+date = df [ [col umn name], where the value of
The validation step ensures the correctness of the created tools. This is achieved by examining whether the abstract tool functions can solve the original problems. Specifically, we offer GPT-4 access to the abstract tool function, with the expectation that it will address the original problems by supplying appropriate arguments to the tool function. The tools that fail to derive the correct answers given the original problems are discarded.
Validation.
Deduplication. To reduce the redundancy in the toolset and improve its diversity, perform deduplication step to streamline the toolset and mitigate potential confusion stemming from redundant tools (e.g., same function names). We organize created tools into groups based on function names and the corresponding number of input arguments. Each group contains tools that have the same function names and the number of input arguments. For groups that contain more than one tool, we prompt GPT-4 to decide on the most comprehensive tool with extensive applications within the groups, using the prompt shown in Appendix C.
2.2 TOOL RETRIEVAL
Retrieving relevant tools from the large constructed toolset is challenging. For better retrieval outcomes, we prompt the LLM to "describe what it needs". During inference, the evaluated LLM is asked to generate the function names J, and the doestrings d, based on the target problem q-- Then CRAFT adopts a similarity measuring strategy that takes into account three key aspects of the created tool tç: (1) The original problem used for creating the tool q: (2) The tool's function name fú (3) The docstring of the function dz For each tool f,, this results in a tuple (q:, fú, d,). We conduct multi-view matching, searching tools via qt ft, and de respectively in the toolset T. Specifically, we have:
Te. arg TopK (sim(g;, 4:) f; E T)
(1)
where arg TopK is a function that returns the indices of the top l elements with the maximum values from a set, sim (.) measures the similarity between two sentences using SimCSE embeddings, and T2 is a list of & tools retrieved by matching problems. We then perform the similar retrieval by matching function names and docstring, obtaining Tú, and Ta, respectively. Next, the three lists of tools are aggregated and ranked by their frequency of occurrences. We then retrieve the three most frequent tools by majority vote. Finally, we filter out those that occur only once, if any. In extreme cases, is also possible that all tools appear only once, i.c. the retrieved tool set is empty, then LLMs would directly perform code generation to solve question without invoking task-specific tools.
After retrieval, the code snippets of tools are added the prompt of LLMs for code generation to solve given question. LLMs can invoke the tools (a.k.a, APIs) embedded in the code. Subsequently, the retrieved tool functions and L.LM-ggeerteed code solutions are instantiated into executable code, and then they are executed to obtain the final predictions.
Summary and Discussion. CRAFT creates a specialized toolset offline, and retrieves useful tools from the toolset in inference time. In toolset creation, we apply iterative problem-sampling strategy based on similarity for diversity, followed by generating code solutions using GPT-4. To ensure the reusability of the created tools, abstract the specific solutions into high-level tools that can tackle various kinds of problems by instructing GPT-4. To ensure the tools' correctness, we evaluate the tools on the original problems and discard those outputting incorrect answers. Finally, we deduplicate
4

---

Published as a conference paper at ICLR 2024
the tools to reduce redundancy, and finally obtain a toolset. In inference, we apply multi-view matching algorithm regarding the target problem, function name, and docstring between those in the toolset to retrieve related tools.
We highlight several advantages of CRAFT. At high level, by leveraging the tool creation paradigm, can effectively utilize the domain-specific data customize the LLMs without extensive fine- tuning, rendering CRAFT' a training-free and plug-and-play approach. Due to CRAFT's flexibility in accommodating various domains and tasks, is broadly applicable across spectrum of problem categories. In the concrete implementation, each tool is instantiated as an executable code snippet and is targeted at small atomic problems, such as identifying the color of an object. This ensures the explainability of the created tools. We can easily incorporate human efforts to examine the problem- atic tools and fix the errors. In addition, this allows for the decomposition of complex problems into multiple manageable steps, facilitating the compositionality of these created tools during inference.
EXPERIMENT
3.1 EXPERIMENTAL SETTING
Evaluation Tasks, Datasets, and Metrics. To demonstrate the versatility of CRAFT, we select three distinct tasks for evaluation, spanning visual question answering (VQA), tabular processing, and mathematical reasoning:
VQA: The goal is to answer questions based on the information available in an associated image. We use three complex visual reasoning datasets, including GQA (Hudson Manning, 2019), OK-VQA (Marino et al., 2019), and A-OKVQA (Schwenk et al., 2022). The GQA problems are more complex and require compositional reasoning to answer, while OK- VQA and A-OKVQA mainly use external real-world knowledge of objects actions in the image. For evaluation, formalize the VQA task as an open-ended generation problem and use the soft accuracy (SAce) metric (Antol et al., 20.5). In addition, observe that L.M-genrraed functions often produce descriptive responses instead of concise phrases, which hurts the exact match between predictions and ground-truth answers. This potentially cause underestimation of the performance, also use the Fl score for evaluation, which is frequently employed in extractive question-answering tasks (Rajpurkar al., 2016).
Tabular Processing: It evaluates an LLM's ability to process structured data in tables. We use TabMWP (Lu et al., 2023), a dataset with each sample containing one table and one corresponding problem in natural language. To handle the task, LLMs should understand the natural language descriptions of the problems, extract relevant information from the accompanying tables, and finally perform calculations based on the extracted information. We use the accuracy based on the exact match to measure model performance.
Mathematical Reasoning: LLMs are expected to solve mathematical problems written in natural language, leveraging both their understanding of textual inputs and complex reasoning capabilities. We use the algebra subset of MATH (Hendrycks et al., 2021), containing 881 challenging competition-leww budget constraint but believe CRAFT'S is equally applicable other math problems. The models' performance is evaluated using accuracy.
algebra problems. Evaluating CRAFT on all subsets goes beyond
Baselines.
We compare CRAFT with baseline methods of four categories:
This line of methods solves downstream problems solely based on the intrinsic reasoning ability of LLMs without access to any external tool. We use the chain- of-thought prompting (CoT) (Wei et al., 2022), which prompts LLMs to generate the rationales before answers without using tools. However, it does not apply to the VQA task since LLMs cannot process visual information without external visual tools.
Basic Reasoning without Tools:
Tool Learning: We compare with approaches that directly leverage existing tools to assist the problem-solving process. In this case, LLMs only learn to use the human hrovieedovided tools without creating and retrieving tools. We compare to two approaches: (1) Vanilla stands for utilizing the most basic tools, such as Python Interpreter for all three tasks, and extra vision models to solve VQA problems. Specifically, the vanilla tool-using method for VQA is ViperGPT (Sur'i is al., 2023), and that for the other two tasks is Program-of- ofoo-ghhtttt reasoning (Chen al.,
5

---

Published as a conference paper at ICLR 2024
Table 1: Distinctions between baseline methods and CRAFT in enhancing LLMSs with created tools.
Tool-Creation Method
Dataset for Create Tools
Reuse Tools?
Tool Base Size
Retrieval-enhaneed?
CREATOR
Test Set
No
No
LAIM
Train Set
CRAFT
Instruction Dataset Train Set
Yes
100; Theoretically Unlimited
Yes
Table 2: The experimental results of CRAFT and four categories of baselines on three tasks. SAcc
denotes soft accuracy, which is widely used for VQA. Fl supplemented to tackle the issue of underestimated performance caused by the descriptive responses of LLMs. Acc denotes the accuracy.
GQA
OK-VQA
A-OKVQA
TabMWP
MATH
GPT-3.5-Turie
Method
SAcc
SAce
SAce
Acc
Acc
Basic Reasoning
CoT
75.2
50.9
Vanilla
35.0
36.9
15.4
24.7
15.6
23.0
80.6
58.2
Tool Learning
External
34.2
37.8
16.8
25.3
14.5
22.9
83.1
41.1
LATM
29.4
30.3
7.8
11.8
6.5
11.4
9.3
30.3
Different Tools
CREATOR
34,3
38.4
16.7
27.3
17.3
25.8
81.0
65.0
SimCSE
36.4
38.8
18.4
28.9
16.8
24.3
83.8
36.7
Alternative Retrieval
BM25
37,9
39.0
13.4
24.3
17.8
26.1
89.2
35.9
This Work
CRAFT
45.4
48.8
33.4
43.0
30.8
40.6
88.4
68.1
2022b). (2) External library: Therefore, also explore the possibility of exploiting external tool functions the Python libraries to enhance the vanilla methods. For VQA, we use Numpy Image (Van der Walt et al., 2014), and Mahotas (Coelho, 2012). For the remaining two tasks, we substitute Scikit Image and Mahotas with Pandas (McKinney al., 201 and SymPy (Meurer al., 2017).
(Harris et al., 2020), SciPy (Virtanen et al., 2020), Scikit-Image
Different LLM-Created Tools: We compare with previous tool creation approaches, including LATM (Cai et al., 2023) and CREATOR (Qian et al., 2023). Specifically, LATM samples 3 exam- ples from the training set and applies GPT-4 to create a tool for the task, which is further verified by 3 samples from the validation set. The created tool is then applied to all test cases. CREATOR creates one specific tool for each test case the inference time. For fair comparisons, we remove the format checking and rectifying process used in the original work and only measure the one-pass accuracy. The distinctions between these two methods and CRAFT are shown in Table 3.1.
Alternative Retrieval Methods: We compare with previous tool retrieval approaches, which focus on the similarity measure between the problem and the API names. We include two prevalent mea- sures, namely SimCSE and BM25 similarity, following Qin et al. (2023b) and Patil et al. (2023) re- spectively. The baseline retrieval methods are also based on our created toolset for fair comparison.
In this work, we implement CRAFT and baselines based on the GPT-3.5-Tures (ChatGPT) backbone because: (1) It is more cost-effective compared to alternatives like GPT-4, with affordable cost and strong performance; (2) The Turbo-0613 version is specially optimized for the tool-lcarning purpose. Conversely, alternative backbone models (c.g.. CodeLlama (Rozière et al., 2023)) demonstrate near-random performance in our setting, which can be attributed to their suboptimal tool-using capabilities. The concrete implementation details are described in Appendix B.
3.2 EXPERIMENTAL RESULTS
We present the results in Table Particularly, we find that directly leveraging tools from external Python libraries fails to improve the performance, and in certain cases, may have detrimental impact (e.g., in mathematical reasoning). This suggests that the relevance of tools affects the performance of augmented LLMs, motivating to construct high-quality tool base that customizes LLMs to each task. We observe that LATM struggles with all datasets and brings negative effects; CREATOR yields notable enhancement in mathematical reasoning task performance, while its impact other datasets appears marginal. This result suggests the necessity of sufficient and diverse tools to tackle problems of various categories in downstream datasets. For tool retrieval baselines, the performances vary across datasets. But general, LLMs do not get substantial enhancement except on TabMWP, posing the need for better retrieval algorithms.

---

Published as a conference paper at ICLR 2024
Table 3: Results of further analysis, encompassing ablation study abstraction and retrieval components, as well as the comparison between ViperGPT and CRAFT with different backbones.
GQA
OK-VQA
A-OKVQA
GPT-3.5-Turbo
SAce
F1
SAce
SAcc
F1
ViperGPT
35.0
36.9
15.4
24.7
15.6
23.0
CRAFT
45.4
48.8
33.4
43.0
30.8
40.6
w/o Abstraction
37.1
39.7
31.0
41.4
28.0
39.3
w/o Problem
42.4
45.8
32.7
42.3
29.8
38.7
w/o Name
36.4
38.3
26.8
35.7
21.7
30.6
w/o Doestring
37.3
39.1
29.8
38.8
25.0
34.0
GQA
OK-VQA
A-OKVQA
GPT-4
SAce
F1
SAce
SAce
F1
ViperGPT
51.4
53.7
36.7
47.2
32.8
42.4
CRAFT
55.6
58.8
39.0
49.1
35.3
44.8
Overall, CRAFT demonstrates superior performance on all datasets, especially on the challenging VQA tasks. Significantly, CRAFT demonstrates notable enhancement over the vanilla baseline, namely ViperGPT, with absolute SAce improvements 10.4, 18.0, and 15.2 observed on the GQA, OK-VQA, and A-OKVQA datasets, respectively. In addition, based the same created toolset, the retrieval approach incorporated in CRAFT demonstrates overall better performance compared alternative ones, which exhibit certain level of performance variance. One exception is the comparison with BM25 on TabMWP. This discrepancy can be attributed to the presence relatively straightforward patterns within this dataset, which do not sufficiently showcase the advantages of our approach in tool retrieval.
4 FURTHER ANALYSIS.
In this section, we conduct an in-depth analysis for CRAFT on VQA datasets. This task is particularly pertinent for assessing the impact of external tool augmentation, given that LLMs lack the capability to directly process images. Thus, it serves key testbed for measuring the influence of external tools.
4.1 DOES ABSTRACTION FACILITATE TOOL USE?
Setup. Abstraction is crucial step in constructing the toolset, converting solutions for specific problems into general-pon purppos tools that applicable diverse problems with common pattern. In this section, we explore its efficacy with an ablation study. To scrutinize this, we establish a control group, where the toolset is created ablating the abstraction step. To ensure compatibility, prompt GPT-4 to assign a distinetive function name and docstring for each solution to facilitate the multi-view retrieval approach for fair comparison.
Results. Table 3 shows clear performance drop when the abstraction step is ablated, confirming its importance. Moreover, comparing abstraction-ablate CRAFT with ViperGPT, improvements are achieved across all three datasets, especially on OK- DK-VQA and A-OKVQA. We identify two potential reasons that can elucidate the improvement. First, the created toolset is large and diverse enough, facilitating the adoption of specific tools without abstraction for addressing new problems. Second, as retrieved tools offer a correct approach to problem-solving, L.MMs can efficiently adapt these strategies address problems.
4.2 EVERY MATCHING IN THE RETRIEVAL TRIPLET EQUALLY IMPORTANT?
Setup. CRAFT retrieves tools based on multi-view matching. We demonstrate its effectiveness in Section 3.2. Next, we respectively ablate problems, function names, and docstring from the matching process to investigate their influence performance.
Results. As demonstrated in Table 3, clear that the removal of any of the three similarity measures from milti-view matching function adversely impacts performance, thereby validating
7

---

Published as a conference paper at ICLR 2024
the rationale behind design strategy. Among them, the function names appear the most important one, resulting in more than 6.6 absolute SAce drop when ablated.
4.3 DOES CRAFT STILL WORK FOR MORE POWERFUL BACKBONE MODELS?
Setup. In previous experiments, CRAFT is implemented using GPT-3.5-Turbo as the backbone. In this analysis, we evaluate CRAFT when using the more powerful GPT-4 as the backbone. Due to the budget limits, only compare CRAFT with the vanilla baseline ViperGPT without tool creation.
Results. The results in Table demonstrate that CRAFT achieves consistently better performance with GPT-4, confirming that CRAFT helpful even with more capable backbone models. However, it's noteworthy that while the improvement of CRAFT on GPT-4 is pronounced, it is less obvious compared to the impact on GPT-3.5-Turbo. We hypothesize that this result is in line with the conclusions of recent work, which finds that LLMs can benefit from the guidance of more capable models while gaining no improvement from the guidance of itself (Fu al., 2023; Wang et al., 2023). The tools, created by GPT-4, may provide comparatively fewer insights for itself, thereby limiting the potential benefits of external tool augmentation.
4.4 CAN CRAFT IMPROVE PERFORMANCE AS THE TOOLSET GETS LARGER?
Setup. A feature of CRAFT distinctive from prior approaches is the extensibility of the toolsets. We examine the utility of extension by manipulating the toolset's size and tracking per- formance trends. To elaborate, the iterative prob- lem sampling strategy detailed in Section 2.1 is
SAcc with Increasing Toolset Size
45
Soft Accuracy
40
35
initialized with total of 11 epochs. In this anal-
30
ysis, the sizes of the toolset are modified through
25
the exclusive inclusion of tools created dis- tinct epochs. We choose tools from the initial epoch, the final epoch, and the epoch in between, resulting in toolset sizes of 0 (no created tool for comparison), 261, 337, and 525, respectively.
GQA
20
OK-VAA
A-OKVQA
15
0
261
337
525
# Tools
Figure 3: The performance of CRAFT improves as the toolset scales up.
Results. The results in Figure 3 show
sistent increase in soft accuracy as the toolset expands across 3 datasets, demonstrating the scalability and potential of CRAFT. The upward trend of soft accuracy continues, suggesting the potential for further improvement of CRAFT as the toolset keeps expanding. Significantly, the most substantial improvement is observed when transitioning from the absence of any created tools to the utilization of 261 tools. This validates the effectiveness of creating the specialized toolset customize LLMs various tasks and domains.
4.5 WHAT IS INSIDE THE TOOLSET?
We analyze the complexity and diversity of the code in toolsets. For complexity, we use the widely adopted cyclomatic complexity (Mc- Cabe, 1994) measure the number of linearly independent paths, with the higher value indicat- ing the code more complicated and requires refactoring to make it more reliable. Good soft- ware should have a complexity of no more than 10. and less complex toolset is desirable since it is less prone to trigger bugs. For diversity, classify each tool into different groups. We use the number of distinct groups as the metric, with a larger number of tool groups indicating wider range of problems that toolset can address.
Table 4: Analysis of cyclomatic complexity and
diversity of the toolsets.
Tabular Process
Mathematics Reasoning
Task
VQA
Avg. Cyclomatic Complexity
2.64
2.07
1.34
525
Classes of Tools
195
23
234
8

---

Published as a conference paper at ICLR 2024
We calculate the complexity using Lizard Python library", and present the average complexity of tools for each task in Table 4. We observe that the created toolsets for 3 tasks exhibit relatively low complexity. indicating that the tools are well-structured and reliable. We then adopt the Louvain community detection method (Blondel et al., 2008), a graph-based community dividing algorithm, to group different tools. As shown in Table 4, for VQA, tabular process, and mathematics reasoning. there are 195, 23, and 234, distinct classes out of 525, 181, and 282 tools respectively. This suggests that the MATH dataset has the most diverse patterns, followed by VQA, while problems in the TabMWP dataset are more homogeneous and can be well-solved using fewer created tools.
5 RELATED WORK
5.1 TOOL LEARNING WITH LLMS
LLMs, when integrated with real-world Application Programming Interfaces (APIs), gain the capability to actively interact with a range of external systems (a. tools) (Parisi et al., 2022; Schick et al., 2023; Tang et al., 2023; Patil et al., 2023; Song et al., 2023; Hao et al., 2023; Wang et al., 2024). The pioncering work connects GPT-3 (Brown et al., 2020) with the web browser to access latest information, and hires human annotators to provide demonstrations of web searching (Nakano et al., 2021). Further research expands upon this concept by encompassing broader range of tools, such as calculators, calendars, interpreter, physical simulator, and maps (Shuster et al., 2022; Paranjape et al., 2023; Liu al., 2023c; Chen al., 2022a; Gao et al.. 2023a; Drori et al., 2022; Pan et al., 2023; Liu et al., 2023b), and explores the application of weakly-supervised methods, such as bootstrap- ping (Parisi et al., 2022; Schick et al., 2023). More recently, progress has been achieved through the process of distilling the tool using the ability of closed-source LLMs (ChatGPT (ChatGPT Plugins)) to the open-source LLMs. The key idea revolves around allowing ChatGPT to produce synthetic data exemplifying the usage of specified APIs. Subsequently, this synthetic data is leveraged for the refinement of open-sourced LLMs (Qin et al., 2023b; Tang al., 2023). In this work, we extend our approach beyond mere dependence existing tools. We adapt LLMs to diverse downstream tasks through the creation of customized tools and the retrieval of relevant tools during inference.
5.2 TOOL CREATION & RETRIEVAL
While the exploration on tool creation and retrieval is relatively limited compared to tool learning with LLMs, identify some preliminary efforts in this domain. For tool creation, Cai et al. (2023) proposes an approach wherein tools are created through the utilization of three training samples, and their efficacy subsequently assessed using three validation samples. Consequently, the resulting toolbase is constrained in quantity. This approach hinges on the assumption that there exists a notable similarity between the distributions of the training and testing data. Consequently, the tools produced can be readily incorporated. Similarly, Qian et al. (2023) adopts a strategy that involves generating tools exclusively based on the provided query. As a result, the created tools lack reusability, thereby undermining the fundamental purpose of tool creation. For tool retrieval, existing research primarily includes pre-selection of human-curated tools tailored to specific problems (Parisi et al., 2022; Tang et al., 2023; Schick et al., 2023; Zhuang et al., 2023), employing heuristic-based methods for tool selection (Shen et al., 2023; Liang et al., 2023), and adopting a straightforward similarity metric between user queries and API names (Qin et al., 2023b; Patil et al., 2023; Xu et al., 2023). In this work, we motivate to create a large tool base that can be effectively utlized on related downstream tasks and address the challenge of retrieving the relevant tools from the large tool base.
6 CONCLUSION
In conclusion, this paper presents CRAFT, a general framework for tool creation and retrieval to generalize L.LMs for diverse domains and tasks. The framework's 's effectiveness is demonstrated through improved performance in challenging tasks, alongside insights into component contributions, constructed toolsets, and scalability.
hhttps://gi httppp:///gittobb.mm/eerrrrr http://ggghhbb..oommer////t/////
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We thank the anonymous reviewers for their suggestions and comments. This research is based upon work supported by U.S. DARPA ECOLE Program No. HR00112390066 and U.S. DARPA ITM Program No. FA8650-23-C-7316. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: visual question answering. In 2015 IEEE International Conference Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 2425-2433 IEEE Computer Society, 2015. doi: 10.1109/CCCV22115.279 URL https: /ooi. wrg./or.oo.....o ICCV. 2015.279.
Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment, 2008 (10):P10008, 2008.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ily Sutskever, and Dario Amodei. Language models few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https: pettps:/pproceedinngg neurips cc/pc//paaa 200per//220/00 Bbfbaac1 Abstaabbbtraaaaaaa html.
6b w00pffcc4946444 11114466771898811111
Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou, Large language models as tool makers. ArXiv, 2023.
ChatGPT, URL https:tttthhhttt openaa comm.
ChatGPT Plugins. URL openai.e.........o..............
Wenhu Chen, Xueguang Ma. Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. CoRR, abs/2211.11588, 2022a. doi: 10.488550arxiv.22111 125888. URL https::/dolt 2211.12583
org
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. CoRR, abs/2211 11s/22 2022b.
Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. Measuring and im- proving chain-of-thought reasoning in vision-language models. arXiv preprint arXiv:2309.04461. 2023.
Luis Pedro Coelho. Mahotas: Open source software for scriptable computer vision. arXiv preprint arXiv: 2012.
Iddo Drori, Sarah Zhang, Reece Shuttleworth, Leonard Tang, Albert Lu, Elizabeth Ke, Kevin Liu, Linda Chen, Sunny Tran, Newman Cheng, et al. A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level. Proceedings of the National Academy of Sciences, 119(32))e123433119, 2022.
10

---

Published as a conference paper at ICLR 2024
Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata. Improving language model negotiation with self-play and in-context learning from ai feedback, 2023.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL: program- aided language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Leurning, ICML 2023, 23-29 July 2023, Honoludu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 10764-10799. PMLR, 2023a. URL https: / pttppppicceeig.neeeee mmrr press / /2t// gao2ff. html.
Shen Gao, Zhengliang Shi, Minghang Zhu, Bowen Fang, Xin Xin, Pengjie Ren, Zhumin Chen, and Jun Ma. Confucius: Iterative tool learning from introspection feedback by casy-to-difficult curriculum. CoRR, abs/2308. 14034, 2023b, doi: 10.48550/ar2v.23000. 140344 URL https: oogli.org//0... 4022g//..4555//aa//
2308.14034 14034.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. 2021.
Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. CoRR, abs/2305. 11544, 2023. doi: 10.48550/ arXiv.2305. 1554. URL https: //oi erg/10......ooo 48550/ aarvvv 23.5.
Charles R Harris, K Jarrod Millman, Stéfan Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sehastian Berg, Nathaniel Smith, et al. Array program- ming with numpy. Nature, (557588877722533733222 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang. Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurlPS, 2021.
Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 6700- 6709. Computer Vision Foundation IEEE, 2019. doi: 10. 10.1109/CVPR.2219..0000 twecff.ttheetccccmm com/ http//oonccctccceeeeeeeeeeeeeeee CWFR___C______ CWC__CCPP_______ //////tmmm.. waapiiinggnnn_d_eeeeee wtwpandddoottoiooooomamm
URL
A WwweeeeWaaasseeeee foraaattt__oot_eeeeeeeeeeeee Rea_-ealllllllo CVPR_ 201220119_____ papperrreppp_____ html.
Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, Yun Wang, Linjun Shou, Ming Gong, and Nan Duan. Taskmatrix.at ai: Completing tasks by connecting foundation models with millions of apis. CoRR, abs/2303.16434, 2023. doi: 10.485500/c2xiv2233.. 16434. URL https: //wii//ii./////// org/ 10. 005448550//// attp/ariivv...///vv 2303..333.mm 16434.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In Computer Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V. 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. ArXiv, 2023a.
Ruibo Liu, Jason Wei, Shixiang Shane Gu, Yen Wu, Soroush Vosoughi, Claire Cui, Denny Zhou, and Andrew M. Dai. Mind's s eye: Grounded language model reasoning through simula- tion. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023b. URL https: /ww.revie..eeeeeeeeoeeoeee net/pdf? 1d=4rXMRooJlai.
Xiao Liu, Hanyu Lai, Hao Yu, Yifan Xu, Aohan Zeng, Zhengxiao Du, Peng Zhang, Yuxiao Dong, and Jie Tang. Webglm: Towards an efficient web-enhanced question answering system with human preferences. In Ambuj Singh, Yizhou Sun, Leman Akoglu, Dimitrios Gunopulos, Xifeng Yan, Ravi Kumar, Fatma Ozcan, and Jieping Ye (eds.), Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, KDD 2023, Long Beach, CA, USA, August 6- 10. 2023, pp. 4549 4560. ACM, 2023c. doi: 00.11455/58800533599999 URL https: //doi org 0. 11 1111111/355500 350330580305555 35999311
11

---

Published as a conference paper at ICLR 2024
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song- Chog-Chhn Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In International Conference Learning Representations (ICLR), 2023.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA: A visual question answering benchmark requiring external knowledge. In IEEE Conference on Computer Vision and Pullern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 3195-3204. Computer Vision Foundation / IEEE, 2019. doi: 0..109/CVPR.2209.00555 URL http: http:////ppcccccss.essssssssssss thecff com/ content http//ooontttteettteteeee 2019 h1019/html/////mmm /aattm//Maain.... VX-VOA_L Visual LitaaAAAiillal_uuestiiio ugg.aaasenneetttooo.o.aatnennnne Btw_BBenhhaaa_eeeiriiin Requiring External_
Knowledge__cps_ 201 9 p_paper. html.
Thomas J McCabe. Software complexity, crosstalk, Journal of Defense Software Engineering, 1994.
Wes McKinney et al. pandas: foundational python library for data analysis and statistics. Python for high performance and scientific computing, 14(9):1-9, 2011.
Aaron Meurer, Christopher Smith, Mateusz Paprocki, Ondfej Čertik, Sergey Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K Moore, Sartaj Singh, et al. Sympy: symbolic computing python. PeerJ Computer Science, 3:3 2017.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. Augmented language models: survey. CoRR, abs/2302.07842,, 2023. doi: 10.48550/a2Xiw.3322.7755 48550/arXiv 2302. 07842.
URL
http://doooooggg1//
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assiited question-answeringg with human feedback. CoRR, abss/112..0933... 2021. URL
OpenAI. GPT-4 technical report. CoRR, abw/2333.08774 2023. doi: 10.48550/aXXiv22003.87774 URL
410.0.44555//1 ar 2303. 08774.
Liangming Pan, Xiaobao Wu, Xinyuan Lu, Anh Tuan Luu, William Yang Wang, Min- Ven Kan, and Preslav Nakov. Fact-checking complex claims with program guided reasoning. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 6981 10704 Association for Computational Linguistics, 2023. doi: 10.18653/ 1l/2023.acl-long..oo URL https : http://oo.oo///////////ooo 18653 htt/////222222 acl- 68
Bhargavi Paranjape, Scott M. Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Túlio Ribeiro. ART: automatic multi-step reasoning and tool-use for large language models. CoRR, abs/2303.090 2023, doi: 00-455500axXx..2330000044 URL https: /ooi. wwog/o../... 48550/arXiv 2303 09014.
Aaron Parisi, Yao Zhao, and Noah Fiedel. TALM: tool augmented language models. CoRR,
abs/2205.122555 2022. doi: (0.48550/axXi.2205.22555 URL https://doil org/10.. 48550/arXiv, 2205. 12255.
Shishir G. Patil, Tianjun Zhang. Xin Wang, and Joseph F. Gonzalez. Gorilla: Large language model connected with massive apis. CoRR, abs/2305. 15334, 2023. doi: 10.48500/aXXi..235555553 URL htwpt:ttt///eeec //t::////o..//// org/ /000 118550/arXivv 2305.15334 15334.
Cheng Qian, Chi Han, Yi Ren Fung, Yujia Qin, Zhiyuan Liu, and Heng Ji. Creator: Disentangling abstract and concrete reasonings of large language models through tool creation. ArXiv, 2023.
Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao. Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu. Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang, Jing Yi, Yuzhang Zhu, Zhenning Dai, Lan Yan, Xin Cong, Yaxi Lu, Weilin Zhao, Yuxiang
12

---

Published as a conference paper at ICLR 2024
Huang, Junxi Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang, Cheng Yang, Tongshuang Wu, Heng Ji. Zhiyuan Liu, and Maosong Sun. Tool learning with foundation models. CoRR, abs/2304.08354, 2023a. doi: 14485500arXx..23344083333 2304. 08354..
URL https://doi. www.:/...rrr.... 4555///5//..////
Yujia Qin, Shi Liang, Yining Ye, Kunlun Zhu, Lan Yan, Ya-Ting Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Marc H. Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis. ArXiv, 2023b.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang, Squad: 100, 000+ questions for machine comprehension text. Jian Su, Xavier Carreras, and Kevin Duh (eds.). Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 2383-2392. The Association for Computational Linguistics, 2016. doi: 10. N..18663/11/166 1264. URL https: /wwii/oiio.//////// org/ 0. 8833 /d1 1264.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Seialom, and Gabriel Synnaeve. Code Ilama: Open foundation models for code. CoRR, abs/2308. 110.0088..99900 2950, 2023. doi: 10.448550/rXxvv2300 12950. URL https:////// /oii .rg...ogggoooooo http://..4855/aaaaecc// 2308 12950.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models teach themselves to use tools. CoRR, abs/2302.00471,, 2023. doi: 10.48550/axxx.220004776. URL https:
//doi wwwp:ooorg/..... 48550/ www.i..ww.3.... 2300.0007555
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mot- taghi. A-OKVQA: A benchmark for visual question answering using world knowledge. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hass- ner (eds.), Computer Vision ECCV 2022 17th European Conference, Tel Aviv, Israel, Oc- tober 23-27, 2022, Proceedings, Part VIII, volume 13668 of Lecture Notes in Computer Sci- ence, pp. 146-162. Springer, 2022. doi: 10. 1007/977 L0000/97--30031-200777 3-031-20074-8) //doi.org 0010:/000000..1077979----------1 1ttp:///1100/1/--33300-------000 0000--3000000------0000000--0000
URL https:
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li. Weiming Lu, and Yueting Zhuang, Hugginggpt: Solving AI tasks with chatgpt and its friends in huggingface. CoRR, abs/2303. 2023. doi: 10.485500/rXiv.2333.. 17580. URL https: /oi. org 10. 4000.88550///00 20/550/axiiv 220..
Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kushal Arora, Joshua Lane, Morteza Behrooz, William Ngan, Spencer Poff, Naman Goyal, Arthur Szlam, Y-Lan Boureau, Melanie Kambadur, and Jason Weston. Blenderbot ployed conversational agent that continually learns to responsibly engage. CoRR, abs/2208.033 000 188, 2022. doi: 10.44550/arriw.2208..33 2208 03186.
de-
URL https::/ /ww. org. 10. 48550/445530000
Yifan Song, Weimin Xiong, Dawei Zhu, Cheng Li, Ke Wang, Ye Tian, and Sujian Li. Restgpt: Con- necting large language models with real-world applications via restful apis. CoRR, abs/2306.06624, 2023. doi: 00448550/aXxi..20066622 URL sttps:///oot 1 2306. 06624.
48550/85550/0
Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. CoRR, abs/2303.081228, 2023. doi: (0.4855WArXxx..2333082222 URL https:
//doi. oro.org/////.ooo 110.g/11..0 00044405555ii 2303. 08128..
D'idac Sur'is, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. 2023.
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. Toolalpaca: Gener- alized tool learning for language models with 3000 simulated cases. CoRR, abs/2306.05 053001, 2023. doi: 10.485500/mxxi.220000555 URL https:/ /doi. org/10...... 4855//10.4855000 arxiv. 2306.05301.
13

---

Published as a conference paper at ICLR 2024
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, A3s/2202.13971, 2023a, doi: 20.4 13971. URL https:/ /doi. org/ 00tg/10.8555/// a 2302. 13971.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog. Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnie, Sergey Edunov, and Thomas Scialom. Llama Open foundation and fine-tuned chat models. CoRR, abs/2307.0922222 2023b, doi: 10.485500arX...2307..2222 URL https: http://oooooooo 10. 48550/ arxiv 2307 09228.
Stefan Van der Walt, Johannes I. Schönberger, Juan Nunez-Iglesias, François Boulogne, Joshua D) Warner, Neil Yager, Emmanuelle Gouillart, and Tony Yu. scikit-image: image processing in python. PeerJ, 2xe453, 2014.
Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods, 17(3):261-272, 2020.
Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. MINT: evaluating llms in multi-turn interaction with tools and language feedback. CoRR, abs/2309. 2023. URL https::p///. /tttps///dee //doi org/rrrgg 0 48550/ a rxiv. 2309. 10691.
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better Ilm agents. arXiv preprint arXiv:2402.0103: 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. volume abs/2201.11900 2022.
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. CoRR, abs/2303.04671, 2023. doi: (0448550/axxiv.233..666 23776. URL https://dde //op::://di.../ org/ 10. 2303. 04671.
Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. On the tool manipulation capability of open-source large language models. CoRR, abs/2305.16504 2023. doi: 10.48550karXiv.2305. 16504. URL https: / doi. www.o.oooooooooooooooooooooooo.o / 10. (..550/arxiv.. 2305.16504, 16504.
Ke Yang, Jiateng Liu, John Wu, Chaoqi Yang, Yi R. Fung. Sha Li, Zixuan Huang, Xu Cao, Xingyao Wang, Yiquan Wang. Heng Ji, and Chengxiang Zhai. Ilm the wizard, then code is the wand: survey on how code empowers large language models to serve as intelligent agents, 2024.
Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fangyuan Zou, Xingyi Cheng, Heng Ji, Zhiyuan Liu, and Maosong Sun. Revisiting -utto--distiibution robustness in NLP: benchmark. analysis, and Iims evaluations. CoRR, abs/2306.046) 2023. doi: (0.48550/a2Xiw.220600ook 11.8. URL https: //////do doii. wttpo...4..////a......oo/
40w550//axiii.. 2306. 64618.
Yuchen Zhuang, Yue Yu, Kuan Wang. Haotian Sun, and Chao Zhang. Toolqa: A dataset for LLM question answering with external tools. CoRR, abs/2306.13000 13304, 2023. doi: 10.488550/rXiv.23366. 13304. URL https:/ //t////o...// org/10 100g/1...0 48550/ arxiv 2306. 2306.13304 13304.
14

---

Published as a conference paper at ICLR 2024
APPENDIX
A LIMITATIONS AND FUTURE WORK
We identify two limitations in this work that are worth future exploration. First, although the basic idea in CRAFT is widely applicable in principle. it is currently based code generation for tool creation. This indicates that CRAFT is only suitable for tasks that can be solved via writing code solutions. We plan expand this scope by exploring the of pseudocode to generalize CRAFT more tasks. Second, the effectiveness CRAFT' is greatly affected by the tool-using ability of backbone models. In our pilot exploration, some open-source models achieve near-random performance in the challenging oollmannpulation setting. Future work includes eliciting the tool manipulation ability in open-source models, such as the pilot exploration in Qin et (2023b).
B CRAFT IMPLEMENTATION DETAILS
Tool Creation. For VQA, we sample problems from general instruction datasets, constructing a comprehensive toolset without relying on strong supervised signals. We adopt LLaVA (Liu al., 2023a), a substantial and diverse collection of vision-related queries, to facilitate the creation of wide array of tools. Since LLaVA is designed instruction dataset based on COCO-2017 (Lin et al., 2014) to complement problem diversity. The construction process follows the procedure Chen et al. (2023), instantiating an interactive process to ask GPT-3.5-Turbo to generate a question-answerr answering pair based on caption of an image, followed by a filtering step. We prompt Turbo to make the answer concise instead conversational. However, some questions generated in this manner are image-independent and can be answered by language models alone. Hence, we apply a filtering strategy by sending each question to Turbo and asking if the question can be directly answered without images. We only select those samples that require visual data for answering. We sample 2,000 problems from the above instruction datasets, with 1,000 being from the primary random sampling epoch and another 1,000 from the subsequent 10 epochs, each contributing 100 problems per epoch. We employ ViperGPT (Sur'is is et al., 2023) to generate specific code solutions. For tabular processing and mathematics reasoning, since there are no high-quality instruction datasets specified for these two tasks, we construct toolsets based on the training split of downstream datasets, i.e. TabMWP and MATH (algebra). For these tasks, we sample 500 problems from the above instruction datasets, 200 from the first random sampling epoch, and another 300 from the following epochs, each yielding 100 problems per epoch.
conversational style, we construct another
Tool Retrieval. In this work, we empirically set the number of retrieved tools 10 for qr 5 for fú, and 10 for d, Corresponding internal code implementations and docstrings will be incorporated into the prompt of the backbone model for subsequent utilization.
C PROMPT
C.1 PROMPT TO FILTER IMAGE-DEPENDENT QUESTIONS IN VQA DATASET
You will be given function named "inm_ qurry"" The function Answers text question using GPT-3 for reasening and inference. Since GPT-3 cannot process visual information, the question must be image-independent.
Then, you will be given a query. You need to decide if this 1lm_query function is able to *directyy solve this query. Directly answer yes or no.
Tips: If the query requires visual information of an image to solve, you should answer no. Otherwise, if the query is image-indeperdent inference task that can be sclved by LLM reasoning or search engine, you should answer yes.
Query: Why isn't Song Bee taking customers?
Answer: yes
15

---

Published as a conference paper at ICLR 2024
Query: Why might of the anchors look angry, and the other Look concerned?
Answer: no
Query:
(query)
Answer:
C.2 EVALUATE THE SPECIFIC GENERATION FOR VQA
Given the question for the visual question answering task: (question) Does the fell follewing predi cted answer have the same meaning as the reference answer in the context of the question? Predicted Answer: (prediction)
Reference Answer: (reference)
You should compare the answers based on your understanding of the task, questicn, and answers, rather than relying on some superficial patterns like word overlap.
Directly answer Yes or No.
C.3 TOoL CREATION
C.3.1 VQA
Rewriting Code to Create Generic Tool Function**
Purpose:** Given query and its corresponding code solution, your task is to rewrite and abstract the code to create general tool function that be appl to similar queries. The tool function should
solve a higher-level quest that encompasses the original query and extend the code's functicnality to make it more versatile and widely applicable.
Consicer following principles:
1. Understand the purpose of the query and infer higher-level question that can be addressed by the teel function.
2. The generic tool function shculd solve queries of the same type, based on commen reasoning steps rather than specifis fic object types.
3. When enhancing the tool functicn's versatility according to the higher-level question, avoid assuming new attributes or methods of the 'ImagePatch" classes.
4. Name the function honestly ensure functionality is not overclaimed.
5. Avoid using redundant and unused variables or intermediate steps in the tool function. Ensure that every step in the code has a purpose and directly contributes
the desired outcome.
6. Replace specific strings or variable names with general variables to enhance the tool's applicability to various querics.
7. Provide docstring and an cxample of how to call the tool function to solve specific query.
8. End your answer with the format 'The final generic tcol with docstring
is: ...' and 'The example te call the tool is: ...!.
Example**
Query: there a backpack to the right the man? Specific code solution:
def (reeuute_oommand[inage ->str:
image_patch ImagePatch (image) man patches image patch.
len (man_patcches)
0:
$ If no man is found, query the image directly with simple_quu _uerry instead of returning long string like "There is no man."
16

---

Published as a conference paper at ICLR 2024
return image__atte saatch.simpan wtw.eshhessmeeeeuurreerr query there backpack to the right of the man?")
pan_ = man_patches [0]
backpack_patches = amage_patch. find ("backpack")
if len bbackpack_patccheeee = 0:
"no"
for backpack_pattat in backpack_patchese
if backpack_ man_pat ch www.choooottttaceeeeeeeeee return "yes" return "no"
Abstract tool:
The final generic tool with docstring is: def check sreck_existence around object objeet_ herizontally (image patch: ImagePatch, object_name str, reference efference_object....ee hiwpivv__horizotaalposses
str, str, query; str) -> str:
"CCccc the existence of an object on either the left or right side of another object.
Args:
image_patch (ImagePatch): The image patch to check. object_ name (str): The name of the object to check for existence. seference_abject.....
reference object name (str): The name of the reference object. relative horizontal positien (str): The relative posittinn
+ velative_horizontal_pcco to the reference object. Options: ["left", "right"]. query (str): The original query to answer.
position of the checked object
Returns:
str: "yes" if the object exists, "no" otherwise.
assert relative_horizontal..oo
in ("left", "right"]
reference patcces if len (reeferencc__pttchhee -- 0:
If na reference object is found, query the image directly with simple_query instead returning long string like "There is no [reference_objert
return image reference_patches
query (query)
reference_patee = reference patches [0]
object_patches image_patch
if len (etteccc_patccheee htw.ppaaach.iilooooteottoooooeee return "no"
0:
for object_paton in
if relative relatiive_horizottaa flag abjeet__atcchhherriiaaaaaccc
horiepottooottllossttttoo == "left":
referresss_aacaa.ooobbbott...... == "right":
elif cel.ative__cciionttalooooee centterrreeeeeee_eeee >
flag = wwwpeoo_paccchttteeeeeeet refferance_stchhhherioomaa
if flag:
return "yes"
return "no"
The example call the tool is: lttpaarenooarrun__ooceeeeeee
(image_patch, "backpack", "man", "right", "Is there a backpack to the right of the man?")
**Bginn
Query: (query) Specific code solution:
[solution]
17

---

Published as a conference paper at ICLR 2024
Abstract tool:
C.3.2 TABULAR PROCESSING
*Rewriiing Code to Create a Generic Tool Function**
*Prppsse:** Given query and its corresponding code solution, your task is rewrite and abstract the code create general tool function that can be applied to similar queries. The tool function should solve a higher-level question that encompasses the original query and extend the code's functionality to make it more versatile and widely applicable.
Consider the following principles:
1. The generic tool function should solve queries of the same type, based commen reasoning steps without mentioning specific object entity terms.
2. Name the function and write the docstring concerning both the core reasoning pattern and data erganization format, without referencing
specific objects.
3. Replace specifia strings or variable names with general variables to enhance the tool 's applicabilityy to various queries. A11 columns names used inside the tool shculd be passed in as arguments.
4. Call the tool function to solve the original query. End your answer with the format '# The final generic tool with docstring is: ...! and The example to call the tool .....
Examplc**
Table+
Name:
Orange candies per bag
Unit: bags
Content:
Stem | Leaf
2, 3,
4
0, 6, 7, 9
| 1, 3,
8
1
+Qucry*
candy dispenser put various numbers of orange candies into bags. How many bags had at least 32 orange candies?
Specific code solution+
"python
import pandas pd def count_ with 32 arange_candienn (dE)
***
This function takes in a pandas dataframe of orange candies per bag,
and returns the number of bags that have at least 32 orange candies.
Args:
df (pandas. (pandas. DataFrame) A pandas DataFrame object containing the number of orange candies per bag.
The dataframe should contain "Stem" and "Leaf" colurns.
Returns:
int: The number of bags that have at least 32 orange candies.
-**
prepare 1ist to caleulate candies in each bag candies []
calculate the total number orange candies in each bag
18

---

Published as a conference paper at ICLR 2024
for i in range (len (df)): stem df (ttem"" [i] [i]]
leaf = df [ ('Leaff']
for j in range (len :
candies. (stem 10 leaf(i])
filter the bags where the tctal number of orange candies is greater a than or equal to 32 filtered = Icandy for candy in candies if candy >= 32]
count the number of items num bags (ei(terered)) return num_bags
data -
"Stem": [2, 3, 4, 5, 6, 7, 81,
"Leaf": [[2, 91, (), [), 10, 91, (0), [1, 9], [5]]
df pmm (data) count with htagggg_wttt_____arageeeaaaaaa (df=df)
Abstr tcel function:
We'rr re creating generic toel function from specitic code that counts the number of bags with at least certain threshold of candies based stem-and-leaf plot. The original code combines the stem and leaf values to calculate the total number of candies in each bag, filters the bags with candies greater than or equal to the threshold value, and counts the number of such bags. We generalize the problem to create flexible function for any stem-and-leaf plct of items and various threshoid values. We replace specific columns, item names, and threshold values with generic variables like stem_col, leaf_col, item_thresheld, and frame "python
The final generic tool with docstring (data_ frame, stem_col, def
leaf_col, iten_threshold))
This funct on takes in pandas DataFrame representing stem-and-leaf plot of groups and threshold value, and returns the number of groups that have values greater than or equal to the threshold.
Args:
fram_ DataFrae))) A pandas DataFrame containing the stem-and-leaf plot of items with columns specified by stem_coi and leaf_col.
stem_col (str): The column
for the values.
leaf_col (str): The column name for the leaf values. item (int): The threshold value for filtering items.
Returns:
intt The number of items with values greater than or equal the threshold.
Initialize the list to calculate items in each group items
Calculate the total value of items in each group for i in range (len (data_framc))
stem data_frame [stem_ccl][ii
leaf data_frame [leaf_ccl]
for j in range (1en (leaf))
itens. append (stem 10 + leaffil)
19

---

Published as a conference paper at ICLR 2024
Filter the items where the tctal value is greater than equai to the threshold filtered = [item for item in items if itom >= iton_threshold)
Count the number of items num_items len (filitered))
return num items
The example call the tool is: data
"Seem": [2, 3, 4, 5, 6, 7, 81,
"Leaf": [[2, 3, 9], [1, [1, [0, 6, 7, 91, [01, [1, 3, 9], [5]]
df pd. rame (data) count_groups_abott hatplpp___bvv__thheeeeeee htwproo__ttrrehho____tttttte (data framo=df, stem_col="Steen Stemm,
:.. lcaf_col="Leaf" item_threshcld-32)
Table
pasta with meat. sauce $6.49
pasta with mushrooms | $9.05
spaghetti and meatballs $7.43
mushroom pizza $9.28
+Query+
How much money does Jayla need to buy 5 orders of pasta with meat sauce and 3 orders of pasta with mushrooms? *Specifcc code solution+
"'python
import pandas as pd
def calculate_total_coo (fenu_df, orders):
***
This function takes in a pandas DataFrame representing a menu table and a dictionary of orders, and returns the total cost of the orders using pandas.
Args:
menu DataFrame) A pandas DataFrame containing menu items and their prices with columns 'Item' and 'Price'. orders (dict): : A dictionary where the keys are menu item names and the values are the number of orders for each item.
Returns:
float: The total cost of the crders.
Initialize the total cost total_ccat 0.0
Iterate through the menu itens and calculate the cost for each ordered item
for item, quantity in orders. orders.items()
Filter the DataFrame for the specific item item_df - menu_df [menu_df[] (ttf('' -- item] if not item item_ empty:::
item_priiie item dfl 'Price') values tosal_cost quantity itee_ttriiii
return total_cost
Example usage:
menu_data
'Item': ["pasta with meat sauce", "pasta with mushrooma", "spaghetti and meatballs", "mushrocm pizza"J, 'Price': 16.49, 9.05, 7.43, 9.28]
20

---

Published as a conference paper at ICLR 2024
@f pd.DataFrame (menu_data)
orders ("pasta with meat sauce"; 5, "pasta with mushroons"; 3) salculate_total_come (menu_df, orders)
Abstrctea tool function:
We re creating generic toel function from specitic code that calcuiates
the total cost of items based table of item prices per unit and identi fy the core reasoning of
dicticnary of item quantities. the specific code is to calculate the total cost based on item prices and quantities for each item, i.e. total_cost unit_price item quantity. The original code iterates through iter. names, filters
the table for each item, and calculates the cost based quantities.
We generalize the problem to create a flexible function. for any table of item prices and a dicticnary of quantitics. We replace specific columns and item names with generic variables like iter_col, price_col, and quanntiiisss We refactor the code with these variables, creating the function anduttitii......sees.....
far ""yyhhn
The final generic tool with decstring is: def calculate_total cost from unit amii_ppices____peeeeeeeeeesss_et and quant (item prices di, item_col, unit price_rcio...oo item item_ quantities):
***
This function takes in pandas DataFrame representing a table of
item prices and dicticnary of item quantities, and returns the total cost of the items based the prices and quantities.
Args:
item_priccs_de (pd. DataFraee) A pandas DataFrame containing iten names and their prices.
item_col (str): The column name for the item names. unit_price_col (str): The column name for the item prices.
item_quantities (dict): A dicticnary where the keys are item names and the values are the quantities of each item.
Returns:
float: The total cost of the itens.
Initialize the total cost total_cost 0.0
Iterate through the item names and calculate the quantity for each item based on quantities
for item_nanee name, quantity in stem_quantities.. items ():
Filter the DataFrame for the specific item name
item_price_dff item_prices_dr pteem_pricce__dl
=
item_name
if not item_price_df.empa
item_price (tn_:rrre_odomittt_rrieeeeeee valuuev
total_cost +- quantity item_price
return total_cost
The example call the tool is: =
'Item': ["pasta with meat sauce", "pasta with mushrooms", "spaghetti and meatballs", "mushroom pizza"), 'Price': [6.49, 9.05, 7.43, 9 9.28]
item peeemppriieeeeeeeeeee di pd. DataFaaFFran rame wtweee_ppie__dddeeeeeeee data)
21

---

Published as a conference paper at ICLR 2024
item quantities ("pasta with meat sauce": "pasta with mushrooms": ctt.aaa_osst..ooouttt...eeee.e "Item", "Price", iton_quantities)
(iter_prices_drr
*Begin!**
<Tbble
===table===
+Query*
===qst===
Specific code solutions
python
--- specific solution
Abstracted tcol function:
C.3.3 MATHEMATICS REASONING
*Rewriting Code to Create Generic Tool Function**
Purpose::** Given table, query and its corresponding code solution, your task is to rewrite and abstract the code to create general tool function that can be applied to similar queries. The tool function should solve higher-level question that encempasses the
original query and extend the code's functionality make it
versatile and widely applicable.
Consider the following principles:
1. The generic tool function should solve queries of the same type, based on commen reasoning steps without mentioning specific object nanes or entity terms.
2. Name the function and write the docstring concerning beth the core reasoning pattern and data organization format, without referencing
specific objects.
3. Replace specific strings or variable names with general variables to enhance the tool's applicability to various queries. All columns names used inside the tcol shculd be passed in arguments.
4. Call the tool function
solve the original query. End your with the format '4 The final generic tool with docstring is: ... and '# The example to call the tcol is: .....
Examplc**
Let (x) =\n\ = (begin \n3x=2 + 2i } x\lle 3, AMMmaa
6 slltext x>3.An) Lnnieend( (endesssss) An\A]Find Sas if the graph of Sy-f (x) is continuous (which means the graph can be drawn without. lifting your pencil from the paper).
Specific code solution: python
from sympy import symbols, Eq, sclve
def find_a(); ***
Finds the value of 'a' that makes the graph of the given piecevise function continuous.
Returns:
float: The value of ta" that makes the graph continuous.
a, symbols x!)
22

---

Published as a conference paper at ICLR 2024
Define the piecewise function pieces left_side 3+x**2
right_side a*x -
Find the value at the point of continuity (x = 3) Jeft_value_at_3 = left left_ sictssiiee.. subs (x, right_value_at_
subs
right_side.sube (x, 3)
Create an equation to solve for 'a' based on the condition of continuity
equation kq(leff_walue__t_3 right_ vaght_value_tt_3)
Solve the equation and return the value of 'a' solution = solve (equation, a)
return sclution[0]
find_a()
Abstreted teel function: assistant
We're creating generic tool function from speci fic code that finds the value of & variable in piecewise function that ensures continuity. The code calcuiates the appropriate value of variable by ensuring that the left and right sides of the piecewise function at the point of continuity are equal. We generalize the problem to create a flexible function for any piecewise function, ensuring continuity at specified point.
ppython
from sympy import symbols, Eq, solve
The final generic tool with decstring is: def aond_ccostant__oocoomtttt contimuity_poiit) **.
(x, constant, funcl, func2,
Find the constant that makes piecewise function cont at given point.
Parameters:
x (symbol): The symbol for the independent variable. constant (symbol): The symbel for the constant.
funcl (function): The first piece of the piecewise function.
func2 (function): The second piece of the piecewise function. continuity_poing (float): The x-value where the two functions should be continuous.
Returns:
float: The constant value that makes the function continuou..
kvaluate the functions the point of continuity
cuncl_at_cont_point funcl.subs subs (x, continuity_point) func2_at_cont_poiin func2.subs (x, continuity_poinc)
Create equation to solve for the constant based on the condition of continuity equation tf
func2_at_cont_poing
Solve the equation and return the value of the constant solution = solve (equation, constant)
return sclution[0]
The example to call the tool is: x, constant symbols ('x ('x constant"
23

---

Published as a conference paper at ICLR 2024
funcl 3*x**2 2
func2 constant**
cont sontiinity_pooint =
inuity (x, constant, funci, func2, continulty_point)
Query: If $2 (3)= and (x) for all Sxs, find Specific code solution:
python
def finé_inverse_val
baase_value base_result, target_result)
***
Finds the value of the inverse function based on the given properties of the original function.
Args:
base_valuc (float): The input value for the original function, f (x). basc_rcsult : The result of the original function, ((basc_value),
target_result :float): The value for which the inverse result needs to be found, {-1) (target_result)..
Returns:
float: The value the inverse function, x, suea that I'1-1) (t-rltttrgtt_eeeeee result) x.
***
current_value base_value current_result base_result
while purrent_resull != target_resulf,
current_value *= 2
current_result x=
return current_value
Set the initial values and find the value of the inverse function
f (-1) (target_result inverse value find inverse_ value (base value=3, (base_ base result=1,
target_resst resget=_e))lt=64
Abstreted teel function:
We creating generie tool function from specific code that finds the
value of an inverse function based on a set of rules for an original function, The original code sclves for the inverse function by
iterating over multiplier rule and updating the corresponding generalize the problem create values of the original function. combination rule with a common flexible function for any linear muitiplier, a base input-output pair, and a target output value for the inverse function. We replace specitic values and maltipliers with generic variables like bbase_nnuut, base_output, target_iinerree_ottpp,, and common_multiplier.
""ythhn
The final generic tool with docstring is:
def find inverse. value linear (base input, jase_output, target_inverse_onte ***
value_lineat
common, commen_multiphe multiplier)
Find the value of the inverse function based on a linear combination
rule for the original function.
Parameters:
base_input (float): The input value for the original function, (x). base base_output (Eloat): The result of the original function, E(base [onase_
24

---

Published as a conference paper at ICLR 2024
target_ inverse output (float): The output value for which the inverse result needs to be found, (-1) starget_inverse_oupp common_multiplier (float): The common multiplier between the original functicn's input and output values.
Returns:
float: The value of the inverse function, x, such that X.
(- itargee_iiverre_outtee
current imput base_inpil current_output base_output
while current_output !- target_invers__outptt
current purrent_input == common cemmon_multiplier
current_output common_mal munnen_
return current_input
The example call the tool is:
(base_input=3, base_output=l,
e target_inverse_ouppi
commen_multiplieres
*Begin!**
Query: ===qst=== Specific code solution:
"'pyhhon
sn-specific solution===
Abstrcted tcol function:
C.4 TOOL DEDUPLICATION
Hore are several tools with similar functionalitics. Your task is to select the most generic one, which can be widely applied and frequently reused across varicus scenarios. Your decision should be based on your understanding of typical use cases of VOA tasks and the
capabilities of the tools, rather than relying on superficial patterns like the frequency of tool names.
Tips:
1. Consider the level of specificity of the strings in the code to assess its generalizabilitt
2. Evaluate the tool's funstionalities and options to determine its versatility.
HH Format ###
Tools are listed below:
No. 0:
Tool 1
No. 1:
Tool 2
No. N: Tool N
Please respend with the numeric number preceding the most general tool using just one token, e.g.: N
25

---

Published as a conference paper at ICLR 2024
IFF Input Add
The tools are:
(
Please provide your answer by entering the numeric number preceding the most general tool using only one token:
C.5 TOOL RETRIEVAL
C.5.1 VQA
Given query, convert it into declarative command and then a brief and concise imperative instructicn.
Next, infer teol functions that can be used based on the instruction. Finally, infer the docstring of the tool functions.
Consider the following principles:
1. The instruction should reflect the action take, rather than emphasizing specific noun phrases. So you should prioritize using general terms like "object', 'people', and "action', and so on, instead of directly saying specific names like "desk", "american president', and "stuffed animal".
2. Use tool function names follewing the format "verb noun' with less than five words. Consider utilizing the most frequently used words in functicn names listed below.
3. The docstring of the tool functicn should be general and abstract, specific the query. Consider utilizing the most frequently used words function docstrings listed belox.
4. End your answer with the format 'The useful functions are: [...]' and 'The final answer is: ...', where 'I...]' is 1ist of useful functions and ....' is the returned answer. The most frequently used words function names: l'object', 'identify', 'check', 'objects', 'find', 'attribute', 'action', 'location', 'determine', 'existence', 'infer', 'type', 'describe', 'property', 'image', 'purpose', 'activity', 'count', 'interaction', 'state']
5.
4
6. The most frequently used words in function docstrings: ('specific', 'object', 'identify', 'check', 'image', 'certain', 'given', 'another', 'objects', 'find', 'type', 'existence', 'attribute', 'determino', 'action', 'possible', 'list', 'two', 'infer', 'number']
Query: What is visible on the desk?
Let's think step by step:
First, the corresponding declarative command of the query is 'Identify fy the visible objects the desk'. After abstracting, the general instruction should be 'Identity the objects on the specific surface.'.
So considering the naming rules of tool functions, the relevant and useful functions could be named as 'identify fybjjcssss or "fcemtifyy_objjctss_o.. _ouurracee....
Finally, we can infer that the docstring of the tool function could be 'Icentify the objects on the specified surface.',
The useful functions are: ('identify__bjeetss) "orcctss_o_susccce"
The final answer Identify the objects on the specified surface.
Query: Which american president is most associated with the stuffed animal seen here? let's think step by step:
26

---

Published as a conference paper at ICLR 2024
First, the corresponding declaritive command of the query is 'Search the american president most associated with stuffed animal seen here', \n'nn\
After abstracting, the general instruction should be 'Search people most associated with the specific object.'.Annnt
So considering the naming rules of tool functions, the relevant and useful functions could be named as An!n"!
ssearch_people. pewpleeassocattedetttte
Finally, we can infer that the docstring of the toel function could be 'Search for people most associated with the specitic object. '. 'ninn The useful functions are: pearrch_popee_assocciii thhhcatttewttt__jjeeeeec \nin") The final answer is: Search for people most associated with specific object.
Query: (query)
Let's think step by step:
C.5.2 TABULAR PROCESSING
Given table and corresponding query, please surmary the task goal and briefly describe the and column of the table. Next, infer generis table processing tool functions that can achieve the task geal.
Finally, infer the docstring of the tool functions.
Consider following principles:
1. Generic tool function names shculd be less than eight words in length. Consider utilizing the most frequently used words in function names listed below.
2. The docstring should summaize the task goal and table format. Be general and abstract, not specific to the query. Consider utilizing the most frequently used words in function docstrings listed below.
3. End your answer with the format 'The useful functiens are: [...]' and 'The final functions and '...'' is the returned answer.
is: ...', where 'I...I' is 1ist useful
4. The most frequently used words in function names: ['count', 'di "diffreence',, 'items', 'total', 'value', 'calculate', 'frequency', 'stem', 'leaf', 'groups', 'table', 'two', 'get', 'item', 'cost', 'specific', 'entities', 'column', 'threshold', 'find', 'group', 'unit', 'probability']
5. The most Exequently used words in function docstrings: ['two', 'number', 'item', 'column', 'items', 'specific', 'frequeney', 'values', 'name', 'total', 'difference', 'valuc', 'groups', 'specified', 'table', 'given', 'row', 'stemandlcaf', 'based', 'plot', 'entities', 'target', 'names']
TTable
Name: Orange candies per pag
Unit: bags
Content:
Stem I Leaf
2 2, 3, 9
5 | 0, 6, 7, 99
| 0
| 1, 3,
+Query+
A candy dispenser put various numbers of orange candies into bags. How many bags had at least 32 orange candies? let's think step by step:
27

---

Published as a conference paper at ICLR 2024
To solve the problem, should count the number bags with at least certain threshold candies based on stem-and-leaf plot. The code should combine stem and leaf values to calculate the total number of candics in each bag, filter the bags with candies greater than or equal to the threshold value, and count the number of such bags. Considering the naming rules of tcel functions, the relevant and useful functions could be named as "count_groups_ abbove_thresholdd inndddiin__ttt in_sten__eatt or above _tt__aaavveeeeeeeee_eeeee baseshooddbbaddddeeeee on Finally, can infer that the docstring the tool function could be
"Given threshold value and pandas DataFrame representing stem-and-leaf plot of groups, count the number of groups that have values greater than or equal to the threshold."
The useful functions are: ('count perouuntgrruuust..t sasvseabbeeeeeeeeeeeeeeeeeeeeeee threshold ii_ stem leaf', abwve_ttoeeseeeeeeeeeeeeeeee thresheld basee based on numeric "osprrii__ommintiioo]=... The final answer Given a threshold value and pandas DataFrame representing a stem-and-lcaf plct of groups, count the number of groups that have values greater than or equal to the threshold.
"count_items above
*Tbble
pasta with meat. sauce $6.49
pasta with mushrooms | $9.05
spaghetti and meatballs $7.43
mushroom pizza $9.28 +Query+
How much money does Jayla need to buy 5 orders of pasta with meat sauce and 3 orders of pasta with mushrooms?
Let's think step by step:
To solve the problem, we should calculate the total cost of items based on table of item prices per unit and a dictionary of item quantitics. The code should calculate the total cost by using the formula total_cost unit_price itantiuantity for each item, iterating through item names, filtering the table for each item, and calculating the cost based on quantities.
Considering the naming rules of teel functions, the relevant and useful
funct inn could be named atteaeeeettttaooofroooooteneeeee
paalculate_ttta__cquee Finally, we can infer that the decstring of the tcol function could be
frem items aadems_addddaee coefficients'.
'Calculate the total cost of the items based on pandas DataFrane
representing table of item prices and dictionary of item quantities,"
The uscful functions are:
["calculate_totallcott crrtst_fro__uit_pppe Baap__nnt_prrics_eee__e and and_quantities', roalcullte_cooallguuuttttttteeee ottpir/ffeeeenittmannooooooeeeee
The final answer is: Calculate the total cost of the iters based on
panças DataFrame representing table of item prices and dictionary of item quantities.
Table.
()
*Query* ()
Let's 's think step by step:
C.5.3 MATHEMATICS REASONING
Given a query, please infer the core mathematical skill for the solution. Next, infer generic mathematical tcol functions that can perform the core skill.
28

---

Published as a conference paper at ICLR 2024
Finally, infer the docstring the tool functions.
Consider the following principles:
1. Generic tool function names shculd be less than eight nathematic terms in length. Consider utilizing the most frequently used words in function names listed below.
2. The docstring should summarize the task goal. not specific to the query. Consider utilizing the most frequently usea words in function docstrings listed below.
Be general and abstract,
3. End your "The final answer is: ...', where '[...]' is list useful functions and '...' is the returned answer.
with the format 'The useful functions are: I...]' and
4. The most frequently used words in function names: ['find', 'calculate', 'sum', 'value', 'expression', 'difference', 'number', 'items'. 'total', 'time', 'target', 'inverse', 'generic', 'constant', 'max', 'squares', 'proportional', 'preduct', 'consecutive', 'evaluate', 'x', 'torm', 'factor', 'largest']
5. The most frequently used words in function docstrings: I'numbor', 'given', 'calculate', 'based', 'two', 'total', 'find', 'value', 'sum', 'time', 'target', 'items', 'certain', 'numbers', 'amount'' 'cost', 'first', 'distance')
Query: Let \* =in' loegin In3x=2 + 2sAltext x 3, Willlaax - 1 alltext x>3. lallend/ccsse An\V/Find $a$ if the graph of $y=f $ continucus (which means the graph can be drawn without lifting your pencil from the paper).
Let's think step by step:
To solve the problem, function expressions the boundary (x=3) The code should substitute x with in beth expressions, equate them, and solve for the unknown variable 'a'.
should ensure the continuity of the function by
equating the
Considering the naming rules of tcel functions, the relevant and useful functions could be named "Eind__cnstant___ cor__contnuity Finally, we can infer that the decatring of the tool function could be
'Find the constant that makes piecewise se function continuous at a given point.'
The useful functions are: ['solve_contt www.cnnnnnouuueeeeeeeeeeeeooeeee se_unttttiooooi for continnitttyll
The final answer is: 'Find the constant that makes a piecewise function continucus at given point.
Query: (3) and (2x) (x) all Sxs, find (-1) (
Let's think step by step:
To solve the problem, we should first understand the relationshis between the given function and its inverse. Then, we need to use the provided information about the function and its properties to deduce the value of the inverse function at given input. The code should analyze the properties to establish a connection (which is a linear relationship here) between the function and its inverse, and subsequent evaluate the result for the speci fic input.
consićering the naming rules of tcel functions, functions could be named as "findiineer___alleellnee 'find inverse_valll pasee_ on propertiess"
relevant and useful or
Finally, we can infer that the decstring of the toel function could be
'Find the value of the inverse function based linear combination rule for the original functicn.'
The useful functions are:
The final answer is: 'Find the value of the inverse function based linear combination rule for the original function.
Query: (query)
Let's think step by step:
29


Reviews:
[{"review_id": "e08cIfMo0P", "reviewer": "Reviewer_A2gB", "summary": "The paper introduces CRAFT, a novel framework for augmenting Large Language Models (LLMs) with specialized tools to tackle complex tasks. CRAFT generates task-specific toolsets and provides a retrieval component, enabling LLMs to offload functions to external modules through code snippets and APIs. This approach overcomes the limitations of general-purpose APIs, offering tailored solutions and improved flexibility. The framework ensures the quality and reusability of tools through validation, abstraction, and deduplication processes. Experiments across various domains, including vision-language, tabular processing, and mathematical reasoning, demonstrate substantial improvements over strong baselines. The paper's in-depth analysis confirms the scalability of CRAFT, the significance of each component, and the reliability and simplicity of the created tools. Ultimately, CRAFT presents a plug-and-play solution, enhancing the adaptability and problem-solving capabilities of off-the-shelf LLMs without requiring finetuning.", "questions": "1. Failure Cases: Can the authors provide specific examples or scenarios where CRAFT may not perform optimally? Insight into challenges or limitations faced by the framework would be valuable for a comprehensive understanding.\n\n2. Baseline Comparison: Could the authors expand on the choice of baselines used for comparison? Including a broader range of existing tools and frameworks (existing SOTA methods) might help in better positioning CRAFT’s contributions.\n\n3. Tool Creation and Retrieval Analysis: How do different types of tools contribute to the performance improvements observed with CRAFT? Additionally, how does the tool retrieval mechanism interact with various tasks?\n\n4. Real-World Application: Are there examples of real-world applications where CRAFT has been applied? Including such examples or results from a user study could provide additional validation for the framework.\n\n5. Tool Abstraction and Deduplication: Could the authors elaborate on the process of abstracting code solutions into reusable snippets and the criteria used for deduplication? Understanding this process in detail would provide clarity on the quality assurance of tools.\n\n6. Handling of Descriptive Responses: The paper addresses potential issues with underestimated performance due to descriptive responses from LLMs. Could the authors provide more details on how this issue is handled or mitigated in CRAFT?\n\n7. Scalability: The paper mentions the scalability of CRAFT. Could the authors provide empirical evidence or a more detailed discussion on how the framework scales with the number of tools and the complexity of tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. CRAFT showcases originality by combining tool learning, code generation, and retrieval to enhance LLMs' capabilities, applying this novel approach across various tasks and domains.\n\n2. The framework is rigorously validated across different tasks, demonstrating substantial improvements and ensuring tool correctness and reusability, reflecting the high quality of the work.\n\n3. The paper is well-structured and clearly written, providing a comprehensive presentation of the CRAFT framework, its applications, and experimental results.\n\n4. CRAFT addresses crucial challenges in augmenting LLMs, offering a significant advancement in the field and demonstrating practical applicability and effectiveness across diverse domains.", "weaknesses": "1. The paper could benefit from a more detailed exploration of scenarios where CRAFT might not perform as expected. Understanding the limitations and potential failure cases of the framework would provide a more balanced view and help guide future improvements.\n\n2. While the paper compares CRAFT to several strong baselines, expanding this comparison to include a wider range of existing tools and frameworks (e.g., SOTA methods in VQA) would strengthen the validity of the claimed improvements. This would also help in positioning CRAFT more clearly in the landscape of existing solutions.\n\n3. The paper could provide a more in-depth analysis of the tool creation and retrieval components of CRAFT. Understanding how different types of tools contribute to performance improvements and how the retrieval mechanism interacts with various tasks would offer valuable insights.\n\n4. While the paper mentions the scalability of CRAFT, providing empirical evidence and a more thorough discussion on how the framework scales with the number of tools and the complexity of tasks would be beneficial.\n\n5. The paper could explore and address potential biases in the tool creation process, especially considering the reliance on GPT-4 for generating code solutions. Ensuring fairness and mitigating biases is crucial for the applicability of CRAFT across diverse scenarios.\n\n6. Including a user study or examples of real-world applications of CRAFT could provide additional validation of the framework's practicality and effectiveness, offering a more comprehensive evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "efa9X7HXdB", "reviewer": "Reviewer_nWn7", "summary": "This paper introduces CRAFT, a novel framework designed to enhance large language models (LLMs) by creating and retrieving task-specific tools. CRAFT creates toolsets first and equips LLMs with a component that retrieves these tools to solve complex tasks. Experiments on vision-language, tabular processing, and mathematical reasoning tasks demonstrate the superiority of this approach over strong baselines. The analysis reveals that performance improvement is consistent when scaling the number of tools and the capability of backbone models, and that the created tools exhibit low complexity and atomicity.", "questions": "Please see the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.\tTraditional approaches to augment LLMs with tools lack flexibility, as they rely on general-purpose APIs. CRAFT addresses this problem by reusing task-related tools, which is more flexible.\n2.\tThis method can adapt off-the-shelf LLMs to new domains and modalities without finetuning.\n3.\tExperiments show that the proposed framework can improve a lot compared to previous approaches.", "weaknesses": "1.\tThe setting of the experiments is a little bit limited. There are many agent benchmarks like MINT, AgentBench, and so on, which focus on the problem-solving capacity of LLMs as agents. The reviewer thinks the work needs to be further verified on broader benchmarks for agents.\n2.\tThe comparison with LATM is a little bit unfair. The toolset created by CRAFT is the output of GPT-4, while the tool used by LATM is created by an inferior model if there is no misunderstood.\n3.\tThe transferability of the toolset should be discussed as I noticed that the toolset for the VQA task and the toolset for the reasoning task are not the same. Maybe the authors can experiment to create a general tool set for all tasks and see what will happen.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tiCV4QnXcv", "reviewer": "Reviewer_3JvW", "summary": "The paper presents CRAFT, a framework for tool creation and retrieval to customize large language models (LLMs) for various tasks and domains. CRAFT creates a specialized toolset by prompting LLMs to generate and abstract code solutions for problems, and validates and deduplicates the tools. CRAFT retrieves relevant tools from the toolset by multi-view matching, and adds them to the prompt of LLMs for code generation. CRAFT improves performance on vision-language, tabular processing, and mathematical reasoning tasks.", "questions": "1. What does \"bug-free\" mean and how do the authors ensure that the generated tools are \"bug-free\"?\n\n2. What is the result of tool generation with GPT-3.5-turbo and other LLMs?\n\n3. Can CRAFT be adapted to programming tasks like HumanEval since it generates \"APIs\"?\n\n4. Can the authors discuss more on the \"pseudocode library\" like can we use a natural language library and how is it different from in-context learning?\n\n5. Can the authors analyze more on the created toolsets like where they might it can be applied/generalized?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a tool generation and tool-using framework for LLMs which is a good attempt to enhance LLMs' capability to solve reasoning tasks by generating programs.\n\n2. Personally I think the idea of a \"pseudocode library\" proposed in future work is cool and meaningful.\n\n3. The experiments on baselines and different LLMs are comprehensive and the result is promising. Basically, I agree with the authors that tool generation puts a high demand on the LLMs' coding ability.\n\n4. The created toolsets are a particularly important contribution to the LLM community.", "weaknesses": "1. The authors mentioned that alternative backbone models like CodeLlama demonstrate near-random performance. Can the authors provide such results (the performance of different LLMs in creating and using tools) in the experiment?\n\n2. I suggest that the author should make the distinction between more specific methods more prominently in the main text (though the difference has been discussed in the experimental setting), such as by creating a table to compare various tool-augmented language model methods, and so on. The current Figure 1 appears to be similar to previous work like LATM, making it difficult to showcase the uniqueness of this article.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)