CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models
Authors: Cheng Qian
Chi Han
Yi Fung
Yujia Qin
Zhiyuan Liu
Heng Ji
Venue: EMNLP 2023 Findings (2023)
Keywords: Large Language Models
Tool Creation
Model Reasoning
TL;DR: We propose CREATOR, a framework enabling Large Language Models to create their own tools for improved performance by disentangling their abstract and concrete reasoning capabilities.

Abstract: Large Language Models (LLMs) have made significant progress in utilizing tools, but their ability is limited by API availability and the instability of implicit reasoning, particularly when both planning and execution are involved. To overcome these limitations, we propose CREATOR, a novel framework that enables LLMs to create their own tools using documentation and code realization. CREATOR disentangles abstract tool creation and concrete decision execution, resulting in improved performance. We evaluate CREATOR on MATH and TabMWP benchmarks, respectively consisting of challenging math competition problems and diverse tabular contents. Remarkably, CREATOR outperforms existing chain-of-thought, program-of-thought, and tool-using baselines. Additionally, we introduce the Creation Challenge dataset, featuring 2K diverse questions, to emphasize the necessity and benefits of LLMs' tool creation ability. Further research demonstrates that leveraging LLMs as tool creators facilitates knowledge transfer, and LLMs exhibit varying levels of tool creation abilities, enabling them to adapt to diverse situations. The tool creation ability revolutionizes the LLM's problem-solving paradigm, driving us closer to the next frontier of artificial intelligence.

CREATOR:
Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models
Cheng Qian', Chi Han', R. Fung', Yujia Qin', Zhiyuan Liu'r, Heng Ji
TTinggha University, PUniversity of Illinois at Urbana-Chanpaign qianc20 @ mails. asinghua.edu.. edu. cn
Abstract
2022). However, there are still limitations ad- dress, such as the inability to handle up-to-date information (Yu and Ji, 2023), provide accurate mathematical results, or reason over long chains of logic (Trivedi et al., 2022; Komeili et al., 2022; Patel et al., 2021; Hendrycks et al., 2021; Lu et al., 2022b).
Large Language Models (LLMs) have made significant progress in utilizing tools, but their ability is limited by API availability and the instability of implicit reasoning, particularly when both planning and execution are involved. propose To overcome these limitations, CREATOR, a novel framework that enables L.LMs create their own tools using documen- tation and code realization. CREATOR disen- langles abstract tool creation and concrete de- cision execution, resulting in improved perfor- mance. We evaluate CREATOR on MATH and TabMWP benchmarks, respectively consisting of challenging math competition problems and diverse tabular contents. Remarkably, CRE- ATOR outperforms existing chain-of of-thought, program-of- thhhought,, and tool-using baselines. Additionally, we introduce the Creation Chal- lenge dataset, featuring 2K diverse questions, to emphasize the necessity and benefits of LLMs' tool creation ability. Further research demon- strates that leveraging LLMs tool creators facilitates knowledge transfer, and L.LMs ex- hibit varying levels of tool creation abilities, en- abling them to adapt to diverse situations. The tool creation ability revolutionizes the LLM's problem-solving paradigm, driving us closer to the next frontier of artificial intelligence. All the codes and data are released!.
To overcome these concerns, researchers have explored equipping LLMs with external tools to alleviate their memory burden and enhance their expertise (Qin et al., 2023). For instance, integrat- ing tools such question-answering systems web search engines enables LLMs to learn how and when to access external resources for problem- solving (Nakano et al., 2021; Schick et al., 2023). Recent studies have also incorporated additional tools for LLMs, such as GitHub resources, neural network models (c.g.. Huggingface library), and code interpreters (e.g.. Python interpreter). aiming to enhance their capabilities (Gupta and Kembhavi, 2022; Surís al., 2023; Shen et al., 2023; Liang ct al., 2023; Lu al., 2023). These tools require LLMs to provide detailed plans before utilizing them to solve complex problems.
However, tool-augmented LLMs still encounter challenges (Chen al., 2022: Gupta and Kemb- havi, 2022; Schick al., 2023; Surís et al., 2023), particularly in the following aspects. (1) Limi- tation in scope: Current approaches focus on a limited number of tools, making it difficult to find an appropriate existing tool for new problem types. (2) Fragility in reasoning: Given that tasks are often complex, reasoning the fly case-by-case can be fragile random errors, while humans can benefit from finding robust commonalities among multiple similar questions.
1 Introduction
In recent years, notable progress has been made in large language models (LLMs) like GPT-3 (Brown et al., 2020), Codex (Chen et al., 2021), PaLM (Chowdhery et al., 2022), LLaMA (Touvron et al., 2023), ChatGPT (OpenAI, 2022), and the latest GPT-4 (OpenAI, 2023). These models exhibit im- pressive capabilities in in-context learning, code generation, and various Natural Language Process- ing (NLP) tasks (Feng al., 2020; Dong al.,
(3) Insufficiency error-handling: Current tool utilization pipelines lack automatic and specific error handling, necessi- tating improvements in accuracy and robustness to ensure reliable execution results.
"Conresponding author. Phttps://githun cow www.//iancceegge/RRATTTTT.
In this paper, we propose a novel approach to ad-

---

ing the cognitive burden of LLMs and disentan- gling their ability perform abstract reasoning (creation of generalizable tools) and concrete rea- soning snreisioo-makingg with details); (3) utilizing code as the medium for tool creation, which is more sensitive to errors, and enabling automatic rectification of tools and decisions based on error tracebacks.
Throwing 40to building, hright relutivn lime repersented
Mi 23122 20E 40. When raach the ground?
External Platforms
Abstraction Quadratic Equation Solving
def sa3ee www.aaidaticcoo.. b, c):
Search
Eugiue
Selver
Dectaton
Conccetization: Calling
1. Use Math Selver do query. 2. Apply Info Extraciar get the answer.
sokre quadh suiic -520
Selection:
WolframAlpha, lnfo Extractor
To evaluate our design's effectiveness, we test CREATOR on two existing benchmarks: MATH (Hendryeks et al.) and TabMWP (Lu et al., 2022a), as well as the Creation Challenge dataset we create.
Execution
Execate the prograta after conrabenating tools and their calls
Quer: Salve (-31 20
Rectification
Letor +
Rectity Creation/ Decision
The MATH dataset contains diverse and challeng- ing math competition problems, while TabMWP in- cludes wide range of tabular contexts for problem- solving. Notably, ChatGPT built on CREATOR achieves remarkable average accuracies of 59.7% and 94.7% on MATH and TabMWP respectively, surpassing the standard chain-of-thought (CoT) (Wei et al., 2022), program-of-thought (PoT) (Chen et al., 2022), and tool-using baselines by significant margins.
Spocoss
+
Render À kowee
Output from Exoc: The time 3,40
Apll
CREATOR (Ours)
Tool-Use
The difference between CREATOR and a gen- eral tool-using framework.
Figure I:
dress these challenges. Rather than treating LLMs users of tools, we empower them to be cre- ators of tools, enabling them to solve problems with higher accuracy and flexibility. We introduce our tool creation framework, CREATOR, which leverages LLMs' ability create and modify tools based on the problem at hand. Figure illustrates the differences between CREATOR and a general tool-using framework. While the tool-using frame- work focuses on reasoning to select and plan API usage, our framework emphasizes diversifying tool choices, disentangling abstract and concrete rea- soning, and improving robustness and accuracy. Specifically, CREATOR consists of four stages:
As existing benchmarks do not specifically evalu- ate tool creation, we further introduce the Creation Challenge dataset, which consists of novel and chal- lenging problems that are inadequately solved us- ing existing tools or code packages. This dataset highlights the necessity and advantages of LLMs' tool creation ability. In addition, we show exper- imental results that provide evidence of how tool creation plays crucial role in promoting knowl- edge transfer across similar queries that possess common core knowledge but differ in specific sce- narios. We also present case studies highlighting the varying levels of tool creation ability observed in LLMs, allowing them to better adapt to diverse problem settings.
Creation: Create generally applicable tools with documentation and realization through abstract reasoning based the problem.
Decision: With available tools, decide when and how to use them to solve the problem.
2 Related Work
Execution: Execute the program, applying the chosen tools to solve the problem.
Large Language Models. Large Language Mod- els (LLMs) have gained attention for their im- pressive performance in handling various NLP tasks, following demonstrations and generating high- texts and codes (Brown et al., 2020; Chen et al., 2021; Chowdhery et al., 2022; Touvron et al., 2023). Prompting methods such chain- of-thought (Wei al., 2022), instruction-followinn (Wang al., 2022b; Longpre et al., 2023; Chung et al., 2022; Touvron et al., 2023; Liu et al., 2023), and verification mechanisms (Fung et al., 2023) have been developed to guide LLMs in problem-
Rectification: Make modifications to tools and decisions based on the execution result.
By introducing these four stages, we aim to bet- ter inspire the LLM's creativity and enhance the paradigm's robustness. This design sets CREATOR apart from traditional tool-using and addresses the three challenges we discussed respectively by (1) leveraging L.MSs to create tools with higher gen- crality, reusability, and variety, rather than relying on a limited number of given APIs; (2) offload-

---

solving and align their behavior with human expec- tations. Our work builds upon these areas, incorpo- rating them into our framework and using them as baselines for complex problem-solving.
Create Utilize Apply Emphasize Reasoning
Method
Tools
Tools
Codes Reusability
Pattern
CoT
Linear
PoT
Linear
Tool Use
Linear
Tool Use and Tool Creation. As an emerging field within NLP, the active interaction of LLMs with environments is facilitated through tools that serve as the medium (Li et al., 2023). Recent stud- ies address constraints of LLMs, such as the lim- ited real-time responsiveness and inaccurate cal- culations, by incorporating external tools (Trivedi et al., 2022; Komeili et al., 2022; Patel al., 2021; Lu et al., 2022b). These studies augment LLMS with tools like scratch pads, search engines, QA systems, and calculators (Nye et al., 2021; Shuster et al., 2022; Schick ct al., 2023) to improve task performance. More recent efforts integrate LLMs' tool-using abilities into pipeline for task planning, tool calling, and result synthesis (Wu et al., 2023; Shen et al., 2023; Liang al., 2023). In contrast, our work goes further by enabling LLMs create tools instead of relying solely on existing tools. As our concurrent works, tool creation ability is also investigated under LATM framework (Cai et al., 2023) and LLM customization (Yuan et al., 2023).
CREATOR
Non-Linear
Table I: A comprehensive comparison of CREATOR with other methods.
ation, decision, execution, and rectification, illus- trated in Figure 2. The utilization of tool creation for problem-solee and aligns with LLMs' innate ability, as illustrated later in Section 5.2. In CREATOR, the main objec- tive of design is to instinctively better inspire their creativity, and facilitate more effective use of it.
inherently straightforward
Previous CoT and PoT methods mainly apply linear reasoning solve target problems, and their task-solving process lacks reusability. However, the tools created in CREATOR can be transferred solve other queries, and the rectification stage in- corporated makes the reasoning process non-linear. We present a comprehensive comparison between CREATOR and other methods in Table 1.
Reasoning and Execution with Program. Rea- soning with programs is an emerging field in NLP, whose goal is to leverage codes to do complicated computational reasoning instead of using natural language thoughts. Chen al. (2022) show that code generation improves performance on math datasets, while Gao et al. (2022); Wang et al. (2022a) further demonstrate the potential of pro- gram reasoning on symbolic and algorithmic bench- marks. These efforts present a code-based chain-of- thought with linear logic but produce no enhanced tools capable of being reused or tested. As the con- cept of tool-using emerges, recent studies begin to incorporate code interpreters external tools (Lu al., 2023; Mialon al., 2023; Wang et al., 2023). However, CREATOR, we use code as the medium for tool creation rather than an external tool. Our framework also excels over PoT as we devise the tool creation stage, code rectification stage, and disentangle the logic in complex reasonings.
3.1 Creation Stage
Implementation Details. In the creation stage of CREATOR, we explicitly instruct LLMs with demonstrative examples to create tools and doc- umentation to solve the problem. The gen- cral prompt text form "##IInstrcctio [INSTRUCTION]Nn [EXAMPLE 1JNn [EXAMPLE 2] Here the instruction text "[INSTRUCTION]" describes the goal and format of the output. Fach demonstration "[EXAMPLE x]" follows format "### Question [QST]\n ### Tool [TOOL]". Each [TOOL] contains documentation text code com- ments. A detailed example of prompt text is shown in Figure 14.
To get these demonstrations, we make a fixed set of demonstrations advance and use them sub- sequently for each task. In specific, we randomly select subset from the training set and prompt the LLM with the text instruction for tool creation for each query. We then correct the errors in these gen- erations (if any) and remove verbose explanations before using them. Although the demonstrations are from the same task as the test queries, they are not required to be semantically similar to test queries, as the main purpose is only to inspire the
3 Design of CREATOR
Distinet from previous frameworks for tool-using, CREATOR leverages the tool creation ability of
LLMs by incorporating four special stages: cre-

---

Query
Given equations 3x+y=17, 5y 12=1 3x+5z=11, what is the value of the sum x
Creation
Decision
Created Tool
Make Decisions about How to Apply the Tool Use the tool to get solution equation system Take the sum over the solutions find the answer
solve, equationsequaaions))
Documentation
Solves a system of Enear equations using himmlyiinaggsooo
Decision
Takes in list of equations in the form of 2D arays:
solution solve, equations(equatioss)
Retims of the solutions to the system.
larget = sum (solution)
Realization
Tool: Code Format
coof const for equations:
solve solve_equat eqque (equations))
coef_append(cel] 1-1)
Decision: Code Format
const.append (eal -1]
sol linaig. scoof, const)
(13,3,0,17], [0,5,1,14), [3,8,5,42]] (er.aciiossssuuuiionnn target no.sun (solutioo)
equations
If Error Occurs:
Success:
EXEC
print Thhe sus of
is", target)
Traceback:
Printed:
Execution
line
The result x +y +2 is 12
Answer
(Error Reasons)
The Answer for this Query is 12.
Rectification
Overview of CREATOR framework with four stages: Creation, Decision, Execution, and Rectification. With an LLM like ChatGPT, we successfully leverage its tool creation ability with code as the medium. In each stage we apply instructions and demonstrations prompts, shown Figures 14 to 16 in Appendices.
Figure 2:
LLM's creativity and regulate its output format.
codes. This [INSTRUCTION] applies both to get demonstrations and to conduct test-time inference, which ensures that the LLM's answer can be eas- ily extracted from printed outputs in subsequent stages. detailed prompt text example is shown in Figure 15.
Ability of Abstract Reasoning. The core impor- tance of the tool creation stage is trigger LLM's ability to employ abstract thinking to alleviate the burden of reasoning during later stages. When LLMs create tools, they effectively use abstraction to address a particular problem type, necessitating a focus on the inherent characteristics of the prob- lem rather than the specific numerical details. For example, in Figure 2, the LLM concentrates solely recognizing the intrinsic nature of the problem and creates tool for solving tion system, disregarding all the numerical details and the specific expression being queried.
Ability of Concrete Reasoning. The decision stage necessitates the LLM's meticulous attention to rules and details for problem-solving, which we refer to concrete reasoning. In Figure 2, the lution obtained from the tool needs be summed for the final answer. This requires the LLM to un- derstand the tool's outputs and relate them the specific query to make an informed decision and derive the correct answer finally. By separating creation from the decision, CREATOR disentangles two phases of the LLM's abilities, which facili- tates smoother elicitation of different aspects of knowledge and improves task performance.
equa-
3.2 Decision Stage
Implementation Details. Similar to the creation stage, we instruct LLMs with demonstrations to decide how to use tools with the same prompt text form. Each demonstration "[EXAMPLE x]" fol- lows "### Question [QST]'n ### Tool [TOOL] ### Solution [SOL]". where [SOL] represents the LLM's decision tool calls in code format. We also derive fixed demonstration set the same way in the creation stage, only that the LLM is now prompted call the given tools instead of creat- ing them, and to print out the final answer with any important information through "print(....)" in
3.3 Execution Stage
The execution stage takes the information from pre- vious stages to exccute the tool leveraging the code interpreter. We do not apply the LLM in this stage, and the created tools and the LLM's decision are concatenated into cohesive code block for exe- cution. The tool is encapsulated within a function in the code block, and the LLM''s decision calls it

---

for problem-solving. During execution, we capture any outputs printed (as we have instructed the LLM in the decision stage) or errors encountered (by in- tercepting error messages in a sub col-process). These information serve as inputs for subsequent stages to determine whether an answer can be obtained or rectifications are needed.
4.1 Experimental Setup
Settings. We select ChatGPT as the base model for all methods due to its exceptional capabilities in code generation, decision-making and logical reasoning. Refer to Appendices A.I for more details. We evaluate CREATOR on two existing datasets: TabMWP, which includes diverse table- related problems, and MATH, consisting of chal- lenging math competition problems. We apply them as they are representative in terms of diversity in data format and difficulty. We also assess the per- formance of our framework on Creation Challenge, comprising 2K data points, explore the impact of tool creation hints on the LLM's s performance. Refer to Appendices A.2 for more details.
3.4 Rectification Stage
Implementation Details. During the rectifica- tion stage, CREATOR has two different options based on the information passed into it. If an error occurs, then the LLM is prompted with demonstrations to rectify the error. Applying a similar prompt format as before, the format of demonstrations "[EXAMPLE x]" now changes to "### Question [QST]in ### Original [ORI]An [REC]",
Baselines. We compare CREATOR against four types of baselines to demonstrate its effectiveness:
### Error [ERR]Nn ### Rectificatio
where we provide the original tool implementation and calling decision in [ORI], offer the error trace- backs [ERR], and concatenate natural language rea- soning on the error with the rectified code in [REC].
Vanilla LLM w/ and w/o CoT: The Vanilla LLM with CoT employs linear reasoning to solve problems, while Vanilla LLM without CoT directly generates the answer.
A detailed illustration of the prompt text is shown in Figure 16.
The LLM utilizes a program to reason through the problem step by step. Besides, we also incorporate rectification into PoT as a stronger bascline for a fair comparison.
PoT:
If the execution is successful, then the answer will be extracted from the captured model's output and compared to the standard answer to measure accuracy.
Tool Use: The LLM utilizes the WolframAlpha API general-purposs tool specialized in cal- culations. It's fair external tool all queries require numerical reasoning to some extent.
Significance. During the rectification process, we provide the LLM with error tracebacks, which offer crucial information for it to identify the er- ror's location and causes. Armed with this guid- ance, the LLM can recover from previous mistakes, adjust its reasoning process, and attempt to solve the problem once again. Subsequent experiments will demonstrate how the inclusion of rectification significantly improves the performance of CRF. ATOR. The success of the rectification stage also showeases the LLM's ability to recognize miscon- ceptions and self-correct.
CREATOR -ettoo---ddtgedd The LLM combines the creation and the decision stage in CREATOR in- stead of disentangling them, which serves as a special baseline for ablation study.
4.2 Creation Challenge
Existing benchmarks are not originally designed to evaluate tool creation, thus unable fully show- case the necessity and advantages brought by the LLM's tool creation ability. Therefore, we intro- duce Creation Challenge to test the LLM'ss problem- solving skills under new scenarios, without existing tools or code packages that can be directly applied. Refer to Appendices B. for details about the data format and construction process.
4 Experiments
To evaluate the effectiveness of CREATOR, we con- duct experiments on two established benchmarks: MATH (Hendrycks et al.) and TabMWP (Lu et al., 2022a). Additionally, we perform experiments newly introduced dataset, Creation Challenge. com- prising 2K diverse questions that are inadequate to solve using existing tools or code packages. This enables us to further demonstrate the necessity and advantages of the LLM''s tool creation ability.
Evaluation
The components of the standard cre- ated tool in each data point of Creation Challenge can serve as valuable hints for the L.M''s tool creation. Therefore, we extend our experiments on Creation Challenge to assess the LLM'ss tool

---

Counting Probability
Itmd. Algebra
Number Theory
Pre- Algebra
Pre- Calculus
Average (weighted)
Method
Setting
Algebra
Geometry
w/o Cof
25.7
25.8
22.4
13.9
18.5
40,9
21.8
25.3
Vanilla
Col
50.9
24.5
17.5
23.2
58.6
16.7
37.9
PoT (wlo Rec.)
w/o CoT
58.2
48.5
35.4
25.8
53.1
66.8
25.0
49.8
CoT
54.0
47.8
32.5
22.3
48.9
64.5
19.9
46.5
w/o CoT
63.8
51.9
35.9
28.6
59.2
70.0
28.2
53.9
PoT (w/ Rec.)
Col
61.4
48.8
34.6
23.7
54.5
67.6
34.6
51.2
CoT
47.3
35.1
27.0
20.5
30.8
56.8
31.4
39.0
Tool Use
w/ Col
55.3
37.8
28.7
20.5
34,8
61.8
26.9
43.0
CREATOR
Demo.
58.0
53.3
34.2
55.7
63.4
33.3
-Entangled
w/o CoT
64.1
55.7
35.9
42.7
61.6
69.0
37.2
57.2
w/ Col
62.7
50,9
33.8
31,4
61.4
68.7
31.4
54.0
CREATOR
w/o Demo. 66.6
53.6
33.8
29.4
$9.8
68.7
34.6
54.9
who CoT
71.5
55.3
41.4
41.9
60.4
71.7
35.3
59.7
(ours)
Col
63.1
58.1
34.6
35.0
61.8
69.7
32.1
55.7
Table 2: The accuracy (%) the test set of MATH dataset leveraging ChatGPT. Rec. represents Rectification.
creation ability with varying levels of hint utiliza- tion. We encourage future research to explore the dataset's potential through more flexible usage.
Successful Execution
Method
Setting
Accuracy
w/o CoT
68.2
99.1
Standard
w/ CoT
75.2
99.3
4.3 Experimental Results
(w/o Rec.)
w/o Cof
80.6
98.5
w/ CoT
80.0
91.2
We present the results MATH, TabMWP, and Creation Challenge respectively in Tables 2 to CREATOR achieves an accuracy of 59.7%, 94.7%. and 75.5% respectively on three tasks, surpass- ing all the best performance in baselines by large margins. To illustrate CREATOR'S 's advantage, we present case study showing how it's better than Tool Use in Figure 3A. For all tasks, disentan- gling the creation and decision stages generally results in better performance, compared to CRE- ATOR- Entangedd For Creation Challenge, we also observe that hints of tool creation can raise the per- formance up to 18.7%. We will further analyze the reasons for this improvement in Section 4.4.
w/o CoT
81.2
99.7
PoT (w/ Rec.)
w/ CoT
87.3
100
w/o CoT
77.6
100
Tool Use
w/ CoT
79.6
w/o CoT
91.6
100
CREAToR-Entangled
w/ Col
93.5
CREATOR (ours)
who Col
90.5
99.7
w/ CoT
94.7
100
Table 3: The accuracy (%) on the set of TabMWP dataset leveraging ChatGPT. Successful Execution in- dicates whether the LLM provides a valid final answer through words codes within the rectification limit.
Figure 3B. we show the adoption of brute-force algorithms and straightforward caleulations when CoT is not applied yields higher accuracy.
4.4 Results Analysis
In contrast, TabMWP involves simpler calcula- tions and more straightforward reasoning paths, promoting consistency between natural language and programming reasoning. Therefore, the ap- plication of CoT enhances performance in these cases. We present more case studies to illustrate it in Appendices C.I.
CoT Incompatible with Codes. Table shows the LLM's performance on MATH problems de- creases consistently when applying CoT under PoT method and CREATOR, and the opposite trend observed for TabMWP. We attribute this difference to the inherent incompatibility between natural lan- guage reasoning and program- based reasoning on challenging problems. MATH problems involve intricate calculations and diverse reasoning paths, leading to conflicts between natural language and programming approaches. When CoT used, the LLM tends to generate programs following natural language reasoning, which compromises the coher- ence and unique advantages of programming. In
CREATOR is Robust to Challenges. Figure 4 il- lustrates the performance of the LLM in relation to difficulty. CREATOR outperforms all the baselines for both tasks and achieves higher accuracy, partic- ularly for difficult problems. This provides com- pelling evidence that CREATOR exhibits greater resilience to challenges.

---

Problem: In the nth row of Pascal's Triangle where the first row is n=0), the arithmetic mean of the elements is 51.2.
the
Creation:
Deciston
In Pascal's Triangle, the row o+1 elemenls, and their sum is 2hhh The arithmelic is then (2/m::(""))nn (n+1)s which is 51.2 Mulliplying both sides by $n+15 and laking the logarilhm:
pascal _aea((n): Initidlization
Eniiiallaation
Earget 51.2
row sue
mus, =1
Take 2oep until natch
Takce Toop and add sue for rango(rux el]:
while True:
WOLFRAM CALL: solve
nean pascals, _rran(() 1f nean target:
#1 nath..cehh(", k)
IAPI Return: (119 Lile:: Result, subpod': l'plaintext: (W(1024
break
Logr21) - (2gg2)))))))))... 16000"1)))
the
Frint the
A the value approximately 6,10003, rupresenns an integer, we round up get n=7
Answer: The valuu is
Problem: How many x between 0 and 100 exclusive satisly both conditions? 1. is multiple of 2. multiple w/o CoT:
Col:
count_rultipp (ultepeesss b): sount
I can use the concept of LCM solve this problem. the tool, can LCM of 4 and to calculate the def count x valuess)
def count nulttoles(stars, end, nue): return (end - 1) mun
in range(3, 181):
(start 3)
if e and 0: count t+ 1
number inlegers between and 100 Uhat multiples of hoth and 6.
len 12 LCM
return count
return count _ww.tiplee(e......
lcn)
nsswer:
Answer:
Figure 3: In subfigure A, show example in which Tool Use reasoning (left) fails, while CREATOR (right) solves successfully as it derives a new tool for the novel question. In subfigure B, we present a case comparing the answer given by CREATOR with and without CoT. Challenging problems in MATH cause conflicts between language and program reasoning.
MATH
Method
Setting
Accuracy
Successful Execution
80
200
w/o CoT
27.9
94.9
Standard
w/ CoT
32.7
99.1
w/o CoT
59.2
93.5
PoT (w/o Rec.)
w/ CoT
60.7
95.7
TV.
TWMMP
PoT (w/ Rec.)
w/o CoT
61.1
98.3
CoT
62.0
98.9
Assamay
EHEETOR-EEnaagled
no hint
64.5
99.2
80
(w/o CoT)
utility hin!
65.8
99.3
70
all hint
75.3
99.5
60
CREATOR (ours)
utility kint
no kint 63.8
67.2
99.1
98.7
50
Grdda
Grodo
Grode
Grado
Grode
Grade 6
Grade
Grade
(w/o CoT)
all
75.7
99.5
(bes)
Sandeed CoT
CHEATOR: (best)
Table 4: accuracy (%) on the Creation Challenge test set leveraging ChatGPT. No hint represents normal CREATOR framework. Utility hint provides hints about the utility the tool, while hint offers additional hints about the possible inputs and outputs of the tool,
Use (best)
Figure 4: Comparison the accuracy of baselines and CREATOR wint. problem difficulty.
leads to improved performance. (2) Availability of Hints: In practical scenarios, guidance is often necessary to harness the LLM's behavior when cre- ating tools. We demonstrate that providing more detailed hints can significantly improve the LLM's performance, as they enable easier implementation of desired tools and eliminate uncertainty and mis- directions in CoT or tool documentation.
Rectification Raises Performance. Figure demonstrates the improvement in the LLM's per- formance achieved through the application of the rectification stage. Results show rectification can increase the accuracy by approximately 10% of the original value, which proves the necessity and rationality of establishing this stage.
Influential Factors of Tool Creation. Tables 2 4 highlight crucial factors affecting the LLM's performance. (1) Separation of Creation and Decision: The separation of these two stages inherently represents the disentanglement of the LLM's abstract and concrete reasoning, which
Further Analysis
In this section, we further show the advantages brought by the LLM's tool creation ability and use case studies to demonstrate different aspects of this ability, which enables them to tackle challenges

---

PoT Improvement:
Similar to previous experiments, we use Chat- GPT the base LLM with unchanged detailed settings. We first test the problems under the normal CREATOR framework respectively. Then, we test the correct tool created under one sce- nario could be applied to the other two, and again test the LLM's performance.
100
Encrose
Nourazy Mi
%5
Incemoo 2.F%
70
82%
GL7
2310
12.
Challenge
CREATOR Improvement
100
Incrose
Nouraey (a)
Incrsoo
765 752
70
5.0%
Results Analysis. The statistics are presented Table 5. Through the application of transferred tools, the LLM's accuracy can be raised by 15.3%. Further analysis shows that 39 sets of queries are positively influenced by this transfer, which high- lights the tool creation ability of the LLM can facilitate knowledge transfer, leading to better performance on clusters of problems that share similar core concepts.
78L7
Challenge
Rectify-1
Rectify-3
Figure 5: The improvement brought by rectification on both PoT and CREATOR. Rectify-N denotes enabling N rounds of rectifications.
Set Queries, Count Data Pieces, Count
300
5.2 Different Levels of LLM's Tool Creation
Tool Create Normal, Ace.
63.0%
We discover in experiments that LLM create tools in different levels without special guidance, which affirms creativity is LLM's intrinsic emerg- ing ability. By inspecting the created tools, we find that they can be categorized into three levels, which provides guidance and reference for future development.
Tool Create with Transfer, Acc.
78.3%
Increase of Acc.
15.3%
Sets Worse with Transfer Sets Better with Transfer
2/ 100
39/ 100
Table Results of tool transfer experiment. Tool trans- fer improves accuracy by up to 15.3%.
1. Enhancement of Existing Tool. First, LLMs demonstrate the capability enhance existing tools by encapsulating an existing tool or API and re- purposing it to serve different needs. The first case of Figure shows how LLM wraps an existing weather query APT into new tool that calculates the average temperature.
with more flexibility and less reasoning burden.
5.1 Facilitation of Knowledge Transfer
One of the main purposes of tool creation lies in its reusability. The content of tools represents the ab- straction of knowledge concepts, so the creation of one tool may help solve problems various scenar- ios that share the same core concept. For instance, a keyword-extrand extraction tool created for sentiment analysis can be applied to other tasks like docu- ment categorization and topic modeling, they all require the identification and extraction of rel- evant keywords for problem-solving. By utilizing the knowledge and logic embedded in the tool, the LLM can transfer its understanding to solve similar problems efficiently with higher performance.
2. Concatenation of Multiple Tools. Second, the LLM can create new tools by organizing multi- plc APIs into pipeline, enabling it to fulfill spe- cific purposes. The second case in Figure 9 shows how the LLM calls two existing APIs three times in the tool for problem-solving.
3. Hierarchical Tool. Third, the LLM can create tools with a clear hierarchy, which establishes clear caller-callec relationships among tools and reduces the burden of repetitive reasoning. The third case in Figure 9 illustrates hierarchical structure where the first tool serves as the callee, while the second tool primarily solves the problem.
Settings. To validate our hypothesis, we con- struct a small set of questions with 300 data points, detailed in Appendices B.2. We divide data points into 100 sets, where all three queries in one set share the same core knowledge concept (key methodology that is universally applicable) but dif- fer in scenario (problem background and specific details inquired).
Conclusion
We propose the concept of automatic tool creation through LLMs and empirically devise CREATOR

---

that harnesses the capabilities of LLMs as tool cre- ators. By disentangling LLM's abstract and con- crete reasoning, CREATOR enables clearer logic and enhances overall performance. Through com- prehensive evaluations on established benchmarks and Creation Challenge, we demonstrate the supe- riority and indispensability of CREATOR compared to existing CoT, PoT, and tool-using approaches. We anticipate our study will inspire the develop- ment of more sophisticated AI systems leveraging LLM's tool creation potential.
creating tools, care must be taken mitigate biases in the demonstrations and instructions,
monitor the tool's performance across stages, and address any disparities that may arise in the
whole generation or rectification process.
Transparency involves making AI systems and their processes understandable and interpretable. When the language model creates tools un- der our framework, it's essential to have trans- parency regarding how those tools are devel- oped. Developers should document any biases limitations associated with the tools created, understand the strengths and weaknesses of the tools and how the decision is reached, and make informed decisions about their application.
Limitations
Our experiment is limited to two established bench- marks, MATH and TabMWP, along with newly introduced dataset, Creation Challenge. However, it is crucial for future research to expand the appli- cation of our framework to encompass broader array of tasks. This will enable a comprehensive assessment of the generalizability of our results, going beyond the scope of our current investiga-
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
tion.
Furthermore, demonstration of the LLM's potential in tool creation is limited in scope. For instance. the current LLM is also capable of creat- ing tools even to build a full project pipeline, but the exccution ability and correctness of its creation still lack proper evaluations and remain question- able. It is incumbent upon future research to delve deeper into the boundaries of LLM's 'ss capabilities and establish clear limits regarding its tool creation potential.
Tianle Cai, Xuezhi Wang. Tengyu Ma, Xinyun Chen, and Denny Zhou, 2023. Large language models as tool makers.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374,
Ethics Statement
Wenhu Chen, Xueguang Ma. Xinyi Wang, and William W Cohen. 2022. Program of thoughts prompting: Disentangling computation from reason- ing for numerical reasoning tasks. arXiv preprint
We consider the following research issues in this paper:
2r1.v:2211.12581
Privacy involves safeguarding sensitive informa- tion and preventing its unauthorized disclosure. With respect to our framework, privacy becomes concern when certain stages require demon- stration examples and clear instructions, which may inadvertently contain sensitive information, or intentionally designed prompt the LLM to leak privacy. Thus, it crucial to ensure that per- sonal or sensitive information not disclosed to the closed-source LLM, and the private infor- mation or knowledge about tool creation in the closed-source LLM should be well-protected. Fairness in AI aims ensure the outputs and decisions made by AI systems do not perpet- uate existing biases or discriminations. When
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. 2022. Scaling cestruction-finctune language models. arXiv preprint 22 1416.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiy- ong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. arXiv preprint arXiv:2301.0022

---

Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xi- aocheng Feng, Ming Gong, Linjun Shou, Bing Qin. Ting Liu, Daxin Jiang, et al. 2020. Codehert: A pre-trained model for programming and natural lan- guages. arXiv preprint arXiv:2002.08155
Pan Lu, Liang Qiu, Kai- Wei Chang, Ying Nian Wu, Song- Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. 2022a. Dynamic prompt learn- ing via policy gradient for semi-structurred mathemat- ical reasoning. arXiv preprint arXiv:2209. 14610.
Yi R. Fung. Tuhin Chakraborty, Hao Guo, Owen Ram- bow, Smaranda Muresan, and Heng Ji. 2023. Norm- sage; Multi-lingual multi-cultural norm discovery from conversations on-the-fly. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.
Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai Kai-Wei Chang. 2022b. A survey of deep learn- ing for mathematical reasoning. arXiv preprint arXiv:2212.10535.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christo- foros Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-YY Yu, Asli Celikyilmar, Edouard Grave, Yann LeCun, and Thomas Scialom. 2023. Augmented language mod- els: survey.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Gra- ham Neubig. 2022. Pal: Program- aided rided language models. arXiv preprint arXiv:2211 10435.
Tanmay Gupta and Aniruddha Kembhavi. 2022. Vi- sual programming: Compositional visual reasoning without training. arXiv preprint 11X1v.2211.1155,,
Retichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu. Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question- answering with human feedback. arXiv preprint arXiv:2112.09332.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. Sort, 2(4):0-6.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. 2021. Show your work: Scratch- pads for intermediate computation with language models. arXiv preprint 12x00:2112.00114.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang. Dawn Song, and Ja- cob Steinhardt. 2021. Measuring mathematical prob- lem solving with the math dataset. arXiv preprint arXiv:2 133.044..
OpenAL. 2022. Chatgpt.
Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2022. augmeeted dialogue generation. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 8460-8478.
OpenAL. 2023. Gpt-4 technical report.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080-2094.
Sha Li, Chi Han, Pengfei Yu, Carl Edwards, Manling Li, Xingyao Wang, Yi Fung. Charles Yu, Joel Tetreault, Eduard Hovy, and Heng Ji. 2023. Defining a new nlp playground. In Proceedings of the 2023 Con- ference Empirical Methods in Natural Language Processing Findings.
Yujia Qin. Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang. Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Yc. Bowen Li, Ziwei Tang. Jing Yi, Yuzhang Zhu. Zhemning Dai, Lan Yan. Xin Cong. Yaxi Lu, Weilin Zhao, Yuxiang Huang, Junxi Yan. Xu Han, Xian Sun, Dahai Li, Jason Phang, Cheng Yang. Tongshuang Wu. Heng Ji, Zhiyuan Liu, and Maosong Sun. 2023. Tool learning with foundation models.
Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, et al. 2023. Taskmatrix. ai: Com- pleting tasks by connecting foundation models with millions of apis. arXiv preprint arXiv:2303. 16434.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning, arXiv preprint arXiv:2304.08485
Shayne Longpre, Le Hou, Tu Vu. Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301. 1..30111188. 13688.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Seialom. 2023. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.0477
Pan Lu, Baolin Peng. Hao Cheng, Michel Galley, Kai- Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jian- feng Gao. 2023. Chameleon: Plug-and-play compo- sitional reasoning with large language models. arXiv preprint 23Xiv:2304.099
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugging- gpt: Solving ai tasks with chatgpt and its friends huggingface. arXiv preprint arXiv:2303. 17580.

---

Kurt Shuster, Jing Xu. Mojtaba Komeili, Da Ju Eric Michael Smith, Stephen Roller, Megan Ung. Moya Chen, Kushal Arora, Joshua Lane, al. 2022. Blenderbot 3: a deployed conversational agent that continually learns to responsibly engage. arXiv preprint 220::2208003188.
Dídac Surís, Sachit Menon, and Carl Vondrick. 2023. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08128
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Mane Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. 2023. Ilama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13971
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Intericav- ing retrieval with chain-of- reasoning for knowledge-intene intensive multi-step questions. arXiv preprint arX:::2212.1000 10509.
Xingyao Wang. Sha Li, and Heng Ji. 2022a. Code4structure: Code generation for few-shot struc- ture prediction from natural language. arxiv.
Xingyao Wang. Hao Peng, Reyhaneh Jabbarvand, and Heng Ji. 2023. Learning to generate from textual interactions. In arxiv.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormo- labashi, Yeganch Kordi, Amirreza Mirzaci, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, An- jana Arunkumar, David Stap, al. 2022b. Super- naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. In Proceedings of the 2022 Conference Empirical Methods in Natu- ral Language Processing, pages 5085-5109
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoe V Le, Denny Zhou, al. 2022. Chain-of-though prompting elicits soning in large language models. In Advances in Neurul Information Processing Systems.
Chenfei Wu, Shengming Yin, Weizhen Qi. Xi- aodong Wang. Zecheng Tang, and Nan Duan. 2023. Visual chatgpt: Talking, drawing and edit- ing with visual foundation models. arXiv preprint 00X.v.200..04671.
Pengfei Yu and Heng Ji. 2023. Self information up- date for large language models through mitigating exposure bias. In arxiv.
Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi R. Fung,
Hao Peng, and Heng Ji. 2023. Craft: Customiz- ing llms by creating and retrieving from specialized toolsets.

---

Appendices
Problem
Suppose you are scientist stadying the spread vinus, You have collacted data on the cumber of new cases reported each day ov 10-day might modeled by polynomial functon of degree 5. You want to fit polynomial function to the data ustng the mathod least squares and predict the number cases on the 11th day.
A Details about Experiment
pertod. suspect that number of
A.1 Model Details.
We employ GPT-turbo-3.5 as the base model for all our experiments. The maximum generation length for all experiments is set to 512, and a tempera- ture of 0.3 is chosen to encourage deterministic generations while maintaining certain degree of diversity, particularly during the creation of tools.
10
42
75
122
187
272
379
510
Solution Sample Tool
def polynoniall prodict day3:
confficients wpp.ooyytttt.g.oop polyyst(x, V,
Feoeeticientall [S]
predict mange(6)) (6)]
Solution Sample Decision
A.2 Dataset Details.
np.arcey ((3,2, 3,,,,,,,,,,,,, 6,,,6,,,7,,,, 9,119,1
y no.arrey (1,,12,23,42, 25,42,55,122 187, 2711179711111
For both the MATH and TabMWP datasets, we evaluate questions that have numerical value an- swers (c.g. integers or decimals). This is due to the complicated format matching problems (e.g. matching of matrices as answers) that may cause bias. The tested questions cover approximately 80% of all and maintain high diversity, making our results still representative. We are planning to update our results on MATH questions ap- plying post-processing soon 2, but some matching problems are still hard to solve.
prediction prodict polyrortsiB(x, y, 11)
Figure 6: An example query and its solution provided in the Creation Challenge dataset.
B.2 Tool Transfer Dataset Details
We create 300 data points in total and divide them into sets of three. Each set of three queries contains three corresponding answers, one standard tool that could be applied in all three scenarios to solve the problem, and three decisions about how to use the tool respectively. Similar to the construction of Creation Challenge, we manually write the seed data, which includes five sets of queries used examples to show the format of each data point, sample demonstration examples from these seeds, and leverage the Text-Davinci-005 to create more data iteratively.
The MATH dataset consists of seven math com- petition problem domains, namely algebra, count- ing and probability, geometry, intermediate algebra, number theory, pre-algebra, and pre-caleulus. Each domain is evaluated separately, and the final metric is computed as the weighted average score. The TabMWP dataset includes a wide range of table information and problems of different difficulty levels, spanning from grade one grade eight.
We present a sample set from the tool transfer dataset we curate in Figure 7. In the set, three different scenarios are provided, with each one con- sisting of a query, sample decision, and answer (not listed). Though the scenarios seem unrelated, they share the same core knowledge which can be transferred. In this case, the core knowledge is the calculation profit. We also provide an example tool that can be applied to all these three scenarios with a corresponding introduction. Note that each set we define actually represents three data points.
B Details about New Datasets
B.1 Creation Challenge Details
We begin by constructing seed dataset that in- volves novel settings and unconventional reason- ing processes. Subsequently, we utilize the Text- Davinci-003 model to expand the dataset in an iter- ative manner. By random sampling from the seed data, we encourage the variety and novelty in the problems and their reasonings.
Figure 6 illustrates sample query and its corre- sponding solution. Each data entry comprises the problem statement, a standard created tool that can be utilized (including utility, input, output, and real- ization), tool-calling decision, and a final answer.
C More about Experimental Findings
C.1 CoT Incompatible with Code
In this section, we will provide more cases further illustrate our arguments made in Sec- tion 4.4 about the conflicts between natural lan- guage thoughts and program thoughts. We con-
phttps://gitte wwt. coom//peaai//aea //rpmnaai//mmme0k/baa//mmmiiiii prmsoek/ Brapbokk/graiig/g/ee hga.enggggrddrrpprgg

---

Sample Tool (Common For Scenarios)
ple problem-solving, In the second case, the natural language thoughts indicate clearly that only simple multiplication should be done, while pure code gen- eration is trapped in a complex and chaotic logic that is prone to error.
The tool is usod calculate the optimal number of units to produce to maximize profit for manufacturing company, takes into account fixed costs, variable costs, selling price, and demand for the product The function uses the formula Profi- (Selling Price* * Quartity) (Vársable Cost Quantiay- - Cost to calculate the profit and returss the optmal quantity to produce.
def calcuiste optinal unitsiseliing price, variable, cest, Fixeed cast, derand):
These two cases further validate the conflicts between natural language thoughts and program thoughts, especially for challenging problems which may possess multiple reasoning paths that differ in suitability for code and natural language.
Caltulate. the profit for each quantity
profits = []
for quantity in range(1, denand=1];
profit (selling price * quantity) (variable cost quant ity) Fixed_eost
profits, appen((prof1t)
Find byyntiity that raxirizes profit
optivsl quantity pep_ts..ndxxeexxtforrtita
quantity
apttiral_quattil
C.2 Different Levels of Tool Creation
Soenario I: Pricing Strategy
Query
compony producas product that has fixed cost of $20,000, variable cost $10 por unit, and demond 10,000 units. compony wants maximize profit and is considerng two pricing strategies. The first strategy sell the product $30 per unit, and second strategy sell prodiact at $35 per unit. W bat is the optimal pricing strategy for the company? ?
We present in this section more details about the different levels of tool creation mentioned in Sec- tion 5.2. We present three cases in Figure 9.
The enhancement of existing tools tool cre- ation is presented in the first case. After the query, the LLM is given an existing API that could be called for a fixed purpose. This mimics the scenario in the real world where an API document is given to let one fulfill particular purpose. At this level, the LLM learns how to create tools first by com- prehensively understanding the existing tool and then transferring this knowledge to a new problem scenario. In this case, the LLM learns how temper- ature should be averaged across several days, and subsequently creates a tool that solves the problem.
Sample Decision
Fixed cost 20860
varishle
dewand 18000
Selling product pen
30
quantity caleulate optinal, units(selling peice_s, variable_cone fised_cest, decind)
profst_1 (sellingg p optinal quantity
(variable cost oprinal cuantity 1) fixed. cost Stenatayy selling product at per unds welling_priee_? 15
a Detameine the optinal pricing stratogy if profit 1 profit 2:
Scenario 2: Production Planning
Query
A manufacturing company produces a product that bas fixed cost of
$10,000, a varaable cost of per unit and selling paice $20 per unit.
The concatenation of multiple tools in tool cre- ation is presented in the second case. In this case, the LLM is given several tools to solve the problem, but the usage of each tool is rather simple to fol- low. This level of tool creation requires the LLM to plan the use of tools in a logical way and organize them with clear logic. Instead of how to call tools to serve different purpose, this level also illus- trates the L.M''s s excellent ability in implementing pipeline to solve specific queries through tool creation.
The company can sell up to 5,000 units of the product at this price. W bat the optimal number units produce maximize profit? Sample Decision
Scenario 3: Capacity Planning Query
A company produces product that has a fixed cost of $50,000, a variable
selling price of $25 per unit The company production capactty of 10,000 units. What is the optimal number of units to produce maximize profit?
$15 per
Sample Decision
Figure 7: An example data point in tool transfer datasct. We provide three scenarios sharing the core know ledge and sample tool that all three scenarios can utilize.
The hierarchy of tool creation is presented in the third case. This not only is the most common phenomenon that observe in the experiment but also represents the most advanced aspect of the LLM's reasoning potential. By creating tools with a clear hierarchy, the LLM is successful in offloading more reasoning burdens and thus solving the problem with higher accuracy. In this case, is prime represents only a "sub-tool", while the main tool solves the problem with more ease by calling it to help count the valid numbers.
trast two additional cases sourced respectively from MATH and TabMWP in Figure 8.
In the case of MATH, the ambiguity of "string manipulation" mentioned in natural language thoughts leads the model to create the tool that finds the hundredth digit in a hard-coding manner, while pure code generation in creating tools can avoid this problem.
Conversely, for TabMWP, CoT helps tool cre- ation by avoiding unnecessary complexities in sim-
Overall, the presented case studies provide valu- able insights into the tool creation abilities of

---

Problem: What's digit bundredths place decimal equivalent 9/160?
MATH
w/o CoT:
w CoT:
can use division sulve this
def decinal, digit(n, d, plece): decinal digft (decinal""
hundredthss place(n, d): calculate the real value decinal hundredths [m]] naturn 1nt (hundredths))
problem. should create a tool that could calculltee the decimal
equivalent of fraction and string manipulation extract the
raturn digit
Answer:
digit in the hundredths place.
Answer: 6
X
Problem: Each drawer in the dresser has 10 blouses. How many blouses are in drawers?
TabMWP
Table: Number of drawers Number of blouses: |10; 20; 3 30; 4
w/o Col:
CoT:
def count blouses (df, nun drawen): the target row
To solve the problem of finding out bow many blouses are 4 drawers, we need to first calculate the number of blouses in one drawer, which is 10, In the tool, we should multiply that by the target number drawers.
def calculate blouses (nun, drawer); given the blouses in drawer blouses, per, draver
drawer row df[ef["
of drawers') ] =* nun, drower) get the blouses for that now nun, blouse drawer rowl of blouses') ]. valuess calculate the total number nun,
nultiply with number of drawers total baal__bouuss blauses_pon drawen
drawer
neturn total, blouses
non,
return total
A Answer: 40
Answer: 160
X
Figure 8: We present cases illustrate the conflicts between program thoughts and natural language thoughts.
Enhancement Existing Tool
Tool Creation:
def average terporaturoccity, N):
Problem: What's the average temperature ShangHai the next seven days?
calculate average temperature total temperature
days
Input:
You have access to the following tool
range(N):
# call the tool for tines
ForecastW eather (city:str, N:int) > dict
weather_de _atrrrdaaaaaa WorecastWeathercciyy i) temperature weather ddaaaaaaaaaa 'tenperature" total_ estal_tenperature + temperature total total_tenperane tetpteeepeereeee
The tool will retum the "UV (ffox"(float) and "temperature" (float) in the given city aftor days.
Concatenation of Multiple Tools
Tool Creation:
def annulus with a r): # calculate the area of two circles CalculateArea(A) areaz Ca calculateArea(") # calculate area annulus annulus abs (areal area2) return
Problem: hat is the radii circle with same area as the annulus formed by two concentric circles with radii R and r? Input: R 10 and r 5. You are given two PIs: ralculateAreatts float) > float CalculateR (area: float) > float
Two Pls calculate the area and radii of circle respectively.
Hierarchy of Tool Creation
return False
How many positive integers less than equal have prime factor that is greater than 4? Input: N 100.
Problem:
return True
def count, tat.ttttretss
count for in range(1, N+1):
Tool Creation:
de-f 1s (ss_prineen)) (n):
for range(5, 1+1): if
prine.
and is_prineej)> count +-
1f 1:
return False
for in range(2, int(n**e.5)+1)
return count
Figure 9: We present three cases to illustrate the idea of the LLM's tool creation from different levels.
i LLMs. However, is important to acknowledge that these studies offer only a glimpse into the vast potential of LLMs in this domain. We encourage future research to explore and harness the full ex-
tent of LLMs' tool creation capabilities, further pushing the boundaries of what can be achieved.

---

D Prompting Details
Instructio
You given malh question.
All the methods we present in our main experi- ments need prompting to formalize the LLM's re- sponse and better inspire its ability.
You should think slep by slep la solve the problem.
Please pur your final numerical
after "Final Answer:"
Qurstiitioon
Point $P$ lios on the line Sx= -3 and is 10 units from the point $(5.2)5. Find product of possible $ys coorriiisntss that satisfy the given conditions
Prompting of CREATOR. We present in Fig- ures 14 16 the general prompting format and the formats of demonstrative examples, as detailed in details in 3. For the creation stage, decision stage, and rectification stage, we apply demonstration ex- amples to enhance the LLM's abstract and concrete reasoning ability, while the execution stage intrin- sically is unrelated to the LLM. We present one demonstrative example about query in MATH, but other tasks including TabMWP and Creation Challenge also follow this prompting format.
seer Thoughts and Answer
Let the y-coordinate of point $P$ be denoted by $ys. Since SPS lies on the Line 5x Since $PS is 10 units from the point $15.2)6, we can use the distance formulaa (t (---5) 224022)222222 Simplifying og equation gives us: sty- 22+61-105
know the coordinates
51-3,y18.
an equation:
sly 22=365
sy 2-1pmi
Sy=00 $y=
Therefor, the possible Sy courdinates that satisfy the given conditions an: s8ş and 5-48.
The product of these valuos is $8 ttimes (-4) =.325.
Final A nswen
40th Question
[Demonstraton Question) suur Thoughts and Answer
Prompting of Baselines. Besides CREATOR, we also apply demonstrative examples in prompting the ChatGPT's CoT, PoT, Tool Use abilities respec- tively, presented in Figures 10 to 13. Similar to the prompting of CREATOR, these prompt formats apply all tasks in the main experiments, includ- ing evaluation on MATH, TabMWP, and Creation Challenge.
[Demonstratio T'ooughts and Answer)
sow Question
(Tagge Question)
suw Thoughts bal and Areaver
(Expocting "Final Answer" in last line)
Figure 10: The instruction and one of the demonstration examples we use when prompting ChatGPT the CoT setting.
Specifically, We separate Tool Use into two parts, the first one aiming to inspire the LLM's ability to call WolframAlpha properly, and the second one aiming to prompt the LLM to retrieve the final answer. For CREATOR setting, the prompts separated according different stages. Note that the execution stage does not need prompting.
Instrucction
You are given math question. You should generate a piace of python code solve the problem. Please show thoughts python codes.
suessttionn
Point SPS lies on the line 3$ and 10 units from the point $15,2)5. Find the product of all possible SysoilleeeS-oodddiiii coordiinss that selisfy the given conditions.
duttinn Code
impart nath
initillze the coordinates
x1
x2
iinitialize distance
:caaculate y-coordinates
y_powcihles neth.sget
s1)**2)
y_posxiblez nath.sqet 22 st)**2)
product y y_possibici possible2
prine("the product all possible y-coordinates is", product
SHAP Quesstioonn
(Demoostraboon Question)
sosoouation Code
(Demonstration Code)
(arged Q Questoon
400 Thoughts Anaver
Figure 11: The instruction and one of the demonstration examples we use when prompting ChatGPT the PoT setting.

---

Instruction
You are grven math question.
have just called W WalframAlphan Ipha API help you solve the question.
Please continue to generate your final numectical answer with the return
from: W WaltramAlpha. API as reference.
There an error return from the APl, you could coutinue you thought
step by step and give your final answer.
Generate "Final A nswee:" in the lest line of with your final numerioal answer.
##AF Instraction
ass Q Question
You math question.
Puint $P$ lits the line Sx and 10 units from the point, ((5.7)..
You could call "'offraAllhhaAA Pl help you solve the question.
Find the product, of all possible rys-iooninates that satisfy the given conditions
seeing question, you should first generate thoughss and think about
how to call the Pl.
ees R Response
Generate "OO.FAMM M:" the line of your response appmpriate
Let the y-coordinate of point $P$ be denoted by $ys. Since $P$ lies on the line Sx 38, we know that the coondinates of $P$ are St-3,y18.
Inguts you'd like to inquiry.
Jeni Quesiion SP$ Find the product of all possible $yS cys-cuordimttes that salisfy the given conditions.
Since $P$ 10 units from the point ((5.2)6, we can use the distance formulla wiiee an equation:
-35 unils point. $15,2)$.
Staqtt 22)1222)2222211111111
We call wolframalpha to salve this equation
aas Resporse
WOLRAMM M: solve
the y-poordinnte point SP$ denoted by $ys. Since SP$ las on the line Sx- 3s, we know that the coordinatos of SP$ $4-3,y1S.
Wolfram/ Return
((1 Uitle: 'Inpul interperlation; 'subpad': ('8 üill:: 1 plaintext: 'ollve
sgrif(-3 5) 22 +ly 2)"2) 10'3), ( 'Resulss', 18 primaryy' 'Irue'
Since SP$ is 10 units from the point $(5.2)s, we use the distance
'subpod' [1 tide!: "plainlex I: =4)., 4, ('8 lille "plaintext! y =8)).... Response Continue
formalla to write an equation:
((-35t( 22----5 22)=00
From the WolframA Ipha retum, we sre that there one two possible values for Sy namely $y=4$ Sy=04, Thorefore, the product possible Sy coondinatos is $-4 lodot 8 3255.
can call wolframalpha jaha solve this equetion
W WOLFE PRAR:: solve 22 "2)0-2)2)==10
Final A nswen
wai Questioon
(Demomsration Question)
Q Question
Wal Resgoree
(Demonstraton Question)
(Demondration Response)
ees it esponse
(Demonstraton Response Same with the First Part)
ese w olframAAppl Retura
#RB Question
Demoonstration Retum Dictionary from Wolfram Ipha AP) eaw R Response Continue
(Target Question)
Wil Response
(Demonstration Finel Answer Extraction)
(Expecting "WOLFRAM" line)
аан Q Question
(Tanget Question)
Figure 12:
The instruction and one of the demonstration examples we use when prompting ChatGPT in the Tool
des R esponse
(LLMM Response exactl the First Part)
#es W A lpha R Return
Use setting. This figure shows the first part about Wol- framAlpha inquiry.
(WolframAIpa) Retur from the First Part) des Response Continue
(Expecting "Final Answer" in the last line)
Figure 13:
The instruction and one of the demonstration examples we use when prompting ChatGPT in the Tool Use setting. This figure shows the second part about answer retrieving.

---

You are given a moth question, and the related tools that may help you solve
the question.
Now you nood to write by thon code to call the tool solve the problem. how Please print out your answer and any information you think it's important.
Read carefully about the tool's parameters,
tool.
sut Question
from the point $(5,2)s, Find the product of all passible sys coordiinss that satisfy the given conditions. soalf Tool
Point SPS lies line 5xm-3$
WHi Instruation
You asked design bython tools solve question.
You can use math, scipy, numpy, sympy... or other packages if becessary:
You should specify the parameters and retums your tool and your tool's
utility in the documestation.
def distance for y1, y2)=
You could
than tool you think they may help solve
Calculates the distance between (81, y1) and (x2, y2).
problem.
return ((2-xx1)**2 (y2-y1)"*2)**
aWN Question
lies the Find the product of all possible $y$ oxordinaaes lhat solisfy the given conditions.
and uni's point $15,215.
possible ne.
1ist all y-coordinates of point
mayy
aWe Tools
distance Formuia(k),
y_diff dise= (center(s)=x))*>
y diff
Calculates the distance between (x),
y2).
return [] el1f y y_diff ==
return (0x2-x1)"12 (22-)1)
return [centor[3]]
else:
def possible_ns(s, dist, center):
return [center[3] y_d2f1 centor[1] y_dirf"
Returns 1ist of all possible y y-coodiinatos of a point that is distance of dist aayy from center' and Lims on the line x '"'
#ewSohution
Given point,
y_dIFf cist [centor(0)xx)=2
dif
(5,2)
return coondinates ys possible ys(r*x, dist-distance, centercenter))
function
elif y_diff return [conter[1]]
Calculate product
passible y-coordinates
[santer[1] y.d., cmnte[s)] y_diffrrn.s)
product =
Chack For the other possible y yocordiinate for y in ys: product +-
aH** Question
emoostration Question)
Print the answen
sHi Tools
print("The product of all possible y-corrdinates is", product)
emonstratoon Tools Created)
surQusttoon
[Demonstration Question)
att Question
#lWTixl
(Target Question)
(Demomáretion Tools Cresied in Crestion Slage)
BMW Fools
amsoolution
[Demonssration Solution Code to Print Uhe nswer)
[Expeding Pylh Code Block)
surstioion
The instruction and one of the demonstration examples we use when prompting ChatGPT in the CRE- ATOR setting. This figure shows the prompts applied the Creation stage.
Figure 14:
[Target Question)
seee Tool
The Tools Generated in Creation Stage) Soluututioon
(Expecting Python Code Block)
The instruction and of the demonstration examples we use when prompting ChatGPT in the CRE- ATOR setting. This figure shows the prompts applied the Decision stage.
Figure 15:

---

Your orteonal codes hare met some ecrors when executing, please rectify
your codes.
You should generate the whole new of code lineluding the tools and
deciston sall), Instead generating a snippet.
whole code block. Pay allestion the conditions in question before moditying.
W your code in py thenonn in" make
Your goel
the question correclly inslasd of just execule the code
sucoessfully.
nestessrry you al geremii whole lool, changing your way
thinking la solve the problem.
stit Question
of and to get where 4 and Pleose calculate the #iigrrila
calculate
...
function calculate suitiply
square
...
nath.agrt (a
return
calculate <(a, b) print( The value of is",
Traosookk (most moent, call lest:
host code.py", c calculate cla, b)
smodules
"rost code0. py
caloulate c
math sqriia b)
NameErro: name 'matth aut Rectification
defined
The math is not defined, so should math module before the tool. In detail, should add 'import moth at the very beconntngg
NodiFy hens to Lmport at the beginndng import math def colculate_c(9,, b):
The Function colculate Mitch 15 the aquare reot suitiply
calculate
print
#utuQustionn
Deronnttraton Question)
Origiaal
[Denonstratoo of and Decisions Previous Stages)
Errar
DDemonstration Exror Traceback from Excecution Stage)
sew Rectification
(Demonstration Rectified Toolss/ Deciston with Thoughts)
anFQuustion
(Targed Queslion)
anrrrigiaall
(Tools and Dexisions Made by L. M in Provious Stagus)
and Earor
(Traxbock from Execution Stage)
200 Rectification
The instruction and one of the demonstration examples we use when prompting ChatGPT in the CRE- ATOR setting, This figure shows the prompts applied in the Rectification stage.
Figure 16:


Reviews:
[{"review_id": "iZkQLGaBz0", "reviewer": "Reviewer_yYH8", "summary": "", "questions": "A. The introduction to the newly created datasets `Creation Challenge` and `Tool Transfer` is somewhat brief and vague. Besides, as a benchmark, how to ensure the correctness of problems and references generated by LLMs? \n\nB. The process of answer extraction is not well-presented, which may affect the reported scores for the MATH dataset. After inspecting the implementation, I think it may not be appropriate to try to match every float number in the printed information. Will it cause false recalls?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your reviews. LLM x Tool is indeed a relatively new topic in NLP and needs more future research. Previous research mostly focuses on tool-using, but we are among the first ones to propose using tool-creation in complex task-solving frameworks and as a measurement of LLM’s creativity. Here are the responses to some of your concerns.\n\n> **Q1:** This paper discusses the different impacts of CoT for helping PoT-like methods in MWP and the MATH dataset, which may be evidence for this concern. I think PoT-like methods may not perform real math reasoning, but leverage a shortcut in solving non-MWP math problems.\n\nThank you for raising this insightful concern. We would like to refer you to Sec. 4.4 where we have discussed the difference between CoT and PoT logic. Though the solving of MATH problem may take a short-cut (e.g. the application of brute-force algorithms), it doesn’t hamper the validity of our framework, because:\ni) straightforwardness and preciseness are the attributes of code as the underlying medium that tool creation is taking advantage of;\nii) large language models have their own way of conducting concrete reasoning through codes [1] in MATH, which may not necessarily be the same as they do in natural language;\nThanks for raising this concern, and we will add these discussions in the next version of our update.\n\n> **Q2:** There lack of a mechanism to verify whether a tool is transferable, or extract transferable tools from extensive solutions\n\nTransferability is a unique novelty of tool creation that differentiates it from PoT or CoT methods. We have provided preliminary evidence from Table 5 that the created tools could be successfully transferred, and the resulting improvement of 15.3% proves it makes the reasoning more robust than simply conducting tool creation to solve the problems.\n\nIn our experiment presented in Sec. 5.1, the choice of which transferred tool to use is decided by the model instead of the human. We also picture that the large language model should possess the ability to discern from mass amounts of tools, as the open-source model now possesses the ability to choose what tools to use from 16,000+ APIs [2]. Therefore, deciding which tools to use from a mass pool is more important than deciding the tool’s transferability itself, because any tool has the potential to be transferred as long as the scenario of queries are similar. Our CREATOR framework contributes to creating tools that possess transferability.\n\nThe tool creation stage in the CREATOR framework necessitates that tools be created and applied in the solution, so these created tools can be directly used as potential sources of transfer, and no additional extraction is needed. We will put more discussion on these issues in the later updated version of this paper.\n\n> **Q3:** The test set of the MATH dataset is not complete as explained in Appendix A.2\n\nThank you for this constructive comment. We haven’t used the entire test set because some answer formats are challenging for automatic evaluation (e.g. the matrix as answer), as explained in Appendix A.2. Following your provided code scripts, we conduct a more comprehensive analysis on all the data points from the MATH test set. Here are the updated results.\n\n|           Method            | Algebra (1187) | Counting & Probability (474) | Geometry (479) | Itmd. Algebra (903) | Number Theory (540) | Pre-Algebra (871) | Pre-Calculus (546) | Average (weighted) |\n| :-------------------------: | :------------: | :--------------------------: | :------------: | :-----------------: | :-----------------: | :---------------: | :----------------: | :----------------: |\n|      Vanilla (w/ CoT)       |     46.25      |            24.47             |     17.33      |        11.30        |        27.96        |       48.22       |        6.96        |       29.18        |\n|   PoT (w/o CoT, w/ Rec.)    |     55.86      |            51.05             |     32.15      |        27.91        |        56.11        |       66.48       |       26.19        |       46.72        |\n|   PoT (w/o CoT, w/o Rec.)   |     48.44      |            45.36             |     31.73      |        24.36        |        50.37        |       60.73       |       22.89        |       41.76        |\n|      Tool Use (w/ CoT)      |     43.13      |            26.16             |     19.42      |        11.85        |        34.81        |       48.45       |        9.52        |       27.62        |\n| CREATOR-Entangled (w/o CoT) |     56.95      |            53.80             |     30.90      |        32.56        |        56.85        |       65.90       |       23.26        |       47.62        |\n|      CREATOR (w/o CoT)      |     62.85      |          **59.28**           |   **37.58**    |      **35.88**      |      **63.70**      |     **70.26**     |     **27.47**      |     **52.74**      |\n\nFor each method, we choose the setting (w/ or w/o CoT) in Table 2 with the highest accuracy for comparison. From the results, we could conclude CREATOR still achieves the highest accuracy on the weighted average and even on all 7 subtasks, which demonstrates the advantage of tool creation and disentanglement of reasoning. It should also be noted that the average for every method is lowered, potentially caused by the same challenging format matching problem (e.g. difficulty in matching the matrices as final answer). We will update all the results above and provide results on all settings in the later version of this paper.\n\n> **Q4:** The novelty and necessity of the new dataset are not well demonstrated. how to ensure the correctness of problems\n\nWe agree on the importance of more clarification on these factors. We would like to refer the reviewer to lines 380 to 388, lines 487 to 492, and appendix B.1 and B.2 for a brief description of how we construct the data point and ensure novelty and diversity during task design. In the following, we elaborate with more details. We will include them in our final revision, and we will continue to improve our clarity in writing.\n\n**Necessity of Creation Challenge:** The motivation behind designing this dataset is that the existing benchmarks can mostly be solved by importing existing tools, and does not require creating novel tools to achieve high scores. Therefore, we have to design a new dataset with creativity being considered an important metric. We compare a question from the existing MATH dataset and our dataset for comparison.\n\n>**MATH Example 1:** What is $62_7+34_5$ when expressed in base 6?\n>\n>**Solution:** ```result_base6 = int(str(int('62', 7) + int('34', 5)), 6)```\n>\n>**Analysis:** The function of int() provided could solve the seemingly complex problem in one line.\n\n>**MATH Example 2:** If $\\mathbf{A}^{-1} = \\begin{pmatrix} 4 & 0 \\\\\\\\ 1 & -1 \\end{pmatrix}$ and $\\mathbf{B}^{-1} = \\begin{pmatrix} 2 & -1 \\\\\\\\ -1 & 3 \\end{pmatrix},$ then find the inverse of $\\mathbf{A} \\mathbf{B}.$\n>\n>**Solution:** ```inverse_of_AB = np.linalg.inv(A_inv @ B_inv)```\n>\n>**Analysis:** The numpy package as an existing tool could also easily solve the problem.\n\n>**Creation Challenge Example:** Suppose Alice deposited \\\\$1000 in a savings account that has an annual interest rate of 3%. Bob deposited \\\\$2000 in a savings account that has an annual interest rate of 5%. They both kept the money in their respective savings account for 10 years. How much more interest did Bob earn, compared to Alice?\n>\n>**Solution:** ```def calc_interest(initial_deposit: float, interest_rate: float, years: int) -> float: …```\n>\n>**Analysis:** The question necessitates the creation of tools for the convenience of reusing, and no existing tool or code package is directly relevant to the problem setting under real-world scenarios.\n\n**Novelty of Creation Challenge:** The Creation Challenge dataset we curated have several novelties (as a more detailed illustration to Appendix B.2, line 840-850):\n1. Creation of new tools is required. The problems in the Challenge cannot be solved using existing tools or code packages directly. The problems are mostly based on real-world scenarios and necessitate a model's careful interpretation and creativity to reach the answer.\n2. Diverse tool creation scenarios. We include various scenarios that necessitate the creation of tools in diverse domains, ranging from math, physics, finance, to problems in everyday life.\n3. Different levels are included. For each data point, we include various meta-information about what tools are required, how the input and output of tools might be, etc. These contribute to varying levels of difficulty for the tool creation challenge, which makes the evaluation on creativity more flexible and fine-grained.\n\n**Necessity of Tool Transfer Data:** The data points in current datasets show great variance among individual questions, and the queries are not clustered based on the specific target knowledge needed, thus making the tests of tool transfer inappropriate. In order to achieve greater control and precision in measuring transferability, we aim to create a small-scale dataset for investigation\n\n**Novelty of Tool Transfer Data:** The tool transfer data we curated possess the following novelties (as a more detailed illustration to Sec. 4.2, line 380-388):\n1. Shared core knowledge: The problems are clustered into a set of three, which share the same core knowledge. This provides a fair test bed for tool transfer, where the interference of the tool’s transferability itself or the ability of tool retrieval can be excluded.\n2. Diverse real-world scenarios: Though the core knowledge is shared, the descriptions of problems in the same set possess diverse scenarios, which makes the transfer of the created tools also challenging to apply. \n\n**Correctness of Both:** Though the creation of problems is automatic with the help of LLMs, we also conduct vigorous supervision from two steps:\n1. Self-consistency: For each problem generated, we feed it back for the model to generate the solution in different styles of reasoning as verifications, including reasoning through PoT and reasoning through CoT. Only the queries and answers that are consistently agreed upon by all verifiers will be retained.\n2. Human supervision: For each retained problem, we also conduct human supervision to ensure the validity of potentially useful tools and the correctness in reasoning about their usage, thus ensuring the correctness of the final answer.\n\n> **Q5:** As a minor issue, the implementation of Tool Use may not be fair enough, since it only performs one time of calculation using the Wolfram API for each exemplar.\n\nWe justify the Tool Use scenarios from the following perspectives:\n1. After investigating the results in our earlier trials (in which we performed tests on randomly chosen queries from MATH), we found the usage of Wolfram API once is enough to solve roughly 85% of the problems as long as the query to API is designed properly.\n2. Wolfram API has already implemented a complex heuristic algorithm for trying to solve a problem over multiple reasoning steps. It also provides a step-by-step solution [3] which further demonstrates its multi-step reasoning ability.\nOur comparison of Tool Use can also be viewed as an illustration that automatic tool creation has the potential to win over complex manual tool creation by experts.\n\n> **Q6:** The process of answer extraction is not well-presented\n\nIn the latest updated version, we apply the evaluation method from the script, which turns the string into sympy expression format and conducts the comparison. This is especially crucial for the evaluation of the MATH dataset, as its queries possess various latex symbols in the answer which may cause confusion during the evaluation.\n\nFor the other two datasets, the answers are all in float numbers, and we extract the answer exactly based on the fixed format of response (e.g. we only retrieve the model’s output after it outputs “Final Answer:”). Through this approach, we guarantee the answer we extract is not arbitrary numbers in the response, but what the model exactly believes in as the final answer.\n\nIn addition, we also manually inspected the answers and verified that no numbers are incorrectly extracted. Though we notice this might not be a formal way, we guarantee there exists no false recall or random output, thus ensuring the reproducibility of our results. We will later update all our results using more formal comparison methods (e.g. the one we now apply for MATH), and state this in a clearer way in Sec. 4.1.\n\n>[1] Chen W, Ma X, Wang X, et al. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks[J]. arXiv preprint arXiv:2211.12588, 2022.\n>\n>[2] Qin Y, Liang S, Ye Y, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis[J]. arXiv preprint arXiv:2307.16789, 2023.\n>\n>[3] Wolfram|Alpha Examples: Step-by-Step Solutions (wolframalpha.com)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The proposed method is well-motivated and intuitive. The idea of tool creation is novel compared to existing work.\n* The proposed method achieves fairly good results on very challenging math problems, i.e., a subset of the MATH dataset.\n* The paper is well-written and easy to follow.", "reasons_to_reject": "* Utilizing a Python interpreter to solve math word problems (e.g., GSM8k, TabMWP) does not raise this concern, but in the MATH dataset, there are problems that are very easy to be solved with a Python program, yet need carefully designed construction for human without external tools. To give a simplified example that does not exist in the MATH dataset, to calculate the sum of 1 to 100, it is fairly easy to list a program that directly sum all the numbers, however, human will adopt the `sum of equivariant series` by understanding the property of the series. In fact, this paper discusses the different impacts of CoT for helping PoT-like methods in MWP and the MATH dataset, which may be evidence for this concern. I think PoT-like methods may not perform real math reasoning, but leverage a shortcut in solving non-MWP math problems. I think it deserves a discussion since this is one of the first works to adopt PoT-like methods in the MATH dataset.\n* The claim that the created tools can be transferred across different questions is questionable. For example, in the prompt provided in the supplemental materials, some created tools containing specific conditions from the problem are obviously not transferable to other problems, such as `solve_equation` for the problem `If $3p+4q=8$ and $4p+3q=13$, what is $q$ equal to?`. Besides, there lack of a mechanism to verify whether a tool is transferable, or extract transferable tools from extensive solutions.\n* The test set of the MATH dataset is not complete as explained in Appendix A.2, which also addresses a possible limitation of the proposed methods. I think it is not that hard to evaluate the full test set, as shown in the [implementation](https://github.com/openai/prm800k/blob/main/prm800k/grading/grader.py).\n* The novelty and necessity of the new dataset are not well demonstrated.\n* As a minor issue, the implementation of `Tool Use` may not be fair enough, since it only performs one time of calculation using the Wolfram API for each exemplar, as shown in the prompt.", "questions_to_authors": "A. The introduction to the newly created datasets `Creation Challenge` and `Tool Transfer` is somewhat brief and vague. Besides, as a benchmark, how to ensure the correctness of problems and references generated by LLMs? \n\nB. The process of answer extraction is not well-presented, which may affect the reported scores for the MATH dataset. After inspecting the implementation, I think it may not be appropriate to try to match every float number in the printed information. Will it cause false recalls?", "paper_topic_and_main_contributions": "This paper proposes a new prompting framework for creating and utilizing functions to solve math reasoning problems with Python programs. It separates abstract and concrete reasoning by designing Python functions to help solve the corresponding problem, then writing code to solve it by execution. Additionally, this paper introduces a dataset for evaluating the ability of LLMs to create tools. Experiments in solving challenging math problems show that the proposed method is effective. Additional analyses show that the created tools can be transferred across different scenarios.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ef0Ga2kLIE", "reviewer": "Reviewer_PJDS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review. We also believe our work will bring benefits to the whole NLP community by bringing out a new framework for complex task-solving through tool-creation, and a new evaluation standard on the large language model’s creativity. Here is the may address your concerns:\n\n> **Q1:** The execution ability and correctness of the LLM's tool creation potential lack proper evaluations and remain questionable. Therefore, the evaluation of the LLM's capabilities may not be comprehensive enough to support the claims made in the paper.\n\nFirst of all, thank you for raising this interesting concern. As we state in the limitation section, in this work we focus on tackling tasks that current evaluation technologies allow us to. For tasks where automatic evaluations are still immature or undeveloped, such as the evaluation of a complex pipeline (e.g. the construction of a webpage, the analysis based on a database, etc.), conducting such evaluations requires developing a whole set of new techniques. This is still an open question and is beyond the scope of this work. Therefore our work focuses on tasks such as math reasoning and tabular reasoning, which follow common practices of prior works [1] [2] [3] and allow for efficient and controlled evaluation.\n\n>[1] Schick T, Dwivedi-Yu J, Dessì R, et al. Toolformer: Language models can teach themselves to use tools[J]. arXiv preprint arXiv:2302.04761, 2023.\n>\n>[2] Paranjape B, Lundberg S, Singh S, et al. ART: Automatic multi-step reasoning and tool-use for large language models[J]. arXiv preprint arXiv:2303.09014, 2023.\n>\n>[3] Qin Y, Hu S, Lin Y, et al. Tool learning with foundation models[J]. arXiv preprint arXiv:2304.08354, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of this paper include its novel approach to empowering Large Language Models to create their own tools, its evaluation on challenging benchmarks, and its demonstration of the benefits of leveraging LLMs as tool creators for knowledge transfer and improved performance. The paper also presents the Creation Challenge dataset, which emphasizes the necessity and benefits of LLMs' tool creation ability. If this paper were to be presented at the conference or accepted into Findings, the main benefits to the NLP community would be the introduction of a new framework for improving LLMs' performance on complex problem-solving tasks, the availability of a new dataset for evaluating LLMs' tool creation ability, and the potential for driving us closer to the next frontier of artificial intelligence. The paper's contributions to NLP engineering experiment and approaches for data- and compute efficiency would also be valuable to the community.", "reasons_to_reject": "The paper acknowledges that the execution ability and correctness of the LLM's tool creation potential lack proper evaluations and remain questionable. Therefore, the evaluation of the LLM's capabilities may not be comprehensive enough to support the claims made in the paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper is about the CREATOR framework, which enables Large Language Models (LLMs) to create their own tools and improve their performance on complex problem-solving tasks. The paper addresses the limitations of current approaches in utilizing a limited number of tools and the fragility of reasoning on the fly case-by-case. The main contributions of the paper include the development and evaluation of the CREATOR framework on challenging benchmarks, the Creation Challenge dataset, and the demonstration of how leveraging LLMs as tool creators facilitates knowledge transfer and drives us closer to the next frontier of artificial intelligence. The paper falls under the category of NLP engineering experiment and approaches for data- and compute efficiency.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eczt7tUJ0P", "reviewer": "Reviewer_DLxW", "summary": "", "questions": "Question A: How does the framework determine the number of tools to create for each query?\n\nQuestion B: Is tool re-use available in your framework? What I’ve understood from your figures is that a new “tool” is generated for every query, and never gets reused. However, (2) on lines 70-74 suggests that your framework should be allowing the system to “benefit from finding robust commonalities among multiple similar questions”. \n\nQuestion C: Section 5 seems to discuss an experiment about tool re-use, but the settings are very unclear. How did the model determine which tools to re-use? Given a query, how does the framework allow the model to access the previously created tools? I wasn't able to find any details about this in section 5.\n\nQuestion D: Figure 2 shows that rectification will provide feedback for the tool creation, but section 3.4 makes it sound like the rectification will only be used to , but isn’t it possible that errors are thrown because of code from the program generated in the execution step. Is that case addressed in your framework?\n\nQuestion E: In Table 3 you show an execution-accuracy gap of between 5%-9%, and I think some insights into the failure cases would be very insightful here. Are the tools not working as intended (semantic errors)? Is the model incorrectly using the tools? Is the model incorrectly translating the problem into the program in the execution step?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your reviews. Large language models and their cooperation with tools is indeed a new topic of interest, and we take a step further from the usage of tools to the creation of tools, which emphasizes the creativity of current large language models, and their creativity’s potential in solving complex problems. Here are the responses to some of your concerns.\n\n> **Q1:** I wasn’t able to find any statistics about the created dataset\n\nPlease refer to our response to Reviewer #1 about the necessity, novelty, and correctness of both the Creation Challenge as a benchmark and the Tool Transfer dataset applied in Sec. 5.1. Here is a more detailed statistic about the Creation Challenge dataset.\n\n|       Types       | Ratio |                       Examples                        |\n| :---------------: | :---: | :---------------------------------------------------: |\n|   Math-Related    |  30%  | geometric calculation, math application problem, etc. |\n| Finance and Money |  28%  |  interest calculator, investment, earning rate, etc.  |\n|      Physics      |  18%  |  problems of fluid, voltage, energy efficiency, etc.  |\n|  Data Processing  |  6%   |    search for value, statistic of variables, etc.     |\n|      Others       |  18%  |       population growth, BMI calculation, etc.        |\n\n\nOur data consist of problems of diverse types and domains, and include the problems that necessitate tool creation to solve under real-world scenarios. We will also include these statistics in a later updated version of this paper, and modify the structure to make our introduction of the dataset clearer.\n\n> **Q2:** The main results of this work don't seem to include tool re-use, leading to a gap between the paper's claims and what actually exists in this work.\n\nWe would like to refer to the experiment in Sec. 5.1 for the testing of reusability of tools created under the CREATOR framework. We have separated the dataset we apply for Table 2-4 and Tool Transfer mainly due to:\n\n1. The main experiments focus more on evaluating language models through distinct and novel questions in order to show tool creation’s strong capacity. Therefore, the tools are naturally less reusable, which is why we created a separate dataset for a more controlled and precise evaluation of transferability especially in Sec. 5.1. (Please also refer to the response to Reviewer #1’s Q4 for more details.)\n\n2. The experiments on transferability contain more questions of the same type, as the three questions in the same set share the same core knowledge, which serves as a fairer test bed. In addition, the Tool Create Normal Acc we present in Table 5 is the result derived applying the CREATOR framework (normal indicates without the transfer of tool), and all the tools in this experiment are created under the CREATOR framework.\n\nThe main experiment and the tool transfer experiment aim to show different aspects of advantages through tool creation, and they are separated for different purposes. We will clarify the separation of different settings at the beginning of Sec. 4 and 5, and further improve our writing in the introduction to avoid misunderstanding in the next version of the update.\n\n> **Q3:** Missing References: LLMs for tool use have been explored in other areas.\n\nThe article [1] represents a category of studies focusing on AI agents and their acquisition of skills. The extension of the skill library under game scenarios introduced in [1] is analogous to the construction of the tool library for retrieval, the core idea of which is more similar to [2] in tool learning. However, our study focuses more on the model’s creativity itself instead of the acquisition of skills from the environment. Thanks for the suggestion. We will add the reference and state the differences in the later updated version of the paper.\n\n> **Question A:** How does the framework determine the number of tools to create for each query?\n\nOur framework does not explicitly determine the number of tools that should be generated. We picture this should be determined by the model after being given the query. Besides, the model should also automatically decide how to apply multiple tools and in what form. In Figure 9 in the Appendix, we showcase the different levels of tool creation, all of which actually involve multiple tools that the model created. For instance, if the model decides to use nested tool calls in its creation, then at least 2 tools must be created in order to fulfill the query’s requirement.\n\n\n> **Question B:** Is tool re-use available in your framework?\n\nThe tools created under the CREATOR framework could be reused, as we have shown in the experiment in Sec. 5.1. The reason for the separation of experiments is stated in the response to Q2.\n\nThe primary purpose of tool transfer is to avoid repetition, and the created tools are particularly helpful when faced with queries with similar core knowledge, such as the solving of quadratic functions, etc. This motivates us to design the scenarios in Sec. 5.1, where we test if the creation of one tool can simply apply to other similar queries.\n\nSince the process of \"creation\" consumes a significant amount of computational resources, reusing tools can reduce computational costs. Its importance lies in the fact that when there are numerous repetitive tasks, or tasks that share the same core knowledge, repeatedly reasoning on the fly is not as efficient as designing a tool and then reusing it. This is also one of the motivations behind tool creation. We will further improve how we present the reuse of created tools and the underlying logic among different experimental settings in the next version of the paper.\n\n> **Question C:** How did the model determine which tools to re-use? Given a query, how does the framework allow the model to access the previously created tools?\n\nIn the experiment, we introduce in Sec. 5.1 (not the pipeline we illustrate in Figure 2, which is for the main experiments), the model is first given only one question and asked to create related tools that could be applied to solve this one question. After the creation, we present the model to the other two scenarios in the same set that share the same core knowledge as the first one. The model is then asked to apply its previously created tools (not specifically tailored for the current two new scenarios) to solve the current two new problems, choosing which tools to reuse and deciding on how to use them all by itself. The transferability is measured by the success of solving both new problems by applying the previously created tools. Our results in Table 5 show how using the transferred tool could even raise the performance than creating a new one.\n\n\n> **Question D:** Isn’t it possible that errors are thrown because of code from the program generated in the execution step. Is that case addressed in your framework?\n\nThanks for bringing our attention to this issue. It is a possible situation and we have taken this situation into consideration. If the tool doesn't raise an execution error, and the only problem lies on the decision call, then in the next round of rectification, the created tool given to the model will remain the same (skip the creation stage in the next iteration), and only modification on decision call will be performed. We will add a more detailed explanation about it in the first paragraph of Sec. 3.4 to avoid the confusion that might be caused.\n\n> **Question E:** In Table 3 you show an execution-accuracy gap of between 5%-9%, and I think some insights into the failure cases would be very insightful here.\n\nFirstly, the error will not be caused by the compilation error of created tools as the validity of the tools is rigorously checked before being given to the model for application.\n\nAfter investigating the errors of wrong execution outputs, we discover the errors can be divided into the following categories, and we present the new case studies as follows:\n\n1.**The erroneous interpretation of the problem:** In this category, the information provided in the problem is wrongly interpreted, resulting in the wrong initial condition during the execution stage.\n\n>**Question:**  Principal Navarro reported the state test scores from some students at her school. How many students scored at least 45 points but fewer than 59 points?\n>\n>**Table:**\n>```\nStem | Leaf \n2 | 0\n3 | 4\n4 | 0, 3, 6\n5 | 1, 1, 5\n6 | 0, 4\n>```\n>**Created Tool:**\n>……\n>\n>**Decision:**\n>```python\n> # Initialize the table\ndf = pd.DataFrame({\n    'Stem': [2, 3, 4, 4, 4, 5, 5, 5, 6, 6],\n    'Leaf': [[0], [4], [0, 3, 6], [0, 3, 6], [0, 3, 6], [1, 1, 5], [1, 1, 5], [1, 1, 5], [0, 4], [0, 4]]\n})\n……\n>```\n>**Analysis:** In this case, the table given in the question is wrongly interpreted. Though the created tool functions as intended, the wrong initial condition provided as the direct interpretation of the table leads to a wrong final answer.\n\n2.**Misalignment between tool’s purpose and decision call:** In this category, the model uses the tool in a way not intended as the tool is created, thus causing chaos in the logic of problem-solving.\n\n> **Question:** Each shelf has 10 hats. How many hats are on 5 shelves?\n>\n>**Table:**\n>```\nNumber of shelves | Number of hats\n1 | 10\n2 | 20\n3 | 30\n4 | 40\n5 | ?\n>```\n>**Created Tool:**\n>```python\ndef calculate_hats_on_shelves(df, num_shelves):\n    \"\"\"\n    This function takes in the number of shelves to calculate the number of hats.\n    \"\"\"\n……\n>```\n>**Decision:**\n>```python\n……\n># Calculate the number of hats on 5 shelves\nnum_hats = calculate_hats_on_shelves(df, 5)\n># Calculate the total number of hats\ntotal_hats = num_hats * 5\n……\n>```\n>**Analysis:** In this case, the tool created is used to calculate the number of hats, which is valid for solving the problem. However, the model instead wrongly thinks the output of the tool is the number of hats on one shelf (which is 10), and further multiplies five to the tool’s output, which increases unnecessary complexity and leads to wrong answers.\n\nFrom the case studies, we discover problems like misinterpretation and misalignment are actually causing more errors than a wrong tool created. This sheds light on a promising future research direction to bridge the gap. Thanks for bringing our attention to this error analysis, and we will include this as an experimental case study in the later updated version of the paper.\n\n>[1] Wang G, Xie Y, Jiang Y, et al. Voyager: An open-ended embodied agent with large language models[J]. arXiv preprint arXiv:2305.16291, 2023.\n>\n>[2] Paranjape B, Lundberg S, Singh S, et al. ART: Automatic multi-step reasoning and tool-use for large language models[J]. arXiv preprint arXiv:2303.09014, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a framework to solve a problem that (in my view) is currently a hot topic in the community. Tool use is a relatively new topic in NLP, and including this paper in the conference will be a benefit to those within the NLP community. Additionally, I believe that methods similar to this may be of interest to other communities as well (eg. neuro-symbolic AI, RL).", "reasons_to_reject": "The authors introduce a new dataset (Creation Challenge) which is underspecified within the paper (1-2 paragraphs in the main body and 1 additional paragraph in the appendix) and therefore difficult to evaluate the validity of some of the findings. It’s possible that I missed something, but I wasn’t able to find any statistics about the created dataset (diversity of tools, types of questions, etc.)\n\nShortcoming #2 Fragility in reasoning (lines 70-74) is a challenge that the authors claim to address in this work (lines 79-80), which leads a reader to expect that their framework will not do \"reasoning on the fly case-by-case\". There seems to be a small scale experiment where tool re-use is allowed, but the experimental setting is very underspecified (for example, how does the model determine which tool to use?) The main results of this work don't seem to include tool re-use (to the best of my understanding), leading to a gap between the paper's claims and what actually exists in this work. If the main results do not include tool re-use then I believe that this work is misrepresenting itself by it's claim of reusing tools, and should be altered before it can be published in this venue.", "questions_to_authors": "Question A: How does the framework determine the number of tools to create for each query?\n\nQuestion B: Is tool re-use available in your framework? What I’ve understood from your figures is that a new “tool” is generated for every query, and never gets reused. However, (2) on lines 70-74 suggests that your framework should be allowing the system to “benefit from finding robust commonalities among multiple similar questions”. \n\nQuestion C: Section 5 seems to discuss an experiment about tool re-use, but the settings are very unclear. How did the model determine which tools to re-use? Given a query, how does the framework allow the model to access the previously created tools? I wasn't able to find any details about this in section 5.\n\nQuestion D: Figure 2 shows that rectification will provide feedback for the tool creation, but section 3.4 makes it sound like the rectification will only be used to , but isn’t it possible that errors are thrown because of code from the program generated in the execution step. Is that case addressed in your framework?\n\nQuestion E: In Table 3 you show an execution-accuracy gap of between 5%-9%, and I think some insights into the failure cases would be very insightful here. Are the tools not working as intended (semantic errors)? Is the model incorrectly using the tools? Is the model incorrectly translating the problem into the program in the execution step?", "paper_topic_and_main_contributions": "This work presents Creator, a framework which utilizes LLMs (ChatGPT in this case) to solve mathematical and tabular word problems. The authors discretize the framework into 4 separate stages: (1) creation of a programmatic tool, (2) deciding how to use the tool for the current problem, (3) executing a program with the chosen tools, (4) rectifying any syntactic errors in the program, or if none then returning the output of the program as the answer.\n\nThe authors evaluate their framework on 2 existing benchmarks (MATH and TabMWP), and compare with existing methods (ChatGPT w/ and w/out chain of thoughts, program of thoughts, ChatGPT + WolframAlpha) finding that their method improves performance above the baselines.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "LLMs for tool-use have been explored in other areas. I don’t think it’s a requirement to include them, but if there is space to here is an example:\n\nWang et al. - Voyager: An Open-Ended Embodied Agent with Large Language Models - https://arxiv.org/abs/2305.16291"}}]

Author rebuttal:

Meta-review: This paper presents an innovative approach to enhance the capabilities of Large Language Models (LLMs) by introducing the CREATOR framework, which allows LLMs to create their own tools using documentation and code realization. The results on MATH and TabMWP benchmarks, as well as the introduction of the Creation Challenge dataset, highlight the potential of this approach to revolutionize problem-solving paradigms. The reviewers concur that the method is intriguing and valuable in the context of LLMs as agents. However, concerns have been raised about the experimental setting and its ability to support the claims made in the paper. In light of these concerns, we recommend accepting the paper to Findings.
Decision: Accept-Findings