Venue: NAACL 2024

# Toolink: Linking Toolkit Creation and Using through Chain-of-Solving on Open-Source Model

 Cheng Qian\({}^{1}\), Chenyan Xiong\({}^{2}\), Zhenghao Liu\({}^{3}\), Zhiyuan Liu\({}^{1}\)

\({}^{1}\)Tsinghua University, \({}^{2}\)Carnegie Mellon University, \({}^{3}\)Northeastern University qianc20@mails.tsinghua.edu.cn

###### Abstract

Large Language Models (LLMs) have demonstrated remarkable progress in utilizing tools, but their closed-source nature and high inference costs pose limitations on their adaptability, necessitating a valid method that leverages smaller, open-sourced models. In this paper, we introduce Toolink, a comprehensive framework that performs task-solving by first creating a toolkit and then integrating the planning and calling of tools through a chain-of-solving (CoS) approach. We first validate the efficacy of Toolink in harnessing the model's creativity and CoS ability on ChatGPT. Subsequently, we curate CoS-GPT, a chain-of-solving dataset designed for tool-using, and finetune the LLaMA-7B model. It results in LLaMA-CoVS, a powerful open-source model with advanced tool-planning and tool-calling capabilities. Evaluation of diverse tasks from BIG-bench demonstrates its CoS ability matches that of ChatGPT while its performance surpasses the chain-of-thought approach. Further studies highlight the generalization of LLaMA-CoS to unseen tasks and showcase its capability in using toolkits not explicitly tailored for the target task, affirming its robustness in real-world scenarios. All codes and data are released1.

Footnote 1: [https://github.com/qianchengo/Toolink](https://github.com/qianchengo/Toolink)

+
Footnote †: footnotemark:

## 1 Introduction

Large Language Models (LLMs) such as Codex [4], ChatGPT [5], and GPT4 [6] have made significant strides in code generation, in-context learning, and logical reasoning. However, they still struggle with precise calculations and accessing current information [8, 9, 10]. To address these issues, research has focused on equipping LLMs with tools such as calculators [1, 11, 12], search engines [13, 14], scratch pads [20], calendars [15], and image retrievers [21] to enhance their capabilities, thus benefiting various tasks including question-answering, math calculations, and long-form generation. Recent studies have also explored how LLMs can devise plans, make decisions, and perform tool invocations [22, 23]. By combining them into a pipeline, these frameworks aim to construct more advanced NLP systems for improved task performance.

However, current tool-using pipelines heavily rely on closed-source models with inaccessible parameters. It poses challenges particularly as follows: i) **Limited adaptability**: The closed-source nature of major LLMs prevents them from customization, resulting in a lack of flexibility to adapt to tasks with specific requirements. ii) **Low efficiency and high inference cost**: Many existing LLMs can only be accessed _online_, which imposes limitations on the inference rate and leads to high expense. iii) **Privacy and security concerns**: Each

Figure 1: An illustration of Toolink, which decomposes tasks via toolkit creation and resolves queries through Chain-of-Solving (CoS). Toolink can be adapted to open-source LLaMA for enhanced tool usage.



query must be submitted to these closed-source LLMs to obtain a tool-using solution, which raises concerns regarding potential privacy breaches and compromises data security.

To address these challenges, we propose Toolink, a comprehensive framework to boost the tool-using ability of open-source models with the help of closed-source models. As shown in Figure 1, Toolink first decomposes the target task by creating a toolkit for problem-solving, and then leverages the open-source model to use tools to answer queries in a chain-of-solving (CoS) approach. Specifically, CoS disentangles the model's reasoning through two stages: _CoS-Planning_, which selects useful tools from the created toolkit and plans their usages based on the specific query; and _CoS-Calling_, which focuses on deriving the answer by performing tool invocations in code format according to the plan devised. To effectively train the open-source model in these abilities, we employ ChatGPT to curate CoS-GPT, a training dataset that aims to inspire the tool-using ability of open-source models through CoS. Specifically, we finetune LLaMA-7B [26] into LLaMA-CoS, which is equipped with strong tool-using capabilities by linking toolkit creation with the chain of problem-solving.

LLaMA-CoS can solve the queries _offline_ without uploading queries to closed-source models, ensuring data security and privacy. Experiments further illustrate that Toolink outperforms the chain-of-thought (CoT) [30] on diverse tasks from BIG-bench [49] and enables LLaMA-CoS to showcase comparable CoS ability to that of ChatGPT. In addition, LLaMA-CoS can generalize to unseen tasks by planning and calling tailored tools, and solve the target task with a toolkit not specifically tailored for it. These findings further affirm our framework's robustness in solving queries under real-world scenarios.

## 2 Related Work

Tool-based enhancement for LLMs.Language models have been enhanced with external tools to improve their expertise. Previous work focused on equipping the LLMs with different tools including a calculator to improve calculation accuracy [58, 47, 46], search engine to injure factual knowledge Carlini et al. (2021); Thoppilan et al. (2022); Schick et al. (2023), Python interpreter to execute programs [45, 46], and retriever to search textual information [19, 20], etc.

More recent studies, such as Hugging GPT [23], Chameleon-LLM [26], VisualGPT [26] and TaskMatrixAI [10], focus on assembling plannings, execution, and reasoning about tools into a robust pipeline. In addition to tool-using, ART [46] builds toolkits based on retrieved tasks from the manually built library, while LATM [11] and CREATOR [31] involve the LLMs' tool-making ability to offload their reasoning burden and raise task performance. In contrast to their prevalent use of closed-source LLMs to leverage tools, Toolbox offers unique advantages of tool use for smaller, open-source models.

Adaptation of open-source models.One research direction focuses on effective tuning of open-source models, including the introduction of lightweight modules such as Adapter [14] and LoRA [14]. These modules are adapted to various model types including LLAMA [26], T5 [24], and other Transformers-based architectures [27], to save computational resources. For instance, GOAT [15] applies LoRA to improve LLaMA's arithmetic calculation ability, while LLaMA-Adapter [10] adopts Adapter and zero-init attention to improve multi-modal task performance.

Other works have investigated how instruction tuning can make open-source models better understand and follow human requirements in both text format [16, 26] and visual domains [15]. More recent works also investigate the curation of instruction following data [19, 28] and construction of open-source tool-using agents [30, 11]. Toolink builds upon the instruction-following paradigm and focuses on tool-using ability through the disentanglement of CoS-Planning and CoS-Calling, which makes learning more efficient.

## 3 Toolink Framework

As shown in Figure 2, Toolink first adopts toolkit creation to break down the target task through generating potential tools for task-solving (SS 3.1).

 

Then, the model links these created tools to address specific queries by selecting pertinent tools from the toolkit, planning their uses, and performing tool invocations (SS 3.2). This new reasoning approach, referred to as chain-of-solving (CoS), not only enables the effective and coherent application of tools but also facilitates the tool-using adaptation on the open-source model (SS 3.3).

### Toolkit Creation

Toolkit creation decomposes a general task into modular and essential tools for problem-solving, facilitating more flexible tool utilization.

Overview.Given the target task \(T\), toolkit creation breaks it down into more manageable components \(t_{1},t_{2},...,t_{n}\) through generating a toolkit \(K_{T}=\{k_{1},k_{2},...,k_{n}\}\), where \(k_{i}(i\leq n)\) represents the tool to solve the subtask \(t_{i}\). We illustrate our approach in Figure 2A, where the target task \(T=\mathtt{Navigate}\) is decomposed into \(t_{1}\) (movement in a single direction) and \(t_{2}\) (change of orientation). Each component is represented by a specific implementation encapsulated within a function tool.

Toolkit Making.We utilize ChatGPT for task decomposition. For each task \(T\), we provide ChatGPT with a task description and a few data samples \(D_{T\text{-sample}}\), expecting them to facilitate the model's understanding of task \(T\)'s objective and identify commonalities among queries. The prompting details are presented in Appendix A and Figure 6. Note that our design requires only a few data points as demonstrations fed into the closed-source ChatGPT, leaving the entire testing set for local processing to maintain privacy.

Tool Details.Each tool \(k_{i}\) within the toolkit \(K_{T}\) is comprised of a concise introduction and its corresponding code implementation. The introduction provides a brief overview of \(k_{i}\)'s utility, inputs, and outputs, facilitating effective planning and calling in subsequent steps.

### Chain-of-Solving

Chain-of-solving (CoS) involves deliberate planning and decision-making for tool invocation, which bridges the gap between toolkit creation and downstream tool use for task query resolution. CoS is disentangled into CoS-Planning and CoS-Calling. This separation allows for a more transparent and interpretable reasoning path, thereby enhancing the applicability of CoS to open-source models.

CoS-Planing.The CoS-Planning stage entails selecting useful tools from a toolkit \(K_{T}\) in response to a specific query of task \(T\). It employs natural language-based reasoning chains, referred to as a _plan_, to determine the most effective way to utilize the selected tools to solve the given query.

In Figure 2B, the model devises strategies for employing tools to update the location and orien

Figure 2: A problem solving chain of Toolink pipeline. We show an example from task Navigate. Toolink first creates a toolkit generally applicable to the task, and then approaches the specific query through CoS, which involves planning and calling of the created tools.

 

tation, with additional initial conditions that serve as a guiding hint. Planning plays a crucial role in establishing a link between toolkit creation and decision-making, thus reducing the cognitive burden associated with tool-use reasoning.

CoS-Calling.The CoS-Calling stage entails the utilization of selected tools and interpretation of the plan into program language to perform tool calls. The plan generated in the previous stage serves as the guidance for program implementation. During the tool execution, all results from the tool invocations are implicitly captured and used to extract the final answer for the given query.

Figure 2C illustrates this process, where the model simulates the entire navigation process using code as the underlying medium. In this example, the model derives the final correct answer, thereby demonstrating a successful CoS-Calling process.

### Open-Source Model Adaptation

Considering the limited adaptability, high inference cost, and privacy concerns posed by closed-source models, we aim to enhance the CoS ability in open-source models. We propose the CoS-GPT, a specialized training dataset that emphasizes the planning and calling of tools, along with code generation. These elements are crucial for boosting the model's CoS ability. The statistics related to CoS-GPT are presented in Table 1. Furthermore, for each specific target task \(T\), we employ \(D_{T^{\text{-sample}}}\) to generate a task-specific dataset. This is achieved by augmenting each sample query with suitable tools, thereby facilitating a more effective training of task \(T\) on the open-source models.

Construction of CoS-Gpt.To enhance the open-source model's skills in applying tools for problem-solving, we construct CoS-GPT from scratch to improve the model's CoS ability from planning, calling, and coding. We include the first two aspects as they are essential for CoS within Toolink, and the last aspect as it serves as the medium for tool-using.

For data points about planning and calling, we enhance the AQUA-RAT Ling et al. (2017), GSM8K Cobbe et al. (2021), and TabMWP Lu et al. (2022a) datasets, comprising graduate-level math problems, numerical reasoning tasks, and diverse table contents, with toolkits. Each query is augmented with a toolkit containing both useful and redundant tools. The model's objective for planning is to select and plan the use of useful tools, while for calling, the objective is to learn how to call the chosen tools through codes. We apply ChatGPT to simulate this process and utilize their responses for dataset construction. Please refer to Appendix E.1 for more details.

Data points for code generation encompass diverse sources, including augmentation from existing datasets, GitHub repositories, and newly generated data, detailed in Appendix E.2. Each query adheres to an instruction-following pattern and aims to enhance the open-source model's understanding of code while expanding its versatility in making informed decisions when performing CoS.

Construction of Task-Specific Data.For each target task \(T\), we construct 200 tool-augmented data points (100 each for plan and call) from the publicly available samples \(D_{T^{\text{-sample}}}\), and use them to tune the open-source model together with CoS-GPT. Similar to the construction process for tool-using data in CoS-GPT, we first augment \(T\) with a toolkit \(K_{T}\). Next, we employ ChatGPT to select useful tools for each query and generate the calling decision. The decision's output is compared against the standard answer, and minor adjustments may be made to ensure the augmented data's validity.

Open-Source Model Finetuning.Together with CoS-GPT, we apply the tool-augmented data points

\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Category** & **Set Name** & **Source** & **Number** \\ \hline \multirow{3}{*}{**Model-Using**} & Tool-Planning & Augmented & 4.4K \\  & Tool-Calling & Augmented & 4.4K \\ \hline \multirow{6}{*}{**Code Generation**} & Python-Simple & New & 2.0K \\  & Python-Specific & New & 2.0K \\  & Math & Augmented & 2.5K \\  & Algorithm & Github & 2.3K \\  & LeetCode & LeetCode & 0.8K \\  & Rectification & Sources Above & 1.6K \\ \hline

- & 20.0K \\ \hline \hline \end{tabular}
\end{table}
Table 1: The statistics about the sources and number of data points in each category of CoS-GPT. _Augmented_ represents augmented from an existing dataset.

\begin{table}
\begin{tabular}{c c} \hline \hline
**Category** & **Task Name** \\ \hline \multirow{2}{*}{**Mathematics**} & Arithmetic, Matrix Shape, Chinese Remainder \\ \hline \multicolumn{2}{l}{**Common Sense**} & Date Understanding, Navigate \\ \hline \multicolumn{2}{l}{**Logical Reasoning**} & Dyck Language, Boolean Expression \\ \hline \multicolumn{2}{l}{**Decomposition**} & Tracking Shuffled Objects \\ \hline \hline \end{tabular}
\end{table}
Table 2: The categories of 8 BIG-bench tasks tested.

 

from all target tasks to finetune the open-source model. We expect the derived tool-augmented open-source model to excel in applying useful tools for problem-solving. By planning and calling through CoS, this model links the created toolkit with specific queries, which realizes the final goal of the Toolkit framework.

## 4 Experiments

To evaluate the effectiveness of Toolink, we first conduct a validation test using ChatGPT. We select eight distinct tasks from the BIG-bench dataset Srivastava et al. (2022) to investigate whether Toolink can effectively leverage ChatGPT's creativity and tool-using capability to improve task performance.

Subsequently, we finetune the open-source LLAMA-7B model following the adaptation process outlined in SS 3.3. This results in LLaMA-CoS, a model capable of linking the created toolkit with specific tool use through CoS. We evaluate the effectiveness of LLaMA-CoS in utilizing tools on the same set of eight tasks and showcase its excellence.

### Validation Evaluation

Settings.To evaluate the effectiveness of Toolink, we conducted a validation test using ChatGPT on eight tasks of diverse categories from BIG-bench, as detailed in Table 2. For each task, we first employ ChatGPT to create a toolkit, outlined in SS 3.1. The total number of tools in the toolkit created for each task is shown in Table 3, with the tool's implementation details provided in Appendix B. Equipped with these tools, ChatGPT is presented with instructions and demonstration examples to perform CoS for problem-solving, detailed in Appendix C.

Baselines.We compare CoS against two baselines: i) **Vanilla** baseline, where ChatGPT directly produces the final answer. ii) **CoT** baseline Wei et al. (2022), where ChatGPT performs chain-of-thought reasoning before providing an answer.

Evaluations.We evaluate the ability of ChatGPT to leverage plans and calls to perform CoS. The accuracy is measured by matching the model's final output to the correct answer. In addition, we also evaluated the individual contributions of CoS-Planning and CoS-Calling separately.

During CoS-Planning, the model is asked to select useful tools and plan their uses given the created toolkit. The planning accuracy is thus measured by the following metric:

\[ACC=\max\{\frac{|K_{\texttt{correct}}|-|K_{\texttt{error}}|}{|K_{\texttt{ correct}}|+|K_{\texttt{error}}|},0\}, \tag{1}\]

where \(|K_{\texttt{correct}}|\) denotes the number of correct (useful) tools in the toolkit selected in the model's generated plan, while \(|K_{\texttt{error}}|\) denotes the number of incorrect (redundant) tools selected.

During CoS-Calling, the model is asked to implement the plan using code as the medium, after the useful tools are selected. The accuracy is thus measured by matching the output from the final execution with the correct answer. Please refer to Appendix D for more details regarding the separated test of CoS-Planning and CoS-Calling.

Results.The results are presented in Table 3. ChatGPT which uses tools through the CoS approach achieves significantly improved performance compared to other baselines, with notable margins of superiority. Further, the accuracy for CoS-Calling and CoS-Planning individually is even higher, indicating successful reasoning in each step of CoS which links toolkit creation with specific uses. These findings affirm the validity of Toolink, establishing a strong basis for its potential transferability to smaller, open-sourced models.

### Experiments on LLaMA-CoS

Our primary objective is to apply Toolink to smaller, open-sourced models. To this end, the models from the LLaMA family Touvron et al. (2023) stand out due to their capability to perform reasoning and generate codes. Considering the affordability of computational resources, we select LLaMA-7B as the representative base model to evaluate the performance of Toolink on open-source models.

Obtaining LLaMA-CoS.We follow the adaptation process outlined in SS 3.3 and finetune LLaMA-7B with CoS-GPT and eight sets of task-specific tool-augmented data. The eight tasks are the same ones as those we test in SS 4.1. Applying the training setting detailed in Appendix F, we derive a powerful variant, LLaMA-CoS, that excels in using tools through CoS.

Settings.We use LLaMA-CoS as the representative finetuned open-source model for testing. Building upon the validation test conducted on ChatGPT, we evaluate LLaMA-CoS's performance on the same set of eight tasks from BIG-bench. We keep all the metrics the same as in SS 4.1 

Baselines.As a comparison to CoS, we employ the chain-of-thought (CoT) reasoning as the baseline. We evaluate both methods under two scenarios: i) Prompting with demonstrations on Alpaca, LLaMA-7B, and ChatGPT, and ii) Finetuning specifically on the LLaMA-7B model. We referred to the LLaMA-7B tuned with CoT data as LLaMA-CoT, while our model, LLaMA-CoS, is specially tuned for tool use through CoS.

Results.We present the results in Table 4. Notably, LLaMA-CoS achieves an impressive average accuracy of 94.74%, outperforming all the CoT baselines, whether tuned or not, by a substantial margin. Compared to ChatGPT, which exhibits strong reasoning and tool-using capabilities under the CoS setting, our tuned model can still achieve comparable performance. These results highlight the effectiveness of CoS in outperforming traditional CoT methods and demonstrate the successful transfer of tool-using abilities from closed-source LLMs to smaller, open-source models.

### Results Analysis

Excellence in Both Planning and Calling.To comprehensively assess the CoS method, we similarly report the individual contribution of CoSPlanning and CoS-Calling in Table 5. Our results demonstrate that CoS-Planning and CoS-Calling separately surpass the performance achieved by CoT-based models on all tasks. This validates the model's proficiency in both stages during CoS and underscores the rationale behind designing CoSPlanning and CoS-Calling to promote effective tool use under the Toolkit framework.

Necessity of Code Training.To evaluate the impact of code generation data in CoS-GPT, we compare the LLaMA-7B tuned with or without them. The results in Figure 3 indicate that LLaMA-CoS

\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline
**Method** & **Model** & **Arith.** & **Date U.** & **Matrix S.** & **Navigate** & **Chinese R.** & **Dyck L.** & **Boolean E.** & **Tracking S.** \\ \hline
**CoT** & Alpaca & 19.89 & 39.76 & 5.62 & 47.11 & 0.0 & 0.0 & 57.46 & 0.44 \\ _(Prompting_ & LLaMA-7B & 39.44 & 33.73 & 12.58 & 39.70 & 0.0 & 2.90 & 50.44 & 14.22 \\ _w/ demo_) & ChatGPT & 79.44 & 68.67 & 80.46 & 87.96 & 0.0 & 19.42 & 75.88 & 40.78 \\ \hline
**CoT** (_Tuned_) & LLaMA-CoT & 50.44 & 49.40 & 70.82 & 71.64 & 0.0 & 35.27 & 62.72 & 28.44 \\ \hline
**CoS** & Alpaca & 17.78 & 7.83 & 3.00 & 48.60 & 7.56 & 1.00 & 94.74 & 6.78 \\ _(Prompting_ & LLaMA-7B & 55.89 & 17.47 & 10.65 & 45.90 & 23.80 & 35.83 & 99.12 & 0.67 \\ _w/ demo_) & ChatGPT & **100.00** & 69.28 & **93.67** & 85.30 & 95.14 & 52.46 & 97.37 & 99.11 \\ \hline
**CoS** (_Tuned_) & LLaMA-CoS & **100.00** & **74.10** & 91.01 & **99.56** & **95.44** & **98.21** & **100.00** & **99.56** \\ \hline \hline \end{tabular}
\end{table}
Table 4: The accuracy (%) of baselines and LLaMA-CoS on the 8 BIG-bench tasks. LLaMA-CoS employs planning and calling of tools, which beats all CoT baselines by large margins and is on par with ChatGPT’s CoS ability.

Figure 3: The improvement of performance when code generation data points are involved during training.

 

trained with code generation data achieves higher accuracy, with an average improvement of 1.4%. This validates the necessity of training on code generation together with CoS ability. By incorporating these data points, the model learns to leverage codes as the medium for tool-using more effectively, which ultimately enhances task performance.

Error Analysis of LLaMA-7B and Alpaca.We discover from the results that the raw LLaMA-7B and Alpaca's performance lags far behind. To provide insights into why they fail to do CoS-Planning and CoS-Calling even with demonstrations, we conduct a detailed error analysis in fig. 4.

Upon analyzing the errors made by both models, we identified two primary tendencies: i) the models tend to learn the pattern but often overlook the utility of the tool and the purpose of the task; ii) they frequently exhibit disarray in reasoning and a misalignment between the tool plan and the tool call. These issues contribute significantly to the subpar performance of both models.

Diverse Usage of Toolkit.Throughout the experiments, LLaMA-CoS exhibits diverse CoS-Calling patterns. It is capable of sequentially calling different tools to achieve a specific purpose, using tools in a non-linear logic (in a loop or with conditions), or performing nested tool calls, where the output from one tool directly serves as the other tool's input. These abilities illustrate the robustness and adaptability of LLaMA-CoS across diverse scenarios. We provide case studies and more details in Appendix G and Figure 5.

## 5 Further Studies

In this section, we show the generalization of LLaMA-CoS to novel tasks and how it can use

\begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Task**} & **If in** & **CoS** & **LLaMA** \\  & **CoS-GPT** & **Stage** & **-CoS** \\ \hline \multirow{3}{*}{**AQUA-RAT**} & \multirow{3}{*}{✓} & _Planning_ & **58.80** & 52.90 \\  & & _Calling_ & 56.12 & **65.94** \\ \hline \multirow{3}{*}{**MATH**} & \multirow{3}{*}{✓} & _Planning_ & **65.83** & 52.61 \\  & & _Calling_ & **50.75** & 43.25 \\ \hline \multirow{3}{*}{**TabMWP**} & \multirow{3}{*}{✓} & _Planning_ & **90.00** & 60.75 \\  & & _Calling_ & **66.00** & 32.75 \\ \hline \multirow{3}{*}{**FinQA**} & \multirow{3}{*}{ ✗} & _Planning_ & **70.51** & 50.15 \\  & & _Calling_ & 22.38 & **32.38** \\ \hline \multirow{3}{*}{** GSMSK**} & \multirow{3}{*}{✗} & _Planning_ & **61.29** & 53.83 \\  & & _Calling_ & **57.25** & 36.50 \\ \hline \hline \end{tabular}
\end{table}
Table 6: The accuracy (%) of CoS-Planning and CoS-Calling on five diverse datasets applying LLaMA-CoS or ChatGPT. Results show that LLaMA-CoS generally beats ChatGPT and is robust to unseen tasks.

Figure 4: Detailed error analysis of Alpaca and LLaMA-7B regarding CoS-Planning and CoS-Calling.

 


Figure 5: Case Studies on the diverse CoS-Calling patterns in the main experiment.

 

toolkits that are not specially tailored for solving the target task. These studies aim to illustrate the robustness of LLaMA-CoS in utilizing tools through planning and calling.

### Generalization to Novel Tasks

The eight evaluation tasks (Srivastava et al., 2022) we previously used all have data points presented during training, despite only leveraging a few publicly available samples. To showcase the generalization ability of LLaMA-CoS, we further test it on two new tasks: FinQA (Chen et al., 2022b) and GSM8K (Cobbe et al., 2021). FinQA involves questions based on financial report data, while GSM8K involves grade school math problems.

Together with AQUA-RAT, MATH, and TabMWP, whose data are presented in CoS-GPT (as detailed in SS 3.3), we randomly select a maximum of 400 test data points from each of the five tasks, ensuring they _do not_ appear in CoS-GPT. We augment each data point with a toolkit, following the method outlined in SS 3.3 regarding how CoS-GPT is constructed. In experiments, we follow the CoS-Planning and CoS-Calling tests outlined in SS 4.1.

We show in Table 6 that LLaMA-CoS achieves high accuracy in both planning and calling stages and could even beat ChatGPT in performance. This affirms the effectiveness and robustness of its CoS ability even applied to unseen tasks. As our tests encompass math, finance, table reasoning, etc, this finding also emphasizes **the robustness of LLaMA-CoS across diverse types of tasks**.

### CoS on Generic Toolkit

We further explore the ability of LLaMA-CoS to use a generic toolkit instead of the one specifically tailored for the target task. In real-world scenarios, toolkits are usually designed to address diverse tasks rather than tailored for a single task. We expect LLaMA-CoS and ChatGPT can apply toolkits borrowed from other tasks to solve the target query in a CoS approach.

To validate this, we source two additional tasks from BIG-bench: Dynamic Counting and Unit Interpretation. For each task, we provide a toolkit created explicitly for the target task or borrowed from another task. Specifically, we pair Dynamic Counting and Unit Interpretation respectively with Dyck Language and Arithmetic.

Under these settings, we evaluate the performance of both LLaMA-CoS and ChatGPT in Table 7 and show that **both LLaMA-CoS and ChatGPT can utilize a generic toolkit borrowed from another task to solve target queries through CoS**. Though the performance still lags, these findings nevertheless confirm our assumption that CoS can increase the robustness of tool-using, and make our Toolink more applicable to real-world scenarios. We present more details in Appendix H.

## 6 Conclusions

We present Toolink, a tool-training framework that effectively applies toolkits to solve problems leveraging small, open-source language models. Toolink offers increased flexibility in adapting to diverse downstream tasks while addressing concerns related to high inference costs and privacy. Our main contributions include i) empirically implementing a framework that can effectively leverage open-source models' tool-using ability, ii) devising the chain-of-solving (CoS) method that links toolkit creation and tool use through robust planning and calling, and iii) releasing the CoS-GPT dataset that aims to enhance the model's CoS capabilities.

Specifically, our LLaMA-CoS model outperforms traditional CoT and achieves a comparable performance to ChatGPT concerning tool-using. We believe our study provides a solid foundation for future researchers to explore and enhance the tool-using capabilities of open-source models.

### Limitations

Our experiments focus on equipping the open-source model with tool-using capabilities through the CoS approach, specifically in planning and calling, while excluding the ability to create toolkits. This limitation arises from the fact that the LLaMA-7B primarily relies on provided demonstrations and lacks the internal creativity required for toolkit cre

\begin{table}
\begin{tabular}{r r c c} \hline \hline
**Task** & **Toolkit Origin** & **LLaMA-CoS** & **ChatGPT** \\ \hline
**Dynamic** & _Raw_ & 97.50 & 80.83 \\
**Cut.** & _From Dyck L_ & 73.30 & 79.17 \\ \hline
**Unit** & _Raw_ & 70.83 & 80.83 \\
**Interp.** & _From Arith._ & 65.83 & 80.00 \\ \hline \hline \end{tabular}
\end{table}
Table 7: The accuracy (%) of ChatGPT and LLaMA-CoS, with toolkit newly created for the target task (_Raw_) or borrowed from other tasks. Our results show that both ChatGPT and LLaMA-CoS can utilize tools not specifically tailored for the target task through CoS.

 

ation. Moreover, the absence of enough training data further hampers the acquisition of this knowledge. We acknowledge this challenge posed by the transfer of the toolkit creation capability from closed-source models and leave it as an avenue for future research.

Additionally, it is important to note that though the tasks tested in our study include diverse toolkits and queries, they are mostly sourced from the BIG-bench dataset. To gain a more holistic understanding of the generalizability of our results, future research should expand the application of Toolink to a broader range of scenarios. This would enable a more comprehensive assessment of the framework's efficacy and applicability.

## Ethics Statement

We consider the following issues in this paper:

* **Privacy** is a crucial aspect to consider when utilizing closed-source models such as ChatGPT and GPT4. These models have the potential to learn sensitive information internally, posing a risk to personal privacy. In contrast, Toolkit addresses this concern by leveraging only a limited number of publicly available samples for toolkit creation, leaving the majority of testing queries blind to closed-source LLMs. This approach reduces the possibility of mishandling data and safeguards user privacy. By minimizing the exposure of sensitive information, Toolbox mitigates the risks associated with privacy breaches when compared to closed-source models.
* **Transparency** is a key aspect that aims to enhance the interpretability and comprehensibility of AI systems from a human perspective. In Toolink, we prioritize transparency through the creation of toolkits that provide clear information about their utility, inputs, and outputs. Additionally, we disentangle the CoS into separate stages of planning and calling, which increases the interpretability of the model's reasoning for users. We also encourage future research to further document the specific scenarios in which our framework exhibits its maximum effectiveness, as well as to outline potential risks involved. This will contribute to a more comprehensive understanding of Toolink and facilitate informed decision-making.
* **Potential Bias** is another critical aspect that we prioritize addressing in our work. We acknowledge that bias and discrimination can inadvertently manifest through problematic examples present in the training data. To mitigate this concern, we adopt a meticulous approach to curate the CoS-GPT dataset, which consists of data points from various sources. We emphasize diversity to minimize the presence of potentially biased patterns during the data construction. Through these efforts, we aim to develop the model's tool-using and CoS ability that promotes equitable and unbiased outcomes, fostering trust and inclusiveness in the application of AI systems.

## References

* A. Amini, S. Gabriel, P. Lin, R. Koncel-Kedziorski, Y. Choi, and H. Hajishirzi (2019)MathQA: towards interpretable math word problem solving with operation-based formalisms. External Links: Link Cited by: SS3.
* S. Borgeaud, C. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. D. Van Den Driessche, J. Lespiau, B. Damoc, A. Clark, et al. (2022)Improving language models by retrieving from trilions of tokens. In International conference on machine learning, pp. 2206-2240. Cited by: SS5.
* T. Cai, X. Wang, T. Ma, X. Chen, and D. Zhou (2023)Large language models as tool makers. External Links: Link Cited by: SS5.
* N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, A. Oprea, and C. Raffel (2021)Extracting training data from large language models. External Links: Link Cited by: SS5.
* M. Chen, J. Tworek, H. Jun, Q. Yuan, H. Ponde de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. (2021)Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: SS5.
* W. Chen, X. Ma, X. Wang, and W. W. Cohen (2022a)Program of thoughts prompting: disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588. Cited by: SS5.
* Z. Chen, W. Chen, C. Smiley, S. Shah, I. Borova, D. Langdon, R. Moussa, M. Beane, T. Huang, B. Routledge, and W. Yang (2022b)Finqa: a dataset of numerical reasoning over financial data. External Links: Link Cited by: SS5.
* K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman (2021) 

2021. Training verifiers to solve math word problems.
* Gao et al. (2022) Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2022. Pal: Program-aided language models. _arXiv preprint arXiv:2211.10435_.
* Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In _International Conference on Machine Learning_, pages 2790-2799. PMLR.
* Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models.
* Khandelwal et al. (2023) Urvash Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In _International Conference on Learning Representations_.
* Liang et al. (2023) Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuiai Lu, Lei Ji, Shaoguang Mao, et al. 2023. Taskmatrix. ai: Completing tasks by connecting foundation models with millions of apis. _arXiv preprint arXiv:2303.16434_.
* Ling et al. (2017) Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 158-167.
* Liu et al. (2023) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. _arXiv preprint arXiv:2304.08485_.
* Liu and Low (2023) Tiedong Liu and Bryan Kian Hsiang Low. 2023. Goat: Fine-tuned llama outperforms gpt-4 on arithmetic tasks.
* Longpre et al. (2023)Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023. The final collection: Designing data and methods for effective instruction tuning. _arXiv preprint arXiv:2301.13688_.
* Lu et al. (2023) Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. Chameleon: Plug-and-play compositional reasoning with large language models. _arXiv preprint arXiv:2304.09842_.
* Lu et al. (2022a) Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. 2022a. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. _arXiv preprint arXiv:2209.14610_.
* Lu et al. (2022b) Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Chang. 2022b. A survey of deep learning for mathematical reasoning. _arXiv preprint arXiv:2212.10535_.
* Nye et al. (2021) Maxwell Nye, Anders Johan Andreasen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. 2021. Show your work: Scratchpads for intermediate computation with language models. _arXiv preprint arXiv:2112.00114_.
* OpenAI (2022) OpenAI. 2022. Chatgpt.
* OpenAI (2023) OpenAI. 2023. Gpt-4 technical report.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in Neural Information Processing Systems_, 35:27730-27744.
* Paranjape et al. (2023) Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. 2023. Art: Automatic multistep reasoning and tool-use for large language models.
* Parisi et al. (2022) Aaron Parisi, Yao Zhao, and Noah Fiedel. 2022. Talm: Tool augmented language models.
* Patel et al. (2021) Arkil Patel, Satwik Bhattacharya, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2080-2094.
* Peng et al. (2023) Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4.
* Pfeiffer et al. (2020) Jonas Pfeiffer, Andreas Ruckle, Clifton Poth, Aishwarya Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020. Adapterhub: A framework for adapting transformers. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 46-54.
* Qian et al. (2023) Cheng Qian, Chi Han, Yi R. Fung, Yujia Qin, Zhiyuan Liu, and Heng Ji. 2023. Creator: Disentangling abstract and concrete reasonings of large language models through tool creation.
* Qin et al. (2023) Yujia Qin, Shihao Liang, Yining Ye, Kunlu Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. ToollIm: Facilitating large language models to master 16000+ real-world apis 

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer.
* Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. _arXiv preprint arXiv:2302.04761_.
* Shen et al. (2023) Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugginggpt: Solving ai tasks with chatopt and its friends in huggingface. _arXiv preprint arXiv:2303.17580_.
* Sheynin et al. (2022) Shelly Sheynin, Oron Ashual, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, and Yaniv Taigman. 2022. Knn-diffusion: Image generation via large-scale retrieval. _arXiv preprint arXiv:2204.02849_.
* Srivastava et al. (2021) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adria Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazary, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amandao Osouras, Ambrose Slone, Ameet Rahane, Ananthharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmuller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angelaa Jiang, Angelica Chen, Anh Vuong, Animeesh Gupta, Anna Gottardi, Antonio Tworelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arnun Kirubarajan, Asher Mulokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykat Erdem, Ayla Karakas, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski, Batthan Ozyurt, Behnam Hedayatnia, Behamen Neyshabur, Benjamin Inden, Benno Stein, Berk Ekemekci, Bill Yuchen Lin, Blake Howald, Cameron Diao, Cameron Doure, Catherine Stinson, Cedrick Argugeta, Cesar Ferri Ramirez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Sirco, Colin Rafifel, Courtney Ashcraft, Cristina Gabacca, Damien Sileeo, Dan Garrettle, Dan Hendrycks, Dan Kolman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Mosegui Gonzalez, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilbooa, David Dolhan, David Draxard, David Jurgens, Debajvooti Datta, Deep Ganguli, Denis Emelin, Denis Klekyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Huples, Diagna Misra, Divjar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Einnice Engeitte Munaysai, Evgenii Zheltonozhskii, Faruse Xia, Fatemeh Siar, Fernando Martinez-Plund, Francesca Happpe, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, German Kruszewski, Giambattista Parascando, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch Lopez, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannanen Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Sherih, Hinrich Schutze, Hiromu Yskura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaechoon Lee, Jaime Fernandez Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Kochon, Jan Thompson, Jared Kaplan, Jarena Radom, Jascha Sohl-Dickstein, Jason Phaga, Jason Wei, Jason Yosinski, Jeketarina Novikova, Jeline Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesusjob Alabi, Jiachep Xu, Jiaming Song, Jilian Tang, Joan Wanveru, John Burden, John Miller, John U. Balis, Jonathan Berant, Jorg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Buodeman, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanelderz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Cory Mathewson, Kristen Chirafullo, Kesenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Liannui Qiu, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colon, Luke Metz, Lutfi Kerem Senel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Battvan, Marco Marelli, Marco Maru, Maria Jose Ramirez Quintana, Marie Tolkiohn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Matyas Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Struttt, Michael Strudbe, Michael Serbrowski, Michele Bevilacqua, Michibir Yasunaga, Mihit Kale, Mike Cain, Mimee Xu, Mirae Suzgun, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozdbeh Gheini, Mukund Varma T, Nanyun Peng, Nathan Chi, Nayeeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nikita Naniga, Niklas Deckers, Nikusa Unninghoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipormollabashi, Peiyuan Liao, Percy Liang, Peter Cheng, Peter Eckersley, Phu Mon Hut, Pinyu Hwang, Piotr Milkowski, Pivush Patili, Pouya Pezeshpour, Priti Oli, Qiazohu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco Delgado, Raphael Millier 

Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronna Tabsorte, Roman Lees Brax, Rosanne Liu, Rown Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rajlan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayau Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideb Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shububh Pachchigar, Shubham Toshiwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thromeyer, Simone Melzi, Siva Reddy, Sacha Priscilla Makin, Soo-Hwan Lee, Spencer Torene, Sriharshan Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tarija Ali, Trasshinotto, Te-Lin Wu, Theo Desordes, Theodore Hochschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Timothy Telleen-Lawton, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Veram Demberg, Victoria Nyami, Vikas Raunask, Vinay Ramseesh, Vinaye Uday Prabhu, Viishakh Pandamukumar, Vivet Srikumar, William Jodus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang HoV, Yunta Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.
* Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford aplaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca).
* Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoor Kulshesththa, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaxiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokom, Will Rusch, Marc Pickett, Pfensh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Engel Morris, Tulsee Doshi, Renclitto Delos Santos, Toju Duke, Johnny Sorakner, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Arroyo, Ravi Rajakumar, Alena Butryna, Matthew Lammy, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022. Lamda: Language models for dialog applications.
* Tuvorno et al. (2023) Hugo Tuvorno, Thibaut Lavril, Gautijer Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillame Lample. 2023. Llama: Open and efficient foundation language models.
* Trivedi et al. (2022) Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. _arXiv preprint arXiv:2212.10509_.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. In _Advances in Neural Information Processing Systems_.
* Wu et al. (2023) Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. 2023. Visual chatgpt: Talking, drawing and editing with visual foundation models. _arXiv preprint arXiv:2303.04671_.
* Zeng et al. (2023)Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yukiao Dong, and Jie Tang. 2023. Agentuning: Enabling generalized agent abilities for llms.
* Zhang et al. (2023)Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. 2023. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. _arXiv preprint arXiv:2303.16199_.

 



## Appendix A Prompt Pattern for ChatGPT Toolkit

We show the pattern of the prompt we apply for the creation of toolkits leveraging GPT-3.5-turbo in Figure 6. The temperature is set to 0.3 to ensure the model clearly follows the instructions while retaining its creativity to a certain extent. The max length during generation is set to 1024. The prompt shown mainly consists of the instruction for toolkit creation, the demonstration of the format, sample public data, and the task description.

## Appendix B Toolkits for tasks from BIG-bench

We show in Figures 8 to 15 the toolkits that GPT-3.5-turbo created leveraging the prompt mentioned in the previous section. Notice that we show the final version of the toolkit, which may contain certain modifications based on human feedback. For instance, in Figure 10, we have integrated addition, subtraction, and hadamard operation into one single tool, as all of them do not change the shape of the given matrix. This will effectively reduce the redundant tools and help the model learn with ease.



of all the tools from the toolkit. We showcase the format of the prompt given to the model in Figure 7.

The accuracy of CoS-Calling is based on the matching of the model's output to the correct answer. For tasks Arithmetic and Chinese Remainder, the accuracy is evaluated in numerical format; for Matrix Shape, the accuracy is evaluated based on the matching of dimensions list; for all other tasks from BIG-bench, the accuracy is based on the matching of strings between the model's output and the correct answer.

## Appendix E Dataset Construction

In this section, we provide more details about how CoS-GPT is constructed. We introduce respectively the construction of tool-using data (including planning and calling) and code generation data. All the data points aim to enhance the open-source model's CoS ability.

### Construction of Tool-Using Data

For each query in AQUA-RAT, GSM8K, and TabMWP, we first utilize ChatGPT to create a diverse set of tools that are potentially relevant to the given query, forming the toolkit. We then provide this toolkit to ChatGPT and allow it to select the most suitable tools. Subsequently, we prompt ChatGPT to generate decision calls based on the selected tools and manually verify the correctness of the resulting outputs. If the final answer is correct,

Figure 6: The pattern of the prompt given to GPT-3.5-turbo to generate the toolkit.

 

we divide ChatGPT's responses into two distinct components, representing the planning stage and the calling stage, which are then individually added to the dataset. In this manner, the validity of our data points can thus be guaranteed.

Throughout these steps of data construction, we also incorporate demonstration examples sampled from the constructed dataset, thereby expanding the dataset in a self-iterative manner. Figure 7 shows detailed information about the format of the query. Besides the query, we also provide the corresponding CoS-Planning or Cos-Calling response and the implementation of the toolkit with useful and redundant tools.

### Construction of Code Generation Data

The code generation data in CoS-GPT are sourced from 6 different venues, including Python-Simple, Python-Specific, Math, Algorithm, LeetCode, and Rectification. The objective behind these categories is to enhance the model's proficiency in problem-solving through code utilization, calling existing packages, applying reasoning, employing algorithms, completing codes of challenging competitions, and engaging in self-rectification.

For Python-Simple and Python-Specific, the former aims to boost the models' ability to solve simple problems using codes, while the latter aims to enhance the model's ability to leverage code packages to solve more complex problems. Both these two sets are generated using ChatGPT. We prompt the model with instructions and demonstrations and gather the code snippets the model generated to solve the given problem.

The queries for the Math set are sampled from the training set of MathQA Amini et al. (2019) and augmented with a code solution based on the given query and reasoning, leveraging ChatGPT. The generated programs are verified to ensure the output answer is the same as the correct one originally, thus ensuring the validity of the augmented data points. The Algorithm set is extracted from the open-source Python algorithm repository, with over 40 categories and more than a hundred diverse algorithms. For each algorithm, we ask ChatGPT to generate a query related to it and use a code snippet to solve the problem. The codes and corresponding queries are then gathered and formed into the instruction-following format.

For the LeetCode set, we directly extract the official open-sourced problems and the code answers from the website and form our data. The Rectification set is gathered from the error codes generated in the five sets before. The error tracebacks and the bad code snippet are fed into ChatGPT, and we leverage it to rectify the codes and generate a

Figure 7: The format of the data (and prompt) for CoS-Planning and Cos-Calling.

 

correct code snippet that can solve the given query successfully. We gather the generated codes and execute them again, retaining only the ones that give a correct answer finally and form the set based on these valid data points.

## Appendix F Main Experiment Setting Details

For our main experiment, we finetuned the LLaMA-7B model on four A100-80G GPUs, with a total batch size of 32 and a learning rate of 1e-5. For the model whose performance we demonstrate in Table 4 and Table 5, its training dataset consists of 1.6K target task-specific data points (8 tasks, 100 for planning and 100 for calling each), 4K tool-using data and 3K code-generation data randomly sampled respectively from the CoS-GPT dataset. We trained the LLaMA-7B on these data for 3 epochs and obtained LLaMA-CoS.

In addition, for the ablation study about the training on codes we perform in SS 4.3, we apply 7K tool-using data and remove all the code-generation data points. We keep all the other settings the same in this study.

## Appendix G Case Studies of Diverse CoS Patterns

In Figure 5, we present three case studies highlighting the diverse nature of LLaMA-CoS in applying planning and calling for tool-using.

Firstly, LLaMA-CoS exhibits the ability to generate sequential plans involving different tools. In the first case, the model simulates the operation on matrices step by step in a linear way and finally gets the correct result.

Secondly, LLaMA-CoS demonstrates proficiency in executing complex tool calls within branch-loop structures. In the second case, the model learns to use different stack operations based on the character met in the expression, and can call the useful tool in a loop structure.

Lastly, the model showcases its competence in performing nested tool invocations. In the third case, the model is able to directly pass the converted hour retrieved from the previous tool as the input parameter for the next tool, which illustrates a successful nested tool call.

These examples serve to show the robustness, versatility, and adaptability of LLaMA-CoS across a wide range of scenarios.

## Appendix H CoS on Generic Toolkit Details

We source two new tasks, Dynamic Counting and Unit Interpretation, from the BIG-bench. We apply all the problems in Dynamic Counting for our test of toolkit generalization. However, for Unit Interpretation, we specifically select the data from LV 1 in order for the tools from task Arithmetic to be properly applied. To ensure fairness, we expand the dataset by interactively sampling new questions with similar patterns from ChatGPT and incorporating them until the dataset reaches its original full size. Note that we only aim to showcase the toolkit's generalization ability and compare the performance of LLaMA-CoS and ChatGPT within this work, so we deem expanding the dataset as fair and reasonable under our settings.

We show the toolkits specially tailored for these two new tasks in Figures 16 and 17. The LLaMA-CoS model we apply is still the model we have trained in the main experiment, detailed in Appendix F. All the other settings, including the ChatGPT applied under our framework, are kept the same as that in the main experiment.

 

* [15] M. C. Bender, J. R. Friedman, and A. S. Thomas, _Theoretical Computer Science_, volume 20 of _Lecture Notes in Computer Science_, pages 3-46. Springer, Berlin, Heidelberg, 1987.
* [16] D. E. Brown, _A Practical Guide to the Philosophy of Philosophy 

* [25] M. C. Bender, R. J. Baxter, and D. K. De Brueck, "The \({}^{3}\) myth for the \({}^{4}\) myth for the \({}^{6}\) myth for the \({}^{7}\) myth for the \({}^{8}\) myth for the \({}^{9}\) myth for the \({}^{10}\) myth for the \({}^{11}\) myth for the \({}^{12}\) myth for the \({}^{13}\) myth for the \({}^{14}\) myth for the \({}^{15}\) myth for the \({}^{16}\) myth for the \({}^{17}\) myth for the \({}^{18}\) myth for the \({}^{19 

* [10] Toolkit for task: Matrix Shape Toolkit for task: Matrix Shape
* Tool 1: multiply: it takes in two lists representing the shape of two matrix, and returns the shape of their product. 'python def multiply(shape1, shape2): if shape1[1] != shape2[0]: raise ValueErrorror("Matrix shapes are not compatible for multiplication.") result_shape = shape1[:-1] + [shape2[-1]] ... returnresult_shape
* Tool 2: kronecker: it takes in two list representing the shape of two matrix, and returns the shape of their Kronecker product. 'python defkronecker(shape1, shape2): if len(shape1) != len(shape2): raise Exception("The number of dimensions of the two matrices is not equal") result_shape = [dim1 * dim2 for dim1, dim2 in zip(shape1, shape2)] returnresult_shape
* Tool 3: sumover_axis: it takes a list representing the shape of the matrix, and the dimension of the axis that is to be sum up. It returns the shape of the resulting matrix. 'python defsum_over_axis(shape, axis): if axis >= len(shape): raise ValueErrorror("Invalid axis dimension.") result_shape = shape[:axis] + shape[axis+1:] returnresult_shape ...
* Tool 4: transpose: it takes a list representing the shape of a matrix to be transposed, and returns the shape of the resulting matrix. 'python deftranspose(shape): result_shape = list(reversed(shape)) returnresult_shape ...
- Tool 5: add_subtract_hadamard: it takes two lists representing the shape of two matrices for add, sbrstract and hadamard, and returns the shape of the resulting matrix. 'python defadd_subtract_hadamard(shape1, shape2): assert shapel == shape2 returnshape1 ...

Figure 10: The toolkit for task Matrix Shape.

 

* [16] Toolkit for task: Navigation
* [17] Toolkit for task: Navigation
* [18] Tool 1: update_orientation: It takes the original orientation(N, E, S or W) and turn direction(left, right or around), and returns the new orientation. It should be used only if not always face forward.
* [19]python def update_orientation(orientation, turn_direction): orientations = ["N", "E", "S", "M"] current_index = orientations.index(orientation) if turn_direction == "left" == new_index = (current_index - 1) % 4 elif turn_direction == "right": new_index = (current_index + 1) % 4 elif turn_direction == "around": new_index = (current_index + 2) % 4 else: raise ValueErrorror("Invalid turn direction.") returnorientations[new_index] ... - Tool 2: update_location: It takes the current location(x, y), orientation(N, E, S or W), and steps, and returns the new location after action. .. python defupdate_location(current_location, orientation, steps): x, y = current_location if orientation == "N": new_location == (x, y + steps) elif orientation == "E": new_location = (x + steps, y) elif orientation == "S": new_location == (x, y - steps) elif orientation == "M": new_location == (x - steps, y) return new_location ....

Figure 11: The toolkit for task Navigation.

 

* [10] Toolkit for task: Dyck Language
* [11] Toolkit for task: Bock Language
* Tool 1: get_closing parenthesis: This tool takes in an opening parenthesis and returns the corresponding closing parenthesis.
* 'python def get_closing_parenthesis(opening): openings = ['(', '[', '{', '<'} closings = [')', ']', '>'] if openning is openings: return closings[opennings.index(opening)] else: return None
* Tool 2: get_opening_parenthesis: This tool takes in an closing parenthesis and returns the corresponding opening parenthesis. 'python def get_opening_parenthesis(closing): openings = ['(', '[', '<'] closings = [')', ']', '>'] if closing in closings: return openings[closings.index(closing)] else: returnNone
* Tool 3: stack_insert: This tool takes in a stack and an element and returns the stack with the element inserted at the top. 'python def stack_insert(stack, element): stack.append(element) returnStack
* Tool 4: stack_pop: This tool takes in a stack and returns the stack with the top element removed. 'python def stack_pop(stack): if len(stack) > 0: stack.pop() return Stack
* Tool 5: stack_pop(Stack): if len(stack) > 0: stack.pop() returnStack
* Tool 6: set_crossing_parenthesis(opening): opennings = ['(', '[', '<'] closings = ['(', '], '>'], '>'], '>'] if openning is opennings: return closings[opennings.index(opening)] else: returnNone
* Tool 7: set_crossing_parenthesis(opening): opennings = ['(', '[', '<'] closings = ['(', ', '<'] closings = ['(', ', ')'], '>'], '>'], '>'] if closing in closings: return openings[closings.index(closing)] else: returnNone
* Tool 8: set_crossing_parenthesis(closing): opennings = ['(', '[', '<'] closings = ['(', ', '<'] closings = ['(', ', ')'], '>'], '>'], '>'] if closing in closings: return openings[closings.index(closing)] else: returnNone
* Tool 9: set_crossing_parenthesis(closing): opennings = ['(', '[', '<'] closings = ['(', ', '<'] closings = ['(', ', ')'], '>'], '>'], '>'] if closing in closings: return openings[closings.index(closing)] else: returnNone
* Tool 1: set_crossing_parenthesis(closing): opennings = ['(', '[', '<'] closings = ['(', ', '<'] closings = ['(', ', ')'], '>'], '>'], '>'] if opening in closings: return openings[closings.index(opening)] else: returnones[closings.index(closing)] else: returnNone
* Tool 1: set_crossing_parenthesis(closing): opennings = ['(', '[', '<'] closings = ['(', ', '<'] closings = ['(', ', ')'], '>'], '>'], '>'] if opening in closings: return openings[closings.index(opening)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else:returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.index(closing)] else: returnones[clos.index(closing)] else: returnones[closings.index(closing)] else: returnones[closings.
 

* [15] Toolkit for task: Tracking Shuffled Objects
* [16] ToolKit for task: Dynamic Counting
* Tool i: get_closing_parenthesis: This tool takes in an opening parenthesis and returns the corresponding closing parenthesis.
* [17] python defget_closing_parenthesis(opening): pairs_open={'(':')','[':']','{':'}','<':'>'} ifopeninginpairs_open: returnpairs_open[opening] else: returnNone
* [18] ...
* Tool 2: find_open_remaining: This tool takes in a sequence of parenthesis and returns the single remaining opening paraenthesis that is not closed.
* [20] -python deffind_open_remaining(parenthesis): parentshesis=parenthesis.split(") pairs_open={'(':')','[':']','{':'}','<':'>'} pairs_close={'(')':'(',']':'[',']':'{','>':'<'} stack=[] forpinparenhesis: ifpinpairs_open: stack.append(p) elifpinpairs_close: ifpairps_close[p]instack: stack.remove(pairs_close[p]) returnstack[p]

Figure 16: The toolkit for task Dynamic Counting.

Figure 15: The toolkit for task Tracking Shuffled Objects.

 

* [16] M. C. D'Auria, R. J. S. Marques, and A. P. Zoller. The role of the \({}^{3}\)-factor in the \({}^{2}\)-factor: a new method for calculating the \({}^{4}\)-factor: a new method for calculating the \({}^{5}\)-factor: a new method for calculating the \({}^{5}\)-factor: b new method for calculating the \({}^{5}\)-factor: c new method for calculating the \({}^{5}\)-factor: d new method for calculating the \({}^{5}\)-factor: c new method for calculating the \({}^{5}\)-factor: d new method for calculating the \({}^{