GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction
Authors: Rui Yang
Lin Song
Yanwei Li
Sijie Zhao
Yixiao Ge
Xiu Li
Ying Shan
Venue: NeurIPS 2023 poster (2023)
Keywords: multimodality
foundation models
tool usage
TL;DR: This paper presents a method how to teach language models to invoke multiply visual foundation models.

Abstract: This paper aims to efficiently enable Large Language Models (LLMs) to use multi-modal tools.
The advanced proprietary LLMs, such as ChatGPT and GPT-4, have shown great potential for tool usage through sophisticated prompt engineering.
Nevertheless, these models typically rely on prohibitive computational costs and publicly inaccessible data.
To address these challenges, we propose the GPT4Tools based on self-instruct to enable open-source LLMs, such as LLaMA and OPT, to use tools.
It generates an instruction-following dataset by prompting an advanced teacher with various multi-modal contexts.
By using the Low-Rank Adaptation (LoRA) optimization, our approach facilitates the open-source LLMs to solve a range of visual problems, including visual comprehension and image generation.
Moreover, we provide a benchmark to evaluate the ability of LLMs to use tools, which is performed in both zero-shot and fine-tuning ways.
Extensive experiments demonstrate the effectiveness of our method on various language models, which not only significantly improves the accuracy of invoking seen tools, but also enables the zero-shot capacity for unseen tools.

GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction
Rui Yang 1f Lin Song2* Yanwei Li Sijie Zhao?, Yixiao Ge Xiu Li', Ying Shan?
Tsinghua Shenzhen International Graduate School, Tsinghua University
Tencent AI Lab *Chinese University of Hong Kong
rayyang01160gmiil.co
romnsongdtencent.cmm
The essential difference between humans and animals is that humans are capable of making and using tools.
-Friedrich Engels
Abstract
This paper aims to efficiently enable Large Language Models (LLMs) to use multi- modal tools. Advanced proprietary LLMs, such ChatGPT and GPT-4, have shown great potential for tool usage through sophisticated prompt engineering. Nevertheless, these models typically rely on prohibitive computational costs and publicly inaccessible data. To address these challenges, we propose the GPT4Tools based self-instruct enable open-source LLMs, such LLaMA and OPT, use tools. We generate an instruction-followinn dataset by prompting an advanced teacher with various multi-modal contexts. By using the Low-Rank Adaptation (LoRA) optimization, our approach facilitates the open-source LLMs to solve a range visual problems, including visual comprehension and image generation. Moreover, we provide benchmark to evaluate the ability of LLMs to use tools, which performed in both zero-shot and fine-tur tenintning ways. Extensive experiments demonstrate the effectiveness of our method on various language models, which not only significantly improves the accuracy of invoking seen tools but also enables the zero-shot capacity for unseen tools. The code and demo have been available at http://hhhub..ommllllaaa-CC/////
Introduction
Recent advances in large language models (LLMs), such as GPT-3 [1], InstructGPP [2], and GPT- 3.5 [3], have demonstrated substantial potential in the area of zero-shot learning and logical reasoning. These models are typically trained large volume of text-only data, primarily sourced from the internet. However, as promising as they may seem, these advanced proprietary LLMs [3, 4] have significant limitations. One of the major hindrances the high computational cost associated with these models, which may not be affordable or accessible to many scenarios. Additionally. these models typically depend on specialized data, such as source code and conversation history, which are not easily available to the public.
Instead of solely focusing language processing, many recent researches [5. 6] attempt to bridge the gap between language models and multi-mode models. Intelligent agents like Visual ChatGPT [5] and MMREACT [6] have made efforts meet this goal by sophisticated prompt engineering. These agents utilize pre-defined template to create instructions that vision-language foundation models can execute. Although these approaches have led impressive results, the primary process of instruction
"Equal contribution. Work done during an internship Tencent. "Coresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Table 1: Comparison of related works. 'LM' is the language model. 'Mechanism' denotes how the
language model learns to invoke tools. 'Unseen' indicates the zero-shot capability on unseen tools.
Method
LM
Mechanism
Teacher
Multi-Modal
Unseen
Lazaridou et al. [10]
Gopher-280B [15]
prompt
X
X
ToolFormer [11]
GPT-J (6B) [16]
self-instruct
x
x
Visual ChatGPT [5]
GPT-3.5 (175B) [3]
prompt
X
V
MMREACT [6]
GPT-3.5 (175B) [3]
prompt
X
GPT4Tools (ours)
Vicuma-13B [12]
self-instruct
V
decomposition is heavily based on GPT-3.5 [3], which is expensive and not publicly available, thus limiting further advancements. In addition, equipping these agents with the capability to use tools requires large amount of data [3]. This brings up open question: how to efficiently enable primitive language model to use multi-modal tools?
To achieve it, different from previous studies [7-11], we explore new perceptive as illustrated Table 1. We propose a simple yet effective method, called GPT4Tools, designed to empower open-source LLMs with the ability use tools via self instruct from advanced LLMs. To be specific, we construct an instruction dataset by prompting advanced teachers, such as GPT-3.5 [3], conditional on visual contents and tool descriptions, which results in great deal of tool-related instructions. Unlike Toolformer [1 I. our method can utilize visual content description to improve data diversity significantly. Furthermore, with the generated cntrructionfollowviin dataset, employ Low-Rank Adaptation (LoRA) to fine-tune the primitive language models including Vicuna [12], LLaMa [13], and OPT [14]. Besides intrinsic language abilities, GPT4Tools-basee language models are also able to solve a variety of visual problems by using tools. The tasks involve visual comprehension and image generation, such as object grounding and segmentation, generating and instructing images, and visual question answering (VQA). The proposed GPT4Tools not only significantly improve the accuracy of LLMs invoke seen tools but also enable the zero-shot capacity for unseen tools in a zero-shot manner.
Furthermore, we propose evaluation metric to assess the effectiveness of LLMs in utilizing tools across diverse tasks. With this metric, two human-curated validation sets are constructed to evaluate the LLMs in zero-shot and fine-tuning ways, offering comprehensive measure of the ability to use tools. To demonstrate the effectiveness of GPT4Tools, we conduct extensive experiments on various language models. The results show the efficacy of teaching LLMs when and how to use tools. Specifically, the Vicuna- 13B fine-tuned on our GPT4Tools achieves 9.3% absolute gains in successful rate over GPT-3.5 [3] that acquires tool priors in context. In addition, the fine-tuned Vicuna-13BB shows a solid capacity to invoke unseen tools, wihich can be comparable to GPT-3.5's success rate.
Our GPT4Tools stands distinct from previous and concurrent studies [5-11] in three ways. First, method enables primitive open-source language models to use tools, eliminating the dependence on advanced proprietary LLMs like ChatGPT. Second, design new approach based multi-modal contexts for self-instruction and augmentation, which significantly promote multi-modal tool usage and can be deployed in different directions. Third, we propose new benchmark to assess the effectiveness of using tools, and our method shows remarkable improvements.
2 Related Work
Vision and Language Model. In the quest achieve multi-modal models capable of addressing both language and vision tasks, several studies [17-22] have explored methods to enable language models to comprehend visual input. These include techniques such transforming images into discrete textual representations [17, 18] or projecting continuous image features into the textual feature space [23-26]. Concurrently, other research has been dedicated to the development of generalist models [19, 20, 22, 21], which permit model to simultaneously input images and text, eliminating the necessity for projection process. For instance, OFA [27] devised a unified Pixel2Pixel [21] decoding architecture applicable to language and object detection tasks. Similarly, converted the outcome of visual comprehension tasks into a series of discrete tokens akin to language tasks. Gato [22] brought together range of vision and control tasks into sequential prediction issue,

---

Image Content
Tool Pocket
person is leaning low their motoreycle on the tracks. A person leaning down on motorcycle as they ride track.
<tool name?:
<usage scenario>, <arguments>
man is nearly sideways while racing motorcycle around track.
OCR
Detection
Generation
OCR
A man with helmet is riding motorcycle it's side.
motorcycle: [179.44, 105.55, 411.64, 220.7] person: (136.26, 77.72, 356.95, 124.75]
Caption
Pose
ChatGPT
Instruction Dataset
Instruction
Do need use tool?
Instruction:
Prefix Prompt.
Human: Provide imago named <image>
Description: <image content>
GPT4Tools
Al: Received.
New input: Suser imput>
Language Model
Suffix Prompt.
LoRA
Response:
Thought: Do need to tool? Yes
Action: <tool name>
Instruction
Response
Action Input: Sarguments>>
I
Observation: Soutput image>
Response
User
Figure I: Diagram of the GPT4Tools. We prompt the ChatGPT with image content and definition of tools in order to obtain a tool-related instruction dataset. Subsequently, we employ LoRA [38] to train open-source LLM on the collected instruction dataset, thus adapting the LLM to use tools.
while UViM [28] and Unified-IO [20] advocated for the learned discrete codes means to unify an array of vision tasks. By contrast, we equip the language model with diverse specialized multi-modal tools to process distinct vision tasks. This approach not only promotes the scalability of the model various tasks but also avoids issue of forgetfulness stemming from repeated fine-tuning.
Instruction Tuning. Recent studies [29, 2, 30-33] have turned out that pre-trained language models could follow natural language instructions and complete various real-world tasks if they are tuned on specific instruction- followii data. Notably, InstructGPT [2], FLAN-TS [31], OPT-IML [33] demonstrated remarkable performance on specific tasks after being fine-tuned with instruction data. In order to release the cost of human-written instructions, Self-Instruction [34] found that the instruction- following capabilities of language models can be enhanced by turning on their own generated instruction data. More importantly, this approach inspired feasible means to improve the zero- and few-shot abilities of language models, i.e., distilling off-the-shelf language models using instructional data from strong GPT-3.5 [3] or GPT-4 [4]. As result, many recent works [12, 35-37] tried to construct excellent language models for various applications based on the LLaMA [13]. For instance, Stanford-Alpan has employed 52K instructions generated by GPT-3.5 [3] to construct an exceptional
dialogue model. LLaVa [37] has adopted GPT-3.5 [3] and GPT-4 [4] to incorporate instruction-
following data related to visual content. In this paper, use GPT-3.5 [3] to construct tool-related instruction datasets, thereby allowing other language models to acquire tool usage capabilities.
Tool Usage In the Natural Language Processing (NLP) community, several arts [7-11] sought to endow language models with the ability to use tools. For instance, Komeili al. [7] proposed to generate conversation responses conditioned on the results of the search engine. LaMDA [9]
created set of tools (comprising an information retrieval system, a calculator, and a translator) to avoid plausible outputs. Lazaridou et al. [10] utilized few-shot prompting on Gopher-280B [15] to enable the search engine to ground output factual and current information. Similarly, Visual ChatGPT [5] and MMREACT [6] prompted ChatGPT to invoke visual foundation models. In addition,

---

ToolFormer [11] used self- instruction and bootstrapping to teach GPT-J (6B) [16] using five tools, which include question and answer system, calculator, search engine, a machine translation system, and a calendar. On the contrary, we focus on using the GPT-3.5 model as powerful teacher to distill off-the-shelf language models and enable them to access many visual models.
3 Method
Large language models (LLMs) [L. 14, 15] have shown remarkable in-context learning abilities. Among them, GPT-3.5 [3] and GPT-4 [4] are proven to effectively perform text-annotation tasks [39] instruct other models to follow instructions of specific domains [40, 35, 12, 37]. Inspired by these findings, we propose to enable off-the-shelf language models to acquire tool usage capabilities by taking GPT-3.5 as powerful teacher. Specifically, we utilize GPT-3.5 generate tools-related instraction-folloinng data, which is then used to tune the language model. This process offers lan- guage models the ability access multi-modal information by invoking visual models. Furthermore, we propose an evaluation metric to assess the tool-use ability of the given language model. In the following, we elaborate on the data generation, instruction tuning, and evaluation metric in turn.
3.1 Dataset Construction
Data Generation. Figure illustrates the process of generating tool-related instruction dataset. Given an image, we construct the image content Xe according the captions and bounding boxes, which is a straightforward means of establishing connections between an image and a language model [17, 18]. Conditioned upon the Xc, we provide the GPT-3.5 [3] (Mr) with tool-related prompt P, whereby attaining a large number of Idtruction-followiine data:
~ (P.
The P. comprises the system message. the definition of tools (<tool name> <usage scenario>, <arguments>), and the suffix prompt which encourages My to generate visual instructions and desired outputs. . the outcome of Mr, consists of N instruction-output pairs {y', ./?....."""" where y; has the format of "<instruction>, <tool name>, <arguments>", and N is the number of defined tools. As each input of MT grounded to the image content Xc, the generated instructions are inherently connected to the image, thus avoiding arbitrary generation. In detail, the Xe consists of ground- captions and bounding boxes with tags corresponding to the images. The rich variability of the image brings up higher diversity of instructions when compared to imagined ones. To provide contrast, we also collect instruction follow-up data without image content Xc. which is similar to ToolFormer [1]]. As depicted in Figure 2, without image context priors, GPT-3.5 tends to generate objects of visual instructions towards a small subset, which is reflected in t-SNE as sparser clusters. On the contrary, instructions generated with image-conditioned prompts are notably informative and diverse due to changes in the image content, reflected in the visualization as denser and more widely distributed results. The language model tuned by mmage-conditioned data is more robust than models without the image content (Table 3).
Data Formation.
Upon the collected raw dataset (70K items), we apply a filtering process to remove duplicate instructions, incorrectly formatted instructions, calls with incorrect tool names, and calls with incorrect tool-arguments formats. This step results in 41K K retained items. Sub- sequently, we transform the retained data into an instruction-response format utiliz- ing standardized template as shown in the bottom-left corner of Figure 1. This procedure produces new dataset, de- noted as Ys.. The instruction component of Ysss incorporates prefix prompt that
-20
-40 -20
w/o image content
W/ image content
Figure 2: t-SNE' visualization for instruction data with and without image content.
encompasses system messages and tool definitions,
'The visual instruction and tool arguments are embedded by Sentence-BEi BERT [41]. We set the temperature for generation without image content 0.9 to avoid constant outcomes.

---

Instruction:
Instruction:
Instruction:
Prefix Pronpt
Prefix Prompt
prompt
Human: Provide an image named exom plg. Deseription: huge crowd of pe oplo sky resort Understand the imag using tools.
Human: Provide an image named exam png. Description The motorey- clist are travelling down the street togother.
Human: Provide image named examm
ple. Description: many people are their laplops working together Understa nd the image using tools.
Understand the image using tools.
Al: Received.
Al: Roceived.
Al: Received.
New imput: Generate real image of ki resort with people and winter clothing
New input: Select the best definition of "sttribute"
New input: Generate new real image of conference room full people war king their laptop computers from this depth map.
from segmentations and text description.
Suffis Prompt.
Suffix Prompt.
Response:
Suffix prompí
Response:
Thought: Do I need to use tool? No
Thought: Do need to use a tool? Yes Action: Predict Depth On Image
Thought: Do I need to use tool? Yes
Al: An attribute can be defined as qual
Action: Sogment the Image
ity, trait, or characteristic inherent to a p erson, thing, or object. the context of database management, refers speci fic value or characteristic of a variable o column within a record. In programmi ng or web development, an altribute defi nes properties, settings characteristics of element markup language or odc.
Action Input: example pig
Action Iaput: exumple. pig
Observation: Siiiput__.ppg
Observation: output I.igg
Thought: Do I need to use tool? Yes
Response:
Action: Generate Image Condition On S egmentations
Thought: Do I need to use a tool? Yes Action: Generate Image Condition On Depth
Action Input: oudpout I.n. A resor with people and winter clothing
Action Input: ougput p.g. The image depicts conferenee room full of people working on their laptop computers.
Observation: owtput_ pnng
Thought: Do I need to use tool? No Al: Result saved as output 2,png
Observation: outpot 2 2.pag
Thought: Do need to use tool? No Al: Result saved output 2.pog
(a) Single-turn sample
(b) Negative sample
(c) Context sample
Figure 3: Samples of the single-tum turn instruction, negative instruction, and contextual instruction.
<image content> that denotes the image content, <user input> that is replaced with the gener- ated visual instruction, and a suffix prompt designed to prompt the language model to reply the user input using given tools. The response in Ys comprises 4 elements: (1) Thought, meaning the model's cognition when to use tools; (2) Action, signifying which tools the model will use or action the model will take; (3) Action Input, representing arguments of the selected tool; and (4) Observation, reflecting outcomes of the used tool. A sample from Y is presented in the Figure 3 (a).
Data Augmentation. Although we have successfully acquired instruction coaructioo-foolowing data related to the tool usage, this simplistic format lacks complexity and depth in both instructions and responses. To tackle this challenge, we augment the generated data from two perspectives:
Negative samples. The generated instructions primarily focus on tool usage, i.c., the decision after the Thought is always "Yes". Consequently, there is potential risk that the fine-tuned model overfits such a decision. When the user instruction is not associated with the tool usage, the fine-tuned model may erroneously execute irrelevant actions by invoking unnecessary tools. To mitigate this issue, we synthesize negative samples Yg by selecting conversation data from the existing dataset [40] and converting them into the required template, as illustrated in Figure 3 (b). By tuning with Y U Yzz. the model can accurately decide when to use tools.
Context samples. The generated instructions adopt standard and fixed single-tune format, which lacks a contextual structure. Thus, as shown in Figure 3 (c), we augment the dataset by cutting off the chain of action. We also randomly select multiple instructions from Y UYs and reformat them into multi-turn conversation data. In this way, we synthesize the contextual instruction-follloiin data Yg. enabling the tuned model call tools within the given context.
So far, we have constructed the tool-related instructional dataset, including positive samples, negative samples, and context samples: Ys = Yś U UYg U UYg.
3.2 Instruction Tuning
Based on the dataset Ys, we tune the off-the-self language model using its original auto-regressive training objective. To make the tuning feasible, we leverage LoRA [38] optimization, which freezes the language model and only optimizes rank decomposition components of the Transformer layers.
5

---

For sequence with L tokens, we compute the probability of the target response X, by:
p(X, Xins))
II pa( Xx, Xinst F14-1); t=1
where Xinst denotes the instruction tokens; and 0 is the trainable parameters. In practice, prefix prompt and suffix prompt are also involved but we here skip them for better readability.
3.3 Evaluation Approach
Numerous benchmarks [42, 43, 8, 44] typically utilize human-annotated datasets to evaluate the performance model. For the purpose of measuring the tool-usage capacity of the language model, we construct an evaluation dataset following the same procedures detailed in $ 3.1 and manually verify the accuracy of each item. This evaluation dataset is partitioned into components: the first part (validation set) has the same ingredients as the training set, encompassing 23 tools; the second part (test set) comprises 8 novel tools absent from the training set. We will use the validation to validate whether the model can adhere to user commands correctly after tuning with the training set. The test set will verify whether the model can generalize to new tools after tuning. Based on the evaluation dataset with N instructions, we design successful rate to measure the model's s performance from three aspects:
Successful Rate of Thought (SR,) measures whether the predicted decision matches the ground- truth decision. is calculated as SR, = & ΣN I(Tz), where т; signifies a singular process. the thought is correct, T(T;) is equal to 1, and 0 otherwise.
Successful Rate of Action (SRact) measures whether the predicted tool name is in agreement with the name of the ground truth tool. It is calculated SRaee + >m' I(a;), where o, denotes the matching process for the tool names. cases where the predicted tool name matches the pre-defined name, I((a;) equals 1, and 0 otherwise.
Successful Rate of Arguments (SRargs) evaluates whether the predicted arguments match the ground-trui arguments. can be calculated using the following equation:
SRargs
Tss where TJ
Σ Th.
Here, denotes sequence of arguments encompassing both the image path and the input text. For instance, ControlNet [45] needs the image path saved conditions (e.g., the pose map, depth map, or segment map) and the input text described user commands. K represents the quantity of arguments 77s- When the argument belongs the image path, Tt. equals if the predicted and ground-truie truth image paths share the same suffix, and 0 otherwise. When the argument is the input text, 17..j is equal to the BLEU score between the predicted and the ground truth text.
Successful Rate (SR) measures whether chain of actions executed successfully, which requires the correctness of thought, tool name, and tool arguments at the same time:
N
SR =
Σ (TT)) I(ass) 1l( > 0.5)
(4)
Additionally, when a procedure comprises two consecutive actions, the SR equals 100% only if both actions are executed correctly.
4 Experiments
4.1
Implementation Details
We employ the ChatGPT following data. Since this study focused on teaching the off-the-sel language models to use tools instead of prompt engineering, we adopted methodology outlined the Visual ChatGPT [5] to construct tool-related prompts. Our tool pocket consists of 31 tools, including the 23 tools defined
[3] as the teacher model to generate the raw instruction-

---

Table Comparison of different language models. The zero-shot prediction is adopted for unseen tools and the models without GoT4Tools.
Validation (seen tools)
Test (unseen tools)
Model
GPT4Tools
SR,
SRact
SRarge
SR
SR,
SRact
SRargs
SR
GPT-3.5 [3]
x
93.5
96.1
78.0
84.8
99.5
99.5
91.5
91.5
(text-davinci-003)
1.1
1.2
0.0
0.0
0.0
0.0
0.0
0.0
OPT-13B [14]
99.4
98.3
89.2
93.2
97.8
89.6
84.0
78.6
20.4
15.7
16.5
3.2
16.1
17.6
21.7
2.0
LLaMa- 13BB--13BBBB [13]
V
77.3
74.9
71.4
66.4
74.2
72.2
70.9
69.9
69.2
25.1
25.2
12.4
84.4
43.7
46.7
26.2
Vicuna- 13B [12]
98.7
97.6
91.4
94.1
98.2
97.0
92.2
90.6
in Visual ChatGPT [5] and 8 extra tools (please refer to Appendix for detailed tool names). During generation, all image information utilized in GPT4Tools is sourced from the training set of COCO [43]. After generation, the training set comprises 71K instruction instruction-response pairs, wherein all instructional data is related to the 23 tools. We divided the human annotated evaluation dataset into two parts: the validation set and the test set. The validation set contains the same tools as the training set, with approximately 50 items associated with each tool. The test set includes tools that are not present in the training set (further details provided in Appendix).
Based on the collected data, we tuned language models (LLaMA [13], Vicuna [12], and OPT [14]) with LoRA [38] technology. Specifically, we equipped the projection layers of query, key, value, and output with LoRA layers. The LoRA attention dimension and scaling alpha were set to 16. While the language model was kept frozen, the LoRA layers were optimized using the AdamW [46]. All models were fine-tuned over 3 epochs, with a batch size 512. The learning rate was set to 3 x 10 10-4, and the maximum length of new tokens was restricted to 2048. Unless otherwise specified, we used Vicuna-13B 13B for the ablation experiments.
4.2 Main Result
Can instruction datasets teach language model using tools? The outcomes of GPT-3.5 [3], OPT- 13B [14], LLaMA-13B [13], and Vicuna-13B [12] presented in Table 2. GPT-3.5 is considered analogous to Visual ChatGPT [5]. Upon prompting GPT-3.5 with tool-associated instructions, it can attain SR of 84.8% the validation set, thereby underscoring its zero-shot ability to follow a standardized format and utilize tools effectively. Notably, OPT-13B fails to invoke tools with the prompts alone. contrast, LLaMA-13B and Vicuna-13B exhibit certain level of comprehension of tool usage, while they still face challenges in executing a chain of actions. Specifically. LLaMA-131 achieves 3.2% SR. which is absolutely lower than SR, SR and SRarge In the case of Vicuna- 13B, its SR is 56.8% less than SR, implying that under a zero-shot setup, Vicuna- 13B displays commendable discernment in determining when to use tools within given context. After fine-tuned with GPT4Tools, there are substantial alterations in the tool invocation competencies of each model. Specifically, the SR of OPT-13B witnessed sharp increase from to 93.2%. Similarly, the SR for LLaMA-13B escalates from 3.2% to 66.4%, and Vicuna- 13B's SR rises from 12.4% to 94.1%. These outcomes unequivocally validate that the GPT4Tools developed this study are indeed effective in instructing language models to use tools.
Can the model be generalized unseen tools after fine-tuning? The right side of Table shows the results when prompting novel tool and corresponding utilization. On the test set, GPT-3.5 attaines 91.5% SR in zero-shot manner. The outcomes for other models, which are fine-tuned on the GPT4Tools and directly invoke tools utilizing prompts, are analogous to those on the validation set. In contrast, models that are fine-tuned on the GPT4Tools dataset exhibit a degree of competence in invoking tools that have not been previously encountered (did not appear in the training set). More specifically, the fine-tuned LLaMA-13BB model achieves superior SR on new tools by a margin of 67.9 67.9% when compared to the original model. The fine-tuned Vicuna-1 13B model demonstrates 90.6% SR on new tools, which comparable to GPT-3.5. This observation indicates that the language model can invoke unseen tools after fine-tuned with GPT4Tools.
7

---

Table Ablation study for data augmentations on the validation set. II"", 'CS', 'NS' denotes image content, context samples, and negative samples, respectively.
Table Ablation study for different model scales on the validation set. 7B and 13B refer to Vicuna- [12] and Vicuna-131 [12] models, respec- tively.
IC
CS
NS
SR,
SRact
SRargs
SR
Model
GPT4Tools
SR,
SR
SRarge
SR
70.0
55.7
51.7
36.9
27.7
15.8
11.5
7B
89.6
89.9
84.5
81.6
V
96,2
94.5
89.8
92.9
97.4
95.7
88.5
91.6
X
69.2
25.1
25.2
12.4
13B
V
98.7
97.6
91.4
94.1
98.7
97.6
91.4
4.3
Ablation Study
Data Augmentation. As depicted in Table 3, we execute series of ablation studies on various tricks implemented during the creation of the dataset. When instructions are not conditioned upon the image content, the SR of the fine-tuned model on the validation set is mere 36.9%. In contrast, when instructions are generated with
90.0
Success
conditioning the image content, the SR the validation set enhanced substantially to 81.6%. This uptick be primarily attributed to the elevated diversity and intricacy of the gen- augmentation erated instructions. Moreover, of the SR to 91.6% is observed upon introduc- ing context samples into the instructions. This finding underscores the fact that partitioning the chain of actions and allocating them to the in- struction and response can strengthen the model's comprehension of the tool. It is noteworthy to mention that with the incorporation of negative samples into the generated instructions, the SR increases to 94.1%. This outcome can be traced back to the propensity of the model, when trained exclusively with positive samples, to bias toward tool invocation. This tendency consequently dimin- ishes the capacity to discern the appropriate cases for tool usage. Adding negative samples equips the model with the ability to determine when to use tools.
85.0
80.0
5R
250
750
1250
1750
Iteration
Figure 4: Performance variation curve with the fine-tuning iteration.
Model Scales. We attempt experiments with models at different scales. The results in Table demonstrate that after fine-tuned on the generated dataset, Vicuna-7B [12] is also capable of invoking tools in fixed format. Specifically, under a zero-shot setting, Vicuna-7B achieves only a 4.5% SR. By contrast, after fine-tuning, it can achieve an SR of 92.9%.
Tuning Iteration. We increase the number of iterations for fine-tuning and present the results in Figure 4. Notably, during the range of iterations from 400 to 800, the model's performance demonstrates substantial fluctuations in tool invocation. However, subsequent to this range, there is steady improvement in SR/, SRacl, SRargs and SR. This indicates that the model progressively adapts to the dataset, enhancing its capability to invoke tools.
4.4 Case Study
Figure 5 presents a comparative analysis our model with Visual ChatGPT [5] and LLaVa [37]. When an image is submitted by the user alongside the instruction "Generate picture of real people based on the edge", Visual ChatGPT delivers image that exhibits weak correlation with the given instruction. Owing to its inability to generate images, LLaVa only returns caption. In contrast, our model produces an accurate result, thereby evidencing that tool-related instruction tuning method proposed in this paper can effectively instruct language models in the correct usage of tools. In Figure we further demonstrate that the Vieuna-130 fine-tnned on GPT4Tools is capable finishing some visual commands by invoking visual tools. This finding indicates that imparting knowledge to

---

Generate picture of people based the edge.
Generate picture real people based the edge.
Generate a picture of real people based on the edge.
The image black and white line drowing peogle standing together small ng. forming up,
group
There people gmo and positions. are cmawing sense unity and camarader leee. T drawing captures the essence group, Show- caseing their happiness and tog- etherness.
Result saved image/1l 8hFidd.ppgg
Imager Loe2 (eea22099.ppp.. pog
ing close
(a) GPT4Tools (ours)
(b) VisuaIChatGPT
(c) LLaVa
Figure 5: Comparison with other models. Our GPT4Tools responds correctly, while Visual Chat- GPT [5] replies with the wrong image, and LLa Va [37] not generate the image.
Generate a man watching besed pose the woman.
Give the sketrh of this image.
Make the image look like cartoon.
Result saved 1mage/3h97111ff.ppm
Result saved imageeboo6fffR..mm
Result saved as imagetcl coadd5 png
Segment the tie in the Image.
Segment the woman in the image.
How many people in this Image?
Result saved as image! aaeff6e2. pag
Result saved as image/ bdfadaef.pmo png
Figure 6: Cases of invoking tools from Vicuna-1333 [12] fine-tuned GPT4Tools.

---

Table Results of GPT4Tools using Top-K related tools.
The total number of tools equals 23.
Model
Retrieval Top-K
;
SRact
SRargs
SR
Vicuma- Ticuma-13B [12]
69.2
25.1
25.2
12.4
w/ GPT4Tools
1
87.0
55.8
57.5
54.0
w/ GPT4Tools
2
93.1
70
69.5
67.8
w/ GPT4Tools
3
95.8
74.4
72.9
73.1
w/ GPT4Tools
23
98.7
97.6
91.4
94.1
language models regarding the tool invocation could potentially be way toward the development of a generalist model. More case studies are presented in the Appendix.
Discussion
Although the proposed GPT4Tools can teach plug-and-play language models use tools effectively, it still has some limitations. As shown in Figure 2 the success rate of all models is not 100%. Thus, further improvements are still necessary for practical applications. Additionally, GPT4Tools teaches the model to explicitly invoke tools using verbose and fixed prompt. This approach decreases the computational efficiency since ineetiin-basee architectures compute the relationships between all tokens. Besides, with the increased number of tools, the prompt length might surpass the limited context length of LLMs. In this case, we can alternatively utilize tool retrieval technique to filter small set of tools and then apply GPT4Tools--iis based LLMs for tool selection and invocation. We employ BM25, based on the user input, to retrieve the top-K tools from the defined 23 tools. As shown in Table 5, SR is only 54% while retrieving the top- tool. When the number of retrieved tools increases to 3, SR is boosted to 73.1%. Although the retrieval strategy can mitigate the reliance on long context models for large number of tools some extent, its SR model. This result can be attributed to the inabilities the retriever. Therefore, in the future, is imperative to build specialized retriever for the tool name retrieval. Moreover, it should be explored how to enable the model to implicitly invoke various tools instead of using the complex prompt. Nevertheless, our GPT4Tools method provides viable approach for equipping language models with the ability to use multi-modal tools.
match the original
Conclusion
In this paper, we introduce GPT4Tools, a novel method that enables open-source language models to utilize multi-modal tools efficiently. Specifically, We construct tool-related instructional dataset by prompting advanced GPT-3.5 conditional on image context. Then, we augment generated data by introducing negative and context samples. Based on the built dataset, we employ LoRA fine-tuning technology to enhance the tool-usage capability of language models, thus allowing them to handle various visual tasks, e.g., visual comprehension and image generation. Moreover, benchmark to assess tool usage accuracy from the decision when to use tools, which tools to use, and arguments of invoked tools. In this benchmark, language models tuned with our GPT4Tools perform comparably to GPT-3.55 on unseen tools. We desire the GPT4Tools to pave the way for equipping open-source language models with the ability to multi-modal tools.
propose
Acknowledgments
This research is partly supported by the National Key R&D Program of China (Grants No. 2020AAA0108302 & 2020AAA0108303), and Shenzhen Science and Technology Project (Grant No. JCYJ20200100 143041798) & Shenzhen Stable Supporting Program (WDZZ22000880200655500 & Shenzhen Key Laboratory of next-generation interactive media innovative technology (Grant No.
ZDS ZDSYYS200100223022000000
10

---

References
[1] Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 1,
[2] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll I.. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurIPS, 2022. 1.
[3]
OpenAI. Chatgpt. attpp://openni... hwt.lppeaaa..cm////. wtmmblog//////oo/oooo/oo//ooollo hhtgg//hhatttt//..// 2023. 1,2,3,4,6,7
[4] OpenAL. Gpt-4 technical report, 2023, 3,
[5] Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671, 2023. 1, 2, 3, 6, 7.8.9
[6] Zhengyuan Yang. Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasa,, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint 23x::233.11381, 2023. 2,
[7] Mojtaba Komeili, Kurt Shuster, and Jason Weston. Internet-augmentedd dialogue generation. arXiv preprint arXiv:2107.07560 2021. 3,3
[8] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In ICLR, 2017. 6
[9] Romal Thoppilan, Danicl De Freitas, Jamic Hall, Noam Shazcer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslic Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.082..2 2022.
[10] Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowice, and Nikolai Grigorev. Internet-augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115. 2022. 3,3
[11] Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves use tools. arXiv preprint aiv: 2023. 3,4
[12] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph Gonzalez, Ion Stoica, Eric Xing. Vicuna: An open-source chat- bot impressing gpt-4 with 90%* chatgpt quality. pttps://lmsyy: wrtes.ogggbbo.o. wrg/blogg//23-0000000/ 00--0-0330000000 vicuna/, 2023. 2, 3, 4, 7, 8, 10
[13] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, arXiv preprint arXiv:2302. 13971, 2023. 2, 7.7
[14] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arxiv:2205.0 00066, 2022. 2, 4,7
[15] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021. 2, 3,4
[16] Ben Wang and Aran Komatsuzaki, GPT-J-6B: Billion Parameter Autoregressive Language Model. /ttp::///ithubb wtwm//iiggolllo/aaaettttaaaa www.essshhtttasafaereeeee twrnt-trrrrroooorrrraa.e..... 2021. 2,
[17] Wenlong Huang. Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In ICML, volume 162 of Proceedings of
Machine Learning Research, pages 9118-9147. PMLR, 2022. 2,
[18] Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of GPT-3 for few-shot knowledge-basse VQA. In AAAI, pages 3081-300 10899. AAAI Press, 2022. 2,4
11

---

[19] Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li. Xiaohua Wang, and Jifeng Dai. Uni- perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. In CVPR. pages 16783-16794, IEEE, 2022. 2
[20] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi- modal tasks. arXiv preprint arXiv:2206. 08916, 2022. 2,
[21] Ting Chen, Saurabh Saxena, Lala Li, Tsung- Yi Lin, David J. Fleet, and Geoffrey E. Hinton. A unified sequence interface for vision tasks. In NeurlPS, 2022.
[22] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint ar arXiv:2205.06177 2022. 2
[23] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-image pre- training for unified uision-lanuud understanding and generation. In ICML, volume 162 of Proceedings of Machine Learning Research, pages 12888-12900 PMLR, 2022.
[24] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597, 2023.
[25] SannBaptistte Alayrac, Jeff Donahue, Pauline Luc, Antoine Micch, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: visual language model for few-shot learning. In NewrIPS, 2022.
[26] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, ct al. Palm-c: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 2023. 2
[27] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: unifying architectures, tasks, and modalities through simple sequence- to-sequence learning framework. In ICMI. volume 162 of Proceedings of Machine Learning Research, pages 23318-23340. PMLR, 2022. 2
[28] Alexander Kolesnikov, André Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, and Neil Houlsby. Uvim: unified modeling approach for vision with learned guiding codes. In NeurlPS, 2022.
[29] Ruiqi Zhong, Kristy Lec, Zheng Zhang, and Dan Klein. Adapting language models for zero-shot learning by meta-tuning on dataset and prompt collections. In EMNLP (Findings), pages 2856-2878, 2021. 3
[30] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned language models zero-shot learners. In ICL.R. OpenReview.net, 2022.
[31] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. Scaling ciatruction-iinetune language models, arXiv preprini 22X1:22100.14116 2022. 3
[32] Stephen H. Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal Nayak, Abheesht Sharma, Taewoon Kim, M. Saiful Bari, Thibault Févry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-David Canwen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, Maged Saeed AIShaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir R. Radev, Mike Tian-Jian Jiang, and Alexander M. Rush. Promptsource: An integrated development environment and repository for natural language prompts. In ACI,, pages 93-104, Association for Computational I Linguistics, 2022.
[33] Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov. Dániel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212. 2022.
[34] Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560. 2022. 3
12

---

[35] Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang. and Tatsunori Hashimoto. Stanford alpaca: An instluctionffolloiing Ilama model. https://gittub.
com/tatsu- laauuullaabaaa satttra-laabsstafoddd.... htabb/sttford_aapccaaa 2023. 3.4
[36] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:334.10599. 2023.
[37] Haotian Liu, Chunyuan Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023, 3, 8,
[38] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation large language models. ICLR, 2022. 3, 5.
[39] Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd-workers for text- annotation tasks, arXiv preprint arXiv:2303. 2023.
[40] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint urXiv:2304.03277, 2023. 4,
[41] Nils Reimers and Tryna Gurevych. Sentence-bert: Sentence emheddings using siamese bert-networks. In EMNLP, pages 3980-3990, 2019.
[42] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. CVPR, pages 248-255, 2009.
[43] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects context. In ECCV, pages 740-755, 2014.
7.7
[44] Haonan Han, Rui Yang, Shuyan Li, Runze Hu,
Li. Ssgd: A smartphone screen glass dataset for defect detection. In ICASSP 2023-2023 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023. 6
[45] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
[46] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
13


Reviews:
[{"review_id": "jzel1ll3Zf", "reviewer": "Reviewer_witA", "summary": "The authors propose GPT4Tools, a new approach to data collection for tool use, along with a novel evaluation method. In this approach, ChatGPT is utilized as a teacher to generate instruction-following data for vision-language tasks related to tools. In addition to the single-turn samples, the authors suggest synthesizing negative samples to instruct the student model on when not to use a tool. They also propose using context samples to guide the student model in utilizing relevant tools based on contextual information. Experimental results indicate that by employing the proposed successful rate(s), open source models like LLaMA-13B and Vicuna-13B achieve significant performance improvements after LoRA fine-tuning. Furthermore, the authors demonstrate a certain level of capability in utilizing unseen tools after the fine-tuning process.", "questions": "Please see the weaknesses section above. Overall I enjoy reading the submission. Since the work mostly discusses instruction data collection procedure, it would be great to expand the discussion between the proposed method and other related fields, such as distillation, other instruction tuning works, etc.", "limitations": "Limitations are included", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The proposed procedure for instruction data collection is straightforward and easy to follow. The performance improvement of the open-source models is evident. The effectiveness of the proposed negative samples and context samples is clearly demonstrated in facilitating downstream tool usage.\n* The proposed evaluation metric is logical and valuable in establishing a standard for future research in this field.\n* The conducted ablation study on data augmentation, model size, and tuning iterations provides valuable insights into understanding the behavior of the model in depth.\n", "weaknesses": "* I believe the work focuses on a straightforward procedure for instruction data collection and a standardized evaluation method. Therefore, I don't identify any significant weaknesses, but rather some points for further discussion:\n  * The process of creating synthetic data from ChatGPT and using it to fine-tune smaller models can be seen as a distillation process. Previous studies have demonstrated that, for many tasks, distilled student language models can outperform their teacher language models across various tasks [A1]. What are the authors' thoughts on the relationship between instruction tuning and distillation in the context of tool use?\n  * Vision tasks often exhibit stronger interdependencies. How effectively can the proposed instruction data collection procedure generalize to other domains, such as manipulating a vast number of APIs for cloud services, etc?\n* I did not find a clear distinction between the two sub-figures in Figure 2. Could the authors emphasize the key area that readers should focus on?\n* There is a typo on line 155: \"single-tune\" should be corrected to \"single-turn.\"\n\n\n[A1] Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. ACL 2023.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** *The process of creating synthetic data from ChatGPT and using it to fine-tune smaller models can be seen as a distillation process. Previous studies have demonstrated that, for many tasks, distilled student language models can outperform their teacher language models across various tasks. What are the authors' thoughts on the relationship between instruction tuning and distillation in the context of tool use?*  \n**A1:** From the perspective of training data, distillation requires input samples to be given and then fed into a network to generate training targets. On the other hand, self-instruction tuning uses a network to generate both input samples and training targets. Therefore, distillation can also be a way to build training targets for self-instruction tuning. However, the most critical challenge in enabling language models to use tools is the lack of diverse input samples, specifically user instructions. Therefore, we leverage the self-instruction technique to utilize advanced language models in generating user instructions based on image content and tool descriptions. In addition, the effectiveness of distillation is limited by the capabilities of the teacher model, which may introduce much noise into the training targets. However, in our GPT4Tools, since we know in advance the toolchains involved in the input samples, we can greatly reduce the noise in the training targets through heuristic rules.\n\n**Q2:** *Vision tasks often exhibit stronger interdependencies. How effectively can the proposed instruction data collection procedure generalize to other domains, such as manipulating a vast number of APIs for cloud services, etc?*  \n**A2:** This is a thoughtful question. In order to enable LLMs to use tools, the following points need to be achieved: judge whether a tool is needed, select the appropriate tool, generate suitable parameters, and interact with the tool in the correct format. For GPT4Tools, we mainly enable the LLM to understand the prompts for toolset definitions and output in the correct format, while selecting the right tools and parameters relies on the own capabilities of models. This allows our method to perform well on unseen toolsets, which is shown in Table 2 and Table 5. We will add more additional tools unrelated vision in the revision.\n\n**Q3:** *A clear distinction between the two sub-figures in Figure 2. Could the authors emphasize the key area that readers should focus on?*  \n**A3:** Figure 2 illustrates the diversity comparison of the collected instructional dataset with and without the image content. Without image context priors, language models tend to generate objects of visual instructions towards a small subset, which is reflected in t-SNE as sparser clusters. In contrast, using image context notably increases diversity, reflected in the visualization as denser and more widely distributed results. Therefore, the language model tuned by image-conditioned data has a higher success rate compared to models without the image content (81.6% v.s. 36.9%)."}, {"review_id": "ooazkequ2c", "reviewer": "Reviewer_VYCC", "summary": "This paper utilizes ChatGPT to generate instruction data for training moderately-sized open-source models to use tools. The authors fine-tune LLMs using LoRA and demonstrate their effectiveness on various visual problems. Additionally, the paper establishes a benchmark for evaluating the LLMs' tool usage ability.", "questions": "please refer to Weaknesses", "limitations": "please refer to Weaknesses", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Using ChatGPT, this paper represents an early effort to consider visual information to produce diverse instruction data that trains moderately-sized open-source models to use tools, resulting in effective learning outcomes. \n\n2. This paper sets a benchmark standard for evaluating the competency of LLMs in tool usage.\n\n3. The results of the experiment demonstrate the efficacy of instruction tuning, as evidenced by the superior performance of the tuned 13B model in comparison to the 175B GPT-3.5 model. Additionally, the study highlights the framework's capacity for generalization to unseen tools.", "weaknesses": "1. It would be intriguing to understand the reasons behind the significant discrepancy in generalization ability between OPT and LLaMA in Table 2.\n2. The prefix prompt used in the study includes both system messages and tool definitions. Although the current context length of open LLMs appears to be sufficient for the 23 tools examined in the paper, it may not be enough if a broader range of tools is considered. Therefore, the present framework would need to address this limitation if it is to accommodate a more diverse range of tools.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** *It would be intriguing to understand the reasons behind the significant discrepancy in generalization ability between OPT and LLaMA in Table 2.*  \n**A1:** Compared to Vicuna and LLaMA, one possible reason is that OPT has weaker capabilities for multi-turn interaction. There are many samples in Validation-II that require multi-turn tool invocations, and OPT demonstrates poor generalization performance on these samples. In contrast, Vicuna, which is optimized through multi-turn dialogues, performs the best. On the contrary, as shown in Table 5 of supplementary material, OPT exhibits good generalization performance on Validation III which only contains single-turn interactions.\n\n**Q2:** *The prefix prompt used in the study includes both system messages and tool definitions. Although the current context length of open LLMs appears to be sufficient for the 23 tools examined in the paper, it may not be enough if a broader range of tools is considered. Therefore, the present framework would need to address this limitation if it is to accommodate a more diverse range of tools.*  \n**A2:** LLMs are currently evolving towards supporting long context, for example Long-form [49] supports 1B token inputs. This makes the prompts for a massive number of tools not an unresolvable issue in the future. In addition, for models with limited context length, we can use tool retrieval techniques to filter out the a small subset of toolsets that may be involved for the user input, and then apply GPT4Tools for inference. To demonstrate effectiveness, we utilized BM25[50] to retrieve the top K tools from 23 tools based on the user input for inference, whose results are shown in Table 14. This strategy can mitigate the reliance on long context models for a large number of tools to some extent. We will provide more experiments on this in the revision.\n\nTable 14: GPT4Tools using Top-K related tools.\n| Model | SR_t | SR_act | SR_args | SR   |\n|-----------------|------|--------|---------|------|\n| Vicuna-13B   | 69.2 | 24.5   | 24.6    | 12.3 |\n| + GPT4Tools (Top-1)  | 87.0 | 55.8   | 57.5    | 54   |\n| + GPT4Tools (Top-2)  | 93.1 | 70     | 69.5    | 67.8 |\n| + GPT4Tools (Top-3)  | 95.8 | 74.4   | 72.9    | 73.1 |\n| + GPT4Tools (All)  | 99.2 | 98.1   | 92.9    | 89.8 |\n\n[49] Ding J, Ma S, Dong L, et al. LongNet: Scaling Transformers to 1,000,000,000 Tokens[J]. arXiv preprint arXiv:2307.02486, 2023.  \n[50] Robertson, S.E., S. Walker, S. Jones, M.M. Beaulieu, M.Gatford. Okapi at TREC-3. TREC-3, (1994), p. 109-126 "}, {"review_id": "GI6hFJ8LTF", "reviewer": "Reviewer_MACc", "summary": "This work propose GPT4Tools, a self-instruction approach to teach large language models to use tools to solve vision tasks. The authors construct an instruction-following dataset by ChatGPT from the combinations of a set of images and 23 image-related tools. In addition, they augment the dataset to enable to capability to not using tools or using tools in multiple turns. Then they apply Low-Rank Adaptation (LoRA) to open-source LLMs like LLaMa and Vicuna to enable them to use tools. They also propose to evaluate the model's performance on tool-using by measureing the success rate from multiple ascepts. The empirical results show that GPT4Tools can successfully utilize the tools and also have capability to use new tools.\nThe study introduces GPT4Tools, a self-instruction method that teaches large language models (LLMs) to utilize tools for solving vision tasks. The authors establish an instruction-following dataset generated by ChatGPT using combinations of images and 23 image-related tools. The dataset is augmented to allow for the capability of non-usage of tools or multi-turn tool usage. The researchers then apply Low-Rank Adaptation (LoRA) to open-source LLMs, like LLaMa and Vicuna, enabling these models to use tools. The paper also proposes an evaluation measure for the model's success rate from multiple aspects. Empirical results reveal that GPT4Tools can effectively use tools and demonstrate the capability to adopt new tools.", "questions": "- Can GPT4Tools effectively do multi-turn planning, given the multi-turn examples in the augmented data? Why does the evaluation data only contain single-turn examples?\n- How scalable is GPT4Tools when using a larger number of tools?\n- (Minor) What the numerical codes (e.g., 179.44, 105.55) represent in the image content shown in Figure 1?", "limitations": "While the authors discuss potential overfitting for longer tuning iterations and the inferior performance of OPT-13B, more detailed discussions regarding the limitations of the proposed evaluation method, the multi-turn capability, and the scalability of GPT4Tools would add value.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "- The paper presents an innovative approach to teaching LLMs to use tools for visual tasks through self-instruction\n- The impact of enabling LLMs to utilize tools could be substantial and pervasive.", "weaknesses": "- The paper falls short in providing a comparison with VisProg[1], which also instructs LLMs to use visual tools through few-shot demonstrations. Although GPT4Tools has the benefit of not utilizing proprietary LLMs in inference, a performance comparison would enrich the paper.\n- The evaluation method appears less compelling because both the training and testing data are generated by using the specific set of tools. Evaluation on standard benchmarks, such as GQA as in [1], could be more convincing.\n- The paper could benefit from enhanced clarity in its descriptions and explanations. Suggestions include:\n    - Clarifying the variables P_t, X_C, Y, possibly through their corresponding parts in Figure 1 or mentioning corresponding examples in the appendix.\n    - The subscript of P_t could be confusing as the total number of tools is defined as N.\n    - The abundant font colors in Figure 1 and Figure 3 may confuse readers, and it would be helpful to indicate which parts are generated by LLMs\n- (Minor) Some crucial details appear to be omitted, such as the process of extracting X_C from images, the image source, and the filtering process of the dataset.\n- (Minor) Typos:\n    - Page 6, line 172: \"and and\"\n    - Page 6, line 173: \"\"items..\"\n\n[1] Gupta, Tanmay, and Aniruddha Kembhavi. \"Visual programming: Compositional visual reasoning without training.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** *How effectively can the proposed instruction data collection procedure generalize to other domains?* \\\n**A1:** Different from VisProg which utilizes few-shot prompt engineering, our method uses a smaller open-source language model and zero-shot manner. As shown in Table 2, our method demonstrates good generalization capability on unseen tools. Furthermore, to further validate efficacy, we conducted ablation studies on GQA following the VisProg by creating a subset for evaluation. As shown in Table 15, the GPT4Tools further improve the accuracy by 3.4% over the VILT-VQA baseline model. We will add more experiments into the revision.\n\nTable 15: Comparison on GQA Dataset.\n **Method** | **Accuracy** |\n|:-------------------:|:--------:|\n VITL-VQA                 | 47.8       | \n GPT4Tools                | 51.2       |   \n\n\n**Q2:** *The paper could benefit from enhanced clarity in its descriptions and explanations.*  \n**A2:** We appreciate the valuable suggestions! Accordingly, we will improve the presentation by clarifying symbols and font colors of figures in the revision.\n\n**Q3:** *Some crucial details appear to be omitted, such as the process of extracting X_C from images, the image source, and the filtering process of the dataset.*  \n**A3**: All images utilized in GPT4Tools are sourced from the training set of COCO [43]. The X_C consists of ground-truth captions and bounding boxes with tags corresponding to the images, which are used to construct prompts for ChatGPT to generate data. We then utilize postprocessing to filter out data with: duplicate instructions, incorrectly formatted instructions, calls with incorrect tool names, and calls with incorrect tool-arguments formats. The comprehensive description about the image source and filtering procedure will be provided in the revision. Meanwhile, the corresponding code will be made available.\n\n**Q4:** *Can GPT4Tools effectively do multi-turn planning, given the multi-turn examples in the augmented data? Why does the evaluation data only contain single-turn examples?*  \n**A4:** Yes, GPT4Tools can perform multi-turn planning. For example, as shown in Figure 7 of supplementary material, the user instruction in a sample \"Generate a man watching a sea based on the pose of the woman.\" requires the following multi-turn planning: detect the woman in the image, extract the pose of the woman, generate an image of a man watching the sea using the pose. In the GPT4Tools, we term these multi-turn planning samples as context samples, which are also included in the validation set.\n\n**Q5:** *How scalable is GPT4Tools when using a larger number of tools?*  \n**A5:** LLMs are currently evolving towards supporting long context, for example Long-form [49] supports 1B token inputs. This makes the prompts for a massive number of tools not an unresolvable issue in the future. In addition, for models with limited context length, we can use tool retrieval techniques to filter out the a small subset of toolsets that may be involved for the user input, and then apply GPT4Tools for inference. To demonstrate effectiveness, we utilized BM25[50] to retrieve the top K tools from 23 tools based on the user input for inference, whose results are shown in Table 14. This strategy can mitigate the reliance on long context models for a large number of tools to some extent. We will provide more experiments on this in the revision.\n\nTable 14: GPT4Tools using Top-K related tools.\n| Model | SR_t | SR_act | SR_args | SR   |\n|-----------------|------|--------|---------|------|\n| Vicuna-13B   | 69.2 | 24.5   | 24.6    | 12.3 |\n| + GPT4Tools (Top-1)  | 87.0 | 55.8   | 57.5    | 54   |\n| + GPT4Tools (Top-2)  | 93.1 | 70     | 69.5    | 67.8 |\n| + GPT4Tools (Top-3)  | 95.8 | 74.4   | 72.9    | 73.1 |\n| + GPT4Tools (All)  | 99.2 | 98.1   | 92.9    | 89.8 |\n\n[49] Ding J, Ma S, Dong L, et al. LongNet: Scaling Transformers to 1,000,000,000 Tokens[J]. arXiv preprint arXiv:2307.02486, 2023.  \n[50] Robertson, S.E., S. Walker, S. Jones, M.M. Beaulieu, M.Gatford. Okapi at TREC-3. TREC-3, (1994), p. 109-126"}, {"review_id": "CJtxhpFIds", "reviewer": "Reviewer_F8mq", "summary": "The goal of this paper is to utilize locally available models, such as vicuna and opt, to learn the tool utilization capability of black-box models like GPT-3.5. They use the large models to generate instructions and samples using image context and apply instruction tuning on these data with the fine-tunable models. Various data augmentation techniques such as negative samples and context samples were also applied. Furthermore, they propose several automated evaluation methods, including the success rate based on the combination of thought, action, and arguments. Experimental results demonstrate significant improvement in tool invocation effectiveness for their fine-tuned models compared to the original backbone models.", "questions": "1. Is there a technical error in Equation 1? where it states that \"Y\" is sampled from the teacher model based on \"X\" to obtain the prompt. However, based on the textual description, Equation 1 should refer to sampling the instruction based on \"X\" and \"P_t.\"\n2. How the image context was constructed?\n3. How is the success rate of thought be computed?", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper proposes a method for generating instructions using image context, which enables the large language model to consider visual information when generating multimodal instructions.\n2. The method of generating negative data has not been considered in previous works.\n3. This paper introduces a novel automated evaluation metric that combines multiple success rates.", "weaknesses": "1. The method proposed in this paper, which utilizes self-instruction techniques of large models to enable tool using capabilities for primitive models, is not novel. Similar techniques have been proposed in several works, such as tool-llama (https://arxiv.org/abs/2304.08354), tool-alpaca (http://arxiv.org/abs/2306.05301), and gorilla (http://arxiv.org/abs/2305.15334). The authors have not made significant contributions to the SFT method.\n\n2. One of the main contributions of this paper, image context, is not thoroughly elaborated in this paper. The self-instruction method seems to be also similar to that of MM-react and LLaVA.\n\n3. For the evaluation metrics, the success rate of thought is not clearly explained. Thought should be a natural languge text with certain diversity, but it is unclear whether the generated thought needs to be identical to the ground truth thought to achieve success rate 1. If they need to be identical, I doubt that the success rate of thought would not be as high as shown in the experimental results, as the model may generate thought that is semantically consistent but not textually identical to the groundtruth. If they do not need to be identical, it would be helpful to provide details on how the success rate of thought is calculated.\n\n4. Some recent strong multimodal baselines, such as HuggingGPT (http://arxiv.org/abs/2303.17580) and gorilla (http://arxiv.org/abs/2305.15334), are missing in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** *The method proposed in this paper, which utilizes self-instruction techniques of large models to enable tool using capabilities for primitive models, is not novel. Similar techniques have been proposed in several works, such as tool-llama (https://arxiv.org/abs/2304.08354), tool-alpaca (http://arxiv.org/abs/2306.05301), and gorilla (http://arxiv.org/abs/2305.15334). The authors have not made significant contributions to the SFT method.*  \n**A1:** We appreciate the comments but believe there exists a big misunderstanding. Our GPT4Tools is the first public work (before all the mentioned works) to enable a primitive language model to use multi-modal tools [51], including demo and codes. Besides, all the mentioned works were pre-printed after the submission deadline. Please note that the mentioned Tool-LLaMa could be [https://arxiv.org/pdf/2307.16789.pdf] instead of the BMTools [https://arxiv.org/abs/2304.08354]. The BMTools relies on the prompt engineering of ChatGPT instead of off-the-shelf language models. By contrast, our GPT4Tools aims to empower open-sourced language models invoking tools through self-instruction tuning. \n\nOur contributions mainly lie in three points:  \n(1) Our method first enables primitive open-source language models to use multi-modal tools, eliminating the dependence on advanced proprietary LLMs like ChatGPT.  \n(2) We design a new approach based on multi-modal contexts for self-instruction and augmentation, which significantly promote multi-modal tool usage and can be deployed in different approaches.  \n(3) We propose a new benchmark to assess the effectiveness of using tools, and our method shows remarkable improvements.  \n\n[51] Yin S, Fu C, Zhao S, et al. A Survey on Multimodal Large Language Models[J]. arXiv preprint arXiv:2306.13549, 2023.\n\n**Q2:** *One of the main contributions of this paper, image context, is not thoroughly elaborated in this paper. The self-instruction method seems to be also similar to that of MM-react and LLaVA.*  \n**A2:** Please note that our GPT4Tools is the concurrent work with MM-REACT and LLaVa. Besides, as far as we known, the MM-REACT is based on prompt engineering instead of self-instructions, and the LLaVa aims to achieve a better VQA model rather than pursue the capability of invoking tools. The LLaVa can be also a tool in our GPT4Tools to enhance the visual understanding. In addition, the image context is consist of the image captions and the coordinates of objects. As presented in Section 3.1 and Table 10 of supplementary material, we use these image contexts to build prompt for ChatGPT and then generate tool-related user instructions. We will clarify it in the revision.\n\n**Q3:** *For the evaluation metrics, the success rate of thought is not clearly explained. Thought should be a natural languge text with certain diversity, but it is unclear whether the generated thought needs to be identical to the ground truth thought to achieve success rate 1. If they need to be identical, I doubt that the success rate of thought would not be as high as shown in the experimental results, as the model may generate thought that is semantically consistent but not textually identical to the groundtruth. If they do not need to be identical, it would be helpful to provide details on how the success rate of thought is calculated.*  \n**A3:** Different from many open-ended tasks, invoking tools need to strictly follow a pre-defined interaction format. The model arbitrarily outputting in unexpected formats, such as wrong tool names and number of parameters, will lead to failure in properly invoking the tools. As presented in Section 3.3, we evaluate the models from three aspects: whether a tool is needed, tool type, and tool parameters. For open-ended text in the tool parameters, we adopt BLEU metric with threshold to assess correctness.\n\n**Q4:** *Some recent strong multimodal baselines, such as HuggingGPT (http://arxiv.org/abs/2303.17580) and gorilla (http://arxiv.org/abs/2305.15334), are missing in this paper.*  \n**A4:** Please refer to Q1. Besides, we evaluate our model with the Visual-ChatGPT in Table 2 and Table 5, which uses the same GPT-3.5 LLM with the HuggingGPT.\n\n**Q5:** *Is there a technical error in Equation 1? where it states that \"Y\" is sampled from the teacher model based on \"X\" to obtain the prompt. However, based on the textual description, Equation 1 should refer to sampling the instruction based on \"X\" and \"P_t.\"*  \n**A5**: As shown in Table 10 of supplementary material, the P_t is the system prefix prompt, which is made up of the basic task definition, the usage description of tools and the placeholder for image context. We utilize ChatGPT to generate diverse user instructions by placing different image contexts into the placeholder of the system prompt. More details will be added into the revision.\n\n**Q6:** *How the image context was constructed?*   \n**A6:** Please refer to Q2. As illustrated in Figure 1 and Table 10 of supplementary material, the image context is consist of the image captions and the coordinates of objects. We use these image contexts to build prompt for ChatGPT and then generate tool-related user instructions. We will clarify it in the revision.\n\n**Q7:** *How is the success rate of thought be computed?*  \n**A7:** Please refer to Q3. As presented in Section 3.3, we evaluate the models from three aspects: whether a tool is needed, tool type, and tool parameters. For open-ended text in the tool parameters, we adopt BLEU metric with threshold to assess correctness."}]

Author rebuttal:

Meta-review: GPT4Tools is proposed: ChatGPT is utilized as a teacher to generate instruction-following data for tool use. In addition to the single-turn samples, negative samples are synthesized to instruct the student model on when not to use a tool. Context samples are used to guide the student model in utilizing relevant tools based on contextual information.

All reviewers agree to accept the paper, though some shared concerns include limited novelty.
Decision: Accept (poster)