SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Based on the collected data, we tuned language models (LLaMA [13], Vicuna [12], and OPT [14]) with LoRA [38] technology. Specifically, we equipped the projection layers of query, key, value, and output with LoRA layers. The LoRA attention dimension and scaling alpha were set to 16. While the language model was kept frozen, the LoRA layers were optimized using the AdamW [46]. All models were fine-tuned over 3 epochs, with a batch size 512. The learning rate was set to  $3 \times 10^{-4}$ , and the maximum length of new tokens was restricted to 2048. Unless otherwise specified, we used Vicuna-13B for the ablation experiments.
Can instruction datasets teach language model using tools? The outcomes of GPT-3.5 [3], OPT-13B [14], LLaMA-13B [13], and Vicuna-13B [12] are presented in Table 2. GPT-3.5 is considered analogous to Visual ChatGPT [5]. Upon prompting GPT-3.5 with tool-associated instructions, it can attain a SR of  $84.8\%$  on the validation set, thereby underscoring its zero-shot ability to follow a standardized format and utilize tools effectively. Notably, OPT-13B fails to invoke tools with the prompts alone. In contrast, LLaMA-13B and Vicuna-13B exhibit a certain level of comprehension of tool usage, while they still face challenges in executing a chain of actions. Specifically, LLaMA-13B achieves  $3.2\%$  SR, which is absolutely lower than  $\mathrm{SR}_t$ ,  $\mathrm{SR}_{act}$ , and  $\mathrm{SR}_{args}$ . In the case of Vicuna-13B, its SR is  $56.8\%$  less than  $\mathrm{SR}_t$ , implying that under a zero-shot setup, Vicuna-13B displays commendable discernment in determining when to use tools within a given context. After fine-tuned with GPT4Tools, there are substantial alterations in the tool invocation competencies of each model. Specifically, the SR of OPT-13B witnessed a sharp increase from 0 to  $93.2\%$ . Similarly, the SR for LLaMA-13B escalates from  $3.2\%$  to  $66.4\%$ , and Vicuna-13B's SR rises from  $12.4\%$  to  $94.1\%$ . These outcomes unequivocally validate that the GPT4Tools developed in this study are indeed effective in instructing language models to use tools.
Can the model be generalized to unseen tools after fine-tuning? The right side of Table 2 shows the results when prompting a novel tool and corresponding utilization. On the test set, GPT-3.5 attains  $91.5\%$  SR in a zero-shot manner. The outcomes for other models, which are not fine-tuned on the GPT4Tools and directly invoke tools utilizing prompts, are analogous to those on the validation set. In contrast, models that are fine-tuned on the GPT4Tools dataset exhibit a degree of competence in invoking tools that have not been previously encountered (did not appear in the training set). More specifically, the fine-tuned LLaMA-13B model achieves a superior SR on new tools by a margin of  $67.9\%$  when compared to the original model. The fine-tuned Vicuna-13B model demonstrates  $90.6\%$  SR on new tools, which is comparable to GPT-3.5. This observation indicates that the language model can invoke unseen tools after fine-tuned with GPT4Tools.
Table 3: Ablation study for data augmentations on the validation set. 'IC', 'CS', 'NS' denotes image content, context samples, and negative samples, respectively.
IC	CS	NS	SRt	SRact	SRargs	SR
70.0	55.7	51.7	36.9
✓	89.6	89.9	84.5	81.6
✓	✓	97.4	95.7	88.5	91.6
✓	✓	✓	98.7	97.6	91.4	94.1
Table 4: Ablation study for different model scales on the validation set. 7B and 13B refer to Vicuna-7B [12] and Vicuna-13B [12] models, respectively.
Model	GPT4Tools	SRt	SRact	SRargs	SR
7B	X	27.7	15.8	11.5	4.5
✓	96.2	94.5	89.8	92.9
13B	X	69.2	25.1	25.2	12.4
✓	98.7	97.6	91.4	94.1
Data Augmentation. As depicted in Table 3, we execute a series of ablation studies on various tricks implemented during the creation of the dataset. When instructions are not conditioned upon the image content, the SR of the fine-tuned model on the validation set is a mere  $36.9\%$ . In contrast, when instructions are generated with conditioning on the image content, the SR on the validation set is enhanced substantially to  $81.6\%$ . This uptick can be primarily attributed to the elevated diversity and intricacy of the generated instructions. Moreover, an augmentation of the SR to  $91.6\%$  is observed upon introducing context samples into the instructions. This finding underscores the fact that partitioning the chain of actions and allocating them to the in
struction and response can strengthen the model's comprehension of the tool. It is noteworthy to mention that with the incorporation of negative samples into the generated instructions, the SR increases to  $94.1\%$ . This outcome can be traced back to the propensity of the model, when trained exclusively with positive samples, to bias toward tool invocation. This tendency consequently diminishes the capacity to discern the appropriate cases for tool usage. Adding negative samples equips the model with the ability to determine when to use tools.
Figure 4: Performance variation curve with the fine-tuning iteration.
Model Scales. We attempt experiments with models at different scales. The results in Table 4 demonstrate that after fine-tuned on the generated dataset, Vicuna-7B [12] is also capable of invoking tools in a fixed format. Specifically, under a zero-shot setting, Vicuna-7B achieves only a  $4.5\%$  SR. By contrast, after fine-tuning, it can achieve an SR of  $92.9\%$ .
Tuning Iteration. We increase the number of iterations for fine-tuning and present the results in Figure 4. Notably, during the range of iterations from 400 to 800, the model's performance demonstrates substantial fluctuations in tool invocation. However, subsequent to this range, there is a steady improvement in SRt, SRact,  $\mathrm{SR}_{\text{args}}$ , and SR. This indicates that the model progressively adapts to the dataset, enhancing its capability to invoke tools.
Figure 5 presents a comparative analysis of our model with Visual ChatGPT [5] and LLaVa [37]. When an image is submitted by the user alongside the instruction "Generate a picture of real people based on the edge", Visual ChatGPT delivers an image that exhibits a weak correlation with the given instruction. Owing to its inability to generate images, LLaVa only returns a caption. In contrast, our model produces an accurate result, thereby evidencing that the tool-related instruction tuning method proposed in this paper can effectively instruct language models in the correct usage of tools. In Figure 6, we further demonstrate that the Vicuna-13B fine-tuned on GPT4Tools is capable of finishing some visual commands by invoking visual tools. This finding indicates that imparting knowledge to
(a) GPT4Tools (ours)
(b) VisualChatGPT
(c) LLaVa
Figure 5: Comparison with other models. Our GPT4Tools responds correctly, while Visual ChatGPT [5] replies with the wrong image, and LLaVa [37] can not generate the image.
Figure 6: Cases of invoking tools from Vicuna-13B [12] fine-tuned on our GPT4Tools.
Table 5: Results of GPT4Tools using Top-K related tools. The total number of tools equals 23.
Model	Retrieval Top-K	SRt	SRact	SRargs	SR
Vicuna-13B [12]	-	69.2	25.1	25.2	12.4
w/ GPT4Tools	1	87.0	55.8	57.5	54.0
w/ GPT4Tools	2	93.1	70	69.5	67.8
w/ GPT4Tools	3	95.8	74.4	72.9	73.1
w/ GPT4Tools	23	98.7	97.6	91.4	94.1
language models regarding the tool invocation could potentially be a way toward the development of a generalist model. More case studies are presented in the Appendix.
Although the proposed GPT4Tools can teach plug-and-play language models to use tools effectively, it still has some limitations. As shown in Figure 2 the success rate of all models is not  $100\%$ . Thus, further improvements are still necessary for practical applications. Additionally, GPT4Tools teaches the model to explicitly invoke tools using a verbose and fixed prompt. This approach decreases the computational efficiency since attention-based architectures compute the relationships between all tokens. Besides, with the increased number of tools, the prompt length might surpass the limited context length of LLMs. In this case, we can alternatively utilize a tool retrieval technique to filter out a small set of tools and then apply GPT4Tools-based LLMs for tool selection and invocation. We employ BM25, based on the user input, to retrieve the top-K tools from the defined 23 tools. As shown in Table 5, SR is only  $54\%$  while retrieving the top-1 tool. When the number of retrieved tools increases to 3, SR is boosted to  $73.1\%$ . Although the retrieval strategy can mitigate the reliance on long context models for a large number of tools to some extent, its SR can not match the original model. This result can be attributed to the inabilities of the retriever. Therefore, in the future, it is imperative to build a specialized retriever for the tool name retrieval. Moreover, it should be explored how to enable the model to implicitly invoke various tools instead of using the complex prompt. Nevertheless, our GPT4Tools method provides a viable approach for equipping language models with the ability to use multi-modal tools.
In this paper, we introduce GPT4Tools, a novel method that enables open-source language models to utilize multi-modal tools efficiently. Specifically, We construct a tool-related instructional dataset by prompting advanced GPT-3.5 conditional on image context. Then, we augment the generated data by introducing negative and context samples. Based on the built dataset, we employ LoRA fine-tuning technology to enhance the tool-usage capability of language models, thus allowing them to handle various visual tasks, e.g., visual comprehension and image generation. Moreover, we propose a benchmark to assess tool usage accuracy from the decision when to use tools, which tools to use, and arguments of invoked tools. In this benchmark, language models tuned with our GPT4Tools perform comparably to GPT-3.5 on unseen tools. We desire the GPT4Tools to pave the way for equipping open-source language models with the ability to use multi-modal tools.
This research is partly supported by the National Key R&D Program of China (Grants No. 2020AAA0108302 & 2020AAA0108303), and Shenzhen Science and Technology Project (Grant No. JCYJ20200109143041798) & Shenzhen Stable Supporting Program (WDZC20200820200655001) & Shenzhen Key Laboratory of next-generation interactive media innovative technology (Grant No. ZDSYS20210623092001004).
[35] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023. 3, 4
[36] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.
[37] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 3, 4, 8, 9
[38] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022. 3, 5, 7
[39] Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:2303.15056, 2023. 4
[40] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023. 4, 5
[41] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In EMNLP, pages 3980-3990, 2019. 4
[42] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255, 2009. 6
[43] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólar, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, pages 740-755, 2014. 6, 7
[44] Haonan Han, Rui Yang, Shuyan Li, Runze Hu, and Xiu Li. Ssgd: A smartphone screen glass dataset for defect detection. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023. 6
[45] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023. 6
[46] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 7