SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3: Success rates on easy questions.
LLM Category	Models	Flight	Coffee	Agenda	Yelp	DBLP	SciREX	GSM8K	Airbnb	Average
Open-Sourced LLMs	LLaMA-2 (13B)	0.0	2.0	0.0	5.0	1.0	0.0	9.0	1.0	2.3
Falcon (40B)	1.0	1.0	2.0	8.0	1.0	0.0	8.0	5.0	3.3
LLaMA-2 (70B)	2.0	6.0	5.0	15.0	0.0	0.0	9.0	4.0	5.1
Closed-Sourced LLMs	ChatGPT	2.0	0.0	0.0	15.0	0.0	2.0	26.0	0.0	5.6
CoT	1.0	1.0	0.0	9.0	0.0	0.0	30.0	0.0	5.1
Tool-Augmented LLMs	Chameleon	30.0	9.0	4.0	8.0	3.0	0.0	27.0	4.0	10.6
ReAct (GPT-3)	61.0	90.0	29.0	77.0	28.0	3.0	32.0	25.0	43.1
ReAct (GPT-3.5)	48.0	81.0	24.0	64.0	23.0	2.0	23.0	29.0	36.8
Table 4: Success rate on hard questions.
LLM Category	Models	Flight	Coffee	Agenda	Yelp	Airbnb	DBLP	SciREX	Average
Open-Sourced LLMs	LLaMA-2 (13B)	1.0	0.0	0.0	4.0	1.0	5.0	1.0	1.7
Falcon (40B)	1.0	0.0	0.0	4.0	1.0	6.0	1.0	1.9
LLaMA-2 (70B)	1.0	0.0	0.0	4.0	1.0	4.0	3.0	1.9
Closed-Sourced LLMs	ChatGPT	2.0	2.3	1.0	0.0	2.0	4.0	3.0	2.0
CoT	0.0	0.8	0.0	1.0	0.0	3.0	5.0	1.4
Tool-Augmented LLMs	Chameleon	3.0	2.3	0.0	0.0	0.0	8.0	0.0	1.9
ReAct (GPT-3)	3.0	10.8	0.0	3.0	0.0	19.0	0.0	5.1
ReAct (GPT-3.5)	5.0	17.7	7.0	8.0	7.0	5.0	8.0	8.2
answering. Additionally, the vanilla LLMs show near-zero performance on different tasks, indicating that there is little overlap between the benchmark data and the LLMs' internal knowledge.
Comparing Different Tool-Use LLMs. Tables 3 and 4 show the results of different methods on the easy and hard questions. ChatGPT and CoT achieve very poor success rates ( $<10$ ) on both easy and hard questions across different tasks. This is expected as the questions in ToolQA cannot be answered solely based on LLMs' internal knowledge and reasoning. Chameleon achieves slightly better performance, with  $10.6\%$  and  $1.9\%$  success rates on easy and hard questions, respectively. This is because Chameleon incorporates tool descriptions and integrates human-induced orderings of these tools in its context, enabling it to comprehend and compose different tools for QA. However, Chameleon cannot take feedback from the execution trace, thus often suffering from infeasible actions or omitted arguments in its generated plans. ReAct is the best-performing model. It can use observations in the execution trace to generate its next action, allowing it to iteratively refine its tool use chain and obtain better success rates.
Easy vs. Hard Questions. Comparing Tables 3 and 4, we observe that all the baselines perform much worse on hard questions. The best method achieves an average success rate of  $43.13\%$  on easy questions, while that number drops to  $8.24\%$  on hard questions. As mentioned in § 3, the hard questions in ToolQA require more tool calls and more complicated compositions. Current tool-augmented LLMs struggle with answering such hard questions, which requires further development of techniques to improve their ability to reason about the task and generate plans for tool use.
GPT-3 vs. GPT3.5. Comparing the different versions of ReAct, we observe that the ReAct (GPT-3) outperforms ReAct (GPT-3.5) on easy questions, yet it shows inferior performance on hard questions. Our hypothesis is that for easy questions, it is more important to learn and follow the format of the tool calls in the context, which GPT-3 is stronger at. For hard questions, the better reasoning and code understanding abilities of GPT-3.5 enables it to come up with "innovative" solutions that never appear in the context, leading to higher success rates. An example can be referred to in § 5.3.
5 Result Analysis and Discussion
We analyze the drawbacks and possible improvements of existing tool-augmented LLMs, taking the best-performed ReAct (GPT-3.5) model on the hard questions of ToolQA as an example.
5.1 Main Error Type I: Argument Errors
By performing comprehensive error analysis, we found that the most common error type when asking LLMs to use tools for QA is argument error — LLMs calling the tools with wrong arguments.
GPT-4 was not included in the evaluation as we have no access to its API.
(a) Incorrect tool calls of ReAct on ToolQA.
(b) Confusion matrix of questions from different resources in ToolQA.
Figure 3: Analysis of incorrect tool calls and incorrect data sources made by ReAct on ToolQA.
For ReAct, this error type makes  $44.56\%$  and  $48.23\%$  out of the 377 and 436 error cases on easy and hard questions respectively, as shown in Figure 3(a). Interestingly, ReAct shows different argument error patterns on easy and hard questions. On easy questions, it tends to make more mistakes on database-related tools. For example, the model commits a total of 120 errors when calling LoadDB, FilterDB, and GetValue tools for easy questions, while this number reduces to 95 for hard questions. On the other hand, when dealing with code-related tools (e.g., SQLInterpreter and PythonInterpreter), ReAct makes nearly  $10\mathrm{x}$  more errors for hard questions than for easy ones. This phenomenon is likely because the solution logic for hard questions is often more complex and cannot be fully inferred from the context alone. Consequently, the LLMs tend to rely on their understanding of code and programming concepts to tackle these intricate questions. In contrast, for easy questions, the LLMs tend to follow the patterns provided in the context, attempting to combine different database operations to arrive at a solution.
5.2 Main Error Type II: Incorrect Data Source
We have conducted an investigation into the data sources preferred by LLMs when answering questions. We found that LLMs also have difficulties in identifying the proper reference corpora answer the questions. This behavior is graphically represented as a confusion matrix in Figure 3(b). Upon examining the figure, it is apparent that for target reference corpora like Flight, Coffee, Airbnb, and Yelp that contain temporal information, LLMs are more likely to query the Agenda corpus for answering questions. Similarly, given that the SciREX knowledge corpora and DBLP graph are both in the scientific domain, LLMs tend to be confused about which source to query when answering scientific questions.
5.3 Main Error Type III: Innovation and Hallucination
For in-context tool-augmented LLMs, it is typical to include descriptions and use-case examples of each tool in the prompt. However, as the problem complexity increases with the number of tools, it becomes challenging to encompass all possible instances of compositional tool use as few-shot exemplars. Consequently, it is vital for LLMs to uncover logical relationships among different tools, which have never been encompassed in the human-provided exemplars, to solve challenging tasks — a process we refer to as "innovation." However, these innovative behaviors are a double-edged sword as they are often accompanied by hallucinations. Figure 4 illustrates this phenomenon with a case study, where LLMs answer hard questions with reference Coffee data. Given the context length constraint, the few-shot exemplar only showcases the basic usage of database operations and the SQL interpreter. For the hard question in Figure 4(a), ReAct (GPT-3) strictly follows the operations displayed in the context, leading to failure. On the contrary, ReAct (GPT-3.5) innovatively identifies the SQL interpreter as a possible alternative to database operations, especially when the latter fails repeatedly. However, such innovations can oftentimes lead to hallucinations. As shown in Figure 4(b), when answering another hard question from the Coffee data, ReAct (GPT-3.5) opts to hallucinate certain observations (highlighted in yellow) that are non-existent in the feedback from tool execution.
5.4 Other Error Types
We manually go through and count all the errors made by the ReAct (GPT-3.5) model and show the errors on both easy and hard questions in Figure 5. In addition to the aforementioned 3 main error types, there are 4 error types that frequently occur:
(a) Innovation
(b) Hallucination
(a) Easy questions.
Figure 5: Error analysis of ReAct on ToolQA.
Figure 4: An example of innovation and hallucination when answering hard questions on Coffee data. Actions and observations shrouded in pink are incorrect, whereas those in green are correct. Terms highlighted in yellow signify hallucinations produced by ReAct (GPT-3.5).
(b) Hard questions.
- Infeasible Actions: The execution of tool calls are infeasible in the environment, often involving new tools that do not exist in the pre-defined tool pool.
- Too Long Context: The encoding of interaction history, observations, and tool-use plans exceed the length limitation of GPT family models, resulting in runtime errors;
- Mis-understanding: The LLMs cannot understand the observations obtained from external interaction and fail to determine the next steps or generate answers;
- Low-Quality Retrieval: This error occurs when the retrieval model fails to extract the relevant information from text corpora, indicating insufficient external knowledge for LLMs to answer questions accurately.
Comparing these error types on easy and hard questions, we find that the overall distribution is similar, though there is a slightly higher rate of hallucination and long-context errors when answering hard questions. This can be attributed to the complexity of hard questions, which often require composing more tools for question answering.
5.5 Potential Misuse of ToolQA
We summarize the potential misuse of the ToolQA dataset and Tool-augmented LLMs as follows:
- If LLMs can be trained or prompted to use external tools, they could be prompted to use tools that extract personal information, propagate malware, or provide misleading information;
- There is potential for future LLMs to over-rely on these external tools, sacrificing their intrinsic reasoning abilities. This can make them less versatile in situations where tool use is not feasible;
Table 5: Performance of tool-augmented open-sourced LLM.
Methods	Coffee-Easy	Coffee-Hard
LLaMA-2 (13B)	2.0	0.0
ReAct (LLaMA-2, 13B)	31.0	6.2
ReAct (GPT-3.5)	81.0	17.7
- As LLMs are prompted to interact with more external systems, the security risks can increase. Malicious actors might find ways to exploit the interactions between LLMs and the external tools they leverage.
5.6 Comparing Open-Source LLMs with ChatGPT
Table 5 shows the comparison of vanilla LLaMA-2, tool-augmented LLaMA-2, and tool-augmented ChatGPT. There is a significant performance gap between vanilla LLaMA-2 and tool-augmented LLaMA-2, which is consistent with what we have observed on closed-source LLMs in Section 4.2. In terms of the ability to use external tools for answering the questions, we found that LLaMA-2 is indeed lagging behind ChatGPT. The prompts tailored for tool-augmented LLMs tend to be complicated and lengthy, containing tool descriptions, few-shot examples, and interaction history with the environment. Such long contexts make it difficult for LLaMA-2 to understand complex instructions hidden inside.
6 Conclusion and Recommendation
We have developed ToolQA, a dataset that assesses the ability of Large Language Models (LLMs) in using external tools for solving complex problems. ToolQA is curated by an automated three-phase process, including reference data collection, template-based question generation, and programmatic answer generation. This pipeline is general and can be expanded to incorporate external knowledge corpora in different domains. We tested both standard LLMs and tool-augmented LLMs on ToolQA. Our experiments showed that even the strongest model achieved limited performance on the hard questions of ToolQA. Our analysis found that current tool-augmented LLMs tend to make errors such as incorrect tool calls and using incorrect data sources. These issues could be potentially addressed by fine-tuning using a collection of tool-use corpora with open-source LLMs. In the future, we plan to collect high-quality tool-use sequences to fine-tune open-source LLMs, and subsequently evaluating their performance on ToolQA.
It is important to note that the reported performance of closed-source LLMs, such as ChatGPT, is based on specific versions of these models. As these closed-source models undergo further development and updates, the results may change accordingly. We advocate two strategies to mitigate the issue: (1) We plan to continuously include the performance of newly released models in our repository, with the collective efforts of both our team and peer researchers of the community. We intend to integrate results from different versions of such product APIs and document the identifiers of each version (e.g., 0314, 0613, etc.) to promote reproducibility. (2) The reported performance of closed-source LLMs should be regarded primarily as a reference, and making comparisons with them is optional. Instead, we encourage the community to use the performance of open-source LLMs as the primary baseline when employing the ToolQA benchmark. This will promote reproducibility and consistency of the evaluation on ToolQA over time.
[1] E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. Falcon-40B: an open large language model with state-of-the-art performance. 2023.
[77] J. Zhang. Graph-toolformer: To empower llms with graph reasoning ability via prompt augmented by chatgpt, 2023.
[78] M. Zhang and E. Choi. SituatedQA: Incorporating extra-linguistic contexts into QA. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7371–7387, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics.
[79] R. Zhang, Y. Yu, P. Shetty, L. Song, and C. Zhang. Prboost: Prompt-based rule discovery and boosting for interactive weakly-supervised learning. arXiv preprint arXiv:2203.09735, 2022.
[80] Y. Zhuang, Y. Li, J. Zhang, Y. Yu, Y. Mou, X. Chen, L. Song, and C. Zhang. ReSel: Nary relation extraction from scientific text and tables by learning to retrieve and select. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 730–744, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics.
Methods	Tool Numbers	Tool Categories	# Tool/Task	Reasoning	Instruction Type	Task
Single-Tool Methods
CoT [65]	1	-	1	Generation	Prompting	QA
Lila [38]	1	math/code	1	Generation	Prompting	MathQA
Program-of-Thought [6]	1	code	1	Generation	Prompting	TabQA
Code4Struct [63]	1	code	1	Generation	Prompting	Event Extraction
PAL [13]	1	code	1	Generation	Prompting	MathQA
MathPrompt [15]	1	code	1	Generation	Prompting	MathQA
ToolFormer [56]	5	Basic	1	Generation	PR & FT	QA
GraphToolFormer [77]	5	Graph	1	Human Info	PR & FT	Graph
Talm [46]	-	Basic	1	Generation	PR & FT	QA
Multi-Tool Methods
WebGPT [40]	10	Web Operation	>1	Feedback	Fine-tuning	QA
HuggingGPT [57]	>10	Vision	>1	Human Info	Prompting	VQA
Chameleon [32]	>10	code, nlp, cv	>1	Human Info	Prompting	ScienceQA, TabQA
GeneGPT [20]	38	NCBI APIs	>1	Generation	Prompting	Gene Tasks
ART [45]	8	code/math/retriever	>1	Human Feedback	Prompting	BigBench
ReAct [76]	3	retriever	>1	Feedback	PR & FT	QA, AlfWorld, WebShop
MM-ReAct [73]	>10	vision	>1	Feedback	Prompting	CV tasks
Visual ChatGPT [69]	>10	vision	>1	Feedback	Prompting	CV tasks
Table 6: A comparison of methods that leverage LLMs for Tool-use.
We list the state-of-the-art related works in tool-augmented LLMs in Table 6. All of them can be categorized into two groups: (1) single-tool methods, that focus on making a single API call perfect in the solution; (2) multi-tool methods, that emphasize more on studying how to compose different tools together to solve a challenging problem. ToolQA is more suitable for the evaluation of the second category to test the inherent logical reasoning behind different tools. Additionally, there exist other notable contributions [64, 25, 59] within the realm of decision-making that specifically emphasize the planning capabilities of expansive language models. These endeavors can be regarded as methods affiliated with tools, wherein the actions within generated plans are analogous to distinct tools utilized for specific purposes.
B Data Sources
B.1 Different Data Source Introduction
- Flight Status (2022-2023)<sup>5</sup> contains almost all flight information of airlines between 2022 and 2023, which is too contemporary for LLMs' internal knowledge.
- Daily Coffee Price (2000-2022)<sup>6</sup> contains the daily price of coffee, ranging from 2000 to 2022, where the information is too contemporary and detailed for LLMs' internal knowledge.
- Yelp Business Data<sup>7</sup> is a subset of Yelp's business data across 8 metropolitan areas in the USA and Canada, where the information is too detailed for LLMs' internal knowledge.
- Airbnb Open Data<sup>8</sup> is a subset of Airbnb activities in New York, where the information is too detailed for LLMs' internal knowledge.
- DBLP Citation Network (V14)<sup>9</sup> constructs the graph based on the records after 2020. The author-author and paper-paper relations are formulated as two separate graphs.
- GSM8k<sup>10</sup> is a dataset of 8.5K high-quality linguistically diverse grade school math word problems. We sample the questions from the error cases made by ChatGPT on the original dataset to make sure that the questions cannot be easily handled with its internal knowledge.
- SciREX<sup>11</sup> is a challenging dataset for document-level information extraction based on a collection of full-length machine-learning scientific papers.
$^{5}$ https://www.kaggle.com/datasets/robikscube/flight-delay-dataset-20182022?select=Combined_Flights_2022.csv
<sup>6</sup>https://www.kaggle.com/datasets/psycon/daily-coffee-price
$^{7}$ https://www.kaggle.com/datasets/yelp-dataset/yelp-dataset?select=yelp_academic_dataset.business.json
<sup>8</sup>https://www.kaggle.com/datasets/arianazmoudeh/airbnbopendra
<sup>9</sup>https://www.aminer.org/citation
10 https://github.com/openai/grade-school-math
11https://github.com/allenai/SciREX
- Agenda is our own synthetic dataset to model the real-world personal agenda data. To avoid the privacy issue, we first create names, events, and dates with ChatGPT and then randomly compose them to form 10000 different records. To create a pure-text personal agenda corpus, we feed each of the records into ChatGPT, containing generated agenda for virtual characters. More Details can be seen in Appendix B.2.
B.2 Generation Details of Agenda Dataset
As mentioned in § 3.2, personal or private data serves as a significant external knowledge source. There exist applications that have been designed with plugins and external tools specifically querying this type of data, such as AI personal assistants on daily agenda. Nevertheless, we recognize that this data often intersects with sensitive areas, and hence, privacy concerns are paramount. To address these issues, we automatically synthesize a personal agenda corpus. This not only ensures that the large language models (LLMs) have not been previously exposed to the data but also eliminates any possibility of them inadvertently memorizing the information within their internal knowledge.
In the synthetically generated personal agenda corpus, each entry follows the pattern: "NAME performs EVENT at TIME on DATE", incorporating key elements such as names, events, dates, and time slots. To begin, we employ ChatGPT to virtually generate these elements. More precisely, we create 100 unique names, 10000 distinctive events each associated with corresponding time slots within a day, and span all possible dates from 01/01/2022 through 12/31/2022. Following this, we commence the random assembly of these generated elements to formulate personal agenda entries. For every event-time pair generated, we randomly select from the pool of 100 names and possible dates to construct each record. This process yields a total of 9,494 unique personal agenda entries. To transform this corpus into an accessible external database for model querying, we transcribe each record into a comprehensible natural language description. Prompts designed for agenda data generation are listed in Appendix F.2.
C Easy Question Templates
C.1 Flights
We design the following 10 templates:
- What was the departure time of the {CARRIER} {NUMBER} flight from {ORIGIN} to {DEST} on {ORIGIN}?
- Was the flight {CARRIER} {NUMBER} from {ORIGIN} to {DEST} cancelled on {ORIGIN}?
- What is the flight number of the {AIRLINE} flight from {ORIGIN} to {DEST} on {ORIGIN}?
- How long was the different between the CRS-recorded departure time and actual departure time of the {CARRIER} {NUMBER} flight from {ORIGIN} to {DEST} on {ORIGIN}?
- How long did {CARRIER} {NUMBER} delay when arrival on {DEST}?
- How many extra minutes did the {CARRIER} {NUMBER} flight take from {ORIGIN} to {DEST} on {ORIGIN}?
- What was the local arrival time of the {CARRIER} {NUMBER} flight from {ORIGIN} to {DEST} on {ORIGIN}?
- What was the CRS-recorded arrival time of the {CARRIER} {NUMBER} flight from {ORIGIN} to {DEST} on {ORIGIN}?
- How long was the flight  $\{\text{CARRIER}\} \{\text{NUMBER}\}$  from  $\{\text{ORIGIN}\}$  to  $\{\text{DEST}\}$  on  $\{\text{ORIGIN}\}$ ?
- How many minutes did the {CARRIER} {NUMBER} flight take to taxi in on {DATE}?
C.2 Coffee
We design the following 8 templates:
- What was the daily coffee price opening on {DATE}?
- What was the lowest coffee price on {DATE}?
- What was the highest coffee price on {DATE}?
- What was the daily coffee price closing on {DATE}?
- What was the trading volume of coffee on {DATE}?
- What was the percentage change in coffee price on {DATE}, based on the difference between the opening and closing prices?
- Was {DATE} a bearish or bullish day for coffee price?
- What was the range of coffee price on {DATE}, based on the difference between the high and low prices?
C.3 Yelp
We design the following 11 templates for the Yelp dataset:
- What is the address of {NAME} in the area of postal code {POSTAL-CODE}?
- What city is {NAME} located in {STATE}?
- What state is {NAME} located in?
- What is the postal code of {NAME} in the area with postal code {POSTAL-CODE}, {CITY}, {STATE}?
- What is the star rating of {NAME} in the area with postal code {POSTAL-CODE}, {CITY}, {STATE}?
- How many reviews does {NAME} receive in the area with postal code {POSTAL-CODE}, {CITY}, {STATE}, received?
- Is  $\{\mathrm{NAME}\}$  still open in the area with postal code  $\{\mathrm{POSTAL-CODE}\}$ ,  $\{\mathrm{CITY}\}$ ,  $\{\mathrm{STATE}\}$ ?
- Does {NAME} require appointment in the area with postal code {POSTAL-CODE}, {CITY}, {STATE}?
- What are the hours of operation for {NAME} in the area with postal code {POSTAL-CODE},
{CITY}, {STATE}?
- What categories does {NAME} belong to, in the area with postal code {POSTAL-CODE}, {CITY}, {STATE}?
- What are the coordinates of {NAME} in the area with postal code {POSTAL-CODE}, {CITY}, {STATE}?
C.4 Airbnb
We design the following 10 templates for easy questions on Airbnb dataset:
- What is the host's name for {NAME} in {NEIGHBOURHOOD}?
- How many days are  $\{\text{NAME}\}$  (id:  $\{\text{ID}\}$ ) available during a year (365 days)?
- What is the room type of {NAME} (id: {ID}) in {NEIGHBOURHOOD}?
- What is the price of  $\{\text{NAME}\}$  (id:  $\{\text{ID}\}$ ) in  $\{\text{NEIGHBOURHOOD}\}$ ?
- What is the minimum number of nights for  $\{\text{NAME}\}$  (id:  $\{\text{ID}\}$ ) in  $\{\text{NEIGHBOURHOOD}\}$ ?
- When did {NAME} (id: {ID}) in {NEIGHBOURHOOD} constructed?
- How many reviews does {NAME} (id: {ID}) in {NEIGHBOURHOOD} have?
- What is the last review date for {NAME} (id: {ID}) in {NEIGHBOURHOOD}?
- What is the review rate number for {NAME} (id: {ID}) in {NEIGHBOURHOOD}?
- What is the average number of reviews per month for  $\{\text{NAME}\}$  (id:  $\{\text{ID}\}$ ) in  $\{\text{NEIGHBOURHOOD}\}$ ?
C.5 SciREX
We design the following 1 templates for easy questions on SciREX dataset:
- What is the corresponding {METRIC} score of the {METHOD} method on {DATASET} dataset for {TASK} task?
C.6 Agenda
We design the following 5 templates for easy questions on Agenda dataset: