SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuchen Zhuang*, Yue Yu*, Kuan Wang*, Haotian Sun, Chao Zhang
College of Computing, Georgia Institute of Technology, Atlanta GA
{yczhuang, yueyu, kuanwang, haotian.sun, chaozhang}@gatech.edu
Large Language Models (LLMs) have demonstrated impressive performance in various NLP tasks, but they still suffer from challenges such as hallucination and weak numerical reasoning. To overcome these challenges, external tools can be used to enhance LLMs' question-answering abilities. However, current evaluation methods do not distinguish between questions that can be answered using LLMs' internal knowledge and those that require external information through tool use. To address this issue, we introduce a new dataset called ToolQA, which is designed to faithfully evaluate LLMs' ability to use external tools for question answering. Our development of ToolQA involved a scalable, automated process for dataset curation, along with 13 specialized tools designed for interaction with external knowledge in order to answer questions. Importantly, we strive to minimize the overlap between our benchmark data and LLMs' pre-training data, enabling a more precise evaluation of LLMs' tool-use reasoning abilities. We conducted an in-depth diagnosis of existing tool-use LLMs to highlight their strengths, weaknesses, and potential improvements. Our findings set a new benchmark for evaluating LLMs and suggest new directions for future advancements. Our data and code are freely available for the broader scientific community on GitHub  $^{2}$ .
Large Language Models (LLMs) have demonstrated superior performance in a myriad of NLP tasks [4, 9, 43, 42, 55, 61]. These models have captured vast amounts of knowledge from enormous and diverse corpora during pre-training. After instruction fine-tuning [10, 44, 2], they have demonstrated impressive capabilities in information-seeking question answering [65, 26]. Despite their remarkable performance, LLMs face several challenges. For example, they are susceptible to hallucinations—generating plausible yet ungrounded information—which can mislead users and affect content integrity [66, 19, 5]. Additionally, they exhibit weaknesses in numerical reasoning, an essential skill in numerous real-life applications [14, 36, 41, 28, 51, 13]. These limitations highlight the need for techniques that can enhance LLMs' question-answering abilities.
Recent research has shown that these issues can be mitigated by augmenting LLMs with external tools, such as retrieval augmentation [58, 17], math tools [56, 76, 32], and code interpreters [13, 63]. For example, a Wolfram math plugin can enhance numerical reasoning [68], and a verified database can mitigate hallucinations by providing up-to-date fact-checked knowledge [49]. However, existing evaluation methodologies struggle to distinguish whether the model is simply recalling pre-trained information or truly utilizing external tools for problem-solving [37]. This challenge arises, in part, because the external data used for evaluation may have already been exposed to LLMs during the pre-training phase [53]. This exposure can lead to a biased evaluation of LLMs' tool-use abilities, as the models could just use their ingrained knowledge and their reasoning abilities, bypassing the use of external tools. As a result, these evaluations cannot accurately reflect the true competency of the
2https://github.com/night-chen/ToolQA
Figure 1: Pre-trained on vast range of corpus, LLMs possess extensive knowledge, which may overlap with evaluation data. This overlap poses a significant challenge to current evaluation methods, as it becomes difficult to discern whether the model is merely recalling pre-trained information or genuinely employing external tools for problem-solving.
models. We need a fair and explicit way to check if LLMs are really good at problem-solving with tools or if they are just using their memorized information.
To fill this gap, we introduce ToolQA, a question answering (QA) benchmark to evaluate LLMs' ability in using external tools for answering questions. ToolQA comprises data from 8 domains and defines 13 types of tools to acquire information from external reference corpora. Each instance in ToolQA consists of a question, an answer, reference corpora, and a list of available tools. ToolQA is unique in that all its questions can be answered only by using appropriate tools to obtain information from the reference corpus. This minimizes the possibility of LLMs answering questions by merely recalling their internal knowledge, and allows for faithfully evaluating LLMs' abilities in using tools.
ToolQA is curated with an automated three-phase process: (1) The first phase, *Reference Data Collection*, involves gathering various types of public corpora including text, tables, and graphs from different domains. These corpora have no overlap with the LLM pre-training data and will serve as reference corpora for tool-based question answering. (2) The second phase is *Human-guided Question Generation with LLMs*. In this phase, we generate questions that can only be answered by using tools over the reference corpora. Our approach is a template-based question generation process, which includes human-guided template generation, template validation, and question instantiation with tool attributes. (3) The third phase is *Programmatic Answer Generation*. This phase produces accurate answers for the generated questions. To ensure answer correctness, we implement operators corresponding to the tools and obtain answers from the reference corpora programmatically. Our three-phase procedure ensures that we generate questions that can only be answered using external knowledge, along with their precise answers. Additionally, the process is highly efficient and requires minimal human labeling efforts.
We conducted experiments using both standard LLMs and tool-augmented LLMs to answer questions in ToolQA. Our findings indicate that ChatGPT and Chain-of-thoughts prompting [65], which rely solely on their internal knowledge, have low success rates of approximately  $5\%$  for easy questions and  $2\%$  for hard questions. In contrast, tool-augmented LLMs such as Chameleon [32] and ReAct [76] perform better by leveraging external tools. For easy questions, the best performance achieved by tool-augmented LLMs is  $43.15\%$ , while for hard questions, the best performance drops to  $8.2\%$ . Our results and error analysis demonstrate that ToolQA is a challenging benchmark for existing tool-augmented LLM methods, especially for its hard questions that require more complex reasoning about tool composition.
2.1 Knowledge-Augmented LLMs
Several prior works aim to enhance LLMs with explicit external knowledge. Specifically, one line of research focuses on retrieval-augmented language models [58, 3, 17, 27, 30, 80, 34, 72], where they use sparse [54] or dense retrieval [22, 16] to extract relevant knowledge from the corpus. These works mainly focus on leveraging free text, without considering multiple types of tools for task solving. On the other hand, Program-of-Thought [6], PAL [13], MathPrompt [15], and
(a) Reference Data Collection
(c) Programmatic Answer Generation
(b) Human-Guided Question Generation
Figure 2: ToolQA, aiming to faithfully evaluate LLMs' abilities to use external tools, curates data through three phases: (a) Reference Data Collection; (b) Human-Guided Question Generation; and (c) Programmatic Answer Generation.
Flight Data Question Templates:
- Did the flight from {Origin} to {Dest} on {Date} get cancelled or diverted? (External Knowledge)  
- What was the flight distance for the flight from {Origin} to {Dest} on {Date}? (Internal Knowledge)  
- Which product on {FlightNumber} has the highest price? (Not Mentioned)
A: def question_gen(table_row): Origin  $=$  table_row["Origin"] Dest  $=$  table_row["Dest"] FlightDate  $=$  table_row["FlightDate"] return question, answer
Code4Struct [63] apply code-based tools to enhance LLMs' abilities in question answering with a focus on tabular and math-related tasks. Several additional works [56, 32, 57] expand the scope of tool utilization by incorporating different types of basic tools (e.g. calculator, calendar, machine translation) to solve complex reasoning tasks. To synergize different functional tools together for problem-solving, LLMs must have advanced planning and memory capabilities. In terms of planning, current methods either enable LLMs to autonomously break down complex tasks into intermediate reasoning steps [45, 24, 47, 50, 65, 75, 31, 76], or encourage LLMs to self-reflect the previous decisions with environmental feedback [39, 76, 35, 8]. Memory capabilities, on the other hand, provide LLMs with opportunities to learn and adapt based on past experiences, whether successes or failures [62]. In addition, several works have extended this line of learning paradigm to other modalities [73, 69] and other domains [20]. Concurrent to our work, there are also several studies [70, 37] that investigate the parametric and nonparametric knowledge from LLMs. A detailed comparison between existing tool-use LLMs can be found in Appendix A.
2.2 Benchmarks on Tool-Augmented LLMs
Earlier tool-augmented LLMs primarily assess single tool usage based on downstream task performance across existing benchmarks. For example, there are works that study how text retrievers augment LLMs' performance on open-domain question-answering [21, 74], fact-checking [60], and timely information benchmarks [7, 23, 78, 12]. Besides, the mathematical reasoning abilities of external calculators and Python interpreters are evaluated using computation-intensive QA datasets [11, 33]. However, these evaluation benchmarks may not faithfully reflect the extent to which models leverage external tools, as some questions could still be correctly answered solely using the internal knowledge of the LLMs. ToolQA attempts to mitigate these issues by selecting data from out-of-scope sources that have not been memorized by LLMs. Concurrent with our work, there are several recent benchmarks for evaluating LLMs' ability in using multiple tools for solving challenging tasks, including API-Bank [29], APIBench [48], and ToolBench [52, 71]. They mainly focus on constructing high-quality tool chains for LLM fine-tuning and evaluating API call trace accuracy against a fixed ground truth trace. In contrast, ToolQA is unique in that it focuses on the open-ended use of tools for question-answering, rather than benchmarking the intermediate process of tool use. Specifically, ToolQA creates tool-based question-answer pairs and assesses whether LLMs can arrive at the correct answer, regardless of the tool chains used.
3 ToolQA Dataset
We curate the ToolQA benchmark to evaluate LLMs' capability in leveraging external tools for question answering. ToolQA consists of data from 8 distinct domains, each instance being a tuple — (question, answer, reference corpora, and tools). The reference corpora are external knowledge sources that can be queried, which can be a text corpus, a tabular database, or a graph. To enable obtaining information from the reference corpora, we have developed 13 tools for text retrieval, database operations, code interpretation, mathematical computations, and more. The questions are designed to simulate real-world information-seeking inquiries. However, they cannot be answered directly with LLMs' internal knowledge, but instead require LLMs to obtain information from the reference corpora via tool use. Table 1 shows the detailed statistics of ToolQA.
To reduce human efforts in generating faithful question-answer pairs to evaluate LLMs' tool-use capabilities, we propose an automatic three-phase process (Figure 2): (1) We first select data from public sources that are unmemorized by LLMs during Reference Data Collection; (2) We adopt Human-Guided Question Generation to steer LLMs to generate valid questions according to predefined templates; (3) We produce accurate answers for the generated questions with Programmatic Answer Generation. We detail the three-phase generation process in the following.
3.2 Reference Data and Tools
To evaluate LLMs' ability in using external tools for question answering, it is crucial to ensure that they cannot directly answer the questions with their internal knowledge. To this end, we collect reference corpora that meet the following criteria (Figure 2(a)): 1) The reference corpora should ideally not overlap with the LLM's pre-training data; 2) The reference corpora should contain context-sensitive facts for generating questions that cannot be directly answered solely based on LLMs' internal knowledge and reasoning abilities; 3) LLMs should be able to obtain all the necessary information from the reference corpora to correctly answer the questions.
Based on these criteria, we define 6 contextual dimensions: temporal, spatial, social, scientific, mathematical, and personal. We collect reference corpora that can yield context-specific questions along one or more of the 6 dimensions. Specifically: 1) Along the temporal dimension, we collect the Flights and Coffee corpora, which contain the latest information that is out of the temporal scope of the LLM's pre-training data. 2) Along the spatial dimension, we collect Yelp and Airbnb, which are two non-text corpora that can yield questions with spatial contexts. 3) Along the mathematical dimension, we collect the questions from GSM8K that ChatGPT cannot answer correctly with its own mathematical reasoning ability; 4) SciREX emphasizes detailed model performances from the scientific domain [18], where GPT family models can easily hallucinate [42]. 5) To incorporate personal data and avoid privacy issues, we synthesize the personal Agenda corpus with ChatGPT with virtual names and events. 6) In addition, we also select data from the most recent DBLP database and create graphs between authors and papers, where social relational knowledge cannot be understood by LLMs currently. Further details can be found in Appendix B.
To obtain information from these reference corpora, we design 13 tools that are available to the LLMs (Table 2). These tools are designed as follows:
Table 1: Dataset Statistics of ToolQA.
Context	Topic	External Knowledge	Easy	Hard
Format	Size	# Templates	# Questions	# Templates	# Questions
Temporal	Flight	Tabular Database	4078318	10	100	10	100
Coffee	Tabular Database	5746	8	100	13	130
Spatial	Yelp	Tabular Database	150346	11	100	10	100
Airbnb	Tabular Database	102599	10	100	10	100
Mathematical	GSM8K	Professional Ability	-	-	100	-	-
Social	DBLP	Graph	553320	10	100	10	100
Scientific	SciREX	Pure-Text Corpus	438	1	100	4	100
Personal	Agenda	Pure-Text Corpus	10000	5	100	5	100
SUM	-	-	-	55	800	62	730
Table 2: Different tools in ToolQA.
Tool Types	# Tools	Tools
Text Tools	2	Agenda Retriever, SciREX Retriever
Database Tools	3	Database Loader, Data Filter, Get Value
Math Tools	1	WolframAlpha Calculator
Graph Tools	4	Graph Loader, Neighbour Checker, Node Checker, Edge Checker
Code Tools	2	Python Interpreter, SQL Interpreter
System Tools	1	Finish
- Text: AgendaRetriever and SciREXRetriever are text retrieval tools. They can retrieve relevant information to a given query from the (synthesized) personal agenda corpus and scientific corpus.
- Database: Database Loader loads data from the local tabular Database. Data Filter can filter the database according to a set of conditions, each of which is composed of a column name, a relation, and a pre-determined value (e.g., "Date=2022-10-15"). Get Value returns all the values under a certain column in the database.
- Math: Calculator is a mathematical tool that treats the input string as a formula and calculates the corresponding result. We use the WolframAlpha API portal as the calculator<sup>3</sup>, which can perform both simple computations (e.g., add, subtraction, multiplication) and complicated operations (e.g., averaging, finding maximum values).
- Graph: Graph Loader loads the graph from local files for future operations. Neighbour Checker lists all the neighbors of the query node in the graph. Node Checker and Edge Checker return the detailed attribute information of the query node and edge, respectively.
- Code: The SQL Interpreter and the Python Interpreter are responsible for interpreting and executing SQL commands and Python code, respectively. They can receive and transform data from other tools, serving as bridges between different tools and the LLM.
- System: Finish parses the feedback from execution and returns the answer to finish the task.
3.3 Human-Guided Question Generation
The question generation phase aims to generate questions that can be answered by using the available tools over the reference corpora. There are two straightforward strategies to generate questions: 1) letting human experts come up with questions about reference corpora, or 2) relying solely on LLMs to generate questions about the reference corpora. However, both strategies have their drawbacks. While human experts can produce high-quality questions, the entire process is labor-intensive, time-consuming, and hard to scale. Depending solely on LLMs may generate unanswerable questions or hallucinate information that does not exist in the reference data. Besides, some of the LLM-generated questions are too easy and can be directly answered with only LLMs' internal knowledge.
To address these challenges, we propose a human-guided LLM generation approach that uses question templates to bridge human guidance and automatic LLM generation [67, 79]. We first ask ChatGPT to generate candidate question templates from reference data, using prompts such as "Generate diverse and challenging template questions that users may have based on the given information." To obtain diverse questions, we generate around 50 template questions for each external data source. We then perform manual validation to select the templates that cannot be answered with LLMs' internal knowledge but become answerable with the reference corpora. We go through all question templates and eliminate those that meet either of the following conditions: (1) Template questions that vanilla ChatGPT can answer based on its internal knowledge with a success rate of over  $50\%$  (e.g., "What is the distance between LAX and SFO?"), where distance information can be memorized by ChatGPT); (2) Template questions posing queries about information not present in the external data (e.g., "What is the average price from LAX to SFO?"), where price information is missing from the flight data). After examining all the templates, we selected 117 most representative and diverse question templates for the entire ToolQA dataset.
After the high-quality question templates are manually selected, we sample values from the reference data to automatically fill into the templates to generate concrete questions. For example, given the template "Did the flight from {Origin} to {Dest} on {Date} get canceled or diverted?", we can sample the values "LAX", "MDW", "01/09/22" from the reference Flight tabular data and fill into the template to form a question: "Did the flight from LAX to MDW on 01/09/22 get canceled or diverted?"
<sup>3</sup>https://products.wolframalpha.com/api
Depending on the difficulty of the questions, we classify them into two classes — easy and hard. Easy questions primarily focus on extracting a single piece of information from external knowledge, thus requiring fewer tools to involve in the solution. Conversely, hard questions require complex operations (e.g., average) and reasoning (e.g., comparison) over multiple information pieces drawn from the reference corpora, requiring more tools and complex reasoning among them. We provide a comprehensive list of both easy and hard question templates in Appendix C and D.
3.4 Programmatic Answer Generation
Our final step is to create accurate answers for the generated questions. To guarantee the validity of these responses, we implement 1) operators, which are functions corresponding to the predefined tools; and 2) tool chains, which are schemas for composing different operators for different question templates. For each question, as we know the true arguments filled into the question template, we can run the tool chains with the corresponding arguments to programmatically extract answers from the reference data. This process enables automatic generation correct answers to questions, even for those questions that involve multi-step reasoning. Figure 2(c) demonstrates this generation process. When answering a generated question with sampled values "Did the flight from LAX to MDW on 01/09/22 get canceled or diverted?", we write Python codes to implement the operators over the reference data, including database loader, data filter, and get-value function. Then, the programmatic pipeline runs a tool chain of these operators to automatically generate the correct answer (details in Appendix E).
We evaluate the performance of the following methods on ToolQA, covering both standard LLMs and tool-augmented LLMs: (1) LLaMA-2 [61] and Falcon [1] are state-of-the-art open-sourced large language models. We directly feed the questions into two versions (13B and 70B) of LLaMA-2 and Falcon (40B) to obtain the predictions; (2) ChatGPT [43]: We directly feed the questions into OpenAI's ChatGPT model (gpt-3.5-turbo) and obtain its response as the final answer. (3) CoT [65, 26]: We use chain-of-thoughts prompting for ChatGPT, adding the prompt "Let's think step by step:" after the question to leverage LLMs' reasoning ability for question answering. (4) Chameleon [32] is a recent method that uses LLMs as a controller to use multiple tools for solving subtasks and has shown promising results in reasoning and QA tasks. When running Chameleon on ToolQA, we set the tool pool to our defined tools in § 3.1. (5) ReAct [76] integrates reasoning with tool use by prompting LLMs to generate interleaved verbal reasoning traces and tool calls. This integration has been shown effective in enhancing LLMs' problem-solving capabilities. We instantiate two versions of ReAct using gpt-3.5-turbo and text-davinci-003.
Different from the existing works that mainly provide task-level few-shot exemplars, we provide tool-level demonstrations. We used 8 demonstrations about how to use tools for QA, ensuring that each tool in the pool is covered at least once by the demonstrations. Such tool-level demonstrations provide a concise tutorial to the LLMs for tool use, covering all tool uses with the LLM context limit. Details about the demonstrations and our prompts are included in Appendix F. To assess the performance of methods on the ToolQA benchmark, we normalize both the ground-truth answers and the model predictions to ensure uniformity in format. Success rates are then computed based on the exact match between these normalized answers. We use a series of rules for normalization: (1) We normalize different time string formats, (e.g., converting "18:06" and "1806.0" to "1806"). (2) For price-related questions, we normalize the units by removing price units (e.g., USD, $); (3) We remove all the punctuations from both the model predictions and ground-truth answers; (4) We normalize the article usage (e.g., a, an, the) via removing all articles from both the model predictions and ground-truth answers; (5) We normalize the white spaces by trimming multiple spaces into single space. As most of the predictions and answers are numerical values or entities, these normalization rules address most of the false negative cases during matching. We evaluate the model's ability against the generated question-answer pairs in an open-ended manner, focusing on whether the model can arrive at the correct answer, regardless of the used tool chains.
Internal Knowledge vs. External Knowledge. From the results in Tables 3 and 4, the vanilla open-sourced LLMs and ChatGPT underperform their tool-augmented counterparts on both easy and hard questions. This is expected, as vanilla LLMs lack access to external information for question