DBQR-QA: A Question Answering Dataset on a Hybrid of Database Querying and Reasoning (2024)

Abstract: This paper introduces the Database Querying and Reasoning Dataset for Question Answering (DBQR-QA), aimed at addressing the gap in current question-answering (QA) research by emphasizing the essential processes of database querying and reasoning to answer questions. Specifically designed to accommodate sequential questions and multi-hop queries, DBQRQA more accurately mirrors the dynamics of real-world information retrieval and analysis, with a particular focus on the financial reports of US companies. The dataset’s construction, the challenges encountered during its development, the performance of large language models on this dataset, and a human evaluation are thoroughly discussed to illustrate the dataset’s complexity and highlight future research directions in querying and reasoning tasks.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 15169–15182 August 11-16, 2024 ©2024 Association for Computational Linguistics
This paper introduces the Database Querying and Reasoning Dataset for Question Answering (DBQR-QA), aimed at addressing the gap in current question-answering (QA) research by emphasizing the essential processes of database querying and reasoning to answer questions. Specifically designed to accommodate sequential questions and multi-hop queries, DBQRQA more accurately mirrors the dynamics of real-world information retrieval and analysis, with a particular focus on the financial reports of US companies. The dataset’s construction, the challenges encountered during its development, the performance of large language models on this dataset, and a human evaluation are thoroughly discussed to illustrate the dataset’s complexity and highlight future research directions in querying and reasoning tasks.

heading: 1 Introduction
text: Question answering (QA) is a fundamental task in the field of Natural Language Processing (NLP). Previous studies have primarily focused on textbased QA (Rajpurkar et al., 2016; Chen et al., 2021a; Gaim et al., 2023). Recently, attention has shifted towards tabular-based QA (Zhang et al., 2020a; Pal et al., 2023) and hybrid QA (Chen et al., 2020; Zhu et al., 2021; Chen et al., 2022). These studies base all experiments on the necessary tables provided as input. However, in real-world scenarios, answering questions like “What is the difference in revenue between Apple Inc. and Meta Platforms, Inc. in 2023?” (Q1) requires two steps: querying and reasoning. Specifically, models need to first query the revenues of the two companies in 2023 and then perform mathematical reasoning to answer the question. To address the gap in previous studies concerning the querying step, we propose the Database Querying and Reasoning Dataset for Question Answering (DBQR-QA).
Taking this a step further, people often ask sequential questions during discussions. For example, a follow-up to Q1 could be “Did that figure increase from 2022?” (Q2). To answer Q2, models must first identify the coreference (i.e., “that” refers to the difference in revenues between the two companies). The next steps involve querying the data for 2022, calculating the difference for that year, and comparing the answers to Q1 and Q2. Another follow-up question might be “Which company had the highest revenue in the Technology Sector in 2023?” This question, while related to Q1, does not require information from Q1 to answer. However, it demands a multi-hop querying capability, as it involves navigating from the industry to specific companies. This is considered a multi-hop query because the model must first identify the technology sector, then find companies linked to it. In DBQR-QA. This paper explores model capabilities in both scenarios.
Previous studies (Zhu et al., 2021; Chen et al., 2021b, 2022) have shown that company financial reports are excellent resources for investigating the proposed task. These reports contain numerous tables and figures, and investors and analysts frequently discuss them. Additionally, comparing reports between companies is a common practice. Following this rationale, we also base our dataset on the financial reports of US companies. Another advantage is that all these data are open access, enabling future research to extend our efforts in both querying and reasoning tasks. Table 1 provides a
15169
comparison of the proposed DBQR-QA with existing datasets. The TAT-QA dataset (Zhu et al., 2021) is designed for hybrid financial tabular and textual data, and the FinQA dataset (Chen et al., 2021b) targets numerical reasoning tasks in finance. ConvFinQA (Chen et al., 2022) introduces the concept of sequential questions based on the FinQA dataset. DBQR-QA advances this line of research by introducing querying tasks and multi-hop QA, moving closer to real-world application scenarios.
In this paper, we detail the construction of DBQR-QA and the challenges encountered in building this dataset. We also investigate the performance of large language models (LLMs) on this dataset and compare its difficulty with that of existing datasets. Furthermore, we provide a human evaluation to highlight issues in both querying and reasoning steps, identifying potential challenges for future research.

heading: 2 Related Work
text: Question-answering has posed a long-standing challenge for language models, prompting the proposal of numerous new datasets to explore and enhance the capabilities of these models. This section presents works associated with the three aspects of our dataset: Tables, reasoning, and conversation.
LLMs have made significant advances in reasoning QA, as demonstrated by the DROP dataset (Dua et al., 2019) for reading comprehension and arithmetic QA, the GSM-8K dataset (Cobbe et al., 2021) for grade-school math word problems, the MMLU benchmark (Hendrycks et al., 2021) for multiple-domain multiple-choice questions, and the NumHG (Huang et al., 2024) for numberfocused headline generation.
Tabular QA is another area that requires reasoning skills. Notable datasets in this field include the TAT-QA dataset (Zhu et al., 2021) for hybrid financial tabular and textual data, the FinQA dataset (Chen et al., 2021b) for numerical reasoning task in finance, and the FeTaQA dataset (Nan et al., 2022) for free-form table QA. Built on top of TAT-QA and FinQA, our dataset expands the scope of reasoning by incorporating querying and reasoning in problem-solving.
Numerous conversational QA datasets target large knowledge bases as they allow for diverse multi-hop questions. Some notable datasets in this domain include SQA (Iyyer et al., 2017) for Wikipedia tables, CSQA (Saha et al., 2018) for reasoning over a knowledge base, and ConvQuestions (Christmann et al., 2019) covering five domains. Non-KB QA datasets also pose significant challenges, for instance, CoQA (Reddy et al., 2019) for machine comprehension and QuAC (Choi et al., 2018) in dialog contexts. Despite extensive research on conversational QA for many years, its tabular and reasoning aspects still require further attention, as evident in ConvFinQA (Chen et al., 2022). This dataset explores the chain of numerical reasoning in single-table conversational QA. Our dataset raises the complexity in table manipulation, cross-question chain of reasoning, and multipletable QA from a financial knowledge graph.

heading: 3 Dataset Construction

heading: 3.1 Overview of DBQR-QA
text: Figure 1 presents an example of the newly proposed DBQR-QA dataset. Concurrently, Figure 2 provides a comparative view with examples from previously established datasets. It is noteworthy
that each question within the TAT-QA dataset only requires a single-step operation. The majority of questions in both the FinQA and ConvFinQA datasets necessitate the execution of one to four steps for calculation, invariably involving the use of two numbers sourced either directly from the data or derived from preceding steps. The proposed DBQR-QA dataset incorporates questions that require both database querying and complex multi-step table manipulations. These tasks are complicated further by a multi-branch chain of reasoning, where each question in the sequence introduces, alters, or eliminates queries, variables, operations, and parameters. This progressive complexity not only challenges the model’s capacity for memorization but also tests its ability to adapt and refine its logic throughout the conversation.
Here is an example of the question and gold label for this question in our dataset:
Question: During the period from 2016 to 2021, what were the top two years Itron reported the highest R&D Expense?
Gold label:

heading: 3.2 Question Preparation
text: The questions in the proposed DBQR-QA were sourced from the TAT-QA (Zhu et al., 2021) and FinQA (Chen et al., 2021b) datasets, which were manually crafted and annotated by financial experts. The limited variety of reasoning operations within these datasets resulted in many questions bearing similarities. We addressed this by grouping similar questions and developing a template-based representation. Specifically, we created 30 text-totemplate examples to identify financial concepts, temporal elements (e.g., years), numerical values, and units (e.g., millions) from a question, then utilized BART (Lewis et al., 2020) to extract these elements and generate templates. For instance, the question “What was the total intangible assets in 2019?” was abstracted to “What was the total [con-
cept] in [year]?” This abstraction process involved calculating string similarity scores, grouping the templates accordingly, and refining them to suit the graph database context, beyond mere tabular data.
Similar to ConvFinQA, our dataset transforms questions from FinQA into a conversational format but stands out by incorporating table manipulation throughout the reasoning process. Different from ConvFinQA which only uses six simple arithmetic operators, such as addition, subtraction, multiplication, and division, our operators include 26 operators in Pandas DataFrame. 1 This setting enables more complex and expressive queries than previous datasets. By leveraging Pandas’ extensive functionality, our approach also provides scalability for a wide range of applications.
Upon establishing the question templates, we populated them with entities (e.g., companies), financial concepts, and numerical data, aligning the financial terminology with the US-GAAP taxonomy2. We prioritized terms based on their occurrence frequency in the questions and selected those represented in the graph to ensure the correctness of the generated answers. We then defined a set of operations and combined them to formulate a program for each question, marking the initial stage of annotation.

heading: 3.3 Automatic-Answer Annotation
text: To utilize the responses annotated by financial experts in TAT-QA and FinQA, we leveraged Re-
1https://pandas.pydata.org/pandas-docs/stable/ reference/api/pandas.DataFrame.html
2https://www.sec.gov/info/edgar/ edgartaxonomies
portKG, a knowledge graph constructed from financial report tables formatted as XBRL documents3. This integration facilitates complex tasks requiring extensive data interlinkage by storing the data within the graph. The graph’s querying mechanism subsequently converts the results into tables, enabling manipulation during reasoning steps. Through ReportKG, automatic-answer annotations for generated questions are accessible. For instance, a question from TAT-QA, “How much revenue came from LinkedIn in 2018?” transforms into “How much profit came from Apple in 2023?” in our dataset. The annotation process in TAT-QA involves extracting a triple (revenue, LinkedIn, 2018) to respond to the question, whereas, in our context, the automatic-answer annotation comprises a triple (profit, Apple, 2023), thus providing a preliminary answer.

heading: 3.4 Answer Verification
text: We employed Amazon Mechanical Turk workers to verify our automatic-answer annotations. Their task was to read the questions and create a program (a series of tabular operations) using data queried from the database. The system then compared their program’s answer against ours. If there was a mismatch, the workers needed to explain which program – or whether the question – was incorrect. This approach mitigated the risk of our interpretation of the questions influencing theirs, as would have been the case if we had requested them to verify our programs.
To control the annotation quality, the eligible workers attained a minimum score of 70% across
3https://xbrl.us/
three qualification tests. Additionally, they offered adequate explanations for all answer discrepancies, demonstrating their ability to identify the issues. We considered a question valid by a majority consensus. We analyzed the workers’ feedback and determined the questions that had been identified as incorrect, such as those involving the possibility of a negative value measuring the “difference” between two quantities (e.g., how much different A is from B), necessitated additional clarification. We finalized minor revisions to the questions, completing our dataset construction.

heading: 4 Dataset Statistics
text: With advancements in NLP, zero-shot learning has emerged as a prominent method in the study of LLMs. Our proposed model, DBQR-QA, is specifically designed for this context, aiming to evaluate the ability of LLMs to generate code for querying and reasoning. Instead of creating a large dataset for supervised learning, we have developed a carefully constructed test set for precise evaluation. In the ConvFinQA benchmark, the test set consists of 434 instances. In line with these benchmarks, DBQR-QA provides a similar scale with 400 QA pairs.
The proposed DBQR-QA is categorized into five subsets based on question type and complexity, and it introduces a structured variety of question types aimed at examing querying and reasoning skills. These categories are crafted to delve into the complex aspects of financial datasets, focusing on distinct objectives and complexities. Below, an introduction to the five unique question types in our dataset is provided, alongside examples in Table 2
to demonstrate their nature and the analytical skills required.
Type 1: Simple Query with Specific Companies (Simple)
This type entails straightforward questions about specific companies, necessitating the direct extraction of data and basic arithmetic for solution derivation. A representative question might involve optimizing financial metrics over a specified period, like identifying a year to exclude to maximize a named insurance company’s average deferred revenues.
Type 2: Complex Query with Unspecified Companies (Complex)
Here, the complexity increases with the companies of interest not specified, incorporating conditional thresholds for financial metrics. The task involves selecting criteria to maximize or optimize a financial parameter among a group of companies. For instance, figuring out the year with the maximum average contractual obligations, based on varying minimum thresholds for purchase obligations.
Type 3: Reasoning Steps Requiring Multiple Tables (Multi-Table)
This category requires synthesizing data from multiple tables to answer questions involving comparative analysis or financial metric aggregation over various periods or conditions. It assesses the ability to navigate and reason with interconnected datasets, such as comparing average earnings per share across different years while accounting for basic versus diluted shares.
Type 4: Multi-hop Query (Multi-Hop) Multi-hop queries necessitate a sequence of logical steps and deductions to conclude. These questions often involve industry-level analysis, such as comparing averages or trends across different criteria or periods. An example question might ask which industry-level criterion results in a higher average net cash provided by operating activities, requiring knowledge of temporal trends and industry characteristics.
Type 5: Instruction QA (Instruction) Instruction-based questions present complex scenarios that guide the analyst through a series of data retrieval and analysis tasks across various dimensions, like time, industry, and financial metrics. These questions mimic real-world data analysis
tasks, requiring high comprehension and the ability to follow multi-step instructions to compare and contrast averages or identify trends among specific company groups.
These five question types collectively offer a thorough framework for querying and reasoning skills with financial datasets, spanning from simple arithmetic to intricate reasoning and multi-step problem-solving.

heading: 5 Experimental Setup

heading: 5.1 Task Design
text: In our experiment, we aim to ask LLMs to answer the given question by (1) querying data from ReportKG and (2) performing mathematical reasoning with the queried data. For querying, we ask LLMs to generate Cypher programs, where Cypher is a graph query language that lets the user retrieve data from the graph. For reasoning, we ask LLMs to generate Python programs by using the Pandas package for reasoning.4

heading: 5.2 Implementation Details
text: In our experiment, we benchmark the performance of Llama 2, PaLM 2, GPT-3.5 and GPT-4. Recognizing that multiple programs can be generated for a single question, we manually evaluate and execute each program to determine its correctness. We classify the outcomes into three categories: pass (producing the correct answer), fail (yielding an incorrect answer), and crash (inability to execute).

heading: 5.3 Manual Evaluation
text: This section describes the rules for manual evaluation. There are two types of answers: text and number. An answer can be a single value or a set (of texts or numbers). Textual answers can be a comparison (higher, lower, or equal) or entities, including financial concepts defined in the taxonomy, companies, persons, industries, and states. There are no other types of textual answers. A human evaluator must disregard other contextual output or any information apart from the answer, whether they are correct or not. If the answer is a set of values, the label and predicted sets must be identical (the orders do not matter), i.e., all values must be in the answer, and the answer must not contain any other values. If the set of values is of particular years or entities, e.g., a set of company revenues
4Please refer to Appendix H for the prompt.
during a period, the predicted values must explicitly describe all correct years or entities.

heading: 5.4 Automatic Evaluation
text: Previous works, such as TAT-QA, FinqA, and ConvFinQA, developed automatic heuristic evaluators to evaluate the outputs. However, the programs generated by LLMs through prompting may include additional information, making it more challenging to compare automatically with absolute certainty. We found three common types of such cases in our dataset. (1) The model outputs the correct answer but in different wording than the label. (2) The model outputs the context that may confuse a heuristic evaluator. (3) The model outputs its chain of thought (even without prompting specifically). To ensure the reliability of automatic evaluation, we implemented and tested three evaluators against human judgments.
The first metric is heuristic and serves as a baseline. The rules are as follows:
1. Single numeric answer: Detect a number in the prediction. There must be only one number (excluding years). The prediction is correct if it matches the label to the second decimal digit.
2. Set of numbers: The counts of the numbers in prediction and the label must be the same. If true, all numbers in the label must have unique matching pairs.
3. Year, a set of year, string, and a set of strings: All the years/strings in the label must be in the prediction.
The second metric compares the reference answer and the prediction using the BERTScore through the models’ embeddings (Zhang et al.,
2020b). We used Microsoft’s DeBERTa XLarge NMLI 5 as the base model, as recommended by the authors, due to its highest correlation with human evaluation. Given the reference answer and the prediction, the BERTScore algorithm computes the accuracy, precision, and F1 scores. We added a binary classification, where the output is 0 if the F1 score is less than 0.5, else 1.
The third metric employs GPT-4 as the evaluator. We first ask the model to compare the generated answer to human annotation (see Appendix B for the prompt). Next, we asked the model to score the answer by modifying the last part of the prompt to: “On a scale of 0 to 10, 0 = not at all and 10 = same, how similar are the two answers? Answer an integer number from 0 to 10 only. Do not explain or output anything else.”

heading: 6 Experimental Result

heading: 6.1 From Querying to Reasoning
text: Table 3 presents our findings. Notably, GPT-4 consistently outperforms other models regardless of question complexity. Additionally, our results demonstrate that the proposed DBQR-QA poses greater challenges to LLMs.
In order to take an in-depth look at the problems that LLMs face when addressing the questions in the proposed dataset, we manually checked the problems that occur in Simple and Complex questions. This testing involves analyzing the output exceptions. We also checked the results in the intermediate steps. The following are the common problems we found:
1. Logical error in the Python script (LP) or Cypher (graph) query (LC): This problem is due to the model misinterpreting the question or making mistakes in its reasoning steps.
5https://huggingface.co/microsoft/ deberta-xlarge-mnli
2. Syntax or runtime error in the Python script (SP) or Cypher query (SC): This problem encompasses various instances, such as the model misunderstanding the languages’ syntax, the code referring to undeclared variables, and the model misinterpreted data structure from the queries or prior operations.
3. Negative output from the difference operation (D): The program outputs a negative answer when asked for differences between numbers.
4. Inverse division (ID): Reversed numerator and denominator in division.
5. Incorrect entity (IE): The query or code refers to irrelevant entities.
6. Manual input (MI): Require manual input.
7. Null handling (N): Fail to handle NULL.
8. No code generated (NC)
Table 4 details the distribution of each error type. It’s worth noting that models exhibit a significant number of errors when querying tables and data using Cypher, while fewer mistakes occur in result calculations.

heading: 6.2 Results of Reasoning
text: From our error analysis in Section 6.1, it is evident that table queries, denoted by LC in Table 4, represent the predominant error type among the ten identified. In this section, we explore a modified experimental setting where models are not required to query tables but are directly provided with tabular data. This adjustment allows us to better assess a model’s ability to generate programs for calculations without the necessity of table querying.
Table 5 shows that model performances improve across all metrics, regardless of the specific model or the complexity of the dataset subset in use. This improvement highlights the frequent failures attributed to incorrect table-querying programs. Notably, the accuracy of GPT-4 with the Complex subset significantly surpasses its performance when tasked with generating a complete program, emphasizing the critical role of correct data querying. However, GPT-4’s performance remains consistent between the Simple and Complex subsets for raw calculations. This consistency suggests that
our dataset does not introduce unrealistic or overly complex calculations, aiming instead to replicate real-world tasks commonly encountered in professional settings, from data querying to calculations.

heading: 6.3 Automatic Evaluation
text: We tested our three automatic evaluators against human annotation. Table 6 shows the models’ performances on the two tasks. The numbers in parentheses are the differences between the models’ predictions and human-annotated labels. Across all the experimental settings, BERTScore is much less predictive of human evaluation than the baseline and GPT-4. We found that the embeddings of numbers are similar to each other regardless of whether or not the numbers are the same. This issue led to the model outputting high F1 scores for incorrect numeric answers. While the baseline evaluator is highly accurate when most answers are wrong, its accuracy drastically drops as it fails to recognize correct answers.
Table 7 summarizes the metrics’ performances by accuracy, precision, recall, and F1 score for binary classification and root mean squared error (RMSE) for continuous score values. The groundtruth values for RMSE are 0 for human-annotated labels indicating incorrect prediction and 1 for the correct ones. Based on the results shown in both tables, we conclude that GPT-4 is the best approach for automatic evaluation due to its low inaccuracy.

heading: 7 Discussion
text: Based on our experimental results, GPT-4 exhibits satisfactory performance in both Simple and Complex subsets. However, its performance is inferior in the remaining subsets. Therefore, in this section, we focus on discussing the Simple and Complex subsets to understand the nuances of LLMs’ performances on the proposed dataset.

heading: 7.1 Coreference Resolution
text: A pivotal facet of conversational question answering is the model’s ability to undertake coreference resolution. For instance, when presented with a
follow-up question, “What about 2021 and 2022?” succeeding the query “Between 2019 and 2020, what was the change in Gibraltar Industries gross profit?”, models must discern that the inquiry pertains to the “gross profit”. To delve into the efficacy of models in this dimension, we executed a turn-based evaluation, the outcomes of which are illustrated in Figure 3.
The data suggests that GPT-3.5 excels during the initial turn of conversation but its performance tapers off in subsequent turns. Conversely, GPT4 showcases commendable results during the first five turns, after which there’s a discernible decline. These observations highlight two key points: firstly, coreference resolution remains a formidable challenge for LLMs. Secondly, the mean turn count for the existing dataset, ConFinQA, stands at a mere 3.67. This duration is inadequate to rigorously assess models like GPT-4. From this vantage point, our proposed DBQR-QA emerges as a more robust benchmark for future studies.

heading: 7.2 Fine-Grained Question Type Analysis
text: We further employ two strategies to categorize the questions into more fine-grained categories. The first method classifies the questions according to the functions adopted by their solution programs,
whereas the second method depends on the judgment of the annotators.
We grouped functions defined for this dataset into six groups as follows:
1. Querying functions (QUER) consist of two functions: One for querying facts regarding a specific company and the other for querying facts associated with nonspecified companies based on particular criteria.
2. Logical functions (LOGI) consist of nine functions for table manipulation. These operations include intersection, union, transpose, conditional replacement, conditional filtering, comparison, merging, excluding, and column stacking.
3. Ordering functions (ORDR) consist of five functions: sorting, reversing the table along the row or column axis, selecting rows or columns, selecting the nth element along the axis, and selecting the first or last k items.
4. Arithmetic functions (CALC) consist of seven functions: addition, subtraction, division, absolute operation, multiplication by a constant, subtraction by a constant, and division by a constant.
5. Aggregation functions (AGGR) consist of two functions: averaging and counting.
6. Listing functions (LIST) consist of two functions: One lists the table’s header, and the other indicates an empty output.
The annotators’ judgment aims to provide a more fine-grained analysis and consists of additional categories as follows:
1. SORT for questions that involve sorting numbers by financial concepts, companies, industries, years, among others.
2. HYPO for questions that ask for calculation in a hypothetical scenario. For example: “What if 2018 was twice what was reported, how much more can be added to 2022 to maintain the 50% margin?”
3. COMP for questions that ask to compare two or more numbers along one axis of a table, for example, companies’ revenues in the year of interest compared to the year prior.
4. NEGA for questions that involve negation, i.e., “not.” For example, “which years can I remove that will not increase the ratio?”
5. DECP for questions that intentionally deceive the model, for example, by referring to a financial concept unrelated to the one asked in the conversation.
6. ELAB for questions that ask to clarify the calculation. For example, “which companies are included in the calculation [of the previous question]?”
We delved deeper into the performance of models across various question categories. The outcomes of this analysis are illustrated in Figure 4. Notably, GPT-4 surpasses its counterparts in the majority of question categories. However, given that there are a mere three questions categorized under Elaboration (ELAB), the differentiation in performance for this type is marginal. Within the array of question types, GPT-4 showcases superior performance on DECP (Deceptive), QUER (Complex Queries), and COMP (Comparison) categories. The annotations for these question types will be included with the proposed MTCQA dataset, offering a foundation for future research to develop specialized solutions targeting each distinct question category.

heading: 7.3 Human Performance
text: Table 8 presents the performance comparison of GPT-4 and humans on existing datasets and the proposed DBQR-QA dataset. The results indicate that accurately answering queries in the DBQR-QA dataset poses greater challenges for both GPT-4 and humans, highlighting the need for future research to enhance performance in either the querying or reasoning aspects of the dataset.

heading: 8 Conclusion
text: By introducing a comprehensive dataset tailored to evaluate the querying and reasoning capabilities of LLMs, the DBQR-QA dataset challenges models with real-world scenarios that require both database queries and multi-step reasoning processes, reflecting the complexities encountered in professional financial analysis. Experimental results reveal that while LLMs like GPT-4 demonstrate potential in certain subsets, significant challenges remain, particularly in subsets requiring advanced reasoning and multi-step querying capabilities. This observation underscores the ongoing need for improvements in models and specialized training to bridge the gap between current capabilities and the demands of real-world data analysis tasks. Lastly, although our dataset focuses on the financial domain, interested researchers and developers can apply the creation and evaluation processes to use cases where tabular reasoning over a database can benefit question-answering. Companies and organizations can also apply our approach to build a QA system to conduct complex analyses of their databases for various purposes, for example, inventory/logistic/resource management, fraud/irregularity detection, system optimization, and supporting information for policy/decisionmaking. In addition to the dataset, all fine-grained annotations for analysis will be shared under the CC BY-NC-SA 4.0 license.6
Limitation
The paper’s limitations can be summarized as follows: First, while our study adopts financial data in line with prior research, this specificity means that our findings are predominantly tailored to financial documents. Explorations into other domains rich in database, such as scientific research or the clinical realm, would be beneficial in subsequent studies.
6Research materials are available at: https://www.ai. iee.e.titech.ac.jp/DBQR-QA/
Second, our primary emphasis is on assessing the capabilities of LLMs in a zero-shot setting. Future investigations might consider broadening the scope to explore our dataset under varied experimental paradigms. Third, consistent with prior works, our dataset is confined to English. It’s worth noting that LLMs might exhibit different behaviors across different languages, and we advocate for this consideration in future endeavors.

heading: Acknowledgement
text: This study is partially based on the results obtained from a project JPNP20006, commissioned by the New Energy and Industrial Technology Development Organization (NEDO).

heading: A Question Generation
text: The questions are all human-generated. However, we used BLOOM 7B7 to find common financial concepts and question templates in the TAT-QA and FinQA datasets. This data helps us decide what concepts and types of questions we should include in the annotation. We gave BLOOM 30 examples for each task and ran the model on all samples in the datasets. Table 9 lists some examples we used. We then search the US GAAP and company-defined taxonomies for the extracted concepts. Table 10 shows the top five concepts found in both datasets. Table 11 shows the top five templates that start with “what,” which account for 82.60% of all questions.
7https://huggingface.co/bigscience/bloom-7b1

heading: B Automatic Evaluation
text: We used the following prompt for our GPT-4 evaluator, which compared the models’ outputs to human annotation:

heading: C Fine-grained Question Type
text: Table 12 and Table 13 show the statistics of the simple and complex sections of the dataset. Note that a question could fit into several fine-grained categories.

heading: D Operations
text: Table 14 summarizes DBQR-QA’s operation extension (n=26) compared to the TAT-QA (n=10), FinQA (n=10), and ConvFinQA (n=6) datasets, where n represents the number of operations.
E Instructions for Annotators
The description of the task given to the annotators is as follows:
1. Read a series of 5 - 10 conversational questions.
2. Build a program that performs numerical operations for each question using our annotation tools.
3. Locate numbers in a table that represent financial concepts mentioned in the questions.
4. Check our pre-annotated program against your submission.
The requirements for the Amazon Mechanical Turk workers are as follows:
1. Basic mathematical reasoning over tabular data (similar to applying formulas in Microsoft Excel).
2. Basic understanding of financial concepts.
3. Latest version of Google Chrome, Safari, or Microsoft Edge running on a desktop PC.
The completion time estimation given to the annotators is as follows:
1. If this is your first time seeing this task, you’re about to take the first test, which:
• consists of 5 questions, • requires you to watch a 4-minute training video
and read additional instructions, • gives you hints on how to build programs, and • takes about 40 minutes on averageto complete.
2. If not, it’s either:
• your second test (5 questions) or • your third test or the actual annotation task (10
questions).
Table 15 summarizes the completion time estimation, also shown to the annotators.
The rejection criteria given to the annotators for assignment submission are as follows:
1. Not giving a clear explanation when prompted (for example, we’ll ask you to identify and explain the problems when your program and ours do not match)
2. Giving up multiple times with no reasonable explanation
3. Having no intention to complete the task (judging by the programs you generated and your explanation)

heading: F Performance Analysis
text: In addition to the model’s performance analysis by question type (defined using human judgment) in Section 7.2, we have conducted an additional analysis that measures the model’s performance by function type, as shown in Table 16.

heading: G GPT-3.5 and GPT-4
text: We evaluated the first two types of questions in our dataset in October 2023. The February 2024 version of the GPT models produced drastically different answers, resulting in significant performance drops. This change in the models’ behavior has been studied previously (Chen et al., 2023), with the results indicating varying degrees of performance changes that can be substantial in some tasks, including mathematics and code generation.

heading: H Prompt for Program Generation
text: The following is the part of the prompt we used for all LLMs tested:

heading: I AI Assistant
text: We utilized GPT-4 to assist with our research and writing. However, the content remains original, and we meticulously reviewed and revised the tool’s output prior to publication.

heading: Future Work
text: Future investigations might consider broadening the scope to explore our dataset under varied experimental paradigms. Third, consistent with prior works, our dataset is confined to English. It’s worth noting that LLMs might exhibit different behaviors across different languages, and we advocate for this consideration in future endeavors