DOCMATH-EVAL: Evaluating Math Reasoning Capabilities of LLMs in Understanding Financial Documents (2024)

Abstract: Recent LLMs have demonstrated remarkable performance in solving exam-like math word problems. However, the degree to which these numerical reasoning skills are effective in realworld scenarios, particularly in expert domains, is still largely unexplored. This paper introduces DOCMATH-EVAL, a comprehensive benchmark specifically designed to evaluate the numerical reasoning capabilities of LLMs in the context of understanding and analyzing financial documents containing both text and tables. We evaluate a wide spectrum of 27 LLMs, including those specialized in math, coding and finance, with Chain-of-Thought and Programof-Thought prompting methods. We found that even the current best-performing system (i.e., GPT-4) still significantly lags behind human experts in solving complex numerical reasoning problems grounded in long contexts. We believe DOCMATH-EVAL can be used as a valuable benchmark to evaluate LLMs’ capabilities to solve challenging numerical reasoning problems in expert domains. § github.com/yale-nlp/DocMath-Eval

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16103–16120 August 11-16, 2024 ©2024 Association for Computational Linguistics
§ github.com/yale-nlp/DocMath-Eval

heading: 1 Introduction
text: Recent advancements in Large Language Models (LLMs) have attracted significant attention due to their capabilities in solving a broad range of tasks (OpenAI, 2022, 2023; Touvron et al., 2023), including math word problems (MWPs) commonly found in academic exams (Wang et al., 2017; Miao et al., 2020; Amini et al., 2019; Cobbe et al., 2021; Hendrycks et al., 2021a; Cobbe et al., 2021; Lu et al., 2023b; Chen et al., 2023b). These MWPs vary from basic arithmetic to advanced algebra, showcasing LLMs’ proficiency in numerical reasoning — a crucial skill for interpreting and manipulating numerical data across various contexts. Despite this progress, there is still a significant gap
∗Equal Contributions.
in understanding the practicality of LLMs’ numerical reasoning in real-world scenarios, particularly in specialized fields such as finance, medicine, and science. As illustrated in Figure 1, these expert domains necessitate LLMs to interpret complex, domain-specific documents, applying numerical reasoning to complex problem-solving (Chen et al.,
16103
2021; Zhu et al., 2021; Zhao et al., 2022; Li et al., 2022). Recognizing this gap, our research focuses on the finance domain (Wu et al., 2023a; Yang et al., 2023b; Callanan et al., 2023). The finance industry often deals with lengthy and data-intensive documents that demand advanced numerical reasoning skills for accurate analysis and decision-making.
We introduce DOCMATH-EVAL, a comprehensive and standardized benchmark that systematically evaluates the numerical reasoning capabilities of LLMs in understanding and interpreting financial documents containing both textual and tabular data. DOCMATH-EVAL encompasses four evaluation sets, each with varying levels of difficulty in numerical reasoning and document understanding. Specifically, We construct a new evaluation set, DMCompLong, from scratch, to examine the LLM’s capabilities in performing complex numerical reasoning over extreme long documents containing multiple tables. We also adapt and re-annotate four existing finance QA benchmarks to develop three additional, less challenging evaluation sets: 1) DMSimpShort based on TAT-QA (Zhu et al., 2021) and FinQA (Chen et al., 2021), necessitates simple numerical reasoning over short document with one table; 2) DMSimpLong based on MultiHiertt (Zhao et al., 2022), necessitates simple numerical reasoning over long document with multiple tables; and 3) DMCompShort based on TAT-HQA (Li et al., 2022), necessitates complex numerical reasoning over short document with one table.
We evaluate a wide spectrum of open- and closed-source LLMs, specifically, 27 models from 17 organizations. This notably includes code-based LLMs (Xu et al., 2023; Luo et al., 2023b; Li et al., 2023a; Tunstall et al., 2023) for enhanced reasoning and programming abilities, as well as LLMs specialized in the finance domain (Xie et al., 2023). Two prompting methods, Chain-of-Thought (CoT) (Wei et al., 2022) and Program-of-Thought (PoT) (Chen et al., 2023a), are adopted for experiments. Our experimental results indicate that while the existing best-performing LLM (i.e., GPT-4) can achieve high performance in a simple setting (i.e., DMSimpShort), it still falls short of human experts in more challenging ones. Specifically, GPT-4 significantly outperforms other open-source LLMs, achieving an accuracy of 41.2% on the most challenging evaluation set (i.e., DMCompLong) when applying PoT prompting. However, it still lags far behind human expert performance, which stands at 76%. This significant gap between LLMs and hu-
man experts underscores the challenges presented by DOCMATH-EVAL. It highlights the need for further advancements in adapting LLMs’ numerical reasoning capabilities for practical application in real-world expert domains.
We conclude our main contributions as follows:
• We introduce DOCMATH-EVAL, a comprehensive benchmark designed to systematically evaluate LLMs’ numerical reasoning ability to understand and interpret financial documents. This includes a newly developed, challenging evaluation set and three adapted evaluation sets for varying difficulty levels.
• We conduct an extensive evaluation encompassing a wide range of LLMs, including those specialized in coding and finance. We also incorporate different prompting methods (e.g., CoT and PoT) to comprehensively assess the capabilities and limitations of existing LLMs in our task.
• Our experimental results reveal a noticeable performance gap compared to human experts in more complex scenarios (e.g., problems requiring complex numerical reasoning over long documents). This highlights the limitations of current LLMs in complex real-world applications and the need for continued advancements.

heading: 2 Related Work

heading: 2.1 Math Word Problems
text: The research community has shown significant interest in the vital role of numerical reasoning skills in LLMs. These skills are vital for models to effectively engage in complex problem-solving. To this end, a wide variety of MWP datasets have been proposed in recent years (Hosseini et al., 2014; KoncelKedziorski et al., 2016; Wang et al., 2017; Ling et al., 2017; Cobbe et al., 2021). More challenging datasets have recently been introduced to enhance diversity (Miao et al., 2020), difficulty (Chen et al., 2023c; Hendrycks et al., 2021b), and adversarial robustness (Patel et al., 2021). However, existing MWP datasets predominantly focus on problems akin to academic exams, with a limited emphasis on real-world scenarios. Addressing this gap, our paper introduces a novel and comprehensive benchmark designed to evaluate LLMs’ abilities in understanding and interpreting mixed-content financial documents through numerical reasoning.

heading: 2.2 Numerical Reasoning over Documents
text: Numerical reasoning over documents requires models to have a deep understanding of context and the ability to derive answers through numerical reasoning (Dua et al., 2019). Applying these models in the finance domain (Xie et al., 2023; Wu et al., 2023a; Yang et al., 2023b) presents additional challenges in terms of interpreting hybrid data (Zhu et al., 2021) and utilizing domain-specific expertise (Chen et al., 2021; Zhao et al., 2023a). Numerous datasets focusing on numerical reasoning within the financial domain have been proposed recently. Two notable benchmarks are TAT-QA (Zhu et al., 2021) and FinQA (Chen et al., 2021), which represent pioneering efforts in studying numerical reasoning in finance, particularly requiring the fusion of tabular and textual content. Building upon TAT-QA, a more challenging dataset named TAT-HQA (Li et al., 2022) was developed, focusing on counterfactual questions in relation to the provided context. Additionally, MultiHiertt (Zhao et al., 2022) focuses on numerical reasoning over longer financial documents, containing multiple tables and longer texts. However, as illustrated in Table 1, these four datasets focus on less challenging scenarios, where either simple numerical reasoning (e.g., calculating average or increasing rate of two metrics) is sufficient, or the input context is short (i.e., one-page document segment with only one table). Furthermore, there is a lack of a standardized benchmark for systematically evaluating models’ performance across varying difficulty
levels in terms of numerical reasoning and document understanding, which is crucial in the era of LLMs.

heading: 3 DOCMATH-EVAL
text: In this section, we first offer a formal definition of the DOCMATH-EVAL task. We then explain the rationale and methodology for adopting Python program as the standardized solution format for DOCMATH-EVAL. Subsequently, we detail the data annotation process used to construct the challenging DMCompLong evaluation set, as well as the data re-annotation process for compiling the other three evaluation sets. Finally, we present human-level performance on each evaluation set in DOCMATH-EVAL. Table 1 describes the basic statistics of four developed evaluation sets. DOCMATH-EVAL contains a total of 5,974 questions with high-quality annotations, featuring varying difficulty levels in numerical reasoning and document understanding.

heading: 3.1 Task Formulation
text: We formally define the task of DOCMATH-EVAL in the context of LLMs as follows: Presented with a numerical reasoning question q and a financial document consisting of textual contents E and structured tables T , the task is to generate the numericvalue answer a:
â = argmax a
PLM(a | q, E, T ) (1)
To obtain the best candidate answer â, we use greedy decoding in all our LLM evaluations.
In the subtasks of DMSimpLong and DMCompLong, due to the length of the document exceeding the maximum input length of LLMs, we first apply retrievers to retrieve the top-n most relevant textual and tabular evidence to form a partial document, while maintaining the original relative order of the evidence within the partial document. This textual content and structured tables in the partial document are then input into the LLMs.

heading: 3.2 Solution Format Standardization
text: We observe that existing finance QA datasets feature solutions in various formats. Specifically, TATQA (Zhu et al., 2021) and TAT-HQA (Li et al., 2022) utilize text, while MultiHiertt (Zhao et al., 2022) employs mathematical expressions, such as 100/3, and FinQA (Chen et al., 2021) uses math programs, such as divide(100,3), for solution annotations. This diversity in annotation formats hinders the development of a unified evaluation framework to assess LLM performance across different benchmarks. Additionally, solutions in text format often lack the precision and unambiguity necessary for computational problem-solving; and solutions in mathematical equations or math programs are less descriptive, with the semantic meaning of each numeric value in the equations sometimes being unclear.
To overcome the aforementioned limitations, in DOCMATH-EVAL, we represent solutions using Python programs, as this format combines the explicitness of code execution with the descriptive power of annotated explanation (in the format of Python comments). Such a unified Python program format supports a standardized and effective evaluation framework for LLM assessment. Specifically, annotators are required to first define variables at the beginning of the Python function, starting with “def solution():”. These variables correspond to the key elements or quantities mentioned in the question or question-relevant content in the documents. Annotators are instructed to assign meaningful names that clearly represent each element. They then write a sequence of Python statements that logically solve the problem, step by step. Additionally, annotators receive a bonus for writing detailed comments, thereby enhancing the code’s readability and understandability. To ensure the accuracy and functionality of the solutions, our annotation interface automatically executes the Python
function. This execution checks that the return type of the answer is either a float or an int and verifies that there are no execution errors.

heading: 3.3 Data Re-Annotation From Public Datasets
text: We re-annotate four existing datasets and incorporate them into DOCMATH-EVAL. Specifically, we re-annotate TAT-QA (Zhu et al., 2021) and FinQA (Chen et al., 2021) for DMSimpShort, MultiHiertt (Zhao et al., 2022) for DMSimpLong, and TAT-HQA (Li et al., 2022) for DMCompShort.
Question Validation and Re-annotation We instruct the annotators to identify and remove questions with incorrect annotations or those whose answers are not numerical. Annotators are then asked to enhance each question by adding a scale descriptor to ensure clarity and specificity. For example, "Question: What is the average payment volume per transaction for American Express? (in billions)". They were also asked to correct any identified errors in the original questions.
Solution Validation and Re-annotation As outlined in Section 3.2, we require annotators to rewrite the original solutions into a unified Python format, standardizing variable names and adding comments to enhance the readability of the solutions. Regarding the supporting evidence annotation, we initially convert the original evidence annotations to our format. We then highlight these evidences in the annotation interface, and direct annotators to verify their correctness.

heading: 3.4 Data Annotation From Scratch
text: In real-world scenarios, financial professionals typically need to handle documents spanning tens of pages, along with problems that require more complex numerical reasoning combined with financial knowledge. However, existing finance-relevant QA benchmarks (Zhu et al., 2021; Chen et al., 2021; Zhao et al., 2022; Li et al., 2022) focus on less challenging scenarios, where either simple numerical reasoning is sufficient, or the input context is short. To bridge this gap, we have developed a new, challenging evaluation set, DMCompLong, from scratch. This set focuses on settings that more closely align with real-world problem-solving scenarios, where models are required to perform complex numerical reasoning over long financial documents for problem solving. The annotation process for DMCompLong is as follows.
Source Document Collection Following previous work (Zhu et al., 2021; Chen et al., 2021; Zhao et al., 2022), we use the quarterly (i.e., Form 10-Q) and annual reports (i.e., Form 10-K) of companies as our source documents, which are publicly available at the open-source database1 of U.S. Securities and Exchange Commission. After collecting all the source documents, we utilize a commercial API2 to extract their textual and tabular content. Subsequently, we apply a heuristic-based method to preprocess these two formats of content. The preprocessed documents are then passed to expert annotators for question annotation.
Data Annotation Given a financial document, annotators are first required to briefly read its content and determine the data points to be used in the question. They must then compose the question and highlight the selected paragraphs or tables as evidence supporting it. Following Chen et al. (2021), we use the paragraph index pi to mark questionrelevant textual evidence in the pith paragraph; and (ti, rj) to mark relevant tabular evidence in the rj th row of the tith column. The same method is applied in the dataset re-annotation process, as detailed in Section 3.3. Finally, the annotators are required to write down the solution to the question in Python program format, as discussed in Section 3.2. We set up a bonus payment system for complex annotations that involve difficult document comprehension and numerical reasoning. Specifically, to increase the difficulty of document understanding, we award bonuses to annotators for questions that necessitate information from: 1) multiple tables, 2) multiple sections, or 3) a combination of tables and textual content. To enhance the challenge in numerical reasoning, we provide bonuses for questions requiring financial expertise or involving complex mathematical operations. If such complex annotations are validated during the quality validation stage, a bonus payment will be added.
Quality Validation We implement a comprehensive quality validation protocol to ensure that each annotated example meets the required standards. For every question annotation, we assign it to another annotator, recognized for their high performance in annotation, to verify its accuracy. This process involves manually locating the question-relevant evidence in the documents
1https://www.sec.gov/edgar/search/ 2https://sec-api.io/
using our retrieval-based search toolkits. They then compare this evidence with the original annotations and correct any errors found. Additionally, validators are tasked with confirming the accuracy of the annotated solutions. We offer bonus payments to annotators for identifying erroneous annotations. Ultimately, 232 of the annotated questions are flagged as erroneous and are subsequently revised. We present the human evaluation scores and interevaluator agreements for a subset of 200 sampled examples. Table 2 demonstrate that DOCMATHEVAL exhibits superior annotation quality and a high degree of inter-annotator agreement.

heading: 3.5 Expert-level Performance Evaluation
text: To provide a rough but informative estimate of the performance of domain-experts on each of DOCMATH-EVAL sets, we invite two professionals with Chartered Financial Analyst licenses for evaluation. Regarding human expert performance on DMSimpShort and DMSimpLong, we report the same results as those in the original papers, with accuracy of 91% and 87%, respectively. For DMCompShort and DMCompLong, We randomly sample 25 examples from each set, asking the expert evaluators to answer the questions individually within a fourhour period. They achieve accuracy of 88% and 80% on DMCompShort (average 84%); and accuracy of 72% and 80% on DMCompLong (average 76%).

heading: 4 Experiment Setup

heading: 4.1 Large Language Models
text: Our goal is to investigate the capabilities of current state-of-the-art LLMs on DOCMATH-EVAL to better understand their strengths and limitations. To
this end, we evaluate a wide range of models:
• General: GPT-3.5&4 (OpenAI, 2022, 2023), Gemini (Google, 2023) Llama-2 (Touvron et al., 2023), Mistral (Jiang et al., 2023), MPT (Team, 2023), WizardLM (Luo et al., 2023b), Yi (01.AI, 2023), Baichuan (Yang et al., 2023a), Aquila (BAAI, 2023), Qwen (Bai et al., 2023), Vicuna (Zheng et al., 2023) Phi-1.5&2 (Li et al., 2023b), and DeepSeek (DeepSeek, 2023).
• Math-specific: WizardMath (Luo et al., 2023a).
• Finance-specific: FinMA (Xie et al., 2023).
• Code-based: StarCoder (Li et al., 2023a), StarChat (Tunstall et al., 2023), CodeLlama (Rozière et al., 2023), WizardCoder (Luo et al., 2023b), and Lemur (Xu et al., 2023).
• Mixture of Experts (MoE): Mixtral of experts (Mistral.AI, 2023).
By default, we use chat or instruct versions for each model, when available, otherwise, we used their base version. Additionally, we select the most recent, largest, and best-performing checkpoint available as of paper submission (i.e, December 13th, 2023). All the model weights of evaluated open-sourced LLMs can be found at HuggingFace Model Hub3.

heading: 4.2 Retrieval
text: We experiment with both dense and sparse retrieval models. For dense retrievers, we specifically experiment with OpenAI Ada Embeddings4 and Contriever (Izacard et al., 2022), while for sparse retrievers, we use BM25 (Robertson et al., 1995). These retrievers are employed in the subtasks of DMSimpLong and DMCompLong to extract the top-n most related textual and tabular evidence from the source document. The extracted evidence is then provided within the LLM input context to answer the given question.

heading: 4.3 Prompting Methods
text: Following recent LLM reasoning benchmark works (Lu et al., 2023a; Chen et al., 2023c), we evaluate two established prompting methods, with examples of prompt illustrated in Figure 2.
3https://huggingface.co/models 4platform.openai.com/docs/guides/embeddings
Chain-of-Thought The CoT method (Wei et al., 2022; Kojima et al., 2022) instructs the LLMs to articulate a step-by-step reasoning process. This leads to a detailed explanation that culminates in the final answer.
Program-of-Thought Different from CoT, the PoT method (Chen et al., 2023a) disentangles computation from the reasoning process by prompting the LLMs to generate a structured program to represent the reasoning process. The final answer is then derived by executing the generated program with an external calculator.

heading: 4.4 Implementation Details
text: The implementation details, including 1) LLM parameter setting, 2) tabular data serialization, and 3) final answer extraction and evaluation are discussed in Appendix A.1.

heading: 5 Experimental Results
text: Given the extensive context length of input document, the main evaluation of DOCMATH-EVAL is conducted under a zero-shot setting, aiming to assess LLMs’ capabilities to generate accurate answers without few-shot demonstrations.

heading: 5.1 Main Results
text: We draw the following findings and conclusions based on the results illustrated in Table 3.
GPT-* Significantly Outperforms Other Opensource LLMs Proprietary models demonstrate the best performance on each evaluation set of DOCMATH-EVAL. Notably, GPT-4 significantly outperforms other LLMs, achieving accuracies of 89.3% on DMSimpShort and 80.0 % on DMCompShort, respectively, when utilizing CoT prompting. In contrast, open-source LLMs lag considerably behind, indicating a substantial need for future efforts in model development to bridge the performance gap.
Significant Performance Gap to Human Expert in the Complex Settings While the current best-performing LLM (i.e., GPT-4) achieves performance comparable to human experts in simple problem settings, we find significant performance gaps in more challenging settings. Specifically, GPT-4 achieves an accuracy of 41.2% on DMCompLong with PoT, which is far behind the human expert performance of 76.0%. This underscores the need for ongoing development in the field of LLMs, particularly in complex problemsolving within expert domains.
Llama-2 Achieves Robust Performance We observe that the Llama-2 models generally outper-
form their variants in DOCMATH-EVAL task. For instance, Llama-2-7B shows superior performance compared to Mistral-7B across all evaluation sets. Furthermore, despite its specialization in the finance domain, PiXiu does not demonstrate competitive performance in DOCMATH-EVAL. These findings suggest that Llama-2 is more versatile and robust for our specific task, indicating the needs for future research on exploring the development of LLMs in specialized domains.

heading: 5.2 Program-of-Thought Analysis
text: We observe that the PoT prompting method consistently improves performance over the CoT method in GPT-* models and code-based LLMs. In contrast, the performance of several general LLMs, such as Mistral and WizardLM degrades with PoT prompting. To better analyze the reasons for these differing performance outcomes, we examine the execution rate of each LLM under PoT prompting, measuring how many of the generated Python programs are executable. Figure 3 in Appendix illustrates the relationship between execution rate and accuracy across different models. It demonstrates that the degraded performance when applying PoT
prompting is attributable to the low execution rate. For instance, although WizardLM achieves competitive performance with CoT, it struggles to consistently generate executable Python solutions, leading to lower accuracy with the PoT prompting.

heading: 5.3 LLM Document Understanding Analysis
text: We develop a metric, DU-F1, designed to gauge the nuanced Document Understanding capabilities of LLMs. Specifically, we apply rule-based methods to extract all explicit values (i.e., those not derived from the computation of other values) present within the CoT or PoT output. Subsequently, a comparative analysis is conducted by juxtaposing these extracted direct values with those obtained from the ground truth Python-format solution. The evaluation criterion employs the F1-score, quantifying the LLMs’ efficacy in evidence extraction. The relationships between DU-F1 and accuracy across different LLMs in CoT and PoT prompting are illustrated in Figure 4 and Figure 5 in Appendix respectively. The final accuracy of LLMs correlates with DU-F1, indicating that enhancing the document understanding abilities of LLMs can improve their overall performance on DOCMATH-EVAL.

heading: 5.4 Error Analysis
text: Table 3 reveals the notable superiority of GPT-* models over other LLMs. Despite this, the accuracy falls short of that achieved by human experts. To further understand the strengths and weaknesses of GPT-*, we undertook an extensive analysis of errors. This analysis centered on 100 randomly selected examples from the dataset where GPT-3.50613 exhibited failure. We pinpoint four common errors prone to occurring in current LLMs. A detailed explanation for each error type is provided in Table 6 in the Appendix. Our error analysis reveals that LLMs are likely to make mistakes in calculations. To disentangle the computational abilities from the final accuracy, we applied an external calculator (Inaba et al., 2023) for CoT output to do computation. Figure 6 illustrates the calibrated results of LLMs with an external calculator.

heading: 5.5 Analysis of Evidence Extraction
text: We analyze the impact of retrieval performance on the final accuracy of LLMs in long document settings. Initially, we evaluate the performance of the retriever model used. As illustrated in Table 4, the Ada embedding achieves the best performance. Specifically, it attains a R@10 of 83% and 69.2%
on DMCompShort and DMCompLong, respectively. As demonstrated in Table 5, improved performance of the retriever module consistently enhances the final accuracy of LLMs in our task. This finding underscores the necessity for future work in developing more advanced information retrieval techniques.

heading: 6 Conclusion
text: This paper presents DOCMATH-EVAL, a comprehensive benchmark tailored to assess LLMs’ capabilities in numerical reasoning and problemsolving, particularly in the realm of financial document analysis. Our comprehensive experiments over 27 LLMs with CoT and PoT prompting methods demonstrate that although the top-performing current model excels in simple problem settings, it falls short of human expert performance in problems requiring numerical reasoning over long contexts. We contend that DOCMATH-EVAL serves as a valuable benchmark for future work on evaluating LLMs’ proficiency in tackling complex numerical reasoning tasks within expert domains.
Ethical Consideration
For the DOCMATH-EVAL annotation, we hired 7 graduate students (5 females and 2 males) majoring in finance-related disciplines, all of whom passed our quality exams in Python and finance. Before beginning the official annotation process, each annotator received a two-hour training session to familiarize themselves with the annotation requirements and learn how to use the annotation interface. For DMCompLong annotation from scratch, we consider the following as a unit task: (1) create one math reasoning question and annotate corresponding supporting evidence, (2) compose a Python-format solution for a given question, and (3) validate two annotated examples. We pay approximately $2.5 for each unit task. On average, an annotator can complete 5 unit tasks per hour after training and practice. For dataset re-annotation from existing datasets (i.e., DMSimpShort, DMSimpLong, DMCompLong), we consider the following as a unit task: (1) validate two questions and their original solutions and (2) convert one original solution to Python format and annotate corresponding supporting evidence. We pay around $1 for each unit task. On average, an annotator can complete 12 unit tasks per hour after training and practice. The hourly rates are in the range of $10 to $15, depending on the different working speeds, which is above the local average wage for similar jobs. We recommend that annotators spend a maximum of 4 hours per day to reduce pressure and maintain a comfortable pace. In total, the approximate working hours to construct DOCMATH-EVAL is 700 hours. The whole annotation work lasted three weeks.
Limitations
In this work, we propose DOCMATH-EVAL and conduct comprehensive analysis of different LLMs’ capabilities in solving knowledge-intensive math reasoning problems in finance domains. However, there are still some limitations: First, our method for extracting final answer from model output (Appendix A.1) is still not perfect. In some cases, this methods fails to locate the answer, leading to the reported accuracy being an approximate lower bound. Moreover, among recently released finance-specific LLMs (Wu et al., 2023b; Yang et al., 2023b; Xie et al., 2023), we only evaluate FinMA, as it is the only work with a checkpoint available at HuggingFace and compatible with the vllm framework. Due to computational resource constraints, we do not
tune LLMs on a large-scale finance-domain data ourselves. However, we believe that training on finance data can help improve LLMs’ capabilities in solving problems in DOCMATH-EVAL.

heading: Acknowledgement
text: We are grateful for the compute support provided by Microsoft Research’s Accelerate Foundation Models Research (AFMR) program. We would also like to thank the anonymous reviewers and area chairs for constructive discussions and feedback.

heading: A Experiment
text: A.1 Implementation Details
LLM Experiment The experiments for opensourced LLMs were conducted using the vLLM framework (Kwon et al., 2023). For all the experiments, we set temperature as 1.0, Top P as 1.0, and maximum output length as 512.
Input Tabular Data Serialization Building on previous work that evaluated LLMs on tablerelevant tasks (Chen, 2023; Zhao et al., 2023b,c), we present our method for processing tabular data in documents. Specifically, we separate headers or cells in different columns using a vertical bar (|), and rows using a newline. This approach allows for the direct feeding of flattened table input into LLMs. In our preliminary study, we discovered that GPT-* and llama-2 can effectively understand such table representations. Nevertheless, we believe future research could explore more effective methods for encoding tabular data.
Final Answer Extraction For LLM with CoT prompting, we adopt the answer extraction pipeline from Chen et al. (2023b) to identify the final answer from the model’s output. For LLM with PoT prompting, we first develop a heuristic method to extract the generated python solution from the model’s output. If this python solution is executable, we execute it to obtain the final answer.
Once we obtain the final answer from model’s output, we compare it with the ground-truth answer for accuracy measurement.

heading: Future Work
text: Nevertheless, we believe future research could explore more effective methods for encoding tabular data. Final Answer Extraction For LLM with CoT prompting, we adopt the answer extraction pipeline from Chen et al. (2023b) to identify the final answer from the model’s output. For LLM with PoT prompting, we first develop a heuristic method to extract the generated python solution from the model’s output. If this python solution is executable, we execute it to obtain the final answer. Once we obtain the final answer from model’s output, we compare it with the ground-truth answer for accuracy measurement