KnowledgeFMATH: Knowledge-Intensive Math Reasoning in Finance Domains (2024)

Abstract: We introduce KnowledgeFMATH, a novel benchmark designed to evaluate LLMs’ capabilities in solving knowledge-intensive math reasoning problems. Compared to prior works, this study features three core advancements. First, KnowledgeFMATH includes 1,259 problems with a hybrid of textual and tabular content. These problems require collegelevel knowledge in the finance domain for effective resolution. Second, we provide expert-annotated, detailed solution references in Python program format, ensuring a highquality benchmark for LLM assessment. We also construct a finance-domain knowledge bank and investigate various knowledge integration strategies. Finally, we evaluate a wide spectrum of 26 LLMs with different prompting strategies like Chain-of-Thought and Programof-Thought. Our experimental results reveal that the current best-performing system (i.e., GPT-4 with CoT prompting) achieves only 56.6% accuracy, leaving substantial room for improvement. Moreover, while augmenting LLMs with external knowledge can improve their performance (e.g., from 33.5% to 47.1% for GPT-3.5) , their accuracy remains significantly lower than the estimated human expert performance of 92%. We believe that KnowledgeFMATH can advance future research in the area of domain-specific knowledge retrieval and integration, particularly within the context of solving math reasoning problems. § github.com/yale-nlp/KnowledgeFMath

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12841–12858 August 11-16, 2024 ©2024 Association for Computational Linguistics
§ github.com/yale-nlp/KnowledgeFMath

heading: 1 Introduction
text: Large language models (LLMs) have been increasingly recognized for their potential for complex problem-solving in real-world scenarios (OpenAI, 2023a; Touvron et al., 2023; Jiang et al., 2023). Solving math reasoning problems has emerged as a
∗Equal Contribution
key method for assessing LLMs’ capabilities (Roy and Roth, 2015; Amini et al., 2019; Cobbe et al., 2021; Chen et al., 2023c), as it demands both understanding contextual information and reasoning over complex logics.
Recent advancements in LLMs have led to remarkable progress in solving fundamental math problems (Wei et al., 2022; Lewkowycz et al., 2022; Chen et al., 2023b; Wang et al., 2023; Luo et al., 2023a; Azerbayev et al., 2024). However, as illus-
12841
trated in Table 1, existing math reasoning benchmarks typically do not require specialized domain knowledge. This becomes a notable shortcoming when considering practical applications of LLMs. Measuring progress in specialized areas such as finance and healthcare typically involves addressing domain-specific and knowledge-intensive problems, which goes beyond the scope of general mathematical reasoning. Recognizing this gap in the existing benchmarks, we focus on the finance domain. We chose this domain because, as illustrated in Figure 1, it often involves scenarios requiring not only basic mathematical skills but also a deep understanding of financial concepts (Yang et al., 2023b; Xie et al., 2023; Wu et al., 2023). Additionally, the finance domain frequently employs tables to represent data (Zhu et al., 2021; Chen et al., 2021; Zhao et al., 2022; Li et al., 2022; Zhao et al., 2023b), which adds another layer of complexity to the knowledge-intensive problem-solving.
We introduce KnowledgeFMATH, the first benchmark tailored for evaluating LLMs in the context of Knowledge-intensive Math reasoning in the Finance domain. The dataset contains 1,259 problems that cover a broad range of finance subareas (e.g., investment analysis, risk assessment, and financial forecasting), with 39.0% of the prob-
lems necessitating data interpretation over tabular data. Each problem is accompanied by detailed, expert-annotated solutions and explanations, providing a comprehensive reference for evaluating the LLMs’ performance. Additionally, we collect and release a comprehensive knowledge bank, which includes detailed definitions and explanations for 1,760 financial terms and concepts, facilitating future research on improving knowledge-intensive problem-solving through knowledge retrieval.
We evaluate a wide spectrum of open-source and proprietary LLMs, specifically, 26 model models from 14 organizations. Notably, this includes math-specific (Luo et al., 2023a), code-based (Xu et al., 2023; Luo et al., 2023b; Tunstall et al., 2023) LLMs, as well as mixture of experts (MoE) LLMs (Mistral.AI, 2023). Two prompting methods, Chain-of-Thought (CoT) (Wei et al., 2022) and Program-of-Thought (PoT) (Chen et al., 2023b), are adopted for experiments. Our experimental results indicate that all evaluated open-source LLMs scored below 24% in accuracy using various prompting methods, including CoT and PoT prompting. Proprietary models perform better, with GPT-4 significantly outperforming other LLMs, achieving an accuracy of 56.6% when applying CoT prompting. However, it still lags far behind
human expert performance in the open-book setting, which stands at 92%. This significant gap between LLMs and human experts demonstrates the challenges of KnowledgeFMATH, highlighting the need for further advancements in LLMs for knowledge-intensive problem-solving capabilities.
Next, we investigate how to integrate domainspecific knowledge to enhance the problem-solving capabilities of LLMs. We investigate various popular knowledge integration strategies and reveal that including question-relevant knowledge into the prompt can consistently improve LLMs’ performance. This provides insights for future work to develop more advanced knowledge-augmented strategies to realize higher performance gains.
Our contributions are summarized below:
• We propose KnowledgeFMATH, the first knowledge-intensive math reasoning benchmark in finance domains, aimed at evaluating LLMs’ abilities in knowledge-intensive math reasoning.
• We conduct comprehensive evaluations using a diverse array of LLMs, uncovering a substantial performance gap between the best-performing LLM (i.e., GPT-4) and human experts.
• We present a detailed analysis on augmenting LLMs with various knowledge integration strategies. This provides valuable insights for future work in knowledge-intensive problem solving.

heading: 2 KnowledgeFMATH
text: In this section, we describe the dataset construction process for KnowledgeFMATH. We begin by constructing a knowledge bank that includes wellformulated definitions of 1,760 financial terms. We then instruct expert annotators to use knowledge terms within the constructed knowledge bank to create knowledge-intensive questions with a hybrid of textual and tabular content.

heading: 2.1 Knowledge Bank Construction
text: We construct a knowledge bank that covers a wide range of 1,760 knowledge terms in the finance domain. It simplifies the creation of knowledgeintensive questions by annotators and enables the exploration of various topics within domain knowledge. The knowledge bank includes financedomain-specific terms (e.g., “exchange rate” and “net present value”) collected from Wikipedia. Each knowledge term is accompanied with their corresponding textual definitions and, where applicable,
mathematical formulas in python format. An example of included knowledge terms is illustrated in Figure 2. We detail the process of 1) knowledge collection, 2) semi-automated knowledge formulation, and 3) knowledge bank update and maintenance in Appendix A.1. It is worth noting that this knowledge bank is versatile and can be applied to a variety of finance-relevant tasks for future research.

heading: 2.2 KnowledgeFMATH Question Annotation
text: For each financial term in the knowledge bank, we instruct annotators to create a corresponding math reasoning question, if applicable. The answer to the composed question should be a numeric value. The annotators are required to adhere to the following guidelines for a successful question annotation:
Question Annotation If the annotators choose to adapt questions from textbooks or the Internet instead of creating their own from scratch, they are asked to adhere to copyright and license regulations, avoiding data from sites prohibiting copy and redistribution. Furthermore, they are required not only to modify the surface-level description of the question but also to change the associated numeric values. In light of the emerging concerns about data contamination in LLMs (Shi et al., 2024; Deng et al., 2024), we instruct annotators to conduct a Google search for each annotated question, ensuring that no similar question appears on the first page of the search results. Additionally, we recognize that many financial problems involve tables, as shown in Figure 1. Such tabular data plays a crucial role in thoroughly understanding financial problems, and it presents unique challenges for LLMs in terms of comprehension and interpretation. Therefore, we encourage and reward annotators to include tables that are relevant and accurately represent the data pertinent to the questions. Finally, out of 1,259 questions, 674 are marked as
having been adapted from existing resources, and 491 are accompanied with tabular data.
Identifying Question-relevant Knowledge After a question is annotated, annotators must identify 1-3 key financial concepts for answering this question. They then search for each term in our constructed knowledge bank. If the term is included, they verify its context and details for relevance. If a term is absent or with low-quality definition, annotators receive a bonus for documenting the term, providing a brief explanation or definition and outlining its relevance to the problem. These identified terms are subsequently added or updated in the knowledge bank, resulting in a total of 346 new inclusions and 83 revisions.

heading: 2.3 KnowledgeFMATH Solution Annotation
text: As illustrated in Table 1, existing math reasoning benchmarks typically represent solutions using text or mathematical equations. However, solutions in text format often lack the precision and unambiguous nature required for computational problemsolving. Solutions in mathematical equations are explicit, but less descriptive, as the semantic meaning associated with each numeric value in the equations can be ambiguous. Moreover, these two formats are less adaptable for use in automated systems due to variations in language and difficulties in semantic parsing and execution.
To overcome these limitations, we use Python programs, starting with “def solution():”, to represent solutions. Such Python program combines the explicitness of code execution with the descriptive power of annotated comments, offering a more effective and adaptable solution representation for complex math reasoning problems. Specifically, annotators are required to first define variables with meaningful names at the beginning of the Python function. These variables correspond to the key elements or quantities mentioned in the textual or tabular content of questions. The annotators then proceed to write a sequence of Python statements that logically solve the problem, step by step. Additionally, annotators are required to write detailed comments, making the code more readable and understandable. To ensure the accuracy and functionality of the Python-format solutions, our annotation interface automatically executes the Python function. This execution checks that the return type of the answer is either a float or an int and verifies that there are no execution errors.

heading: 2.4 Data Quality Validation
text: We conduct a comprehensive validation protocol to ensure the high quality of the annotated data. For each annotated question, we first assign another annotator to validate whether: 1) the question is meaningful and grammatically correct, 2) the associated knowledge terms are accurately annotated and complete, 3) the Python-format solution is logically correct and easy to understand. Validators are asked to revise examples that do not meet these standards. We also report the human evaluation scores and inter-evaluator agreements over 200 sampled examples. As illustrated in Table 2, KnowledgeFMATH has a high annotation quality.

heading: 2.5 Data Statistics and Dataset Release
text: Table 3 describes the basic statistics of KnowledgeFMATH, with topic-type distribution shown in Figure 4 in Appendix. We randomly divide the dataset into two subsets: validation and test. The validation set contains 259 examples and is intended for model development validation. The test set comprises the remaining 1,000 examples and is designed for standard evaluation. To prevent data contamination (Shi et al., 2024; Sainz et al., 2023; Deng et al., 2024), the answer for the test set will not be publicly released. Instead, we will develop and maintain an online evaluation platform, allowing researchers to evaluate models and participate in a leaderboard. Following recent LLM reasoning benchmarks (Chen et al., 2023c; Yue et al., 2023; Lu et al., 2024), the main evaluation of KnowledgeFMATH is conducted under a zero-shot setting on the test set to assess LLMs’ capabilities to generate accurate answers without fine-tuning or few-shot demonstrations on our benchmark.

heading: 2.6 Human-level Performance Evaluation
text: To provide a rough but informative estimate of human-level performance by non-experts and experts on KnowledgeFMATH, we randomly sampled 50 examples from the validation set. We enroll two experts, both with the CFA license, and two nonexperts to individually solve these questions.
We first evaluate their performance in a closedbook setting, where the evaluators do not have access to the internet or textbooks and are required to finish the 50 questions within three hours. The nonexpert evaluators achieve accuracy of 54% and 62% (average 58%), and the expert evaluators achieve accuracy of 76% and 70% (average 73%).
We then transition to an open-book setting, where the evaluators are asked to use the internet and textbooks to correct their initial errors. This setting is designed to assess how external knowledge resources could enhance human problem-solving abilities and accuracy. The non-expert evaluators improved their accuracy to 86% and 82% (average 84%). Similarly, the expert evaluators improved the accuracy to 94% and 90% (average 92%).

heading: 3 Evaluated Systems

heading: 3.1 Large Language Models
text: We evaluate following LLMs on KnowledgeFMATH:
TODO
• General: GPT-3.5&4 (OpenAI, 2022, 2023a), Gemini-Pro (Google, 2023), Llama-2 (Touvron et al., 2023), Mistral (Jiang et al., 2023), MPT (Team, 2023), Falcon (Almazrouei et al., 2023), WizardLM (Luo et al., 2023b), Yi (01.AI, 2023), Baichuan (Yang et al., 2023a), Phi-1.5 (Li et al., 2023), and DeepSeek (DeepSeek, 2023).
• Math-specific: WizardMath (Luo et al., 2023a).
• Code-based: CodeLlama (Rozière et al., 2023), WizardCoder (Luo et al., 2023b), and Lemur (Xu et al., 2023).
• Mixture of Experts (MoE): Mixtral of experts (Mistral.AI, 2023).
By default, we use chat or instruct versions for each model, when available, otherwise, we used their base version. Additionally, we select the most recent, largest, and best-performing checkpoint available as of paper submission (i.e, December 10th, 2023). All the model weights of evaluated open-sourced LLMs can be found at HuggingFace Model Hub1. The implementation details (i.e., LLM parameter setting, tabular data serialization, and final answer extraction and evaluation) are discussed in Appendix B.1.

heading: 3.2 Prompting Methods
text: Following recent LLM reasoning benchmark works (Lu et al., 2024; Chen et al., 2023c), we
1https://huggingface.co/models
evaluate two established prompting methods, with examples of prompt illustrated in Figure 3.
Chain-of-Thought The CoT method (Wei et al., 2022; Kojima et al., 2022) instructs the LLMs to articulate a step-by-step reasoning process. This leads to a detailed explanation that culminates in the final answer.
Program-of-Thought Different from CoT, the PoT method (Chen et al., 2023b) disentangles computation from the reasoning process by prompting the LLMs to generate a structured program to represent the reasoning process. The final answer is then derived by executing the generated program with an external calculator.

heading: 4 Experimental Results

heading: 4.1 Main Results
text: Table 4 illustrates the performance of the evaluated LLMs using CoT and PoT prompting methods on the test set of KnowledgeFMATH. From this, we draw the following conclusions:
GPT-* significantly outperforms other opensource LLMs Proprietary models demonstrate the best performance on KnowledgeFMATH. Notably, as illustrated in Table 4, GPT-4 significantly outperforms other LLMs, achieving an accuracy of 56.6% on KnowledgeFMATH with CoT. In contrast, open-source LLMs significantly lag behind. Furthermore, our case study in Table 7 shows that while GPT-* models are capable of performing complex mathematical calculations, other LLMs often fail to understand financial terms, leading to incorrect mathematical expressions. This highlights a critical need for future development efforts to close the performance gap.
Substantial Discrepancy in Performance Between Human Experts and LLMs Even the best-performing LLM, GPT-4, performs much worse than human experts. For instance, the accuracy of GPT-4 using the CoT prompting method stands at 56.6%, falling short of the 92% accuracy achieved by expert evaluators in the open-book setting. This gap highlights the critical need for further advancements in LLMs, especially in tackling complex problem-solving tasks within specialized domains that are knowledge-intensive.
Analysis of Open-sourced LLMs Among opensource LLMs with CoT prompting, Mixtral MoE
achieves the best performance, demonstrating the effectiveness of applying a mixture of experts framework. Moreover, WizardMath also performs well as it is further instruction-tuned to learn mathematical reasoning. Moreover, among opensource LLMs with PoT prompting methods, Lemur achieves better performance than its backbone (i.e., Llama-2), demonstrating the effectiveness of tuning LLMs on code-based tasks for enhanced reasoning and coding capabilities.

heading: 4.2 Program-of-Thought Analysis – LLMs’ Ability to Generate Executable Programs
text: We observe that the PoT prompting method consistently improves performance over the CoT method in GPT-* models and code-based LLMs. In contrast, the performance of several general LLMs, such as Mistral and WizardLM degrades with PoT prompting. To better analyze the reasons for these differing performance outcomes, we examine the execution rate of each LLM under PoT prompting, measuring how many of the generated Python programs are executable. Figure 6 illustrates the relationship between execution rate and accuracy across different models. It demonstrates that the degraded performance when applying PoT prompting is attributable to the low execution rate. For instance, although WizardLM achieves competitive performance with CoT, it struggles to consistently generate executable Python solutions, leading to lower accuracy with the PoT prompting approach.

heading: 4.3 Case Study and Error Analysis
text: In Table 4, we observe that GPT-* models significantly outperform other LLMs. Notably, the latest GPT-4 version achieves an accuracy of 56.6% using CoT prompting, closely approaching the nonexpert human-level performance in the close-book setting (i.e., 58%). To gain a deeper insight into the capabilities and limitations of GPT-* on our dataset, we conducted a comprehensive error analysis and case studies. This was based on 100 randomly sampled examples from KnowledgeFMATH development set where GPT-3.5-1106 exhibited failures. We identify four common mistakes that the current LLMs are likely to make (i.e., misinterpretation of required knowledge, computation error, table misunderstanding, and question misunderstanding). We provide detailed examples and explanations for each error type in Table 6 in Appendix. Moreover, we also present case study of model output from various LLMs with CoT prompting, as shown in
Table 7 in the Appendix.

heading: 5 Knowledge Augmentation Analysis
text: In this section, we provide a comprehensive analysis to understand the performance of LLMs and the quality of knowledge incorporated into the input context, aiming to provide insights for future work on knowledge augmentation in LLMs to solve knowledge-intensive tasks.

heading: 5.1 Evaluated Knowledge-Augmented Method
text: We develop and evaluate various knowledgeaugmented approaches. For each setting, we include the definition of question-relevant knowledge terms within the prompts (Figure 5 in Appendix).
• Oracle: To investigate the headroom in knowledge augmentation, we use an oracle setting, where the ground-truth knowledge terms associated with the question (Section 2.2) are included.
• LLM as Knowledge Base: Recent work (Petroni et al., 2019; Kang et al., 2023) demonstrates that
LLMs themselves can effectively serve as knowledge bases. Therefore, we prompt LLMs to first identify the financial terms required to answer the question. They then generate definitions of each identified knowledge term using the inherent data memorization capabilities.
• Knowledge Retrieval: We use the question as the retrieval query to the constructed knowledge bank. We investigate 1) BM25 as sparse retriever and 2) OpenAI Ada Embedding2 as dense retriever to retrieve the top-n question-relevant knowledge terms from knowledge bank.
• LLM-Instructed Knowledge Retrieval: While the method of using “LLM as Knowledge Base” can effectively identify the knowledge required to answer a question, it is likely to produce knowledge definitions that are not entirely accurate (Chen et al., 2023a; Peng et al., 2023). To address this issue of unfaithfulness, we harness the power of external knowledge retrieval
2https://platform.openai.com/docs/guides/ embeddings, we use the text-embedding-ada-002 version.
for obtaining more trustworthy knowledge definitions. Specifically, instead of using the original question as the retrieval query, we utilize each knowledge term along with its definition generated from the “LLM as Knowledge Base”. This approach provides a more informative and semantically similar basis for knowledge retrieval.
• LLM as Retrieval Re-Ranker: Recent studies have demonstrated LLMs’ competitive capabilities in re-ranking retrieved candidates to output a more precise list (Sun et al., 2023). Therefore, in this setting, we first use retriever in “Knowledge Retrieval” to retrieve top-3n candidates. Subsequently, we prompt LLMs to select top-n most relevant knowledge terms from this candidate set.

heading: 5.2 Experimental Results
text: As illustrated in Table 5, improving the questionrelevance of incorporated knowledge can consistently improve the LLMs’ performance. Specifically, LLMs equipped with retrieved knowledge from Ada Embedding consistently outperform those using retrieved knowledge from BM25. This is due to the more advanced capabilities of the Ada Embedding-based retriever. Among different LLMaided retrieval strategies, LLM-Instructed Knowledge Retrieval achieves the best performance, demonstrating the effectiveness of using refined queries for knowledge retrieval. Nevertheless, it is worth noting that even when incorporated with the ground-truth knowledge (i.e., the oracle setting), GPT-3.5 still performs much worse than human
experts in close-book setting (i.e., 92.0%). This highlights the need for future work on developing more advanced domain-specific knowledge integration methods. Table 8 and Table 9 in Appendix C present a case study on effectiveness of various knowledge integration strategies.

heading: 6 Related Work
text: The development of general-purpose intelligent systems is significantly dependent on the foundational aspect of mathematical reasoning, a topic that has garnered considerable attention in the academic community. As illustrated in Table 1, researchers have proposed a wide spectrum of math reasoning datasets that cater to a variety of educational levels, ranging from elementary school to college. However, these math reasoning benchmarks typically do not require specialized domain knowledge, a notable shortcoming when considering the practical applications of LLMs. Therefore, recent work has investigated the LLMs’ capabilities in knowledge-intensive problem solving. For example, Chen et al. (2023c) collected a theorem-driven question-answering dataset, designed to evaluate AI models’ ability to apply theorems in solving challenging science problems. Contemporary to our work, MMMU (Yue et al., 2023) and MathVista (Lu et al., 2024) include examples that require complex visual reasoning in expert domains.

heading: 7 Conclusion
text: This paper introduces KnowledgeFMATH, aimed at assessing LLMs in knowledge-intensive math reasoning. Our comprehensive evaluations of 26 LLMs, using both CoT and PoT prompting methods, identify significant areas where LLMs need to enhance their specialized knowledge for complex problem-solving in expert domains. Additionally, our knowledge augmentation analysis indicate that integrating domain-specific knowledge can improve LLMs’ problem-solving abilities. We believe this research provides valuable insights for future work in advancing LLMs in complex problemsolving within expert domains.
Limitations
In this work, we propose KnowledgeFMATH and conduct comprehensive analysis of different LLMs’ capabilities in solving knowledge-intensive math reasoning problems in finance domains. However, there are still some limitations: (1) Our method for
extracting final answer from model output (Appendix B.1) is still not perfect. In some cases, this methods fails to locate the answer, leading to the reported accuracy being an approximate lower bound. (2) In our experiment, we regard tables in the question as textual input (Appendix B.1). However, in real-world scenarios, tabular data might appear as images, where people cannot obtain its textual content directly. In these cases, OCR tools to extract table content (Du et al., 2020) or LLMs with vision capabilities (OpenAI, 2023b; Yue et al., 2023; Lu et al., 2024) may be required. (3) Due to computational resource constraints, we do not tune LLMs on a large-scale finance-domain data ourselves. However, we believe that training on finance data can help improve LLMs’ capabilities in solving knowledge-intensive financial problems.

heading: Acknowledgement
text: We are grateful for the compute support provided by Microsoft Research’s Accelerate Foundation Models Research (AFMR) program. We would also like to thank the anonymous reviewers and area chairs for constructive discussions and feedback. Hongjun Liu and Chen Zhao are supported by Shanghai Frontiers Science Center of Artificial Intelligence and Deep Learning, NYU Shanghai.

heading: A KnowledgeFMATH Dataset
text: A.1 Knowledge Bank Construction Knowledge Collection To construct a knowledge bank, we first collect knowledge relevant to the finance domain from Wikipedia using “finance” and
“economics” as key search terms. After collecting the raw financial data, we adopt comprehensive heuristics, embedding-based methods to remove duplicates. This procedure ensures the uniqueness of each knowledge term in our bank.
Automatic Knowledge Formulation To enhance the adaptability and usability of the knowledge bank, we incorporate a two-step automatic knowledge formulation process, making each piece of collected knowledge standardized and distilled into a clear, concise format. The primary motivation for using automatic knowledge formulation is cost efficiency and effectiveness. We have observed that GPT-* models are adept at handling this straightforward task with minimal bias, as this process does not involve the addition of extraneous knowledge. We first prompt GPT-3.5 to reformulate the gathered information for each financial term into a concise, paragraph-long textual definition. Since some financial terms come with mathematical definitions, we address the issue of varied formula formats in the original sources (e.g., LaTeX and HTML). We instruct GPT-4 to transform these formulas into a unified python program format. Figure 2 illustrates an example collected in the knowledge bank.
Knowledge Bank Update and Maintenance After formulating knowledge using LLMs, during the dataset annotation stage (Section 2.2), we dynamically update and maintain the constructed knowledge bank, incorporating new knowledge that, although not initially covered, is essential for answering the annotated questions. Additionally, we remove any duplicate entries identified by the annotators. We eventually collect 1,760 pieces of
financial knowledge in the knowledge bank, with 52% of the terms including Python-formatted mathematical definitions.

heading: B Experiment Setup
text: B.1 Implementation Details LLM Experiment The experiments for opensourced LLMs were conducted using vLLM framework (Kwon et al., 2023). For all the experiments, we set temperature as 1.0, Top P as 1.0, and maximum output length as 512. For questions involving tabular data, we converted the tables into Markdown format for model input.
Final Answer Extraction For LLM with CoT prompting, we adopt the answer extraction pipeline from Chen et al. (2023c) to identify the final answer from the model’s output. For LLM with PoT prompting, we first extract the generated python solution from the model’s output. If this python solution is executable, we execute it to obtain the final answer. Once we obtain the final answer from model’s output, we compare it with the groundtruth answer for accuracy measurement.
Tabular Data Serialization Following previous work on table-relevant tasks (Chen, 2023; Zhao et al., 2023c), we use Markdown format to present tabular data in math reasoning problems. In our preliminary study, we discovered that GPT-* and llama-2 can effectively understand such table representations.
C Experimental Results and Analysis
TODO

heading: Future Work
text: We believe this research provides valuable insights for future work in advancing LLMs in complex problemsolving within expert domains. Limitations
In this work, we propose KnowledgeFMATH and conduct comprehensive analysis of different LLMs’ capabilities in solving knowledge-intensive math reasoning problems in finance domains. However, there are still some limitations: (1) Our method for
extracting final answer from model output (Appendix B. 1) is still not perfect. In some cases, this methods fails to locate the answer, leading to the reported accuracy being an approximate lower bound. (2) In our experiment, we regard tables in the question as textual input (Appendix B. 1). However, in real-world scenarios, tabular data might appear as images, where people cannot obtain its textual content directly. In these cases, OCR tools to extract table content (Du et al. , 2020) or LLMs with vision capabilities (OpenAI, 2023b; Yue et al. , 2023; Lu et al. , 2024) may be required. (3) Due to computational resource constraints, we do not tune LLMs on a large-scale finance-domain data ourselves. However, we believe that training on finance data can help improve LLMs’ capabilities in solving knowledge-intensive financial problems