FinTextQA: A Dataset for Long-form Financial Question Answering (2024)

Abstract: Accurate evaluation of financial questionanswering (QA) systems necessitates a comprehensive dataset encompassing diverse question types and contexts. However, current financial QA datasets lack scope diversity and question complexity. This work introduces FinTextQA, a novel dataset for long-form question answering (LFQA) in finance. FinTextQA comprises 1,262 high-quality, source-attributed QA pairs extracted and selected from finance textbooks and government agency websites.Moreover, we developed a Retrieval-Augmented Generation (RAG)-based LFQA system, comprising an embedder, retriever, reranker, and generator. A multi-faceted evaluation approach, including human ranking, automatic metrics, and GPT-4 scoring, was employed to benchmark the performance of different LFQA system configurations under heightened noisy conditions. The results indicate that: (1) Among all compared generators, Baichuan2-7B competes closely with GPT-3.5-turbo in accuracy score; (2) The most effective system configuration on our dataset involved setting the embedder, retriever, reranker, and generator as Ada2, Automated Merged Retrieval, Bge-Reranker-Base, and Baichuan2-7B, respectively; (3) models are less susceptible to noise after the length of contexts reaching a specific threshold. The dataset is publicly available 1.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6025–6047 August 11-16, 2024 ©2024 Association for Computational Linguistics
Accurate evaluation of financial questionanswering (QA) systems necessitates a comprehensive dataset encompassing diverse question types and contexts. However, current financial QA datasets lack scope diversity and question complexity. This work introduces FinTextQA, a novel dataset for long-form question answering (LFQA) in finance. FinTextQA comprises 1,262 high-quality, source-attributed QA pairs extracted and selected from finance textbooks and government agency websites.Moreover, we developed a Retrieval-Augmented Generation (RAG)-based LFQA system, comprising an embedder, retriever, reranker, and generator. A multi-faceted evaluation approach, including human ranking, automatic metrics, and GPT-4 scoring, was employed to benchmark the performance of different LFQA system configurations under heightened noisy conditions. The results indicate that: (1) Among all compared generators, Baichuan2-7B competes closely with GPT-3.5-turbo in accuracy score; (2) The most effective system configuration on our dataset involved setting the embedder, retriever, reranker, and generator as Ada2, Automated Merged Retrieval, Bge-Reranker-Base, and Baichuan2-7B, respectively; (3) models are less susceptible to noise after the length of contexts reaching a specific threshold. The dataset is publicly available 1.
1 Introduction
The growing demand for financial data analysis and management has led to the expansion of artificial intelligence (AI)-driven question-answering (QA) systems (Wu et al., 2023). These systems not only enhance customer service but also assist in risk management and personalized stock recommendations (Yuan et al., 2021, 2023c,b). The
* Co-corresponding Author 1https://huggingface.co/datasets/GPS-Lab/
FinTextQA
intricate nature of financial data, with its domainspecific terminologies, concepts, and the inherent uncertainty of the market and decision-making processes, demands a deep understanding of the financial domain to generate accurate and informative responses (Confalonieri et al., 2021). In this context, long-form question answering (LFQA) scenarios become particularly relevant as they require models to demonstrate a broad spectrum of sophisticated skills, including information retrieval, summarization, data analysis, comprehension, and reasoning (Fan et al., 2019).
In the general domain, there are several LFQA datasets available, including ELI5 (Fan et al., 2019), WikiHowQA (Bolotova-Baranova et al., 2023) and WebCPM (Qin et al., 2023). However, it is important to note that there is currently no LFQA dataset specifically tailored for the finance domain. Existing financial QA benchmarks often fall short in addressing question complexity and variety by primarily on sentiment analysis and numerical calculation, as comprehensive paragraph-length responses and relevant document retrievals are often required to answer intricate, open-domain questions (Han et al., 2023). To address these challenges, we intro-
6025
duce a new dataset, FinTextQA, which comprises LFQAs from finance-related textbooks and government agency websites to assess QA models on general finance and regulation or policy-related questions. FinTextQA consists of 1,262 high-quality, source-attributed question-answer pairs and associated document contexts. It contains six question types with an average text length of 19.7k words, curated from five rounds of human screening. This dataset is pioneering work in integrating financial regulations and policies into LFQA, challenging models with more demanding content.
In addition to introducing the dataset, we conduct comprehensive benchmarking of state-of-theart (sota) models on FinTextQA to provide baselines for future research. Current LFQA systems frequently solely rely on fine-tuning pre-trained language models such as GPT-3.5-turbo, LLaMA2 (Touvron et al., 2023), Baichuan2 (Yang et al., 2023a), etc., which often fail to provide detailed explanations or effectively handling complicated finance questions (Yuan et al., 2023a). In response, we opt for the Retrieval-augmented generation (RAG) framework, as illustrated in Figure 2. By processing documents in multiple steps, RAG systems can pre-process and provide the most relevant information to LLMs, enhancing their performance and explanation capabilities (Guu et al., 2020).
We believe this work, by introducing the first LFQA financial dataset and conducting comprehen-
sive benchmark experiments on the dataset, marks a milestone in advancing the comprehension of financial concepts and enhancing assistance in this field: FinTextQA offers a rich and rigorous framework for building and assessing the capabilities of general finance LFQA systems. Our experimental analysis not only highlights the efficacy of various model configurations but also underscores the critical need for enhancing current methodologies to improve both the precision and explicability of financial question-answering systems.
2 Related Work
2.1 Long-Form Question Answering (LFQA) Compared to the conventional question-answering task (Bouziane et al., 2015; Mao et al., 2023; Yang et al., 2023b; Cao et al., 2021; Cheng et al., 2023d, 2024), the goal of LFQA is to generate comprehensive, paragraph-length responses by retrieving and assimilating relevant information from various sources (Fan et al., 2019). This poses a significant test for current Natural Language Processing (NLP) and Artificial Intelligence (AI) models, given their limited understanding and learning capacities (Thompson et al., 2020; Cao et al., 2022; Zhang et al., 2023; Cheng et al., 2023b,c; Cheng, 2021).
Several LFQA datasets are available in general domain, including ELI5 (Fan et al., 2019), WikiHowQA (Bolotova-Baranova et al., 2023), and We-
bCPM (Qin et al., 2023). In the financial domain, some QA datasets have been developed. However, none of them addresses LFQA. While these datasets like FinQA (Chen et al., 2021) and TATQA (Zhu et al., 2021) address specific scopes such as numerical reasoning, they do not touch upon general LFQA tasks. In addition, FIQA (Maia et al., 2018) only provides short-context documents, which may not adequately represent real-life scenarios and have limited industry applicability. Although FinanceBench (Islam et al., 2023) does cover a wider scope, it only offers 150 open-source question-answer pairs, while the question complexity and answer do not satisfy real-life LFQA scenarios.
2.2 Retrieval-Augmented Generation (RAG)
RAG frameworks represent a significant advancement in LFQA, incorporating external knowledge sources and In-Context Learning (ICL) for efficient information retrieval and application. By combining diverse documents into comprehensive prompts, RAG enables language models to generate contextually informed responses without task-specific retraining (Lewis et al., 2020).
The evolution of RAG involves three principal stages: Naive RAG, Advanced RAG, and Modular RAG. Naive RAG offers improvements over traditional language models by providing costefficient indexing, retrieval, and generation, albeit with certain constraints. Advanced RAG addresses these limitations by integrating refined indexing and retrieval techniques, optimizing data handling, and introducing strategic Retrieval and post-retrieval processes. Its capabilities include fine-tuning domain-specific embedders, employing dynamic ones for improved context comprehension, and applying reranker and prompt compression during post-retrieval processes (Ilin, 2023). Modular RAG represents a further advancement from traditional NLP frameworks by introducing specialized modules for similarity-based retrieval, fine-tuning, and problem-solving. It also incorporates innovative modules like Search and Memory (Cheng et al., 2023a), Fusion (Rackauckas, 2024), and Routing to customize RAG for specific applications and improve search and retrieval operation (Gao et al., 2023). The ongoing evolution of RAG demonstrates its potential to revolutionize information retrieval and adaptability in language model systems within the fast-evolving field of computa-
tional linguistics. In this study, we choose to assess the effectiveness of Modular RAG with the rewrite, retrieve, re-rank, and read modules following previous work (Gao et al., 2023).
3 The FinTextQA Dataset
3.1 Data Sources The data in FinTextQA are sourced from wellestablished financial literature and government agencies, such as expert-authored question-answer pairs from recognized finance textbooks: Bank Management and Financial Services (BMFS), Fundamentals of Corporate Finance (FCF), and The Economics of Money, Banking, and Financial Markets (EMBFM). Additionally, crucial information regarding financial regulations and policies is incorporated from esteemed websites such as the Hong Kong Monetary Authority (HKMA) 2, European Union (EU)3, and the Federal Reserve (FR)4. Question types encompass various domains, spanning concept explanation and numerical calculation to comparative analysis and open-ended opinionbased queries.
3.2 Selection of Policy and Regulation Data In textbooks, questions are typically straightforward, and evidence (i.e., citations) can be easily found within each chapter. However, in policies and regulations, some of the questions draw from multiple sources and may not directly align with the documents in our dataset. This poses a challenge for the model to provide accurate answers. Additionally, policy and regulation data often require deeper analytical thinking and interpretation, demanding a robust reasoning ability from the QA system. Given the complexity and importance of financial regulations and policies, we have implemented a thorough two-step verification process to ensure the relevance and accuracy of the QA pairs and the associated regulation and policy documents:
1. Evidence identification: Initially, annotators are tasked with locating relevant evidence (aka citations and references) for each questionanswer pair within the dataset. Any questions that cannot be feasibly linked to a valid citation or reference were promptly excluded from consideration;
2https://www.hkma.gov.hk/eng 3https://european-union.europa.eu/index_en 4https://www.federalreserve.gov
2. Relevance evaluation: Another distinct group of annotators evaluates the coherence and connectedness between the question, context, and answer for each entry. Using a grading scale from 1 to 5, they ensure high standards of relevancy. Only entries with a score exceeding 2 across all three variables are included in the final dataset.
Initially, we collected 300 regulation and policy question-answer pairs with related document contexts. After careful data quality control, 240 pairs were retained. The data selection process resulted in a dataset demonstrating strong relevance among answer-context (3.91), question-answer (4.88), and question-context (4.54), indicating its high quality and dependability. Further details of human evaluation can be found in Appendix A.1.
3.3 Dataset Statistics
FinTextQA contains 1,262 QA pairs, with 1,022 pairs from finance textbooks, accounting for 80.98% of the dataset, and 240 pairs from policies and regulations, accounting for 19.02% of the dataset. We randomly split the dataset into training, validation, and test sets following a 7:1:2 ratio for model fine-tuning and evaluation. Table 1 presents data distribution across different sources.
Table 1 illustrates the distribution of these questions, representing various aspects of financial regulations and policies. The European Commission subset comprises 12 questions focused on transaction regulation and its interpretations. The Federal Reserve subset, containing over 190 questions, addresses topics such as banking regulations, monetary policy strategies, and international banking operations. The Hong Kong Monetary Authority subset contains 38 questions covering anti-money laundering, counter-terrorist financing ordinance, and credit card business regulations, etc.
FinTextQA consists mainly of compound questions, where each primary question includes 2-3 related sub-questions. This hierarchical format introduces more complexity for question understanding and reasoning. These sub-questions come in different forms, leading to a variety of interrogative words as illustrated in Figure 3. Our analysis shows that 36.98% of the questions start with "what", making it the most common starting word, followed by "how" at 19.63%, "why" at 12.59%, and "can" at 6.81%. The diversity in the types of interrogative words enriches the dataset, providing a more thorough test of large language models’ ability to read and understand text.
3.4 Comparison to Existing Datasets
Table 2 shows a comparison of LFQA datasets, not limited to finance. FinTextQA stands out with an average question length of 28.5 words, answers of 75 words, and notably extended document contexts, averaging 19,779.5 words. These extensive contexts, segmented into chapters or sessions, are designed to enhance retrieval tasks. Furthermore, FinTextQA covers a broad scope, including multiturn, numerical, finance domain, and open-ended questions. It contains the most complex questions and longest answers alongside the widest scope, as compared with other finance QA datasets. Further details of question types can be found in Appendix A.4.
4 Benchmarks on FinTextQA
4.1 RAG-based LFQA system
We employ the modular RAG as discussed in (Gao et al., 2023) and follow the guidelines outlined in LlamaIndex5 to construct the RAG-based LFQA
5https://www.llamaindex.ai
system. As shown in Figure 2, this LFQA system consists of four modules: embedder, retriever, reranker, and generator. The first three modules together serve to find relevant information (aka evidence or citations) from contexts. The last module synthesizes responses using the retrieved information. Each module can be implemented by different models, and the combinations of models for all modules constitutes the system’s configurations. The selection of the models for each module is
summarzied as follows: The Embedder Module The role of the embedder module is to convert human language into a vector representation that can be understood and processed by computers. In our experiments, we adopt four popular embedding models that have achieved high rankings on the Hugging Face leaderboard, including (1) BAAI’s Bge-small-en-v1.5 (Xiao and Liu, 2023), (2) NLPer’s Gte-large (Li et al., 2023), (3) LLMRails’ Ember-v16, and (4) OpenAI’s Ada27. The Retriever Module The retriever module forms the backbone of our experiment by searching and retrieving relevant context related to a given question. We explore three retriever methods, including Auto Merging Retriever (AMR) (Liu, 2023), (2) Sentence Window Retriever (SWR) (llamaindex, 2023), and a simple vector-based retriever approach. AMR organizes documents into a hierarchical tree system with parent nodes’ contents distributed among child nodes. This enables users to determine the relevance of the parent node based on its child nodes’ relevance to the query. SWR fetches context from a custom knowledge base by
6https://huggingface.co/llmrails/ember-v1 7https://platform.openai.com/docs/guides/
embeddings
considering a broader context and retrieving sentences around the most relevant sentence. This leads to the generation of higher-quality context. Finally, the vector-based retriever approach simply searches for related context through a vector index. The Reranker Module The primary objective of rerankers is to refine the retrieved information by repositioning the most pertinent content towards the prompt edges. To accomplish this, we examine the influence of three rerankers on the overall system performance: (1) LLMRerank (Fajardo, 2023), (2) Bge-Ranker-Base8, and (3) All-Mpnet-Basev2(Song et al., 2020). The Generator Module The generator module first consolidates the query and relevant document context prepared by the former modules into a wellstructured and coherent prompt. These prompts are then fed to a LLM to generate final responses. To evaluate the performance of various LLMs, we include six sota models, including (1) Qwen-7B (Bai et al., 2023), (2) Baichuan2-7B (Yang et al., 2023a), (3)LLaMA2-7B (Touvron et al., 2023), (4) GPT-3.5-turbo, (5) Solar-10.7B (Kim et al., 2023a), and (6) Gemini-Pro (Team et al., 2023).
4.2 Experimental Settings
To ensure a thorough understanding of each model within every module in a controlled manner, we systematically tested all configurations of models in each module in the RAG-based LFQA system to determine the optimal one. All configurations are evaluated on two sets of experiments - one where the generators were fine-tuned using the training set of FinTextQA, and another without such finetuning. Note that Gemini-Pro remains a private model and is thus excluded from the fine-tuning process.
8https://huggingface.co/BAAI/ bge-reranker-base
To understand the robustness of the best systems, we select the three highest-ranking configurations based on their performance with generators in their base form. This criterion ensures a fair comparison with Gemini-Pro. We then evaluate the performance of these systems under conditions of increased noise by incrementally adding numbers of documents from one to three.
Hyperparameter settings involved in the experiments are set as follows: Retrievers For AMR, we define three levels of chunk sizes: 2048 for the first level, 512 for the second, and 128 for the third. For the SWR method, we set the window size to 3. For all retrievers, the similarity top k value was set to 6. Rerankers We set the LLMRerank batch size to 5, and the top n values of LLMRerank, BgeReranker-Base, and All-Mpnet-Base-v2 to 4. Generator We use AzureOpenai’s API 9 to access GPT-3.5-turbo and GPT-4-0314 for GPT series models. Google VertexAI API10 is used to access Gemini-Pro. The LLaMA2, Baichuan2, and Qwen models are all used in their 7B versions, while the Solar model is accessed in the 10.7B version. Finetuning of open-source models is carried out on the training set of FinTextQA. For GPT-3.5-turbo, we adopt the fine-tuning methods in Azure AI Studio11, setting the batch size to 2, learning rate multiplier to 1, and epochs to 5.
GPTQConfig (Frantar et al., 2022) is used to load the Qwen-7B model in 4-bit, the GenerationConfig (Joao Gante, 2022) for Baichuan2-7B in 4-bit, and the BitsAndBytesConfig (Belkada, 2023) for LLaMA2-7B and Solar-10.7B in 4-bit. We employ LoRA for LLaMA2-7B, Baichuan2-7B, Qwen-7B,
9https://azure.microsoft.com/en-us/products/ ai-services/openai-service
10https://cloud.google.com/vertex-ai/docs/ reference/rest
11https://oai.azure.com
and Solar-10.7B, with the rank set to 1, alpha set to 32, and dropout at 0.1. Prefix token lengths are set to 2048, learning rate to 1.0e-3, batch size to 2, and maximum input and target length to 2048. All fine-tuning efforts are performed using 12 NVIDIA RTX3090 GPUs for 10 epochs.
4.3 Evaluation Methods
4.3.1 Evaluation of Individual Modules
Embedders, Retrievers, and Rerankers. In recent studies, such as (Sottana et al., 2023; Cao et al., 2024; Liu et al., 2023; Ye et al., 2023) and (Kim et al., 2023b), the GPT-4 evaluator has been extensively tested. To evaluate the performance of these modules and their combined performance in evidence generation, we use GPT-4 to analyze the relevance between questions and retrieved citations (aka. evidence). In detail, GPT-4 is asked to grade the question-evidence relevance on a fivepoint Likert scale. The average score, referred to as the ‘GPT-4 score’, is calculated for overall performance evaluation. The prompt used for GPT4-aided evaluation is shown in Appendix A.2. Generators. To evaluate the performance of generators, we employ automatic metrics comprising matching-based measures such as ROUGE-1, ROUGE-2, and ROUGE-L (Lin, 2004), as well as the BLEU score (Papineni et al., 2002). However, since prior research (Zheng et al., 2023) shows that matching-based metrics may overestimate performance in long sequences, we also use the GPT4 evaluation method mentioned above to assess evidence-answer relevance. In addition, we report the ratio of unanswered questions in the responses (e.g., cases when models return "can not provide answer based on the content).
4.3.2 Overall Evaluation of the RAG-based LFQA System
ROUGE (Lin, 2004) and BLEU (Papineni et al., 2002)) are used to automatically measure the overall system performance. The GPT-4 scoring method is used to evaluate the answers from helpfulness, relevance, accuracy, depth, and creativity. Additionally, we invite three annotators to rank top-performing answers from all tested models and compare them with the ground truth answers, capturing human perception and assessing subjective response quality. Further details of human evaluation can be found in Appendix A.1.
4.4 Results
Embedders, Retrievers, and Rerankers Table 4 shows the GPT-4 score of different embedders, retrievers, and rerankers, which constitute the evidence generation pipeline. It also shows at the end the best-performing evidence-generation module combinations. We observe that the highestperforming embedding model is Ada2, achieving a score of 4.586, followed by Ember-v1 (4.486) and Bge-Small-en-v1.5 (4.455) with similar scores. Gte-Large lagged with a noticeable gap with a score of 4.261. Among the retrievers we assess, AMR outperforms the rest with an impressive score of 4.492. SWR ranks second at 4.466, while the simple vector-based approach has the lowest performance with a score of 4.358.
Among the rerankers, Bge-Reranker-Base performs the best, achieving a competitive score of 4.489. LLMRerank ranks second with a score of 4.469, followed by All-Mpnet-Base-v2 with a score of 4.383.
The evidence generation modules together, we observe that the combination of AMR, Ada2, and Bge-Reranker-Base yields the highest score of 4.622, followed by the combination of SWR, Ada2, and Bge-Reranker-Base/All-Mpnet-Base-v2, with a score of 4.620. The marginal differences in performance among these leading combinations indicate that a variety of configurations are capable of yielding satisfactory outcomes for evidence generation. Generators Table 5 shows the comparison of different generators, contrasted by their base form and fine-tuned form. Although the fine-tuned models have a decreasing loss (from 2.5 to 0.1), they do not have significant improvement. They demonstrate slightly lower performance in terms of GPT-4 score, ROUGE-L, and BLEU scores. However, fine-tuned models have less unanswered questions, showing better understanding capabilities than their base forms.
We also observe that while Gemini-Pro shows high numeric scores, it struggles the most in generating contextually relevant responses. Conversely, Baichuan2-7B demonstrates the best prompt com-
prehension ability. GPT-3.5-turbo experiences more difficulty with contextual understanding, affecting its overall performance, while LLaMA2 has minimal context-related problems. However, LLaMA2 generates instances of simply rephrasing prompts, resulting in reduced accuracy scores. RAG-based LFQA System Table 3 shows the performance comparison of RAG-based LFQA system with generators in their base forms. We observe that system with the top-3 performing configurations incorporate GPT-3.5-turbo and Baichuan27B as generators. In contrast, system configurations using the Gemini-Pro generator yield suboptimal performance in terms of accuracy. Meanwhile, we observe that system employing LLaMA2-7B as generators show the lowest ROUGE and BLEU scores among all the configurations tested. More results on the performances of different generators in the RAG systems are provided in Appendix A.3.
The top-scoring system configurations comprise (1) GPT-3.5-turbo, AMR, Ada2, and LLMRERanker (noted as system 1); GPT-3.5-turbo, AMR, Ember-v1, and Bge-Reranker-Base (system 2); (3) Baichuan2-7B, AMR, Ada2, and BgeReranker-Base (system 3).
Table 6 shows the annotator-ranked preference of these top system configurations. We notice that some model-generated answers obtain higher average rankings than corresponding ground truths. For instance, Answer 2, produced by system 3, attains an average ranking of 2.11, outperforming the ground truth (2.19). Further investigation into annotator feedback reveals that annotators favor Answer 2 because it gives accurate responses while providing additional details. Answer 3, generated by system 1 performs the worst with the highest average ranking (3.10). Answer 4, generated by system 2, achieves an average ranking of 2.60. Best System Configuration in Multi-Document Settings Figure 4 shows the performance of the
three best-performing system configurations when given different numbers (n = 1 to 3) of documents. We observe a consistent pattern from the results: as the number of input documents increases, all system performance tend to decline. However, exceptions are also noted. For instance, the scores for certain instances with three documents marginally surpasses those with two documents in system 2 when compared to the accuracy score. Further investigation shows that the performance is dependent on the total context words of the input. When the number of context words reaches about 34k words, adding more input documents exerts a less marginal effect on system performance.
Performance of Generators on Regulation and Textbook Dataset Table 11 presents the performance results of all the generators on regulationbased and textbook-based questions. For the regulation and textbook datasets, the Rouge-1 scores are 0.317 and 0.270, respectively; the Rouge-2 scores are 0.146 and 0.084, respectively; the Rouge-L scores are 0.215 and 0.170; the BLEU scores are 0.074 and 0.031; and the GPT-4 scores are 4.238 and 4.293. These results indicate that the LFQA system performs better on the policy/regulation dataset. Since the textbook dataset consists of questions derived from textbook exercises, each question is closely tied to the text content. After annotating the data, 60 policy/regulation data items were removed to ensure that each remaining question had a corresponding answer in the document. Therefore, the quality of the dataset does not affect the performance of the LFQA system. Based on our current findings, the textbook dataset is more challenging.
Performance of Generators on Different Question Types Concerning overall performance on different question types, the models excelled at answering open-ended questions but struggled with numerical ones (Table 10). Specifically, GPT-3.5
Turbo demonstrated the strongest performance in open-ended questions, while Llama2-7B underperformed. Baichuan2-7B showed versatility in handling various question types, but its competence in addressing numerical questions was subpar, placing it third. Remarkably, among all models, Qwen7B excelled in numerical ability, securing the top position across all metrics.
5 Conclusion
This study presents FinTextQA, an LFQA dataset specifically designed for the financial domain. The dataset is comprehensive, covering complex financial question systems and including queries on financial regulations and policies. This makes it a valuable resource for further research and evaluation of RAG modules and large language models. We also introduce a robust evaluation system that leverages human ranking, automatic metrics, and GPT-4 scoring to assess various facets of model performance. Our results suggest that the most effective combination of models and modules for finance-related LFQA tasks includes Ada2, AMR, Bge-Reranker-Base, and Baichuan2-7B.
Limitations
Despite its expert curation and high quality, FinTextQA contains a relatively smaller number of QA pairs compared to larger AI-generated datasets. This limitation could potentially affect the generalizability of models trained on it when applied to broader real-world applications. High-quality data are challenging to acquire, and copyright restrictions often prevent sharing. Therefore, future research should concentrate on data augmentation and the development of innovative methods to address data scarcity. Expanding the dataset by incorporating more diverse sources and exploring advanced RAG capabilities and retrieval frameworks could also be beneficial.
Ethical Statement
In this study, we uphold rigorous ethical standards and endeavor to mitigate any potential risks.
• While constructing our dataset, we meticulously ensure that all data are acquired through lawful and ethical means. Adhering to the Fair Use principle, the dataset is exclusively utilized for academic research purposes and is strictly prohibited from commercial exploitation.
• We bear the responsibility of openly sharing the interface, dataset, codes, and trained models with the public. Nonetheless, there exists a possibility of malicious misuse of these resources. For instance, our models could be employed to generate responses without appropriately crediting the information source. We are committed to ensuring their ethical use and guarding against any malicious or harmful intent.
• We are dedicated to mitigating bias, discrimination, or stereotypes during annotation by systematically excluding any questionable examples. To achieve this, we provide thorough training to annotators using 20 samples until they achieve an average accuracy score of 3.8 out of 5. We continually assess their performance throughout the annotation process. Additionally, we provide compensation of $114 per day to annotators until the completion of the annotation task.
Acknowledgements
This work was supported by the National Natural Science Foundation of China (No. 62306257) and the Guangzhou Municipal Science and Technology Project (No. 2024A04J4390). The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the National Natural Science Foundation, or the Guangzhou Government.
This document is not intended as investment research or investment advice, or a recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction.
Finally, we are also grateful to HSBC Lab and HSBC Global Payment Solutions Department for their substantial financial support.
References Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang,
Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.
Younes Belkada. 2023. Bitsandbytes.
Valeriia Bolotova-Baranova, Vladislav Blinov, Sofya Filippova, Falk Scholer, and Mark Sanderson. 2023.
Wikihowqa: A comprehensive benchmark for multidocument non-factoid question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5291–5314.
Abdelghani Bouziane, Djelloul Bouchiha, Noureddine Doumi, and Mimoun Malki. 2015. Question answering systems: survey and trends. Procedia Computer Science, 73:366–375.
Meng Cao, Long Chen, Mike Zheng Shou, Can Zhang, and Yuexian Zou. 2021. On pursuit of designing multi-modal transformer for video grounding. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9810– 9823.
Meng Cao, Haoran Tang, Jinfa Huang, Peng Jin, Can Zhang, Ruyang Liu, Long Chen, Xiaodan Liang, Li Yuan, and Ge Li. 2024. Rap: Efficient text-video retrieval with sparse-and-correlated adapter. arXiv preprint arXiv:2405.19465.
Meng Cao, Tianyu Yang, Junwu Weng, Can Zhang, Jue Wang, and Yuexian Zou. 2022. Locvtp: Videotext pre-training for temporal localization. In European Conference on Computer Vision, pages 38–56. Springer.
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, et al. 2021. Finqa: A dataset of numerical reasoning over financial data. arXiv preprint arXiv:2109.00122.
Tingting Cheng. 2021. A multidimensional analysis: speaking style of learner speech across proficiency levels. In Proceedings of the 35th Pacific Asia Conference on Language, Information and Computation, pages 149–159, Shanghai, China. Association for Computational Lingustics.
Xin Cheng, Di Luo, Xiuying Chen, Lemao Liu, Dongyan Zhao, and Rui Yan. 2023a. Lift yourself up: Retrieval-augmented text generation with self memory. arXiv preprint arXiv:2305.02437.
Xuxin Cheng, Bowen Cao, Qichen Ye, Zhihong Zhu, Hongxiang Li, and Yuexian Zou. 2023b. Ml-lmcl: Mutual learning and large-margin contrastive learning for improving asr robustness in spoken language understanding. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6492– 6505.
Xuxin Cheng, Zhihong Zhu, Bowen Cao, Qichen Ye, and Yuexian Zou. 2023c. Mrrl: Modifying the reference via reinforcement learning for nonautoregressive joint multiple intent detection and slot filling. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10495– 10505.
Xuxin Cheng, Zhihong Zhu, Hongxiang Li, Yaowei Li, Xianwei Zhuang, and Yuexian Zou. 2024. Towards multi-intent spoken language understanding via hierarchical attention and optimal transport. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17844–17852.
Xuxin Cheng, Zhihong Zhu, Wanshi Xu, Yaowei Li, Hongxiang Li, and Yuexian Zou. 2023d. Accelerating multiple intent detection and slot filling via targeted knowledge distillation. In The 2023 Conference on Empirical Methods in Natural Language Processing.
Roberto Confalonieri, Ludovik Coba, Benedikt Wagner, and Tarek R Besold. 2021. A historical perspective of explainable artificial intelligence. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 11(1):e1391.
Andrei Fajardo. 2023. Llm reranker demonstration (great gatsby).
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. Eli5: Long form question answering. arXiv preprint arXiv:1907.09190.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929–3938. PMLR.
Wookje Han, Jinsol Park, and Kyungjae Lee. 2023. Prewome: Exploiting presuppositions as working memory for long form question answering. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8312–8322.
Ivan Ilin. 2023. Advanced rag techniques: an illustrated overview.
Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. 2023. Financebench: A new benchmark for financial question answering. arXiv preprint arXiv:2311.11944.
Sylvain Gugger Joao Gante. 2022. [link].
Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, et al. 2023a. Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling. arXiv preprint arXiv:2312.15166.
Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. 2023b. Prometheus: Inducing fine-grained evaluation capability in language models. arXiv preprint arXiv:2310.08491.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
Jerry Liu. 2023. Auto merging retriever.
Junling Liu, Ziming Wang, Qichen Ye, Dading Chong, Peilin Zhou, and Yining Hua. 2023. Qilin-med-vl: Towards chinese large vision-language model for general healthcare. arXiv preprint arXiv:2310.17956.
llamaindex. 2023. Sentence window retriever.
Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. Www’18 open challenge: financial opinion mining and question answering. In Companion proceedings of the the web conference 2018, pages 1941–1942.
Yangjun Mao, Jun Xiao, Dong Zhang, Meng Cao, Jian Shao, Yueting Zhuang, and Long Chen. 2023. Improving reference-based distinctive image captioning with contrastive rewards. arXiv preprint arXiv:2306.14259.
Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang, et al. 2023. Webcpm: Interactive web search for chinese long-form question answering. arXiv preprint arXiv:2305.06849.
Zackary Rackauckas. 2024. Rag-fusion: a new take on retrieval-augmented generation. arXiv preprint arXiv:2402.03367.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2020. Mpnet: Masked and permuted pretraining for language understanding. Advances in Neural Information Processing Systems, 33:16857– 16867.
Andrea Sottana, Bin Liang, Kai Zou, and Zheng Yuan. 2023. Evaluation metrics in the era of gpt-4: reliably evaluating large language models on sequence to sequence tasks. arXiv preprint arXiv:2310.13800.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
Neil C Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel F Manso. 2020. The computational limits of deep learning. arXiv preprint arXiv:2007.05558.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and Hong Lin. 2023. Ai-generated content (aigc): A survey. arXiv preprint arXiv:2304.06632.
Shitao Xiao and Zheng Liu. 2023. [link].
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023a. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305.
Bang Yang, Meng Cao, and Yuexian Zou. 2023b. Concept-aware video captioning: Describing videos with effective prior information. IEEE Transactions on Image Processing.
Qichen Ye, Junling Liu, Dading Chong, Peilin Zhou, Yining Hua, and Andrew Liu. 2023. Qilinmed: Multi-stage knowledge injection advanced medical large language model. arXiv preprint arXiv:2310.09089.
Zhiqiang Yuan, Junwei Liu, Qiancheng Zi, Mingwei Liu, Xin Peng, and Yiling Lou. 2023a. Evaluating instruction-tuned large language models on code comprehension and generation. arXiv preprint arXiv:2308.01240.
Zixuan Yuan, Hao Liu, Renjun Hu, Denghui Zhang, and Hui Xiong. 2021. Self-supervised prototype representation learning for event-based corporate profiling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 4644–4652.
Zixuan Yuan, Junming Liu, Haoyi Zhou, Denghui Zhang, Hao Liu, Nengjun Zhu, and Hui Xiong. 2023b. Lever: Online adaptive sequence learning framework for high-frequency trading. IEEE Transactions on Knowledge and Data Engineering.
Zixuan Yuan, Yada Zhu, Wei Zhang, and Hui Xiong. 2023c. Earnings call analysis using a sparse attention based encoder and multi-source counterfactual augmentation. In Proceedings of the Fourth ACM International Conference on AI in Finance, pages 331–339.
Peiyan Zhang, Haoyang Liu, Chaozhuo Li, Xing Xie, Sunghun Kim, and Haohan Wang. 2023. Foundation model-oriented robustness: Robust image model evaluation with pretrained models. arXiv preprint arXiv:2308.10632.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685.
Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. arXiv preprint arXiv:2105.07624.
A Appendix
A.1 Evaluate Human Performance In our study, we’ve made it a priority to closely consider the human aspect of performance, covering everything from how we gather data to how we evaluate the answers produced. Our team of annotators, all of whom hold master’s degrees with their education conducted in English, play a crucial role in this process.
To accurately identify citations and references, we’ve laid out a detailed five-step annotation process. At the outset, we provide our team of three annotators with a benchmark example of what a correct citation looks like (shown in Table 9). This serves to clarify several critical criteria: how well the answer fits with the context (Groundedness), how relevant the answer is to the question asked (Answer Relevance), and how the question relates to the context provided (Context Relevance).
After this initial step, the annotators first conduct a practice round involving 20 data samples. Here, they compare their citation identifications against gold standard annotations. We then score their findings and give them feedback to help refine their skills. Once they’ve shown they’ve got a good handle on the process, they move on to four more rounds, each with an increasing number of data samples to work through. By the end of this process, after completing 300 tasks, our annotators are well-versed in our annotation standards, ensuring a high level of accuracy in our data collection and analysis.
In each round, we randomly select 10% of the samples for evaluating annotator performance (Table 8). To minimize potential biases, we engage another three annotators to rate relevance and accuracy using a 5-point Likert Scale. Table 7 presents the performance of annotators, revealing that the average scores are above 4 after the first round’s training. Context Relevance and Answer Relevance scores are above 3.
The relevance scores in the 5th round are comparatively lower due to the difficulty in finding citations in many pairs. Ultimately, we remove 60 pairs with relevance scores lower than 2 or those lacking citations in the document, retaining 240 pairs for further analysis. This rigorous evaluation and annotation process ensures the quality of FinTextQA.
During the answer evaluation phase, annotator competence is measured through their performance
Ground Truth Citation 1. Below the notification thresholds, the Commission should be able to require the notification of potentially subsidised concentrations that were not yet implemented or the notification of potentially subsidised bids prior to the award of a contract, if it considers that the concentration or the bid would merit ex ante review given its impact in the Union. 2. The Commission may request the prior notification of any concentration which is not a notifiable concentration within the meaning of Article 20 at any time prior to its implementation where the Commission suspects that foreign subsidies may have been granted to the undertakings concerned in the three years prior to the concentration. Such concentration shall be deemed to be a notifiable concentration for the purposes of this Regulation. 3. By way of derogation from paragraph 2 of this Article, Articles 21 and 29 shall apply from 12 October 2023. Annotator Citation "1. This Regulation shall enter into force on the twentieth day following that of its publication in the Official Journal of the European Union. 2. It shall apply from 12 July 2023. 3. By way of derogation from paragraph 2 of this Article, Articles 47 and 48 shall apply from 11 January 2023and Article 14(5), (6) and (7) shall apply from 12 January 2024. 4. By way of derogation from paragraph 2 of this Article, Articles 21 and 29 shall apply from 12 October 2023." Score: 3/5 Feedback: Only select part of the citations, which can not fully answer the question.
Table 8: An example of scoring evidence found by annotators
in three TOEFL reading tests, ensuring strong reading comprehension skills. Proceeding to the ranking of generated answers, several responses — including ground truth answers — are presented without revealing their origin. If ground truth answers rank too low, the evaluation is considered unsuitable; if ranked within the top two among four responses, the evaluation is considered appropriate.
Annotators analyzed 253 question-answer (QA) pairs by comparing the correct answers to the top 3 answers generated by leading long-form question answering (LFQA) systems, specifically Baichuan2-7B and GPT-3.5 Turbo. This analysis produced four different types of answers for preference ranking. Taking individual differences into account, the agreement scores (Krippendorff’s Alpha) for these rankings are: 0.728, 0.647, 0.679,
and 0.706, resulting in an overall agreement score of 0.750. This indicates a high level of consistency.
A.2 Prompt of GPT-4-aided Evaluation Figure 5 shows the prompt we use to ask GPT-4 to evaluate the relevance and accuracy of modelgenerated answers in our experiments.
A.3 Experiment Results of Different Generators in RAG Systems
Table 12 - 17 shows a systematic performance comparison of RAG systems with different models in each module.
A.4 Example of Question Types Table 18 - 23 shows the samples of QA pairs in each question type.
Institute European Union Document REGULATION (EU) 2022/2560 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL Question Are transactions signed between 12 July 2023 and 12 Oc-
tober 2023 (and implemented on 12 October 2023 or later) subject to mandatory notification under Regulation EU 2022/2560?
Answer Notifiable concentrations under Article 20 of Regulation EU 2022/2560 for which the agreement was concluded on 12 July 2023 or later but which have not yet been implemented on 12 October 2023, will need to be notified pursuant to Article 21 of Regulation EU 2022/2560 and are subject to the standstill obligation under Article 24 of Regulation EU 2022/2560. By contrast, the notification obligation does not apply to concentrations for which the agreement was concluded on 12 July 2023 or later but which are implemented before 12 October 2023. Notifying Parties are encouraged to engage in pre-notification contacts, in principle as of September 2023, in advance to facilitate the submission of notifications as from 12 October 2023. Citation Notifiable concentrations under Article 20 of Regulation EU 2022/2560 for which the agreement was concluded on 12 July 2023 or later but which have not yet been implemented on 12 October 2023, will need to be notified pursuant to Article 21 of Regulation EU 2022/2560 and are subject to the standstill obligation under Article 24 of Regulation EU 2022/2560. By contrast, the notification obligation does not apply to concentrations for which the agreement was concluded on 12 July 2023 or later but which are implemented before 12 October 2023. Notifying Parties are encouraged to engage in pre-notification contacts, in principle as of September 2023, in advance to facilitate the submission of notifications as from 12 October 2023. Groundedness 5 Answer Relevance 5 Context Relevance 5
Table 9: A Sample of Ground Truth Annotations
Question Type Numerical Reasoning Amount 63 Question Critics have charged that compensation to top managers in the
United States is simply too high and should be cut back. For example, focusing on large corporations, Mark Parker, CEO of Nike, earned about $47.6 million in 2016. Are such amounts excessive? In answering, it might be help- ful to recognize that superstar athletes such as LeBron James, top entertainers such as Taylor Swift and Dwayne Johnson, and many others at the top of their respective fields earn at least as much, if not a great deal more.
Answer "How much is too much? Who is worth more, Mark Parker or LeBron James? The simplest answer is that there is a market for executives just as there is for all types of labor. Executive compensation is the price that clears the market. The same is true for athletes and performers. Having said that, one aspect of executive compensation deserves comment. A primary reason executive compensation has grown so dramatically is that companies have increasingly moved to stock-based compensation. Such movement is obviously consistent with the attempt to better align stockholder and management interests. In recent years, stock prices have soared, so management has cleaned up. It is sometimes argued that much of this reward is due to rising stock prices in general, not managerial performance. Perhaps in the future, executive compensation will be designed to reward only differential performance, that is, stock price increases in excess of general market increases.
Table 18: An Example of Numerical Reasoning QA Pairs
Question Type Multi-turn Amount 356 Question Who owns a corporation? Describe the process whereby the own-
ers control the firm’s management. What is the main reason that an agency relationship exists in the corporate form of organization? In this context, what kinds of problems can arise?
Answer In the corporate form of ownership, the shareholders are the owners of the firm. The shareholders elect the directors of the corporation, who in turn appoint the firm’s management. This separation of ownership from control in the corporate form of organization is what causes agency problems to exist. Management may act in its own or someone else’s. best interests, rather than those of the shareholders. If such events occur, they may contradict the goal of maximizing the share price of the equity of the firm.
Table 19: An Example of Multi-turn QA Pairs
Question Type Finance Domain Knowledge Amount 795 Question What is a pro forma statement of cash flows and what is its purpose? Answer A pro forma statement of cash flows estimates the borrower’s
Question Type Comparative Analysis Amount 392 Question Suppose a company has a preferred stock issue and a common
stock issue. Both have just paid a $2 dividend. Which do you think will have a higher price, a share of the preferred or a share of the common?
Answer The common stock probably has a higher price because the dividend can grow, whereas it is fixed on the preferred. However, the preferred is less risky because of the dividend and liquidation preference, so it is possible the preferred could be worth more, depending on the circumstances.
Table 21: Example of Comparative Analysis QA Pair
Question Type Open-minded Amount 102 Question Suppose you were the financial manager of a not-for-profit busi-
ness (a not-for-profit hospital, perhaps). What kinds of goals do you think would be appropriate?
Answer Such organizations frequently pursue social or political missions, so many different goals are conceivable. One goal that is often cited is revenue minimization; that is, provide whatever goods and services are offered at the lowest possible cost to society. A better approach might be to observe that even a not-for-profit business has equity. Thus, one answer is that the appropriate goal is to maximize the value of the equity.
Table 22: Example of Open-minded QA Pair
Question Type Cause and Effect Analysis Amount 688 Question Last month, Central Virginia Power Company, which had been
having trouble with cost overruns on a nuclear power plant that it had been building, announced that it was temporarily suspending payments due to the cash flow crunch associated with its investment program. The company stock price dropped from $28.50 to $25 when this announcement was made. How would you interpret this change in the stock price (that is, what would you say caused it)?
Answer The stock price dropped because of an expected drop in future dividends. Since the stock price is the present value of all future dividend payments, if the expected future dividend payments decrease, then the stock price will decline.
Table 23: Example of Cause and Effect Analysis QA Pair
Question Refer to the previous exercise. Which type of foreign market intervention must the central bank of Colombia conduct to keep the exchange rate at a level where the currency is not under- or overvalued in terms of PPP? Baichuan2-7B To answer the above question, we need to know the current exchange rate of the Colombian peso relative to the U.S. dollar. Assuming the current exchange rate is not under- or overvalued in terms of PPP, the central bank of Colombia should conduct sterilized intervention to maintain this exchange rate. Sterilized intervention involves selling foreign assets (e.g., dollars) and purchasing domestic assets (e.g., pesos) to offset the effect of the intervention on the money supply. If the central bank sells dollars and buys pesos, the money supply in pesos will increase while the money supply in dollars will decrease. This will cause the value of the peso to appreciate relative to the dollar, maintaining the current exchange rate. GPT-3.5-turbo Based on the content provided, the information necessary to answer the question is not present. The content does not mention the central bank of Colombia or provide any information about the exchange rate of the Colombian currency. Therefore, we cannot determine which type of foreign market intervention the central bank of Colombia must conduct to keep the exchange rate at a level where the currency is not under- or overvalued in terms of PPP. Ground Truth To eliminate the overvaluation in terms of PPP, the exchange rate for the Columbian peso needs to decline. The central bank of Colombia should undertake an unsterilized foreign market intervention in which it will sell the domestic currency (COP) and buy foreign reserves, in order to increase reserves of its banking system, decrease the domestic interest rate, and shift the expected return on domestic currency denominated assets curve to the left.
Table 24: An Example of Unanswered Questions. We compare answers generated by GPT-3.5-turbo and Baichuan27B with the same embedder, retriever, and Reranker.

heading: Future Work
text: In addition to introducing the dataset, we conduct comprehensive benchmarking of state-of-theart (sota) models on FinTextQA to provide baselines for future research. Current LFQA systems frequently solely rely on fine-tuning pre-trained language models such as GPT-3. 5-turbo, LLaMA2 (Touvron et al. , 2023), Baichuan2 (Yang et al. , 2023a), etc. , which often fail to provide detailed explanations or effectively handling complicated finance questions (Yuan et al. , 2023a). In response, we opt for the Retrieval-augmented generation (RAG) framework, as illustrated in Figure 2. By processing documents in multiple steps, RAG systems can pre-process and provide the most relevant information to LLMs, enhancing their performance and explanation capabilities (Guu et al. , 2020). We believe this work, by introducing the first LFQA financial dataset and conducting comprehen-
sive benchmark experiments on the dataset, marks a milestone in advancing the comprehension of financial concepts and enhancing assistance in this field: FinTextQA offers a rich and rigorous framework for building and assessing the capabilities of general finance LFQA systems. Our experimental analysis not only highlights the efficacy of various model configurations but also underscores the critical need for enhancing current methodologies to improve both the precision and explicability of financial question-answering systems. 2 Related Work
2. 1 Long-Form Question Answering (LFQA) Compared to the conventional question-answering task (Bouziane et al. , 2015; Mao et al. , 2023; Yang et al. , 2023b; Cao et al. , 2021; Cheng et al. , 2023d, 2024), the goal of LFQA is to generate comprehensive, paragraph-length responses by retrieving and assimilating relevant information from various sources (Fan et al. , 2019). This poses a significant test for current Natural Language Processing (NLP) and Artificial Intelligence (AI) models, given their limited understanding and learning capacities (Thompson et al. , 2020; Cao et al. , 2022; Zhang et al. , 2023; Cheng et al. , 2023b,c; Cheng, 2021). Several LFQA datasets are available in general domain, including ELI5 (Fan et al. , 2019), WikiHowQA (Bolotova-Baranova et al. , 2023), and We-
bCPM (Qin et al. , 2023). In the financial domain, some QA datasets have been developed. However, none of them addresses LFQA. While these datasets like FinQA (Chen et al. , 2021) and TATQA (Zhu et al. , 2021) address specific scopes such as numerical reasoning, they do not touch upon general LFQA tasks. In addition, FIQA (Maia et al. , 2018) only provides short-context documents, which may not adequately represent real-life scenarios and have limited industry applicability. Although FinanceBench (Islam et al. , 2023) does cover a wider scope, it only offers 150 open-source question-answer pairs, while the question complexity and answer do not satisfy real-life LFQA scenarios. 2. 2 Retrieval-Augmented Generation (RAG)
RAG frameworks represent a significant advancement in LFQA, incorporating external knowledge sources and In-Context Learning (ICL) for efficient information retrieval and application. By combining diverse documents into comprehensive prompts, RAG enables language models to generate contextually informed responses without task-specific retraining (Lewis et al. , 2020). The evolution of RAG involves three principal stages: Naive RAG, Advanced RAG, and Modular RAG. Naive RAG offers improvements over traditional language models by providing costefficient indexing, retrieval, and generation, albeit with certain constraints. Advanced RAG addresses these limitations by integrating refined indexing and retrieval techniques, optimizing data handling, and introducing strategic Retrieval and post-retrieval processes. Its capabilities include fine-tuning domain-specific embedders, employing dynamic ones for improved context comprehension, and applying reranker and prompt compression during post-retrieval processes (Ilin, 2023). Modular RAG represents a further advancement from traditional NLP frameworks by introducing specialized modules for similarity-based retrieval, fine-tuning, and problem-solving. It also incorporates innovative modules like Search and Memory (Cheng et al. , 2023a), Fusion (Rackauckas, 2024), and Routing to customize RAG for specific applications and improve search and retrieval operation (Gao et al. , 2023). The ongoing evolution of RAG demonstrates its potential to revolutionize information retrieval and adaptability in language model systems within the fast-evolving field of computa-
tional linguistics. In this study, we choose to assess the effectiveness of Modular RAG with the rewrite, retrieve, re-rank, and read modules following previous work (Gao et al. , 2023). 3 The FinTextQA Dataset
3. 1 Data Sources The data in FinTextQA are sourced from wellestablished financial literature and government agencies, such as expert-authored question-answer pairs from recognized finance textbooks: Bank Management and Financial Services (BMFS), Fundamentals of Corporate Finance (FCF), and The Economics of Money, Banking, and Financial Markets (EMBFM). Additionally, crucial information regarding financial regulations and policies is incorporated from esteemed websites such as the Hong Kong Monetary Authority (HKMA) 2, European Union (EU)3, and the Federal Reserve (FR)4. Question types encompass various domains, spanning concept explanation and numerical calculation to comparative analysis and open-ended opinionbased queries. 3. 2 Selection of Policy and Regulation Data In textbooks, questions are typically straightforward, and evidence (i. e. , citations) can be easily found within each chapter. However, in policies and regulations, some of the questions draw from multiple sources and may not directly align with the documents in our dataset. This poses a challenge for the model to provide accurate answers. Additionally, policy and regulation data often require deeper analytical thinking and interpretation, demanding a robust reasoning ability from the QA system. Given the complexity and importance of financial regulations and policies, we have implemented a thorough two-step verification process to ensure the relevance and accuracy of the QA pairs and the associated regulation and policy documents:
1. Evidence identification: Initially, annotators are tasked with locating relevant evidence (aka citations and references) for each questionanswer pair within the dataset. Any questions that cannot be feasibly linked to a valid citation or reference were promptly excluded from consideration;
2https://www. hkma. gov. hk/eng 3https://european-union. europa. eu/index_en 4https://www. federalreserve. gov
2. Relevance evaluation: Another distinct group of annotators evaluates the coherence and connectedness between the question, context, and answer for each entry. Using a grading scale from 1 to 5, they ensure high standards of relevancy. Only entries with a score exceeding 2 across all three variables are included in the final dataset. Initially, we collected 300 regulation and policy question-answer pairs with related document contexts. After careful data quality control, 240 pairs were retained. The data selection process resulted in a dataset demonstrating strong relevance among answer-context (3. 91), question-answer (4. 88), and question-context (4. 54), indicating its high quality and dependability. Further details of human evaluation can be found in Appendix A. 1. 3. 3 Dataset Statistics
FinTextQA contains 1,262 QA pairs, with 1,022 pairs from finance textbooks, accounting for 80. 98% of the dataset, and 240 pairs from policies and regulations, accounting for 19. 02% of the dataset. We randomly split the dataset into training, validation, and test sets following a 7:1:2 ratio for model fine-tuning and evaluation. Table 1 presents data distribution across different sources. Table 1 illustrates the distribution of these questions, representing various aspects of financial regulations and policies. The European Commission subset comprises 12 questions focused on transaction regulation and its interpretations. The Federal Reserve subset, containing over 190 questions, addresses topics such as banking regulations, monetary policy strategies, and international banking operations. The Hong Kong Monetary Authority subset contains 38 questions covering anti-money laundering, counter-terrorist financing ordinance, and credit card business regulations, etc. FinTextQA consists mainly of compound questions, where each primary question includes 2-3 related sub-questions. This hierarchical format introduces more complexity for question understanding and reasoning. These sub-questions come in different forms, leading to a variety of interrogative words as illustrated in Figure 3. Our analysis shows that 36. 98% of the questions start with "what", making it the most common starting word, followed by "how" at 19. 63%, "why" at 12. 59%, and "can" at 6. 81%. The diversity in the types of interrogative words enriches the dataset, providing a more thorough test of large language models’ ability to read and understand text. 3. 4 Comparison to Existing Datasets
Table 2 shows a comparison of LFQA datasets, not limited to finance. FinTextQA stands out with an average question length of 28. 5 words, answers of 75 words, and notably extended document contexts, averaging 19,779. 5 words. These extensive contexts, segmented into chapters or sessions, are designed to enhance retrieval tasks. Furthermore, FinTextQA covers a broad scope, including multiturn, numerical, finance domain, and open-ended questions. It contains the most complex questions and longest answers alongside the widest scope, as compared with other finance QA datasets. Further details of question types can be found in Appendix A. 4. 4 Benchmarks on FinTextQA
4. 1 RAG-based LFQA system
We employ the modular RAG as discussed in (Gao et al. , 2023) and follow the guidelines outlined in LlamaIndex5 to construct the RAG-based LFQA
5https://www. llamaindex. ai
system. As shown in Figure 2, this LFQA system consists of four modules: embedder, retriever, reranker, and generator. The first three modules together serve to find relevant information (aka evidence or citations) from contexts. The last module synthesizes responses using the retrieved information. Each module can be implemented by different models, and the combinations of models for all modules constitutes the system’s configurations. The selection of the models for each module is
summarzied as follows: The Embedder Module The role of the embedder module is to convert human language into a vector representation that can be understood and processed by computers. In our experiments, we adopt four popular embedding models that have achieved high rankings on the Hugging Face leaderboard, including (1) BAAI’s Bge-small-en-v1. 5 (Xiao and Liu, 2023), (2) NLPer’s Gte-large (Li et al. , 2023), (3) LLMRails’ Ember-v16, and (4) OpenAI’s Ada27. The Retriever Module The retriever module forms the backbone of our experiment by searching and retrieving relevant context related to a given question. We explore three retriever methods, including Auto Merging Retriever (AMR) (Liu, 2023), (2) Sentence Window Retriever (SWR) (llamaindex, 2023), and a simple vector-based retriever approach. AMR organizes documents into a hierarchical tree system with parent nodes’ contents distributed among child nodes. This enables users to determine the relevance of the parent node based on its child nodes’ relevance to the query. SWR fetches context from a custom knowledge base by
6https://huggingface. co/llmrails/ember-v1 7https://platform. openai. com/docs/guides/
embeddings
considering a broader context and retrieving sentences around the most relevant sentence. This leads to the generation of higher-quality context. Finally, the vector-based retriever approach simply searches for related context through a vector index. The Reranker Module The primary objective of rerankers is to refine the retrieved information by repositioning the most pertinent content towards the prompt edges. To accomplish this, we examine the influence of three rerankers on the overall system performance: (1) LLMRerank (Fajardo, 2023), (2) Bge-Ranker-Base8, and (3) All-Mpnet-Basev2(Song et al. , 2020). The Generator Module The generator module first consolidates the query and relevant document context prepared by the former modules into a wellstructured and coherent prompt. These prompts are then fed to a LLM to generate final responses. To evaluate the performance of various LLMs, we include six sota models, including (1) Qwen-7B (Bai et al. , 2023), (2) Baichuan2-7B (Yang et al. , 2023a), (3)LLaMA2-7B (Touvron et al. , 2023), (4) GPT-3. 5-turbo, (5) Solar-10. 7B (Kim et al. , 2023a), and (6) Gemini-Pro (Team et al. , 2023). 4. 2 Experimental Settings
To ensure a thorough understanding of each model within every module in a controlled manner, we systematically tested all configurations of models in each module in the RAG-based LFQA system to determine the optimal one. All configurations are evaluated on two sets of experiments - one where the generators were fine-tuned using the training set of FinTextQA, and another without such finetuning. Note that Gemini-Pro remains a private model and is thus excluded from the fine-tuning process. 8https://huggingface. co/BAAI/ bge-reranker-base
To understand the robustness of the best systems, we select the three highest-ranking configurations based on their performance with generators in their base form. This criterion ensures a fair comparison with Gemini-Pro. We then evaluate the performance of these systems under conditions of increased noise by incrementally adding numbers of documents from one to three. Hyperparameter settings involved in the experiments are set as follows: Retrievers For AMR, we define three levels of chunk sizes: 2048 for the first level, 512 for the second, and 128 for the third. For the SWR method, we set the window size to 3. For all retrievers, the similarity top k value was set to 6. Rerankers We set the LLMRerank batch size to 5, and the top n values of LLMRerank, BgeReranker-Base, and All-Mpnet-Base-v2 to 4. Generator We use AzureOpenai’s API 9 to access GPT-3. 5-turbo and GPT-4-0314 for GPT series models. Google VertexAI API10 is used to access Gemini-Pro. The LLaMA2, Baichuan2, and Qwen models are all used in their 7B versions, while the Solar model is accessed in the 10. 7B version. Finetuning of open-source models is carried out on the training set of FinTextQA. For GPT-3. 5-turbo, we adopt the fine-tuning methods in Azure AI Studio11, setting the batch size to 2, learning rate multiplier to 1, and epochs to 5. GPTQConfig (Frantar et al. , 2022) is used to load the Qwen-7B model in 4-bit, the GenerationConfig (Joao Gante, 2022) for Baichuan2-7B in 4-bit, and the BitsAndBytesConfig (Belkada, 2023) for LLaMA2-7B and Solar-10. 7B in 4-bit. We employ LoRA for LLaMA2-7B, Baichuan2-7B, Qwen-7B,
9https://azure. microsoft. com/en-us/products/ ai-services/openai-service
10https://cloud. google. com/vertex-ai/docs/ reference/rest
11https://oai. azure. com
and Solar-10. 7B, with the rank set to 1, alpha set to 32, and dropout at 0. 1. Prefix token lengths are set to 2048, learning rate to 1. 0e-3, batch size to 2, and maximum input and target length to 2048. All fine-tuning efforts are performed using 12 NVIDIA RTX3090 GPUs for 10 epochs. 4. 3 Evaluation Methods
4. 3. 1 Evaluation of Individual Modules
Embedders, Retrievers, and Rerankers. In recent studies, such as (Sottana et al. , 2023; Cao et al. , 2024; Liu et al. , 2023; Ye et al. , 2023) and (Kim et al. , 2023b), the GPT-4 evaluator has been extensively tested. To evaluate the performance of these modules and their combined performance in evidence generation, we use GPT-4 to analyze the relevance between questions and retrieved citations (aka. evidence). In detail, GPT-4 is asked to grade the question-evidence relevance on a fivepoint Likert scale. The average score, referred to as the ‘GPT-4 score’, is calculated for overall performance evaluation. The prompt used for GPT4-aided evaluation is shown in Appendix A. 2. Generators. To evaluate the performance of generators, we employ automatic metrics comprising matching-based measures such as ROUGE-1, ROUGE-2, and ROUGE-L (Lin, 2004), as well as the BLEU score (Papineni et al. , 2002). However, since prior research (Zheng et al. , 2023) shows that matching-based metrics may overestimate performance in long sequences, we also use the GPT4 evaluation method mentioned above to assess evidence-answer relevance. In addition, we report the ratio of unanswered questions in the responses (e. g. , cases when models return "can not provide answer based on the content). 4. 3. 2 Overall Evaluation of the RAG-based LFQA System
ROUGE (Lin, 2004) and BLEU (Papineni et al. , 2002)) are used to automatically measure the overall system performance. The GPT-4 scoring method is used to evaluate the answers from helpfulness, relevance, accuracy, depth, and creativity. Additionally, we invite three annotators to rank top-performing answers from all tested models and compare them with the ground truth answers, capturing human perception and assessing subjective response quality. Further details of human evaluation can be found in Appendix A. 1. 4. 4 Results
Embedders, Retrievers, and Rerankers Table 4 shows the GPT-4 score of different embedders, retrievers, and rerankers, which constitute the evidence generation pipeline. It also shows at the end the best-performing evidence-generation module combinations. We observe that the highestperforming embedding model is Ada2, achieving a score of 4. 586, followed by Ember-v1 (4. 486) and Bge-Small-en-v1. 5 (4. 455) with similar scores. Gte-Large lagged with a noticeable gap with a score of 4. 261. Among the retrievers we assess, AMR outperforms the rest with an impressive score of 4. 492. SWR ranks second at 4. 466, while the simple vector-based approach has the lowest performance with a score of 4. 358. Among the rerankers, Bge-Reranker-Base performs the best, achieving a competitive score of 4. 489. LLMRerank ranks second with a score of 4. 469, followed by All-Mpnet-Base-v2 with a score of 4. 383. The evidence generation modules together, we observe that the combination of AMR, Ada2, and Bge-Reranker-Base yields the highest score of 4. 622, followed by the combination of SWR, Ada2, and Bge-Reranker-Base/All-Mpnet-Base-v2, with a score of 4. 620. The marginal differences in performance among these leading combinations indicate that a variety of configurations are capable of yielding satisfactory outcomes for evidence generation. Generators Table 5 shows the comparison of different generators, contrasted by their base form and fine-tuned form. Although the fine-tuned models have a decreasing loss (from 2. 5 to 0. 1), they do not have significant improvement. They demonstrate slightly lower performance in terms of GPT-4 score, ROUGE-L, and BLEU scores. However, fine-tuned models have less unanswered questions, showing better understanding capabilities than their base forms. We also observe that while Gemini-Pro shows high numeric scores, it struggles the most in generating contextually relevant responses. Conversely, Baichuan2-7B demonstrates the best prompt com-
prehension ability. GPT-3. 5-turbo experiences more difficulty with contextual understanding, affecting its overall performance, while LLaMA2 has minimal context-related problems. However, LLaMA2 generates instances of simply rephrasing prompts, resulting in reduced accuracy scores. RAG-based LFQA System Table 3 shows the performance comparison of RAG-based LFQA system with generators in their base forms. We observe that system with the top-3 performing configurations incorporate GPT-3. 5-turbo and Baichuan27B as generators. In contrast, system configurations using the Gemini-Pro generator yield suboptimal performance in terms of accuracy. Meanwhile, we observe that system employing LLaMA2-7B as generators show the lowest ROUGE and BLEU scores among all the configurations tested. More results on the performances of different generators in the RAG systems are provided in Appendix A. 3. The top-scoring system configurations comprise (1) GPT-3. 5-turbo, AMR, Ada2, and LLMRERanker (noted as system 1); GPT-3. 5-turbo, AMR, Ember-v1, and Bge-Reranker-Base (system 2); (3) Baichuan2-7B, AMR, Ada2, and BgeReranker-Base (system 3). Table 6 shows the annotator-ranked preference of these top system configurations. We notice that some model-generated answers obtain higher average rankings than corresponding ground truths. For instance, Answer 2, produced by system 3, attains an average ranking of 2. 11, outperforming the ground truth (2. 19). Further investigation into annotator feedback reveals that annotators favor Answer 2 because it gives accurate responses while providing additional details. Answer 3, generated by system 1 performs the worst with the highest average ranking (3. 10). Answer 4, generated by system 2, achieves an average ranking of 2. 60. Best System Configuration in Multi-Document Settings Figure 4 shows the performance of the
three best-performing system configurations when given different numbers (n = 1 to 3) of documents. We observe a consistent pattern from the results: as the number of input documents increases, all system performance tend to decline. However, exceptions are also noted. For instance, the scores for certain instances with three documents marginally surpasses those with two documents in system 2 when compared to the accuracy score. Further investigation shows that the performance is dependent on the total context words of the input. When the number of context words reaches about 34k words, adding more input documents exerts a less marginal effect on system performance. Performance of Generators on Regulation and Textbook Dataset Table 11 presents the performance results of all the generators on regulationbased and textbook-based questions. For the regulation and textbook datasets, the Rouge-1 scores are 0. 317 and 0. 270, respectively; the Rouge-2 scores are 0. 146 and 0. 084, respectively; the Rouge-L scores are 0. 215 and 0. 170; the BLEU scores are 0. 074 and 0. 031; and the GPT-4 scores are 4. 238 and 4. 293. These results indicate that the LFQA system performs better on the policy/regulation dataset. Since the textbook dataset consists of questions derived from textbook exercises, each question is closely tied to the text content. After annotating the data, 60 policy/regulation data items were removed to ensure that each remaining question had a corresponding answer in the document. Therefore, the quality of the dataset does not affect the performance of the LFQA system. Based on our current findings, the textbook dataset is more challenging. Performance of Generators on Different Question Types Concerning overall performance on different question types, the models excelled at answering open-ended questions but struggled with numerical ones (Table 10). Specifically, GPT-3. 5
Turbo demonstrated the strongest performance in open-ended questions, while Llama2-7B underperformed. Baichuan2-7B showed versatility in handling various question types, but its competence in addressing numerical questions was subpar, placing it third. Remarkably, among all models, Qwen7B excelled in numerical ability, securing the top position across all metrics