Venue: NAACL 2025

# CFinBench: A Comprehensive Chinese Financial Benchmark for Large Language Models

Ying Nie\({}^{1}\)  Binwei Yan\({}^{1}\) Tianyu Guo\({}^{1}\) Hao Liu\({}^{1}\) Haoyu Wang\({}^{1}\) Wei He\({}^{1}\)

**Binfan Zheng\({}^{2}\) Weihao Wang\({}^{3}\) Qiang Li\({}^{3}\) Weijian Sun\({}^{2}\) Yunhe Wang\({}^{1}\) Dacheng Tao\({}^{4}\)**

\({}^{1}\)Huawei Noah's Ark Lab  \({}^{2}\)Huawei GTS   \({}^{3}\)Huawei Group Finance

\({}^{4}\)Nanyang Technological University

{ying.nie, yanbinwei, tianyu.guo, yunhe.wang}@huawei.com

###### Abstract

Large language models (LLMs) have achieved remarkable performance on various NLP tasks, yet their potential in more challenging task like finance, has not been fully explored. In this paper, we present CFinBench: a meticulously crafted, the most comprehensive evaluation benchmark to date, for assessing the financial knowledge of LLMs under Chinese context. In practice, to better align with the career trajectory of Chinese financial practitioners, we build a systematic evaluation from 4 first-level categories: (1) _Financial Subject_: whether LLMs can memorize the necessary basic knowledge of financial subjects, such as economics, statistics and auditing. (2) _Financial Qualification_: whether LLMs can obtain the needed financial qualified certifications, such as certified public accountant, securities qualification and banking qualification. (3) _Financial Practice_: whether LLMs can fulfill the practical financial jobs, such as tax consultant, junior accountant and securities analyst. (4) _Financial Law_: whether LLMs can meet the requirement of financial laws and regulations, such as tax law, insurance law and economic law. CFinBench comprises 99,100 questions spanning 43 second-level categories with 3 question types: single-choice, multiple-choice and judgment. We conduct extensive experiments on a wide spectrum of representative LLMs with various model size on CFinBench. The results show that GPT4 and some Chinese-oriented models lead the benchmark, with the highest average accuracy being 66.02%, highlighting the challenge presented by CFinBench. All the data and evaluation code are open sourced at [https://cfinbench.github.io/](https://cfinbench.github.io/).

+
Footnote †: Corresponding Author

## 1 Introduction

Recently, there has been a significant advancement in LLMs, exemplified by the representative models like ChatGPT [5], GPT4 [7], LLaMA [8], Baiduan [10], InternLM [11] and ChatGLM [12], _etc_. At the same time, the corresponding evaluation works for LLMs are flourishing and a series of evaluation benchmarks have been proposed like MMLU [1], C-Eval [13], Xiezhi [14] and AGIEval [15], _etc_. These benchmarks have been instrumental in catalyzing the progress of LLMs, as they enable the quantitative assessment of advanced knowledge and complex reasoning abilities.

Finance is the backbone of modern society, playing a vital role in facilitating economic growth and prosperity [16]. However, mastering the intricacies of financial knowledge is challenging for individuals, due to its intricate nature and dynamic environment. Therefore, endowing LLMs with financial knowledge is essential as it can provide significant convenience and insight to humanity. For example, BloombergGPT [26], which possesses 50 billion parameters, exhibits superior performance across multiple financial tasks. Similarly, FinMA [10] is crafted by fine-tuning LLaMA [8] with the financial instruction data. Of course, a comprehensive financial evaluation benchmark is also essential for financial LLMs.

Several benchmarks have been introduced for better evaluating financial LLMs. FLUE [16] first introduces the financial benchmark across 5 NLP tasks in English domain, and its successor, FLARE [10], further extends it with financial time-series reasoning task like stock price movements forecasting. In addition to English domain, benchmarks in Chinese are another one of significant importance. BBETCFLEB [11] presents the first Chinese financial evaluation benchmark, which includes 6

datasets covering both understanding and generation tasks. FinEval Zhang et al. (2023) builds a collection of 4,661 single-choice questions including 4 first-level categories: finance, economy, accounting, and certificate. However, FinEval is constrained by its limited size, and the category coverage is also inadequate for capturing the real-world financial scenarios. Moreover, the BBT-CFLEB, which targets the basic NLP tasks in finance, struggles to provide sufficient challenges for the increasingly advanced large language models.

In this paper, we present CFinBench: a meticulously crafted, the most comprehensive evaluation benchmark to date, for assessing the capabilities of LLMs on Chinese financial tasks. The design philosophy of our benchmark aligns with the career progression trajectory of financial practitioners, which can be likened to a 'leveling up' in a game. Specifically, it begins with mastering the required foundational knowledge in financial subjects, followed by obtaining the necessary qualified certifications, and subsequently honing skills through practical experience in industry applications. Last but not least, compliance with financial laws and regulations is also a crucial aspect. As described in Figure 1, we include 4 first-level categories: (1) _Financial Subject_: examining whether LLMs can memorize the foundational knowledge in financial subjects, such as political economy, statistics and macroeconomics, _etc_. (2) _Financial Qualification_: examining whether LLMs can obtain the necessary qualified certifications for financial practitioners, such as tax practitioner qualification, futures practitioner qualification and fund practitioner qualification, _etc_. (3) _Financial Practice_: examining whether LLMs can fulfill the specific tasks in financial jobs, such as junior banking professional, asset appraiser and junior statistician, _etc_. (4) _Financial Law_: examining whether LLMs can comply with the financial laws and regulations, such as securities law, insurance law and economic law, _etc_. CFinBench are primarily sourced from the mock exams and financial reports freely available on the Internet. To enhance the quality and diversity of the benchmark, and mitigate the problem of data contamination, we perform a series of rigorous data processing pipelines, encompassing data cleaning, internal and external de-duplication, LLM-assisted rephrasing, option shuffling, and multi-round human-in-the-loop cross-validation. CFinBench comprises 99,100 questions spanning 43 second-level categories with 3 question types: single-choice, multiple-choice and judgment.

We conduct extensive experiments on a wide spectrum of representative LLMs with various model size on CFinBench. The results show that GPT4 and some Chinese-oriented models like Qwen Bai et al. (2023), Yi Young et al. (2024), and XuanYuan Zhang and Yang (2023), _etc_. lead the benchmark, with the highest average accuracy being 66.02%, highlighting the challenge presented by CFinBench. It also indicates that there is still significant room for improvement in current Large Language Models (LLMs) within the Chinese financial domain.

## 2 Related Work

Financial LLMsThe advent of ChatGPT OpenAI (2021) marks a significant milestone in natural language processing (NLP), demonstrating

Figure 1: CFinBench comprises 4 first-level categories and 43 second-level categories, which are more align with the career trajectory of Chinese financial practitioners.

 

the remarkable capabilities of large language models with billions of parameters across a diverse range of tasks. This progress is further amplified by the release of GPT4 (OpenAI, 2023), which exhibits even greater generalization abilities. There have also been studies that are dedicated to adapting LLMs to the financial domain. BloombergGPT Wu et al. (2023), the first proprietary LLM comprising 50 billion parameters, has been meticulously tailored for the financial domain. The successors like FinGPT Yang et al. (2023b) and FinMA Xie et al. (2023) introduce the financial LLMs based on fine-tuning LLaMA, by means of low-rank adaptation or full-parameters. Also, the Chinese-oriented financial LLMs like DISCFinLLM Chen et al. (2023a), CFGPT Li et al. (2023b), XuanYuan Zhang and Yang (2023) and YunShan Wang et al. (2023), _etc_. have also demonstrated excellent performance across multiple Chinese financial tasks. At the same time, to objectively and quantitatively measure the capabilities of LLMs, a comprehensive and thorough evaluation benchmark is crucial.

Finnacial BenchmarksFLUE Shah et al. (2022) first introduces a financial evaluation benchmark across 5 NLP tasks in English context, and its successor, FLARE Xie et al. (2023), further includes financial time-series reasoning task like stock movement prediction. The FinBen Xie et al. (2024) reorganizes 35 public English datasets across 23 financial tasks into three spectrums of difficulty. For the Chinese domain, BBT-CFLEB Lu et al. (2023) presents the first Chinese financial evaluation benchmark, which includes 6 datasets covering both understanding and generation tasks. By integrating FLARE Xie et al. (2023) and BBT-CFLEB Lu et al. (2023), ICE-FLARE Hu et al. (2024) enables the evaluation of bilingual financial tasks. CFBenchmark Lei et al. (2023) assesses the text processing capabilities across recognition, classification, and generation tasks. CGCE Zhang et al. (2023b) incorporates 200 general and 150 finance-specific question-answering questions. The recent study of CFLUE Zhu et al. (2024) builds the datasets tailored for both knowledge assessment and application assessment. By the way, the most similar works to ours are FinanceIQ Zhang and Yang (2023) and FinEval Zhang et al. (2023a). FinancelIQ Zhang and Yang (2023) encompasses 10 primary categories, including economists, securities professionals and actuaries, _etc_. with a total of 36 subcategories and 7,173 single-choice questions. FinEval Zhang et al. (2023a) includes four primary categories, _i.e_., finance, economy, accounting, and certificate, with a total of 34 subcategories and 4,661 single-choice questions. However, the two benchmarks are limited in size. In contrast, the proposed CFinBench is more comprehensive, comprising a more reasonable dimension of financial abilities, with a total of 99,100 questions across 3 question types. The detailed comparison of the CFinBench with other Chinese-oriented financial benchmarks is summarized in Table 1. It is worth highlighting that we are not a aggregation of existing benchmarks, but rather a significant supplement from the perspective of professional practitioners in terms of advanced knowledge and complex reasoning, which, in conjunction with preceding benchmarks Shah et al. (2022); Lu et al. (2023); Zhang et al. (2023a); Lei et al. (2023); Zhu et al. (2024), foster the development of financial LLMs.

## 3 CFinBench

In this section, we first introduce the overall design principle of the proposed benchmark, then we elaborate the taxonomy of CFinBench including financial subject, financial qualification, financial practice, and financial law. The detailed process of data construction including data sources, data processing and human cross-validation is also presented at the end.

\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Benchmark** & **\#Test Questions \#Categories \#Question Types** & **Task** \\ \hline BBT-CFLEB Lu et al. (2023) & 20,416 & 6 & 4 & Basic NLP \\ CGCE Zhang et al. (2023b) & 150 & 4 & 1 & QA \\ CFBenchmark Lei et al. (2023) & 3,917 & 8 & 3 & Basic NLP \\ \hline FinEval Zhang et al. (2023a) & 4,661 & 34 & 1 & Advanced Knowledge \\ FinancelQ Zhang and Yang (2023) & 7,173 & 36 & 1 & Advanced Knowledge \\ Ant-FinEva Group (2023) & 8,445 & 33 & 1 & Advanced Knowledge \\ CFLUE Zhu et al. (2024) & 16,522 & 33 & 6 & Advanced Knowledge \& Application \\ \hline
**CFGnBench** & 99,100 & 43 & 3 & Advanced Knowledge \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison of the proposed CFinBench with other Chinese-oriented financial benchmarks.

 



### Overview

The motivation of CFinBench is to evaluate the financial knowledge of large language models in the context of Chinese. Inspired by predecessors (Huang et al., 2024; Fei et al., 2023; Zhang et al., 2023), we also focus on the advanced knowledge and complex reasoning abilities, which, compared to traditional NLP capabilities, pose a greater challenge to the increasingly advanced LLMs of today. In practice, we build CFinBench based on the real-world examination questions used in China for assessing financial professionals across multiple dimensions. We include 3 question types: single-choice, multiple-choice and judgment, as exemplified in Figure 2. Compared with the single-choice questions alone in most existing works (Zhang et al., 2023; Zhang and Yang, 2023; Clark et al., 2018; Group, 2023; Huang et al., 2024), a broader range of question types can more comprehensively assess the capabilities of LLMs. Specifically, for single-choice questions, each question has four options, with only one correct answer. For multiple-choice questions, each question has four or five options, with at least two correct answers. For judgment questions, each question requires a direct judgment of whether the statement is correct or wrong.

### Taxonomy

In the evaluation of large language models, a diverse array of tasks is often preferred to comprehensively assess their capabilities. A hierarchical evaluation framework enables a more nuanced understanding of the abilities of LLMs. Instead of categorizing the financial tasks solely based on their subjects (Zhang et al., 2023; Zhang and Yang, 2023), we thoroughly explore the characteristics of Chinese financial system, and reorganize the financial tasks into more reasonable categories. Specifically, the process starts with acquiring fundamental knowledge in financial subjects, followed by passing essential professional qualifications, and subsequently refining skills through practical experience in industry applications. Additionally, adherence to laws and regulations is also a critical aspect. In practice, we include 4 first-level categories and 43 second-level categories, which are summarized in Figure 1.

* **Financial Subject**: The purpose of the financial subject is to test whether LLMs can memorize the essential foundational knowledge in financial subjects. Specifically, 11 subjects are included: Political Economy, Western Economics, Microeconomics, Macroeconomics, Industrial Economics, Public Finance, International Trade, Statistics, Auditing, Economic History, and Finance. These subjects provide a comprehensive framework for understanding the intricacies of economic systems, market structures, and financial institutions.
* **Financial Qualification**: The objective of the financial qualification is to examine whether LLMs can obtain necessary qualified certifications for finance professionals. We include 8 qualifications: Tax Practitioner Qualification, Futures Practitioner Qualification, Fund Practitioner Qualification, Real Estate Practitioner

Figure 2: Examples of 3 types of questions. English translations are shown in blue for better readability.



Qualification, Insurance Practitioner Qualification, Securities Practitioner Qualification, Banking Practitioner Qualification, and Certified Public Accountant (CPA). By obtaining these qualifications, professionals can enhance their knowledge and skills in areas such as financial analysis, risk management, and financial planning.
* **Financial Practice**: Financial practice is to examine whether LLMs can fulfill the specific tasks in practical financial jobs. We include 13 jobs: Junior/Intermediate Auditor, Junior/Intermediate Statistician, Junior/Intermediate Economist, Junior/Intermediate Banking Professional, Junior/Intermediate Accountant, Tax Consultent, Asset Appraiser, and Securities Analyst. These practices involve the application of financial concepts and techniques to real-world problems, requiring professionals to possess a deep understanding of financial markets, instruments, and regulations.
* **Financial Law**: The purpose of financial law is to test whether LLMs can comply with financial laws and regulations. Specifically, it includes 11 exams of laws: Tax Law I/II, Tax Inspection, Commercial Law, Securities Law, Insurance Law, Economic Law, Banking Law, Futures Law, Financial Law and Civil Law. These laws provide the legal foundation for financial transactions, investments, and operations. Proficiency in financial laws can reduce the occurrence of illegal activities.

### Data Construction

#### 3.3.1 Data Sources

Our dataset is primarily sourced from publicly accessible channels like mock exams on internet, public books on economics and law, announcements and financial reports of listed companies, financial articles and news, _etc_.

#### 3.3.2 Data Processing

The collected data come in various formats, including PDF, EPUB, Microsoft Word documents and web pages. Documents in PDF format and EPUB format are parsed into text using PyMuPDF1 and EbookLib2 respectively. We standardized all single-choice questions to have exactly four options, excluding those with fewer options and randomly removing excess wrong options from those with more than four. Similarly, for multiple-choice questions, to maintain uniformity, we only retain questions with four or five options. Footnote 1: [https://github.com/pymupdf/PyMuPDF](https://github.com/pymupdf/PyMuPDF)

Footnote 2: [https://pypi.org/project/EbookLib/](https://pypi.org/project/EbookLib/)

Following predecessors Yuan et al. (2021); Penedo et al. (2023); Wei et al. (2023); Huang et al. (2024), all the collected questions go through a standard data preprocessing pipeline including cleaning and de-duplication. For data cleaning, we first remove non-Chinese paragraphs with the inexpensive n-gram models like fastText Joulin et al. (2016). Then a series of filtering rules and heuristics are performed, such as only keeping lines with valid punctuation, discarding consecutive newlines and whitespace characters, or removing unsemantic and garbled lines. For data de-duplication, we adopt MinHash algorithm Broder (1997) for internal de-duplication and de-duplication with external public data Zhang et al. (2023); Zhang and Yang (2023); Group (2023); Information (2023); Lu et al. (2023); Zhu et al. (2024). To enhance data diversity and mitigate data contamination problem, we also adopt the strategy of question rephrasing based on GPT4 Wang et al. (2022); Xu et al. (2023). We observe that the collected raw data exhibits a significant class imbalance, with a notable scarcity of judgment questions and a substantial surplus of single-choice questions. To address this issue, we prompt GPT4 to rephrase a portion of the single-choice questions into judgment questions, while maintaining semantic consistency, as exemplified in Figure 3 (a). Furthermore, to mitigate the problem of data contamination, we first randomize the option order Berglund et al. (2023). In practice, this includes both random shuffle and 'farthest option swapping', where the correct option is exchanged with the incorrect option that is farthest away. Subsequently, we prompt GPT4 to rephrase the questions based on the shuffled options, similarly preserving semantic consistency, as exemplified in Figure 3 (b).

#### 3.3.3 Human Cross-Validation

In order to ensure the quality of the benchmark, we establish a financial team of more than 10 people with professional financial backgrounds and conduct three rounds of manual verification of the rephrased questions. Specifically, we first spend about 2 months on the first round of verification on the rephrased questions. During this period, 

about 20,000 questions with obvious quality are discarded. About 70,000 questions with high quality and high confidence in the correctness of the answers are retained. In addition, about 35,000 questions for which it is difficult to determine the correctness of the answers in a short time are reserved for the next round of verification. In the second round of verification, we spend about another month carefully proofreading and verifying the questions, and produce about 30,000 high-quality questions. Finally, the final approximately 100,000 questions undergo a third round of random sampling and verification over 10 days to further improve the quality. The statistics of the final dataset are summarized in Appendix D.

## 4 Experiments

### Setup

Data SplitWe randomly split the questions into a development set, a validation set, and a test set within each second-level category. The development split per category consists of three examples to facilitate few-shot evaluation. A portion of the development examples are also annotated with detailed explanations to enable few-shot chain-of-thought settings Wei et al. (2022). The validation set and test set are divided in a ratio of 2:8, where the validation set is for hyperparameter tuning and the test set is for full evaluation.

Inference DetailsWe employ the OpenCompass Contributors (2023) framework to perform model inference. Specifically, during the generation process, we set both the temperature and the top \(p\) to 1.0, and employ greedy decoding. The input token length is limited to 2048, and the output token length is limited to 64, which is sufficient for the questions of choice and judgment. Right truncation is performed for input prompts exceeding the length limitation. All models are inferred in both zero-shot and three-shot settings, which are exemplified in Appendix C.

Evaluation MetricsWe adopt accuracy to measure the match between model prediction and gold answer. Specifically, for single-choice questions, if multiple valid options are predicted by the model, we only select the first option as the final answer predicted by the model. For multiple-choice questions, if any of the options predicted by the model are not among the gold answer, we directly classify it as wrong. Otherwise, we score it based on the number of predicted answers (out of a full score of 1). Considering that the judgment questions are relatively simple and usually score higher, we appropriately reduce the weight of the judgment questions to calculate the final score more reasonably, _i.e_., \(final=0.4\times single+0.4\times multiple+0.2\times judgment\).

### Models

To give a comprehensive view of the status of LLMs in a Chinese financial context, we evaluate a wide spectrum of large language models, as depicted in Table 2. Specifically, open-source LLMs from various families are included, includ

Figure 3: Examples of question rephrasing. English translations are shown in blue for better readability. In each example, the top is the original question, and the bottom is the rephrased question.

 

ing Llama (Touvron et al., 2023; Meta, 2024), Qwen Bai et al., 2023), ChatGLM Zeng et al., 2022; Du et al., 2022), Baichuan Yang et al., 2023a), InternLM Team, 2023; Cai et al., 2024), Phi Gunasekar et al., 2023; Li et al., 2023c; Abdin et al., 2024), DeepSeek (DeepSeek-AI, 2024), XuanYuan Zhang and Yang, 2023), FinMA Xie et al., 2023), Gemma Team et al., 2024), Tiggerot Chen et al., 2023b), Skywork Wei et al., 2023), Yi Young et al., 2024), Mistral Jiang et al., 2023), DISC-FinLLM Chen et al., 2023a) and CFGPT Li et al., 2023b). We classify models into different categories according to their size. Considering the legal issues, we only report the results of two API-based LLMs, _i.e_., ChatGPT (_gpt-3.5-turbo-0125_) and GPT4 (_gpt-4-turbo_).

### Results

In Table 2, we report the 0-shot and 3-shot accuracy of each first-level category on the test split under the answer-only setting. As can be seen, the Chinese-oriented Qwen2.5-72B Bai et al., 2023) lead the benchmark, with an average accuracy reaching 66.02%. In addition, the accuracy of some

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \hline Model & Subject & Qualification & \multicolumn{2}{c}{Practice} & Law & Average \\  & 0-shot & 3-shot & 0-shot & 3-shot & 0-shot & 3-shot & 0-shot & 3-shot & 3-shot \\ \hline _API Call_ & & & & & & & & & & \\ \hline GPT4 & 58.62 & 58.47 & 58.34 & 57.09 & 57.56 & 56.37 & 55.20 & 55.15 & 57.43 & 56.77 \\ ChatGPT & 39.58 & 40.86 & 43.15 & 42.56 & 40.86 & 40.51 & 38.18 & 38.83 & 40.44 & 40.69 \\ \hline \multicolumn{10}{l}{_Size > 65B_} \\ \hline Qwen2.5-72B & 66.32 & 67.17 & 64.67 & 65.44 & 65.74 & 66.91 & 63.25 & 64.57 & 65.00 & 66.02 \\ XuanYuan2-70BBase\({}^{*}\) & 53.56 & 59.96 & 53.36 & 56.27 & 53.55 & 58.05 & 48.27 & 52.48 & 52.19 & 56.69 \\ Lidama3-70B & 50.75 & 56.27 & 47.52 & 52.35 & 45.17 & 51.24 & 44.62 & 49.26 & 47.02 & 52.28 \\ DeepSeek-67B-Base & 45.76 & 52.03 & 44.61 & 50.57 & 43.95 & 49.16 & 42.87 & 47.01 & 44.30 & 49.69 \\ Tigerbot-70B-Base & 43.22 & 52.13 & 43.15 & 48.42 & 40.32 & 46.00 & 38.56 & 45.87 & 41.31 & 48.11 \\ \hline \multicolumn{10}{l}{_Size \(\approx\) 30B_} \\ \hline Qwen2.5-32B & 61.77 & 63.31 & 62.03 & 64.44 & 60.42 & 62.25 & 58.61 & 61.17 & 60.71 & 62.79 \\ Yil1.5-34B & 58.62 & 61.44 & 58.30 & 60.91 & 57.26 & 59.75 & 55.16 & 58.55 & 57.34 & 60.16 \\ \hline \multicolumn{10}{l}{_I0B < Size < 20B_} \\ \hline Qwen2.5-14B & 56.18 & 59.55 & 58.37 & 62.08 & 56.64 & 60.13 & 55.58 & 58.71 & 56.69 & 60.12 \\ InternLM2.5-20B & 52.41 & 54.97 & 54.74 & 57.59 & 53.07 & 56.42 & 51.24 & 54.29 & 52.87 & 55.82 \\ XuanYuan-13B-Base\({}^{*}\) & 40.87 & 44.32 & 41.64 & 47.68 & 42.30 & 46.56 & 41.73 & 45.75 & 41.64 & 46.08 \\ Phi3-14B-Instruct & 44.95 & 42.46 & 46.12 & 43.60 & 44.26 & 40.83 & 42.17 & 39.61 & 44.38 & 41.63 \\ Baichuan2-13B & 29.16 & 41.13 & 34.25 & 44.19 & 31.27 & 40.63 & 31.44 & 40.05 & 31.53 & 41.50 \\ Skywork-13B & 34.66 & 39.24 & 37.78 & 43.88 & 36.22 & 40.90 & 36.39 & 41.38 & 36.26 & 41.35 \\ DISE:FinLLM-13B\({}^{*}\) & 37.42 & 39.46 & 42.59 & 38.81 & 38.32 & 39.89 & 39.09 & 38.35 & 39.36 & 39.13 \\ Tigerbot-13B-Base & 32.83 & 34.09 & 35.36 & 39.21 & 33.09 & 35.87 & 33.75 & 35.52 & 33.76 & 36.17 \\ Llama2-13B & 30.24 & 32.15 & 31.42 & 35.18 & 29.35 & 31.92 & 29.48 & 34.33 & 30.12 & 33.40 \\ \hline \multicolumn{10}{l}{_SB \textless{Size < 10B_}} \\ \hline Qwen2.5-7B & 56.27 & 58.41 & 54.13 & 56.28 & 55.25 & 58.47 & 53.53 & 56.26 & 54.80 & 57.36 \\ YunShan-7B\({}^{*}\) & 52.65 & 53.00 & 52.61 & 51.79 & 53.33 & 52.77 & 52.52 & 52.23 & 52.78 & 52.45 \\ InterLM2.5-7B & 52.31 & 54.67 & 50.26 & 53.22 & 51.49 & 53.85 & 49.55 & 51.62 & 50.90 & 53.34 \\ Vi1.5-9B & 47.85 & 50.51 & 50.45 & 51.03 & 48.26 & 49.20 & 46.41 & 47.03 & 48.24 & 49.44 \\ Owen1.5-7B & 46.26 & 49.18 & 47.97 & 50.28 & 46.52 & 47.93 & 44.66 & 46.04 & 46.35 & 48.36 \\ ChatGLM3-6B-Base & 46.56 & 46.41 & 47.52 & 49.45 & 46.56 & 48.20 & 43.62 & 45.05 & 46.07 & 47.28 \\ XuanYuan-6B-Base\({}^{*}\) & 39.99 & 41.65 & 44.30 & 45.87 & 42.70 & 43.91 & 41.70 & 42.81 & 42.17 & 43.56 \\ Llama3.1-8B & 33.54 & 40.38 & 34.87 & 41.25 & 32.54 & 39.47 & 35.24 & 41.27 & 34.05 & 40.59 \\ Bialuchan-7B & 30.22 & 37.23 & 35.56 & 41.35 & 31.30 & 37.33 & 29.59 & 37.49 & 31.67 & 38.35 \\ Mistral-7B & 29.46 & 35.63 & 29.11 & 37.56 & 28.75 & 35.87 & 28.39 & 34.34 & 28.93 & 35.85 \\ CFGPT1-sft-7B-Full\({}^{*}\) & 32.45 & 33.73 & 34.41 & 33.92 & 32.80 & 32.93 & 33.48 & 35.05 & 33.29 & 33.91 \\ FinMA-7B\({}^{*}\) & 23.71 & 22.74 & 24.92 & 25.86 & 22.71 

other models like Yi1.5-34B (Young et al., 2024), XuanYuanZ-70B-Base (Zhang and Yang, 2023) and GPT4 (OpenAI, 2023) also exceed 56%. In the size range of 10B-20B, Qwen2.5-14B (Bai et al., 2023), InternLM2.5-20B (Cai et al., 2024) and XuanYuan-13B-Base (Zhang and Yang, 2023) are in the lead, with average accuracy exceeding 46%. Notably, in the size range of 5B-10B, Qwen2.5-7B (Bai et al., 2023) and the Chinese finance-specific model YunShan-7B (Wang et al., 2023) is in the absolute leading position, with an accuracy of over 52%, which is even higher than the accuracy of some 70B models. Also, Yi1.5-9B (Young et al., 2024), Qwen1.5-7B (Bai et al., 2023) and ChatGLM3-6B-Base (Zeng et al., 2022; Du et al., 2022) also achieve the accuracy over 45%. During the size range of less than 5B, Qwen2.5-3B (Bai et al., 2023) is the only model that achieves an accuracy of more than 45%. The other models like Phi3.5-3.8B-Instruct (Abdin et al., 2024) and YunShan-1.5B (Wang et al., 2023) also achieve the accuracies of more than 39%. In conclusion, there is still significant room for improvement.

### Analysis

**Few-shot examples are helpful in most cases.** As we can see from Table 2, the performance of most models demonstrates improvement when some examples are provided. However, in the case of Phi3-14B-Instruct, the zero-shot setting outperforms the few-shot setting. We guess that this is because the models have acquired the ability to fully understand the questions without the need for examples during the pre-training or fine-tuning. The introduced examples may mismatch with their training methodology, which leads to the decrease in accuracy (Gu et al., 2024; Li et al., 2023a).

**Scaling up the model size usually results in better performance.** In Table 2, as the model size increases, the accuracy of Qwen2.5-3B, Qwen2.5-7B, Qwen2.5-14B, Qwen2.5-32B and Qwen2.5-72B increases accordingly, \(i\)._e_., 45.06%, 57.36%, 60.12%, 62.79% and 66.02% respectively at the 3-shot setting. Similarly, the accuracy of Yi1.5-3dB is increased by 10.72% over Yi1.5-9B at the 3-shot setting. However, this does not mean that increasing the model size will definitely improve the performance.

**Domain specific pre-training and fine-tuning are helpful.** Impressively, two finance-specific models XuanYuan (Zhang and Yang, 2023) and YunShan (Wang et al., 2023) achieve very competitive accuracy, demonstrating a better grasp of financial knowledge. This can be attributed to the fact that both models are mixed with high-quality financial corpus during pre-training and fine-tuning. We think the same is true for the general base models such as Yi (Young et al., 2024) and Qwen (Bai et al., 2023) that perform well in Table 2.

Accuracy improves with LLMs' iteration.We visualize how the accuracies of 4 first-level categories change as the LLM is iteratively updated (Qwen, Qwen1.5, Qwen2, Qwen2.5 and ChatGLM, ChatGLM2, ChatGLM3). As can be seen from Figure 4, with the improvement of training method and the enrichment of training corpus, the accuracy will often be significantly improved, which also highlights the rationality and importance of CFinBench.

The performance of chat models.To better engage in natural conversation, the chat version are often derived from base model by alignment techniques (Ouyang et al., 2022), such as supervised finetuning (SFT) and reinforcement learning from human feedback (RLHF). As observed in Table 3, the accuracy of some models' chat version is improved when compared to the base version, such as Qwen2.5-32B, Gemma2-2B, Baichuan2-13B, _etc_. At the same time, the accuracy of some models' chat version have declined, such as Yi1.5-34B and ChatGLM3-6B. The varying alignment strategies of the models lead to different results.

Figure 4: The 0-shot accuracy changes with the iterations of LLMs.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline Model & \multicolumn{2}{c}{Base} & \multicolumn{2}{c}{Chat} \\  & 0-shot & 3-shot & 0-shot & 3-shot \\ \hline Yi1.5-34B & 57.34 & 60.16 & 58.99 & 57.48 \\ Qwen2.5-32B & 60.71 & 62.79 & 61.62 & 64.48 \\ InternLM2.5-20B & 52.87 & 55.82 & 54.35 & 55.27 \\ Baiduchan2-13B & 31.53 & 41.50 & 40.74 & 44.60 \\ ChatGLM3-6B & 46.07 & 47.28 & 30.79 & 27.27 \\ Gemma2-2B & 22.56 & 31.13 & 30.47 & 35.81 \\ \hline \hline \end{tabular}
\end{table}
Table 3: The 0/3-shot average accuracy (%) of base model and chat model on the test split.

 

Conclusion

In this paper, we present CFinBench, the most comprehensive evaluation benchmark to date, for assessing the financial domain knowledge of LLMs under Chinese context. We improve the quality and diversity of the data and mitigate the issue of data contamination through a series of processes, including data cleaning, internal and external de-duplication, LLM-assisted question rephrasing, option shuffling, and multiple rounds of human cross-validation. Four first-level categories are included in CFinBench: financial subject, financial qualification, financial practice, and financial law, which are more align with the career trajectory of financial practitioners in China. The CFinBench comprises 99,100 questions spanning 43 second-level categories with 3 question types: single-choice, multiple-choice and judgment. We conduct extensive evaluations on a wide spectrum of LLMs with various model size. The results show that there is still significant room for improvement for current LLMs in the Chinese financial domain.

## Ethical Statement

All data utilized in this study primarily originate from publicly accessible channels that have been processed by our professional experts. Also, It is important to note that all datasets within CFinBench carry low ethical risks, with stringent measures in place to ensure the absence of any sensitive or personally identifiable information. In order to promote the research progress, our data and evaluation code are open sourced to the community under the Apache-2.0 license3.

Footnote 3: [https://www.apache.org/licenses/LICENSE-2.0](https://www.apache.org/licenses/LICENSE-2.0)

## Limitations

CFinBench focuses on Chinese financial system and is therefore not suitable for assessing financial knowledge in other countries. It provides an important basis for evaluating the LLMs' mastery of Chinese financial knowledge. However, since we have open sourced the questions and answers of the validation split, if they are improperly used to train the large language model, the accuracy of the model may be falsely high on the validation split. Therefore, the results on benchmark are just a reference. The true quality of the model depends on the performance of the user in the practical scenario.

## References

* Abdin et al. (2024) Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. _arXiv preprint arXiv:2404.14219_.
* Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. _arXiv preprint arXiv:2309.16609_.
* Berglund et al. (2023) Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. 2023. The reversal curse: LLms trained on" a is b" fail to learn" b is a". _arXiv preprint arXiv:2309.12288_.
* Broder (1997) Andrei Z Broder. 1997. On the resemblance and containment of documents. In _Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171)_, pages 21-29. IEEE.
* Cai et al. (2024) Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. 2024. Internlm2 technical report. _arXiv preprint arXiv:2403.17297_.
* Chen et al. (2023a) Wei Chen, Qiushi Wang, Zefei Long, Xiangvin Zhang, Zhongtian Lu, Bingxuan Li, Siyuan Wang, Jiarong Xu, Xiang Bai, Xuanjing Huang, et al. 2023a. Disc-fillm: A chinese financial large language model based on multiple experts fine-tuning. _arXiv preprint arXiv:2310.15205_.
* Chen et al. (2023b) Ye Chen, Wei Cai, Liangmin Wu, Xiaowei Li, Zhankuan Xin, and Cong Fu. 2023b. Tigerbot: An open multilingual multitask llm. _arXiv preprint arXiv:2312.08688_.
* Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyrind Tarjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. _arXiv preprint arXiv:1803.05457_.
* Contributors (2023) OpenCompass Contributors. 2023. Opencompass: A universal evaluation platform for foundation models. [https://github.com/open-compass/opencompass](https://github.com/open-compass/opencompass).
* DeepSeek-AI (2024) DeepSeek-AI. 2024. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. _Preprint_, arXiv:2405.04434.
* Du et al. (2022) Zhengixiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. GIm: General language model pretraining with autoregressive blank infilling. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 320-335 

Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Ge. 2023. Lawbench: Benchmarking legal knowledge of large language models. _arXiv preprint arXiv:2309.16289_.
* Group (2023) Ant Group. 2023. Ant-fin-eva. [https://github.com/alipay/financial_evaluation_dataset/](https://github.com/alipay/financial_evaluation_dataset/).
* Gu et al. (2024) Zhouhong Gu, Xiaoxuan Zhu, Haoning Ye, Lin Zhang, Jianchen Wang, Yixin Zhu, Sihang Jiang, Zhuozhi Xiong, Zihan Li, Weijie Wu, et al. 2024. Xiezh: An ever-updating benchmark for holistic domain knowledge evaluation. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 38, pages 18099-18107.
* Gunasekar et al. (2023) Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio Cesar Teodoro Mendes, Allie Del Giorno, Sivakanth Goppi, Mojan Javaherji, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. 2023. Textbooks are all you need. _arXiv preprint arXiv:2306.11644_.
* Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. _arXiv preprint arXiv:2009.03300_.
* Hu et al. (2024) Gang Hu, Ke Qin, Chenhan Yuan, Min Peng, Alejandro Lopez-Lira, Benyou Wang, Sophia Ananiadou, Wanlong Yu, Jimin Huang, and Qianqian Xie. 2024. No language is an island: Unifying chinese and english in financial large language models, instruction data, and benchmarks. _arXiv preprint arXiv:2403.06249_.
* Huang et al. (2024) Yunzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. 2024. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. _Advances in Neural Information Processing Systems_, 36.
* Information (2023) East Money Information. 2023. Openfindata. [https://github.com/open-compass/OpenfinData/](https://github.com/open-compass/OpenfinData/).
* Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. _arXiv preprint arXiv:2310.06825_.
* Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Herve Jegou, and Tomas Mikolov. 2016. Fasttext. zip: Compressing text classification models. _arXiv preprint arXiv:1612.03651_.
* Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. _Advances in neural information processing systems_, 35:22199-22213.
* Lei et al. (2023) Yang Lei, Jiangtong Li, Ming Jiang, Junjie Hu, Dawei Cheng, Zhijun Ding, and Changjun Jiang. 2023. Cfbenchmark: Chinese financial assistant benchmark for large language model. _arXiv preprint arXiv:2311.05812_.
* Li et al. (2023a) Haonan Li, Yixuan Zhang, Fajit Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023a. Cmmlu: Measuring massive multitask language understanding in chinese. _arXiv preprint arXiv:2306.09212_.
* Li et al. (2023b) Jiangtong Li, Yuxuan Bian, Guoxuan Wang, Yang Lei, Dawei Cheng, Zhijun Ding, and Changjun Jiang. 2023b. Cfgpt: Chinese financial assistant with large language model. _arXiv preprint arXiv:2309.10654_.
* Li et al. (2023c) Yuanzhi Li, Sebastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023c. Textbooks are all you need ii: phi-1.5 technical report. _arXiv preprint arXiv:2309.05463_.
* Lu et al. (2023) Dakuan Lu, Hengkin Wu, Jiaqing Liang, Yipei Xu, Qianyu He, Yipeng Geng, Mengkun Han, Yingsi Xin, and Yanghua Xiao. 2023. Bbt-fin: Comprehensive construction of chinese financial domain pre-trained language model, corpus and benchmark. _arXiv preprint arXiv:2302.09432_.
* Meta (2024)Meta. 2024. Llama3. Https://llama.meta.com/llama3.
* OpenAI (2021) OpenAI. 2021. GitHub://www.openai.com/chatgptt.
* OpenAI (2023) OpenAI. 2023. GPT-4 technical report. _Preprint_, arXiv:2303.08774.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in neural information processing systems_, 35:27730-27744.
* Penedo et al. (2023) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeildi, Baptiste Pannier, Ebensam Almazrouei, and Julien Launay. 2023. The refinedweb dataset for falcton llm: outperforming curated corpora with web data, and web data only. _arXiv preprint arXiv:2306.01116_.
* Shah et al. (2022) Raj Sanjay Shah, Kunal Chawla, Dheeraj Eidhani, Agam Shah, Wendi Du, Sudheer Chava, Natraj Raman, Charese Smiley, Jiao Chen, and Diyi Yang. 2022. When flue meets flang: Benchmarks and large pre-trained language model for financial domain. _arXiv preprint arXiv:2211.00083_.
* Shiller (2013) Robert J Shiller. 2013. _Finance and the good society_. Princeton University Press.
* Team et al. (2024) Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Riviere, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. _arXiv preprint arXiv:2403.08295_.
* Team (2023) InternLM Team. 2023. Internlm: A multilingual language model with progressively enhanced capabilities.

 

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. _arXiv preprint arXiv:2302.13971_.
* Tuovron et al. (2023b) Hugo Tuovron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Wang et al. (2022) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Aliisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language models with self-generated instructions. _arXiv preprint arXiv:2212.10560_.
* Wang et al. (2023) Yunhe Wang, Hanting Chen, Yehui Tang, Tianyu Guo, Kai Han, Ying Nie, Xutao Wang, Hailin Hu, Zheyuan Bai, Yun Wang, et al. 2023. Pangu-\(\pi\): Enhancing language model architectures via nonlinearity compensation. _arXiv preprint arXiv:2312.17276_.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. _Advances in neural information processing systems_, 35:24824-24837.
* Wei et al. (2023) Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lu, Rui Hu, et al. 2023. Skywork: A more open bilingual foundation model. _arXiv preprint arXiv:2310.19341_.
* Wu et al. (2023) Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. Bloomberggpt: A large language model for finance. _arXiv preprint arXiv:2303.17564_.
* Xie et al. (2024) Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, et al. 2024. The finben: An holistic financial benchmark for large language models. _arXiv preprint arXiv:2402.12659_.
* Xie et al. (2023) Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. 2023. Pixiu: A large language model, instruction data and evaluation benchmark for finance. _arXiv preprint arXiv:2306.05443_.
* Xu et al. (2023) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Davin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. _arXiv preprint arXiv:2304.12244_.
* Yang et al. (2023a) Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, et al. 2023a. Baichuan 2: Open large-scale language models. _arXiv preprint arXiv:2309.10305_.
* Yang et al. (2023b) Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang. 2023b. Fingpt: Open-source financial large language models. _arXiv preprint arXiv:2306.06031_.
* Young et al. (2024) Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. 2024. Yi: Open foundation models by 01. ai. _arXiv preprint arXiv:2403.04652_.
* Yuan et al. (2021) Sha Yuan, Hanyu Zhao, Zhengxiao Du, Ming Ding, Xiao Liu, Yukuo Cen, Xu Zou, Zhilin Yang, and Jia Tang. 2021. Wudaocorpora: A super large-scale chinese corpora for pre-training language models. _AI Open_, 2:65-68.
* Zeng et al. (2022) Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-130b: An open bilingual pre-trained model. _arXiv preprint arXiv:2210.02414_.
* Zhang et al. (2023a)Liwen Zhang, Weige Cai, Zhaowei Liu, Zhi Yang, Wei Dai, Yujie Liao, Qianru Qin, Yifei Li, Xingyu Liu, Zhiqiang Liu, et al. 2023a. Fineval: A chinese financial domain knowledge evaluation benchmark for large language models. _arXiv preprint arXiv:2308.09975_.
* Zhang et al. (2023b) Xuanyu Zhang, Bingbing Li, and Qing Yang. 2023b. Cgc: A chinese generative chat evaluation benchmark for general and financial domains. _arXiv preprint arXiv:2305.14471_.
* Zhang and Yang (2023)Xuanyu Zhang and Qing Yang. 2023. Xuanyuan 2.0: A large chinese financial chat model with hundreds of billions parameters. In _Proceedings of the 32nd ACM International Conference on Information and Knowledge Management_, pages 4435-4439.
* Zhong et al. (2023) Wanjun Zhong, Ruixing Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023. Agieval: A human-centric benchmark for evaluating foundation models. _arXiv preprint arXiv:2304.06364_.
* Zhu et al. (2024)Jie Zhu, Junhui Li, Yalong Wen, and Lifan Guo. 2024. Benchmarking large language models on cflue-a chinese financial language understanding evaluation dataset. _arXiv preprint arXiv:2405.10542_.

 

More experiments

Results on chain-of-thought setting.To further explore the models' reasoning capabilities, in addition to the answer-only (AO) setting, we also perform some experiments on the chain-of-thought (COT) setting Kojima et al. (2022); Wei et al. (2022). Evaluation on COT setting requires the model to generate explanations for a given question and then give the final answer based on the generated explanations. Specifically, we obtain the explanations examples only for multiple-choice questions manually by professional financial practitioners. The experimental results are reported in Table 4.

As observed in Table 4, the models achieve comparable or lower average accuracy than in the answer-only setting. This suggests that COT prompting does not necessarily improve results, which is also evidenced in other benchmarks like FinEval Zhang et al. (2023) and C-Eval Huang et al. (2024), _etc_.

Comparison with other similar benchmark.FinEval Zhang et al. (2023) is another representative benchmark for evaluating the Chinese financial advanced knowledge of LLMs. In Table 5, we report the few-shot accuracy on CFinBench and FinEval with various LLMs. It can be seen that the accuracy of the highest Yi1.5-34B reaches nearly 87%, while ours is around 60%. Likewise, the lowest InternLM2-7B accuracy is still over 62%, while ours is only around 43%. This all suggests that CFinBench is more challenging and better able to distinguish the performance of different models.

## Appendix B Analysis on Data Contamination

Since we publicize the answers of the validation set, we imagine a data contamination scenario: what would happen if the LLM was trained using part of the validation set? We conduct an experiment here. Specifically, we randomly select 60% of the data from the validation set and fine-tuned InternLM2.5-1.8B for 2 epochs based on LoRA (the modules of query, key, value and out are chosen as the target, the rank is set to 32). The 0-shot results on validation and test split are reported in Table 6. As we can see, data contamination on validation split will lead to the accuracy improvements on both validation and test. Therefore, it is crucial to avoid the contamination of evaluation data during the model training process.

## Appendix C Prompt examples

We list the prompt examples utilized in the evaluation process, including zero-shot and few-shot in answer-only scenarios ( Figure5 and Figure6), zero-shot and few-shot in chain-of-thought scenarios ( Figure7).

## Appendix D Data statistic

In Table 7, we enumerate the comprehensive statistical information of the dataset.

 



\begin{table}
\begin{tabular}{l c c c c c} \hline  & GPT4 & \multicolumn{2}{c}{Qwen1.5-72B} & Qwen1.5-32B & Qusen1.5-7B & ChatGLM3-6B & InternLM2-7B \\ \hline CFinBench & 56.77 & 60.16 & 58.56 & 57.64 & 48.36 & 47.28 & 43.65 \\ FinEval & 70.13 & 86.79 & 83.93 & 84.36 & 72.55 & 63.08 & 62.03 \\ \hline \end{tabular}
\end{table}
Table 4: The 0-shot and 3-shot average accuracy (%) of multiple-choice questions on the test split under the answer-only (AO) setting and chain-of-thought (COT) setting.

Figure 5: Examples of zero-shot prompts in answer-only setting. English translations are shown in blue for better readability.

\begin{table}
\begin{tabular}{l c c c c c} \hline  & Subject & Qualification & Practice & Law & Average \\ \hline Val (W/O) & 32.52 & 35.43 & 33.71 & 34.24 & 33.98 \\ Test (W/O) & 30.60 & 36.81 & 31.02 & 33.56 & 33.00 \\ Val (W) & 75.13 & 82.79 & 76.22 & 80.15 & 78.57 \\ Test (W) & 40.62 & 47.21 & 40.89 & 42.68 & 42.85 \\ \hline \end{tabular}
\end{table}
Table 6: The 0-shot accuracy (%) of IntermLM2.5-1.8B on validation and test split with or without data contamination.



* [164] M. C. D. A. K. S. R. G. Wilson, and J. E. H. Stewart, "A new approach to the determination of the mass of a single-photon resonance in the electron-positron channel", _Phys. Rev. Lett._**83**, 205902(R):070801, doi:10.1103/PhysRevLett.83.205902, arXiv:hep-ph/9903201
* [165] M. C. D. A. K. S. R. G. Wilson, and J. E. H. Stewart, "The mass of a single photon resonance at high energies", _JHEP_**01**, 043 (2001):013, doi:10.1007/JHEP01(2001)013, arXiv:hep-ex/010101

[MISSING_PAGE_POST]

 

* [138] M. C. B. K. S. A., and D. J. E. T. Jones. The distribution of the number of clusters in a cluster. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 45-67. ACM, 2009.
* [139] R. G. K. S. A., and P. W. H. L. van der Zijlstra. Cluster-based clustering for cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [140] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Clustering and clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [141] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [142] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [143] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [144] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [145] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [146] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [147] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [148] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [149] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [150] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [151] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [152] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [153] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [154] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [155] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [156] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [157] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [158] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [159] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [160] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [161] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [162] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [163] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [164] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [165] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [166] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [167] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [168] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [169] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [170] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [171] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [172] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [173] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [174] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [175] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [176] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [177] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [178] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [179] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [180] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [181] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [182] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [183] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [184] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10. ACM, 2009.
* [185] Y. Li, X. Zhang, and Q. Wang. Cluster-based clustering: Cluster-based clustering. In _Proceedings of the 2009 ACM SIGKDD international conference on Knowledge Discovery and data mining_, pages 1-10. ACM, 2009.
* [18 



\begin{table}
\begin{tabular}{l c c c c} \hline \hline Category & Single-Choice & Multiple-Choice & Judgment & All \\ \hline
**Subject** & _3302_ & _1889_ & _3915_ & _9106_ \\ \hline Political Economy (\(\text{\sc Fil