Venue: NAACL 2025

# FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark

for Large Language Models

Xin Guo\({}^{1}\), Haotian Xia\({}^{2}\), Zhaowei Liu\({}^{1}\), Hanyang Cao\({}^{1}\), Zhi Yang\({}^{1}\), Zhiqiang Liu\({}^{1}\),

**Sizhe Wang\({}^{1}\), Jinyi Niu\({}^{3}\), Chuqi Wang\({}^{2}\), Yanhui Wang\({}^{4}\), Xiaolong Liang\({}^{5}\), Xiaoming Huang\({}^{5}\), Bing Zhu\({}^{6}\), Zhongyu Wei\({}^{3}\), Yun Chen\({}^{1}\), Weining Shen\({}^{2*}\), Liwen Zhang\({}^{1*}\)**

\({}^{1}\)Shanghai University of Finance and Economics, \({}^{2}\)University of California, Irvine

\({}^{3}\)Fudan University, \({}^{4}\)Ping An Technology (Shenzhen) Co., Ltd.

\({}^{5}\)Tencent YouTu Lab, \({}^{6}\)HSBC Lab, HSBC, Shanghai, China

{zhang.liwen}@shufe.edu.cn, {weinnings}@uci.edu

Corresponding authors.These authors contributed equally to this work.

###### Abstract

Large language models have demonstrated outstanding performance in various natural language processing tasks, but their security capabilities in the financial domain have not been explored, and their performance on complex tasks like financial agent remains unknown. This paper presents FinEval, a benchmark designed to evaluate LLMs' financial domain knowledge and practical abilities. The dataset contains 8,351 questions categorized into four different key areas: Financial Academic Knowledge, Financial Industry Knowledge, Financial Security Knowledge, and Financial Agent. Financial Academic Knowledge comprises 4,661 multiple-choice questions spanning 34 subjects such as finance and economics. Financial Industry Knowledge contains 1,434 questions covering practical scenarios like investment research. Financial Security Knowledge assesses models through 1,640 questions on topics like application security and cryptography. Financial Agent evaluates tool usage and complex reasoning with 616 questions. FinEval has multiple evaluation settings, including zero-shot, five-shot with chain-of-thought, and assesses model performance using objective and subjective criteria. Our results show that Claude 3.5-Bonnet achieves the highest weighted average score of 72.9 across all financial domain categories under zero-shot setting. Our work provides a comprehensive benchmark closely aligned with Chinese financial domain. The data and the code are available at [https://github.com/SUFE-AIFLM-Lab/FinEval](https://github.com/SUFE-AIFLM-Lab/FinEval)

## 1 Introduction

With the development of the financial industry, its integration with large language models has become increasingly close. The financial sector needs large language models to process massive amounts of data, predict market trends, and optimize decision-making processes, thereby helping financial institutions enhance efficiency and reduce risks. This integration requires large models to possess critical capabilities in areas such as financial academic knowledge, industry knowledge, financial security, and financial agents. Financial academic knowledge necessitates that models have a foundational understanding of finance, serving as the baseline for applying large language models in the financial domain. Financial industry knowledge considers language interactions in practical application scenarios, requiring large models to have generalization capabilities across different contexts. Financial security involves various aspects of privacy for individuals and enterprises, which is a priority for the financial industry. Meanwhile, financial agents represent complex tasks within financial scenarios, involving numerous terms and methods that make them difficult for ordinary people to navigate.

There are several well-established benchmarks for evaluating English and Chinese foundation models, such as MMLU Hendrycks et al. (2021), BIG-bench Srivastava et al. (2022) and GAOKAOBench Zhang et al. (2023). Nevertheless, there are some significant drawbacks to these benchmarks for financial tasks: they only cover a small portion of the financial scenarios, and are not widely applicable in real-world financial circumstances. In addition, there are other benchmarks specifically designed to focus on advanced LLMs abilities that become apparent as the scale increases, such as hard math problem-solving Hendrycks et al. (2021), and coding Chen et al. (2021). Additionally, there are financial-specific benchmarks such as FinQA Chen et al. (2021), FinanceIQ Duxiaoman-DI (2023) and CFLUE Zhu et al. (2024). Although these benchmarks contribute differently to financial tasks, they are all hindered by their limited applicability in real-world situations, narrow scopes, and inability to adequately

capture the complexities of financial reasoning. As mentioned in He et al. (2024), with the rapid development of LLMs, they are gradually acquiring the ability to handle complex tasks, but there are still privacy and security issues. Ding et al. (2024) indicates that LLMs still face challenges in managing complex trading tasks in the financial domain. Therefore, for tasks like financial security and financial agent, which are more closely integrated with real-world financial scenarios and require higher standards, appropriate datasets are needed to assess LLM capabilities. Financial security emphasizes that large models must ensure the protection of personal information and cybersecurity in real-world financial applications. Meanwhile, financial agent highlights the need for large models to possess strong information processing and reasoning capabilities in the complex and dynamic financial market, as well as the ability to flexibly use various financial tools to decompose and solve complex financial tasks. Therefore, evaluating LLMs' capabilities in financial security and financial agent tasks is crucial for the financial domain.

We introduce FinEval, an extensive benchmark designed to evaluate the practical capabilities of large language models in the financial domain, with a particular focus on financial security and financial agent tasks within the Chinese context. The dataset contains 8,351 questions distributed divided into major domains: Financial Academic Knowledge, Financial Industry Knowledge, Financial Security Knowledge and Financial Agent, as illustrated in Figure 1. Financial Academic Knowledge comprises 4,661 multiple-choice questions spanning 34 subjects like finance and accounting, testing the theoretical foundation of models. Financial Industry Knowledge, with 1,434 questions, targets real-world financial practices, covering areas such as investment research. Financial Security Knowledge is assessed through 1,640 questions, covering eleven financial security tasks, including Security Analysis, Vulnerability Protection, etc. These questions evaluate the comprehensive capabilities of large language models in terms of security from multiple dimensions. Finally, Financial Agent consists of 616 questions, assessing the performance of large language models under complex information in real financial markets across three major dimensions and seven tasks. Our experiments evaluated various models, such as Claude 3.5-Sonnet, GPT-4o, Qwen2.5-72B-Instruct, and XuanYuan3-70B-Chat, to demonstrate their capabilities in various financial tasks. These models were assessed through zero-and few-shot standard prompting, as well as chain-of-thought prompting(Wei et al., 2022)).

In a series of experiments conducted under a zero-shot setting to evaluate LLM performance on financial knowledge, Claude 3.5-Sonnet performed the best among over 19 models, with the highest weighted average score (72.9). GPT-4o also demonstrated strong capabilities, particularly in financial security, achieving a notable score of 81.8. In addition, the open-source model Qwen2.5-72B-Instruct outperformed Claude 3.5-Sonnet in financial security and matched GPT-4o with a score of 81.8, making it a highly competitive model in this domain. However, this level of accuracy and similarity implies that there is still significant room for improvement in the field of finance for all LLMs.

In summary, our main contributions include:

* We introduced Financial Security Knowledge and Financial Agent based on academic and industry knowledge in finance, creating the first comprehensive dataset for evaluating financial security and agent tasks in the financial domain. The emergence of FinEval addresses the shortcomings of existing financial evaluation benchmarks, providing a comprehensive and in-depth assessment system for evaluating large language models in the financial sector.
* Our work innovatively adds a comparison of the capabilities of large models with those of ordinary individuals and experts in the financial domain, providing a valuable reference for the study of large models' capabilities in finance. The average result shows that large models have surpassed the level of ordinary individuals (30.1) in financial capabilities, but there is still a gap compared to financial experts (85.9), indicating that there is room for improvement in the capabilities of large models within specialized fields.
* Our dataset includes financial academic knowledge derived from publicly accessible mock exam questions, as well as financial industry knowledge compiled and totally rewritten by professionals in the financial field from various publicly available financial websites. Financial security knowledge is adapted from SecEval (Li et al., 2023) and developed in collaboration with domain experts with over five 

years of work experience. The questions for financial agents are manually created by finance experts. Answers are provided by GPT-4o and have undergone multiple rounds of review by financial experts. To better benefit the research community, our dataset will be made publicly available.

## 2 Related Work

General BenchmarkCurrent general benchmarks primarily focus on conventional tasks such as natural language understanding, text generation, logical reasoning, programming skills, professional knowledge, and multi-turn dialogues. These benchmarks rarely address tasks in the financial domain, particularly the security issues that are highly valued in finance and complex tasks like those involving agents. There are several well-established benchmarks for evaluating English and Chinese foundation models, including MMLU Hendrycks et al. (2021), HELM Liang et al. (2022), AGIEval Zhong et al. (2023), CLUE Xu et al. (2020), and C-Eval Huang et al. (2023). Other benchmarks focus on large language models' advanced abilities, like hard math problem-solving Hendrycks et al. (2021) and coding Chen et al. (2021), which become more apparent as model scale grows. TruthfulQA Lin et al. (2021) measures the authenticity of language models when answering questions. BIG-bench Srivastava et al. (2022) evaluates language models across various domains. CBLUE Zhang et al. (2021) is a collection of language understanding tasks in the biomedical field, including named entity recognition and information extraction. GAOKAO-Bench Zhang et al. (2023) gathers questions from the Chinese Gaokao examination to evaluate the language comprehension and logical reasoning abilities of LLMs. Similarly, AGIEval Zhong et al. (2023) assess the performance of foundation models on human-centric standardized exams, such as college entrance exams.

Financial BenchmarkHowever, the availability of benchmarks specifically catering to the financial domain remains limited, and current research mainly focuses on the financial academic and financial industry sectors. FLUE Shah et al. (2022), ConvFinQA Chen et al. (2022), BBT-CFLEB Lu et al. (2023), and FinQA Chen et al. (2021) in the English domain all focus solely on knowledge-based question answering. In the Chinese domain, FinanceIQ Chen et al. (2021) also emphasizes knowledge-based questions. CFLU Zhu et al. (2024) provides questions and NLP tasks related to Chinese financial knowledge, but its actual evaluation tasks are still limited to the knowledge level and do not include important topics such as financial security, nor do they delve into more complex agent-related tasks designed for financial business scenarios. Additionally, there is no comparison

Figure 1: FinEval is divided into four parts:Financial Academic Knowledge, Finance Industry Knowledge, Financial Security Knowledge and Financial Agent. The number of each sub-dataset is indicated after the corresponding name.

 

between large models and ordinary individuals or experts in the financial domain, making it difficult to accurately assess the true capabilities of large models in financial scenarios.

## 3 FinEval Benchmark

### Overview

We introduce FinEval, a benchmark specifically designed for evaluating large models in the Chinese financial domain. Building on academic and industry knowledge, we further upgrade our focus to address important security and agent tasks in real-world applications within the financial sector.

Financial academic knowledge and financial industry knowledge encompass the fundamental concepts of the financial domain, particularly the subject-specific questions in academic knowledge and various investment recommendations or other specific tasks in industry knowledge. Financial security knowledge is crucial for LLMs in the financial sector, as it involves all aspects related to user or enterprise information. LLMs must possess robust security capabilities to address the various challenges faced by the financial industry. Financial agent tasks involve complex decisions and operations that go beyond simple information processing, requiring LLM to have a deep understanding of financial data and the ability to analyze decisions.

As for the question type, financial academic knowledge and security knowledge datasets primarily consist of multiple-choice questions. The multiple-choice questions follow a format similar to that in Hendrycks et al. (2021). In financial industry knowledge, we differentiate between semi-open questions (where the answers typically consist of a few words or phrases, or are selected from specific options) and open questions (which require long text responses) through objective short-answer questions and subjective open-ended questions. For the financial agent dataset, the questions are designed as open-ended questions focused on agent-specific tasks. The answers to these open-ended questions are usually long text passages that encompass various outputs, such as task steps, strategies, and results, making them more suitable for complex financial scenarios. Examples of all questions can be found in Appendix B.

### Data Collection

#### 3.2.1 Data source and data quality

Our financial academic knowledge data mainly comes from publicly accessible mock exams and adaptations of questions from certification exams or printed textbooks. The financial industry knowledge data is collected and adapted from various financial websites. All of this data is gathered and adapted by professionals in the financial field, ensuring there are no copyright or other issues. The financial security knowledge data is adapted from SecEval, with the adaptation and annotation work completed by financial experts with over five years of work experience. Similarly, the financial agent data is produced by these experts in finance, using GPT-4o for answer generation and undergoing multiple rounds of review by them.

In terms of data quality, the dataset is collected and adapted by eight postgraduate students with backgrounds in statistics and finance. Three financial experts, specializing in evaluation logic and content, strictly adhere to data quality requirements and are responsible for data quality checks. They manually select and filter questions based on multiple dimensions, including content, direction, logic, and difficulty. After the data adaptation and annotation are completed, the three quality checkers review all the data. Only when all three quality checkers reach a consensus on all aspects of the data is it retained; otherwise, it needs to be re-adapted or annotated. Similarly, the financial security data is adapted and reviewed by the three financial experts. As a result, a high-quality FinEval dataset is obtained.

#### 3.2.2 Data Processing

The academic knowledge multiple-choice questions in FinEval primarily consist of PDF files, with most sourced from exercise sets in various textbooks, mock exams from different certifications, and past exam questions. All questions in the financial academic knowledge have been processed and refined to include only four options. The multiple-choice questions for financial security knowledge are handled in a similar manner, with financial experts also retaining only four options. The objective short-answer and subjective open-ended questions in financial industry knowledge, as well as the complex open-ended questions in financial agent, are all answered by GPT-4o and reviewed by domain experts. All of the above questions are ultimately 

converted into a structured format.

For subjects involving mathematical formulas in financial academic knowledge, we convert them into standard LaTeX format, which is a typesetting system commonly used for creating high-quality documents, particularly in academic and technical fields. LaTeX allows us to express mathematical expressions directly using text format. Approximately 100 questions were handled for each subject. Examples can be found in Appendix B

### Statistics

The questions in FinEval contains four parts: Financial Academic Knowledge consists 4661 questions and 34 distinct subjects, which are subsequently classified into broader categories, including Finance, Economy, Accounting, and Certificate. Financial Industry Knowledge consists of 1434 questions and 10 specific directions, which are further categorized into three specific scenarios: Investment research, Investment Advisor, and Financial Operations. Financial Security Knowledge consists of 11 specific directions with 1640 questions, which consists of four specific scenarios: Software and Application, Network and System Protection, Security Analysis and Vulnerability Protection. Finance Agent consists of 7 categories of tasks and 616 detailed tasks in total.The task is divided into 3 different aspects: Reasoning and Planning, Longterm Memory and Tool Application. All the detail tasks and their broader categories can be found in the Appendix C, as well as the number of questions included in each task.

## 4 Experiments

Our following experiments show the evaluation results of diverse LLMs on FinEval to analyze their performance and provide baselines for future usage of FinEval.

### Experimental Setup

In this section, we will outline the experimental setup utilized to evaluate the performance of LLMs on financial academic knowledge, financial industry knowledge, financial security knowledge and financial agent. To gauge the adaptability of these large language models, we conducted zero-shot and five-shot with Chain of Thought (CoT). Additionally, we provide specific examples on how to design the prompts in Appendix B. Due to limitations related to funding and other factors, we extracted 20% of the total data as the test set for the evaluation of the final results.

We selected accuracy as the metric for multiple-choice questions in financial academic knowledge and financial security knowledge. In financial industry knowledge, we use Rouge-L Lin (2004) as the evaluation metric. Agent tasks are typically more complex, comprehensive, and open-ended, with the outputs of large models being longer and more flexible. In such cases, using Rouge-L or other objective evaluation metrics may not accurately assess the quality of the model's output. Therefore, we introduced GPT-4o as a judge model. The judge model scores the responses based on predefined prompts by analyzing several aspects, including the semantic relevance, coherence, logic, and overall quality of the output in relation to the task requirements. GPT-4o is capable of un

Figure 2: Examples of financial security and financial agent. For better readability, the English translation is displayed below the corresponding Chinese text. Additional examples can be found in Appendix B.

 

derstanding and evaluating the nuances of longer, more diverse responses, making it well-suited for assessing the quality of outputs in complex, multi-dimensional scenarios. At the same time, we also used Claude 3.5-Sonnet and Gemini1.5-Pro as the judging models to compare with GPT-4o. The judging results of Claude 3.5-Sonnet and Gemini1.5-Pro, as well as the comparison of the scoring results of the three judging models, can be found in Tables 26, 27, and 28 in Appendix D, respectively. The judging prompt for scoring evaluation can be found in Appendix B.

As a result, our model evaluation encompasses four types of scenarios: zero-shot prompting, five-shot prompting, zero-shot CoT prompting and five-shot CoT prompting. Due to the higher complexity of financial agent tasks, we only conduct evaluations on the financial agent data under the zero-shot prompting setting.

### Models

To achieve a comprehensive understanding of the state of LLMs in the context of the Chinese language, we conducted an evaluation of 19 high-performing LLMs that can process Chinese input. The detailed information about these LLMS participating in the evaluation can be found in Appendix A.

Closed-Source Models:In the realm of closed-source models, we evaluated six leading, high-performance LLMs provided by three organizations, including GPT-4o (OpenAI, 2024b) and GPT-4o-mini (OpenAI, 2024a) from OpenAI, Claude 3.5-Sonnet (Anthropic, 2024) from Anthropic, and Gemini1.5-Flash and Gemini1.5-Pro (Team, 2024a) from Google.

Open-Source Models:For open-source models, we evaluated seven mainstream LLMs capable of understanding and generating Chinese, including Baichuan2-13B-Chat (Baichuan-inc, 2023), Yi1.5-9B-Chat (01.AI, 2024), Yi1.5-3dB-Chat (01.AI, 2024), ChatGLM3-6B (THUDM, 2023), GLM-4-9B-Chat (GLM et al., 2024), InternLM2-20B-Chat (Team, 2023), InternLM2.5-20B-Chat (Team, 2024b), Qwen2.5-7B-Instruct (Qwen, 2024b), Qwen2.5-72B-Instruct (Qwen, 2024a), etc.

Financial Domain Models:In financial domain, we evaluated five representative LLMs tailored for the financial tasks, including DISC-FinLLM (Fudan-DISC, 2023), FinGPTV3.1 (AI4Finance-Foundation, 2023), CFGPT2-7B (TongjiFinLab, 2024), XuanYuan2-7OB-Chat (Duxiaoman-DI, 2024a) and XuanYuan3-7OB-Chat (Duxiaoman-DI, 2024b).

### Results

We evaluated the models in four settings: zero-shot, five-shot, zero-shot CoT, and five-shot CoT. However, due to the extensive variety, high difficulty, and complexity of financial agent tasks, we only evaluated financial academic knowledge, financial industry knowledge, and financial security knowledge in the other three settings, with these results available in Appendix D. In the zero-shot setting, we present the results of 19 models participating in the evaluation across the four independent tasks.

Table 1 showcases the abilities of 19 models under the setting of zero-shot. Among them, Claude 3.5-Bonnet and GPT-4o has shown outstanding capabilities with an weighted average score exceeding 70 and performing the best in all four task categories. When comparing models across three categories, we find that overall, closed-source models outperform open-source models, which in turn outperform models specialized in the financial domain. Among the open-source general and financial models, Qwen2.5-72B-Instruck ranks the highest with an weighted average score of 69.4. Following closely are Qwen2.5-7B-Instruct, Yi1.5-34B-Chat and XuanYuan3-7OB-Chat. This demonstrates the outstanding capabilities of these open-source models in the financial domain. Furthermore, it can be observed that general models rank relatively higher compared to financial models, while except XuanYuan3-7OB-Chat, other fine-tuned models rank relatively lower, indicating that general models perform better in financial domain and suggesting their superior task generalization abilities. In our five-shot CoT setting, Qwen2.5-72B-Instruct performed exceptionally well, ranking first compared to its third-place ranking in the zero-shot setting. The financial large model XuanYuan3-7OB-Chat also climbed from ninth place in the zero-shot setting to first place. This trend is also observed in some later-released open-source large models, such as GLM-4-9B-Chat. This indicates that open-source large language models can enhance their performance through prompt optimization guidance





### Contrast Analysis

To better evaluate the capabilities of large language models and to make meaningful contributions to model research, we organized a competition between these models, ordinary people, and financial experts. In our dataset, we selected relatively easier multiple-choice questions as competition topics. Considering that some question-and-answer items may be difficult for ordinary individuals and that a limited number of questions would not be representative, we extracted 20% from the financial academic knowledge and financial security knowledge used for evaluation, totaling 260 questions for testing.

For the ordinary participants, we randomly selected three undergraduate students who had no prior exposure to financial or security-related knowledge. For the expert responses, we also randomly selected three experts in the financial field with more than five years of work experience to participate in the answering. To ensure the validity of the results, all participants involved in the testing have not been exposed to any FinEval questions. We present the comparison results between the large language models, ordinary individuals, and financial experts in Table 2. It can be seen that in both the financial academic knowledge and financial security knowledge test results, the performance of closed-source, general, and financial domain models far exceeds that of ordinary individuals, with closed-source models and some general models achieving notably high performance. However, the overall results of the large models still have a gap compared to the experts, with the best models showing nearly a 10% difference from expert results. Nevertheless, certain models, such as GPT-4o and Qwen2.5-72B-Instruct, have demonstrated capabilities in safety that are very close to expert levels, with a gap of around 5%, reflecting the current emphasis on safety in large models.

## 5 Error Analysis

To further identify the shortcomings of large language models in financial knowledge and tasks, we analyze the errors made by the model during the testing process. Financial Agent includes a series of relatively open-ended, subjective tasks that place high demands on the model, and the model displayed a diverse and rich variety of error types when completing more complex tasks. Three types of errors were identified for answers that did not receive full scores from the evaluation model: Logical Reasoning Error, Contextual Misunderstanding, and Ambiguity Handling Weakness. Logical Rea

\begin{table}
\begin{tabular}{l l c c c c} \hline \hline
**Source** & **Category** & **Financial Academic** & **Financial Security** & **Average** \\ \hline Human & Ordinary individual & 35.1 & 26 & 30.1 \\ Express & **84.9** & **86.8** & **88.9** \\ \hline Closed-source & Clinkb 3-5-Semet & 73.9 & 78.1 & 76 \\  & GPT-4e & 71.5 & 81.8 & 76.7 \\ General & Openv 5-72B-Interst & 69.7 & 81.8 & 75.8 \\ Yil 5-34B-Clust & 59.5 & 76 & 67.8 \\ Financial & XuanWanX-70B-Chart & 55.2 & 74.4 & 64.8 \\ CFPGT-7B & 53.9 & 65.1 & 99.5 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance comparison across ordinary individuals, experts and LLMs (selected with top 2 model’s results within each category).

\begin{table}
\begin{tabular}{l l r r r r r} \hline \hline
**Model** & **Size** & **Finnacial Academic** & **Financial Industry** & **Financial Security** & **Financial Agent** & **Weighted Average** \\ \hline Claude 3.5-Somnet & unknown & **73.9** & 60.6 & 78.1 & **79.3** & **72.9** \\ GPT-4o & unknown & 71.5 & **61.3** & **81.8** & 73.9 & 71.9 \\ Qwen2.5-72B-Invstruct & 72B & 69.7 & 54.4 & **81.8** & 68.4 & 69.4 \\ Gemini1.5-Prub & unknown & 68.3 & 60.5 & 77.8 & 72.8 & 69.2 \\ GPt-d mini & unknown & 62.4 & 61.1 & 79.1 & 72.9 & 66.2 \\ Gemini1.5-Flash &unknown & 62.1 & 61.2 & 77.5 & 70.9 & 65.6 \\ Quenz2.5-7B-Instruct & 7B & 62.7 & 48.3 & 71.7 & 66.7 & 62.3 \\ Vi1.5-34B-Chart & 34B & 59.5 & 49.6 & 76.0 & 66.0 & 61.5 \\ XuanYunx-70B-Chart & 70B & 55.2 & 52.0 & 74.4 & 63.9 & 59.1 \\ InternalM2.5-20B-Cast & 20B & 54.7 & 53.2 & 74.1 & 63.1 & 58.9 \\ GLM4.4-9B-CHart & 9R & 54.7 & 53.1 & 73.1 & 60.2 & 58.4 \\ InternalM2.20B-Cast & 20B & 54.7 & 50.3 & 73.1 & 60.9 & 58.0 \\ Vili 5-9B-Chiat & 9R & 55.0 & 44.7 & 71.4 & 61.1 & 56.9 \\ XuanYunx-70B-Chart & 70B & 52.8 & 46.6 & 68.0 & 61.7 & 55.4 \\ CFQPT2.7B & 7B & 53.9 & 50.2 & 65.1 & 50.9 & 55.3 \\ Baschang-13B-Chart & 13B & 41.1 & 50.2 & 61.6 & 55.7 & 47.8 \\ ChatGLM3-6B & 6B & 38.9 & 48.6 & 48.2 & 49.6 & 43.2 \\ DISCF-FinLM & 13B & 39.1 & 46.1 & 25.2 & 41.8 & 37.8 \\ FuGPIV3.1 & 6B & 25.3 & 36.1 & 22.7 & 31.2 & 27.1 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Average zero-shot scores across four evaluated categories. We report the results under zero-shot setting for four categories and one final weighted average: Financial Academic Knowledge, Financial Industry Knowledge, Financial Security Knowledge and Financial Agent. For the scoring criteria for each section, please refer to Section 4.1. As for the details of the models involved in the evaluation, you can refer to Table 3 in Appendix A.



soning Error occurs when the model fails to draw correct logical conclusions. Contextual Misunderstanding happens when the model misses or misinterprets relevant context. Ambiguity Handling Weakness refers to the model's inability to handle unclear or ambiguous questions properly. Detail examples can be found in Appendix B. We selected ten representative models and randomly sampled 27 erroneous responses for each model on each task. The results are shown in Figure 3. Each bar in the chart represents the proportion of a specific type of error among all errors made by a particular model.

Across all the erroneous responses, the average proportion for open-source models in Logical Reasoning Error, Contextual Misunderstanding, and Ambiguity Handling Weakness are 29.0%, 41.6%, and 29.4%, respectively. In comparison, the corresponding average proportion for closed-source models are 21.3%, 37.6%, and 41.1%. This suggests closed-source models are better at reasoning and understanding long texts, but more prone to semantic issues and hallucinations. This could be due to their larger parameter sizes, which, while improving reasoning and comprehension, also lead to over-interpretation of ambiguous content. Specifically, the results show that open-source models are more likely to encounter Logical Reasoning Errors and Contextual Misunderstanding, while closed-source models are more prone to Ambiguity Handling Weakness. For Logical Reasoning Errors, models struggle with multi-step reasoning tasks, often misusing formulas or making calculation mistakes. In Contextual Misunderstanding, models fail to connect distant parts of the text. For Ambiguity Handling Weakness, models have trouble interpreting vague information, sometimes leading to hallucinations. They often include unnecessary tools or data in their responses, indicating a need for better semantic understanding and relevance filtering.

## 6 Conclusion

This study introduces FinEval, a benchmark for evaluating large language models' capabilities in the financial domain. Unlike previous financial evaluation benchmarks, FinEval delves deeper into financial security and financial agent, covering pressing security issues and complex agent tasks in the financial field, assessing models' security and their ability to handle complex tasks. Our results indicate that Claude 3.5-Sonnet performs the best among the 19 models evaluated, but it still faces challenges with more complex tasks. While it surpasses ordinary individuals, it has not reached the level of human experts. This study illustrates that although large language models have made certain breakthroughs in the financial domain, they still require a more in-depth and detailed understanding to enhance their task generalization capabilities in a wider and more complex diverse financial market environment. As an important benchmark for future research on large language models in the financial field, FinEval provides a structured frame

Figure 3: Error analysis results of ten models. Each bar represents the proportion of a specific type of error among all errors made by a particular model. The sum of the values of the three bars for a model equals to 1, representing the total error distribution for that model.

 

work for measuring and improving the capability of large language models, contributing to the development of evaluation benchmarks in the Chinese financial domain.

## Limitations

Although FinEval has become a relatively comprehensive evaluation benchmark in the financial domain, encompassing a wide range of financial tasks, we acknowledge its limitations. With the ongoing emergence and iteration of data formats such as image, audio, and video, there is an increasing amount of multimodal data in the financial domain, which presents a limitation for the current FinEval. This limitation highlights the need to develop a multimodal evaluation dataset as the next focus for FinEval. We will recruit more specialized financial personnel to collect additional multimodal data related to the financial domain (especially financial chart data, which is particularly important), to evaluate a broader range of multimodal large language models, ensuring that FinEval remains a comprehensive benchmark in the financial field.

## Acknowledgments

This research is supported in part by the National Social Science Fund of China 22BTJ031 (Liwen Zhang). We sincerely appreciate the assistance and guidance provided by Professor Xiao Cao and Associate Professor Min Min from the School of Finance at Shanghai University of Finance and Economics, as well as Shen Wang from the Experimental Center at Shanghai University of Finance and Economics.

## References

* [1]M. Al-H. (2024) YYi: open foundation models by 01.ai. Note: [https://github.com/01-ai/Yi-1](https://github.com/01-ai/Yi-1) Cited by: SS1.
 

Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yixai Zhang, Jiayi Lei, et al. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. _arXiv preprint arXiv:2305.08322_.
* Li et al. (2023) Guancheng Li, Yifeng Li, Wang Guannan, Haoyu Yang, and Yang Yu. 2023. SecEval: A comprehensive benchmark for evaluating cybersecurity knowledge of foundation models. [https://github.com/XuanwuAI/SecEval](https://github.com/XuanwuAI/SecEval).
* Liang et al. (2022) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soyltu, Michihiro Yasunaga, Yuan Zhang, Deepak Narayanan, Yuhui Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. _arXiv preprint arXiv:2211.09110_.
* Lin (2004) Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In _Text Summarization Branches Out_, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
* Lin et al. (2021) Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. _arXiv preprint arXiv:2109.07958_.
* Lu et al. (2023) Dakuan Lu, Jiaqing Liang, Yipei Xu, Qianyu He, Yipeng Geng, Mengkun Han, Yingsi Xin, Hengku Wu, and Yanghua Xiao. 2023. Bbt-fin: Comprehensive construction of chinese financial domain pre-trained language model, corpus and benchmark. _arXiv preprint arXiv:2302.09432_.
* OpenAI (2024a) OpenAI. 2024a. Gpt-4o mini: advancing cost-efficient intelligence. _[https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/_](https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/_).
* OpenAI (2024b) OpenAI. 2024b. Hello gpt-4o. _[https://openai.com/index/hello-gpt-4o/_](https://openai.com/index/hello-gpt-4o/_).
* Qwen (2024a) Qwen. 2024a. Qwen2.5-72b-instruct. [https://github.com/QwenLM/Qwen2.5](https://github.com/QwenLM/Qwen2.5).
* Qwen (2024b) Qwen. 2024b. Qwen2.5-7b-instruct. [https://github.com/QwenLM/Qwen2.5](https://github.com/QwenLM/Qwen2.5).
* Shah et al. (2022) Raj Sanjay Shah, Kunal Chawla, Dheeraj Eidhani, Agam Shah, Wendi Du, Sudheer Chava, Natraj Raman, Charese Smiley, Jiao Chen, and Diyi Yang. 2022. When flue meets flang: Benchmarks and large pre-trained language model for financial domain. _arXiv preprint arXiv:2211.00083_.
* Srivastava et al. (2022) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abuwal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adria Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. _arXiv preprint arXiv:2206.04615_.
* Team (2024a) Gemini Team. 2024a. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.
* Team (2023) InternLM Team. 2023. Interlnm: A multilingual language model with progressively enhanced capabilities. [https://github.com/InternLM/InternLM](https://github.com/InternLM/InternLM).
* Team (2024b) InternLM Team. 2024b. Internlm: A multilingual language model with progressively enhanced capabilities. [https://github.com/InternLM/InternalM](https://github.com/InternLM/InternalM).
* THUDM (2023) THUDM. 2023. Chatglm3-6b. [https://github.com/THUDM/ChatGLM3](https://github.com/THUDM/ChatGLM3).
* TongjiFinLab (2024) TongjiFinLab. 2024. Cfgpt2: Chinese financial assistant with large language model. [https://github.com/TongjiFinLab/CFGPT](https://github.com/TongjiFinLab/CFGPT).
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H Chi, Quoc V Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. In _Advances in Neural Information Processing Systems_.
* Xu et al. (2020) Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weiljan Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan. 2020. CLUE: A Chinese language understanding evaluation benchmark. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 4762-4772, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Zhang et al. (2021) Ningyu Zhang, Mosha Chen, Zhen Bi, Xiaozhuan Liang, Lei Li, Xin Shang, Kangping Yin, Chuanqi Tan, Jian Xu, Fei Huang, et al. 2021. Clube: A chinese biomedical language understanding evaluation benchmark. _arXiv preprint arXiv:2106.08087_.
* Zhang et al. (2023) Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xiepeng Qiu. 2023. Evaluating the performance of large language models on gaokao benchmark. _arXiv preprint arXiv:2305.12474_.
* Zhong et al. (2023) Wanjun Zhong, Ruixiang Cui, Yaduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023. Agricultural: A human-centric benchmark for evaluating foundation models. _arXiv preprint arXiv:2304.06364_.
* Zhu et al. (2024) Jie Zhu, Junhui Li, Yalong Wen, and Lifan Guo. 2024. Benchmarking large language models on cflue-a chinese financial language understanding evaluation dataset. _arXiv preprint arXiv:2405.10542_ 

Evaluated Models Overview

We list the models we evaluated in this paper in Table 3.

## Appendix B Examples for FinEval

We list the examples of FinEvalin this paper in Figure 4, 5, 6, 7, 8, 9, 10, 11, 12. In which, Figure 4, 5, 6, 7 are examples of zero-shot, five-shot, zero-shot CoT, and five-shot CoT, respectively. Figure 8 is an objective short-answer question, and figure 9 is a subjective open-ended question. Figure 10 is an example in LaTeX format. Figure 11, 12 are examples of two subclasses (API invocation and task planning) under the theme of Financial Agent. Figure 13, 14 are evaluation prompts for API invocation and task planning respectively. Figure 15, 16, 17, 18 are examples of typical error LLM makes in solving financial agent tasks.

## Appendix C Detailed Statistics of FinEval

Table 4 presents detailed information on the four sections of FinEval.

## Appendix D Other results

### Five-shot Results Analysis

In the five-shot setting across three financial tasks, Claude 3.5-Sonnet achieved the highest overall weighted average score (73.2). GPT-4o excelled in Financial Security Knowledge with a score of 81.8, while Qwen2.5-72B-Instruct performed strongly in Financial Security as well, achieving 80.2. (Table 5).

### Zero-shot CoT Results Analysis

In the zero-shot CoT setting across three financial tasks, Claude 3.5-Sonnet achieved the highest overall weighted average score (72.4). Qwen2.5-72B-Instruct led in Financial Academic Knowledge (76.5), while GPT-4o excelled in Financial Security Knowledge (81.8). (Table 6).

### Five-shot CoT Results Analysis

In the five-shot CoT setting, Qwen2.5-72B-Instruct achieved the highest overall weighted average score (73.6), leading in Financial Academic Knowledge (75.7) and performing well in Financial Security (78.5). GPT-4o excelled in Financial Security with a score of 81.2. (Table 7).

### Comprehensive Results for Individual Financial Tasks Across Four Settings

Evaluation results for finance academic knowledge under four settings are shown in Table 8, 9, 10 and 11. Evaluation results for finance industry knowledge including objective and subjective questions under four settings are shown in Table 12, 13, 14, 15, 16, 17, 18 and 19. Evaluation results for finance security knowledge under four setting are shown in Table 20, 21, 22 and 23. Evaluation result for finance agent under zero-shot setting is shown in Table 24.

### Results Analysis

The performance of various open-source models varies under FinEval, and we analyze the reasons as follows:

(1)First, FinEval is quite challenging, closely integrated with professional knowledge and realworld business scenarios, providing a more realistic reflection of the various models' true capabilities.

(2)Second, most financial LLMs on the market are derived from general base models that have been trained, rather than being directly trained base models. This can lead to some loss in capability.

(3)Third, the general closed and open-source models that perform well in FinEval evaluations are recognized as stronger LLMs in the current LLM field, so it is normal for financial LLMs, such as the XuanYuan series, not to perform exceptionally well.

(4)Finally, many of the underperforming financial LLMs come from various universities or small AI organizations, which typically utilize LoRA fine-tuning. These models often lack vast computing resources, leading to poorer fine-tuning results. Moreover, these financial LLMs have not undergone any updates or iterations, making their subpar performance reasonable.

### Comparative Analysis Under Different Evaluation Methods

#### d.6.1 Correlation analysis between human evaluation and GPT evaluation

In this section, we will demonstrate that GPT evaluation can replace human evaluation in large-scale testing by analyzing the correlation between human evaluation and GPT evaluation results. Table 25 is the Spearman correlation matrix of human evaluation scores and GPT evaluation scores for Gemini 1.5-Pro across seven agent tasks, each task 

samples 20% of its entire data for testing. The resulting matrix is derived from the human evaluation result matrix and the GPT evaluation result matrix. The diagonal elements represent the correlation of scores for the same task under both evaluation systems, while the off-diagonal elements represent the correlation of scores for different tasks. The correlation coefficients for human evaluation scores and GPT evaluation scores for each task are 0.66, 0.84, 0.68, 0.85, 0.85, 0.63, and 0.55, with an average correlation coefficient of 0.72. The similarity for each task are 0.70, 0.63, 0.70, 0.96, 0.67, 0.74, and 0.93, with an average similarity of 0.76. It can be seen that the GPT evaluation scores have a high correlation with human evaluation scores, indicating that GPT can replace human scoring in large-scale testing.

#### d.6.2 Evaluation Results Using Different LLMs as Judges

We employed Claude-3.5-Sonnet and Gemini-1.5-Pro as judges to score Agent tasks according to expert evaluation criteria in order to compare their differences with GPT-4o. We conducted tests on five closed-source LLMs that participated in the FinEval evaluation. The results of Claude-3.5-Sonnet are shown in Table 26, the results of Gemini-1.5-Pro are shown in Table 27, and the comparative

\begin{table}
\begin{tabular}{l l c c c c} \hline \hline
**Category** & **Model** & **Creator** & **Parameter** & **Access** & **Version Date** \\ \hline
**Closed-Source** & GPT-4o & OpenAI & undisclosed & API & 2024.5 \\  & GPT-4o-mini & OpenAI & undisclosed & API & 2024.7 \\  & Gemini1.5-Flash & Google & undisclosed & API & 2024.5 \\  & Gemini1.5-Pro & Google & undisclosed & API & 2024.5 \\  & Claude 3.5-Sonnet & Anthropic & undisclosed & API & 2024.3 \\ \hline
**Open-Source** & Qwen2.5-DInstruct & Alibaba Cloud & 7B & Weights & 2024.9 \\  & Quven2.5-7ZB-Instrcut & Aliabba Cloud & 72B & Weights & 2024.9 \\  & ChatGLM3-6B & Tsinghua \& Zhipu.AI & GB & Weights & 2023.10 \\  & GLM-4-9B-Chat & Tsinghua \& Zhipu.AI & 9B & Weights & 2024.6 \\  & Yi1.5-9B-Chat & 01.AI & 9B & Weights & 2024.5 \\  & Yi1.5-34B-Chat & 01.AI & 34B & Weights & 2024.5 \\  & InternalLM2-20B-Chat & Shanghai AI Lab \& SenseTime & 20B & Weights & 2024.1 \\  & InternDL2.5-20B-Chat & Shanghai AI Lab \& SenseTime & 20B & Weights & 2024.8 \\  & Baichuan2-13B-Chat & Baichuan & 13B & Weights & 2023.12 \\ \hline
**Financial** & XuanVana3-7OB-Chat & Duxiaoman-DI & 70B & Weights & 2024.9 \\  & XuanYuan2-7OB-Chat & Duxiaoman-DI & 70B & Weights & 2024.3 \\  & DISC-FinLLM & FudanDISC & 13B & Weights & 2023.10 \\  & CFGPT2-7B & TongjiFinLab & 7B & Weights & 2024.8 \\  & FinGPTv3.1 & AIAFinance-Foundation & 6B & Weights & 2023.10 \\ \hline \hline \end{tabular}

* The total owners’ equity of Company was a 20 million yuan at the beginning of 2015, and the loss was 2 million yuan in 2015 and 3 million yuan in 2016. The pre-tax profit from 2017 to 2019 was 0 million cash year, and the company achieved a pre-tax profit of 8 million yuan in 2020. The legal surplus reserve shall be withdrawn at 10%oft the net profit. The board of directors of the company proposed the annual distribution profit of 500,000 yuan in 2020, but it has not yet been submitted to the general meeting of shareholders for consideration. Assuming that the company does not have other tax adjustments other than making up losses and other matters resulting in changes in owners’ equity from 2015 to 2020, the applicable corporate income tax rate is 25%. The total owners' equity of Company A at the end of 2020 is ....illion yuan.

[MISSING_PAGE_POST]

ep=0pt]
* [noitemsep=0pt]
* [nosep=0pt]
* [noitemsep=0pt]
 

results of the three judging LLMs are shown in Table 28. From the comparison of the results, it is evident that the three judging LLMs performed differently. GPT-4o was stricter in its evaluation, while Gemini-1.5-Pro was more lenient, and Claude-3.5-Sonnet fell between the two.

Figure 5: An instance of five-shot evaluation. The red text denotes the response automatically generated by the model, with the preceding text being the input prompt. English translations for the related Chinese text are provided beneath.

 


* [16] A. J. Laird, D. P. Kingma and Y. Bengio, "Adam: a method for stochastic optimization," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, volume 2084 of _Lecture Notes in Computer Science_, pages 357-396, Cham, 2015.
* [17] M. C. Davis, R. G. LeCun, S. K. Lee, and T. Darrell, "A new approach to minimizing the maximum likelihood estimation problem," _IEEE Transactions on Information Theory and Applications_, vol. 10, no. 1, pp. 115-120, Jan 2006.
* [18] B. E. Hinton, I. Goodfellow, and Z. Jia, "Deep learning with deep neural networks," _arXiv preprint arXiv:1503.03087_, 2015.
* [19] H. Li, X. Zhang, and Q. Weinberger, "Learning generative adversarial nets by reducing the dimensionality of data," _Journal of Machine Learning Research_, vol. 10, no. 1, pp. 103-118, Mar 2015.
* [20] W. Liu, X. Wang, and Q. Weinberger, "Deep learning with convolutional neural networks," _CoRR_, abs/1512.03087, 2015.
* [21] F. Ma, J. Sun, and Q. Weinberger, "Deep learning with convolutional neural networks," _Neural Networks_, vol. 10, no. 1, pp. 103-113, Feb 2015.

[MISSING_PAGE_POST]

 

* [26] M. C. Bender, R. J. K. Sears, and A. D. Lafferty. The role of the human-level system in social science: a review. In _Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 357-380, 2004.
* [27] G. F. Hinton, P. H. Friedman, and E. Hinton. Visualizing machine learning with applications to image processing. _Journal of Machine Learning Research_, 1(1):1-10, 2004.
* [28] T. Hastie. An introduction to neural networks. _Neural Networks_, 1:1-2, 1996.
* [29] Y. Bengio. Generative adversarial nets for large scale datasets. _arXiv preprint arXiv:1503.03017_, 2015.

[MISSING_PAGE_POST]

 

Figure 8: An example of objective short-answer questions in Financial Sentiment Analysis. For better readability, the English translation is displayed below the corresponding Chinese text.

Figure 9: An example of subjective open-ended questions in Financial Investment Advice. For better readability, the English translation is displayed below the corresponding Chinese text.

 


* [1] A. SNN(Sigma^2), [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [21], [22], [23], [24], [25], [26], [27], [28], [29], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [20], [21], [22], [23], [24], [25], [27], [28], [29], [21], [22], [24], [25], [26], [27], [28], [29], [20], [22], [23], [24], [25], [27], [28], [29], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [21], [22], [23], [24], [25], [27], [28], [29], [22], [29], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [23], [24], [25], [28], [29], [20], [22], [23], [24], [25], [26], [27], [28], [29], [21], [22], [23], [24], [25], [27], [28], [29], [23], [24], [25], [29], [26], [28], [29], [21], [22], [23], [24], [25], [27], [28], [29], [20], [22], [23], [24], [25], [26], [27], [28], [29], [20], [22], [29], [21], [22], [23], [24], [25], [26], [27], [28], [29], [20], [22], [23], [24], [25], [27], [28], [29], [20], [22], [29], [21], [22], [23], [24], [25], [26], [27], [28], [29], [21], [22], [23], [24], [25], [26], [27], [28], [29], [20], [22], [23], [24], [25], [26], [27], [28], [29], [21], [22], [23], [24], [25], [26], [27], [28], [29], [20], [22], [23], [24], [25], [26], [27], [28], [29], [21], [22], [23], [24], [25], [26], [27], [28], [29], [20], [22], [23], [24], [25], [26], [27], [28], [29], [21], [22], [23], [24], [25], [26], [27], [28], [29], [20], [22], [23], [24], [25], [26], [27], [28], [29], [21], [22], [23], [24], [25], [26], [27], [28], [29], [20], [22], [23], [24], [25], [2 

* [10] M. C. B. K. and R. S. W. A., _Theoretical Methods for the Study of Simulation in Practice: An Introduction to Data Mining_. Springer Science & Business Media, vol. 327, pp. 45-68.
* [11] J. D. L., _A Practical Guide to Machine Learning_. Springer Science & Business Media, vol. 39, no. 1, p. 105-118.

[MISSING_PAGE_POST]

 

Figure 14: The evaluation prompt of task planning by financial agent evaluation.

Figure 13: The evaluation prompt of API invocation by financial agent evaluation.

 

Figure 15: An example of error model(GPT-4o) encountered while solving a financial text summarization problem related to handling long texts. The expected answer was to include both the entity announcing the policy and the policy content, but the model’s output only focused on the latter part of the material, addressing the policy content, while neglecting the entity that announced the policy at the beginning.

 

Figure 16: An example of a logical reasoning error made by a model when solving a financial CoT task.The model made an error in the 5th step by applying the wrong formula to calculate the average profit rate, which led to an incorrect result.

 

Figure 17: An example of a contextual misunderstanding made by a model when solving a API utilize task.The model failed to correctly understand the relationship between the original question and the follow-up question. In its response, it unnecessarily modified the API, invoked the SetAlert API, and set up a monitoring task.

 

Figure 18: The model failed to grasp the key points in the open-ended instructions provided in the question. The solution offered by the model lacked focus on the number of attendees and the meeting topic, instead outputting overly broad and less relevant content such as scheduling and risk management, which had limited practical value.

 



\begin{table}
\begin{tabular}{l l c r}
**Component** & **Category** & **Subject** & **\#Questions** \\ \hline Financial Academic Knowledge & Finance & Finance (\%) & 159 \\  & & Insurance (\%) & 113 \\  & & Investments (\%) & 145 \\  & & Central Banking (\%) & 119 \\  & & Financial Markers (\%) & 142 \\  & & Monetary Finance (\%) & 160 \\  & & Corporate Finance (\%) & 138 \\  & & International Finance (\%) & 88 \\  & & Financial Engineering (\%) & 105 \\  & & Commercial Bank Finance (\%) & 96 \\ Economy & Macroeconomics (\%) & 137 \\  & & Microeconomics (\%) & 136 \\  & & Econometrics (\%) & 83 \\  & & Statistics (\%) & 140 \\  & & Political Economy (\%) & 104 \\  & & International Economics (\%) & 135 \\  & & Public Finance (\%) & 139 \\ Accounting & Accounting (\%) & 120 \\  & Auditing (\%) & 137 \\  & Financial Management (\%) & 130 \\  & Cost Accounting (\%) & 148 \\  & Economic Law (\%) & 96 \\  & Tax law (\%) & 143 \\  & & Advanced Financial Accounting (\%) & 77 \\  & Intermediate Financial Accounting (\%) & 112 \\  & & Management Accounting (\%) & 83 \\  & & corporate Strategy and Risk Management (\%) & 134 \\ Certificate & China Actuary (\%) & 144 \\  & & Certified Practising Accountant (\%) & 140 \\  & & Certified Management Accountant (\%) & 124 \\  & Fund Qualification certificate (\%) & 252 \\  & & Futures Practitioner Qualification Certificate (\%) & 153 \\  & & Banking Practiver Qualification Certificate (\%) & 420 \\  & & Securities Practitioner Qualification Certificate (\%) & 109 \\  & All & 4661 \\ \hline Financial Industry Knowledge & Investment Research & Financial Sentiment Analysis (\%) & 205 \\  & & Financial Text Classification (\%) & 174 \\  & & Financial Text Summarization (\%) & 299 \\  & Investment Advisor & Financial Client Portrait (\%) & 120 \\  & & Marketing Script Recommendations (\%) & 150 \\  & Financial Investment Advice (\%) & 124 \\  & Financial Operations & Financial Event Extraction (\%) & 85 \\  & & Causal Event Extraction (\%) & 83 \\  & & Relationship Extraction (\%) & 103 \\  & & Negative Entity Extraction (\%) & 91 \\  & All & & 1434 \\ \hline Financial Security Knowledge & Software and Applications & Appside (\%) & 100 \\  & & Sftwrasafe (\%) & 100 \\  & & Memsire (\%) & 100 \\  & Network and System Protection & Netwaskate (\%) & 102 \\  & & Syssafe (\%) & 99 \\  & & Website (\%) & 306 \\  & Security Analysis & Crystase (\%) & 100 \\  & Vulnerability Protection & Malware (\%) & 101 \\  & & Penest (\%) & 432 \\  & & Reveng (\%) & 100 \\  & All & Vantho (\%) & 100 \\ \hline Finance Agent & Reasoning and Planning & FinRAG (\%) & 100 \\  & Long-term Memory & FinCor (\%) & 100 \\  & & FinTaskK (\%) & 100 \\  & & FinDog (\%) & 88 \\  & & FinDoc (\%) & 100 \\  & Tool Application & APIUtil (API\%) & 68 \\  & & APIFind (API\%) & 60 \\  & All & & 616 \\ \hline FinEval & All & & 8351 \\ \end{tabular}
\end{table}
Table 4: The FinEval dataset provides specific subdivisions in every category.





\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Model** & **Financial Academic** & **Financial Industry** & **Financial Security** & **Weighted Average** \\ \hline Claude 3.5-Sonnet & 74.4 & 59.6 & 78.1 & **72.4** \\ GPT-4o & 72.1 & **62.3** & **81.8** & 72.3 \\ Quenz-5.7ZB-Instract & **76.5** & 57.2 & 72.8 & 72.1 \\ Gemini-1.5-Prb & 68.2 & 59.7 & 77.4 & 68.6 \\ InternalMZ-5.20B-Chat & 67.3 & 57.5 & 73.1 & 66.7 \\ GPT-4o mini & 61.9 & 61.8 & 79.5 & 65.6 \\ GIAL+9B-Chat & 66.2 & 58.1 & 70.4 & 65.6 \\ Gemini-1.5-Flash & 61.2 & 57.7 & 77.1 & 63.9 \\ Yil1.5-34B-Chat & 65.0 & 54.4 & 63.0 & 62.6 \\ Openz-2.5-7Bsstruct & 69.7 & 52.9 & 50.4 & 62.5 \\ Yil1.5-9B-Chat & 61.9 & 57.4 & 65.7 & 61.9 \\ InterallMZ-20B-Chat & 60.3 & 54.3 & 63.3 & 59.8 \\ XuanYunaz-7.0B-Chat & 57.2 & 57.0 & 66.0 & 59.0 \\ XuanYunaz-7.0B-Chat & 57.2 & 54.0 & 63.4 & 57.9 \\ CFGFPT-2R & 57.7 & 48.3 & 63.3 & 57.2 \\ Background-13B-Chat & 48.1 & 53.6 & 57.6 & 51.1 \\ CatchGLMS-6B & 44.6 & 52.5 & 51.2 & 47.5 \\ DISC:FinkLM & 45.0 & 40.9 & 35.0 & 42.1 \\ FinGPTy3.1 & 29.3 & 39.5 & 32.6 & 31.9 \\ \hline \end{tabular}
\end{table}
Table 6: Average zero-shot CoT scores across three evaluated categories and one final weighted average: FAK (Financial Academic Knowledge), FIK (Financial Industry Knowledge), FSK (Financial Security Knowledge), and WA (Weighted Average)

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Model** & **Financial Academic** & **Financial Industry** & **Financial Security** & **Weighted Average** \\ \hline Claude 3.5-Sonnet & 74.4 & 59.6 & 78.1 & **72.4** \\ GPIT-4o & 72.1 & **62.3** & **81.8** & 72.3 \\ Qven2.5-7ZB-Instract & **76.5** & 57.2 & 72.8 & 72.1 \\ Gemini-1.5-Prb & 68.2 & 59.7 & 77.4 & 68.6 \\ InternalMZ-5.20B-Chaat & 67.3 & 57.5 & 73.1 & 66.7 \\ GPTT-4o mini & 61.9 & 61.8 & 79.5 & 65.6 \\ GIAL+9B-Chat & 66.2 & 58.1 & 70.4 & 65.6 \\ Gemini-1.5-Flash & 61.2 & 57.7 & 77.1 & 63.9 \\ Yi1.5-34B-Chat & 65.0 & 54.4 & 63.0 & 62.6 \\ Owen2.5-7BS-Insstruct & 69.7 & 52.9 & 50.4 & 62.5 \\ Yin1.5-9B-Chat & 61.9 & 57.4 & 65.7 & 61.9 \\ InternallMZ-20B-Chat & 60.3 & 54.3 & 63.3 & 59.8 \\ XuanYunaz-7.0B-Chat & 57.2 & 57.0 & 66.0 & 59.0 \\ XuanYunaz-7.0B-Chat & 57.2 & 54.0 & 63.4 & 57.9 \\ CFGFPT-2R & 57.7 & 48.3 & 63.3 & 57.2 \\ Blackburn-13B-Chat & 48.1 & 53.6 & 57.6 & 51.1 \\ ChatGLMS-6B & 44.6 & 52.5 & 51.2 & 47.5 \\ DISC:FinkLM & 45.0 & 40.9 & 35.0 & 42.1 \\ FineGTy3.1 & 29.3 & 39.5 & 32.6 & 31.9 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Average five-shot CoT scores across three evaluated categories and one final weighted average: FAK (Financial Academic Knowledge), FIK (Financial Industry Knowledge), FSK (Financial Security Knowledge), and WA (Weighted Average)

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Model** & **Financial Academic** & **Financial Industry** & **Financial Security** & **Weighted Average** \\ \hline Claude 3.5-Sonnet & **74.4** & **61.9** & 79.8 & **73.2** \\ GTPT-4o & 72.1 & 59.3 & **81.8** & 71.8 \\ Queen2.5-7ZB-Instract & 71.0 & 56.9 & 80.2 & 70.3 \\ Gemini1.5-Prb & 68.2 & 57.6 & 76.8 & 68.1 \\ GPTT-4o mini & 61.9 & 60.7 & 81.2 & 65.8 \\ Gemini1.5-Hash & 61.2 & 60.1 & 79.1 & 64.8 \\ Qwen2.5-7B-Instruct & 64.8 & 54.9 & 71.4 & 64.4 \\ Vi1.5-34B-Chat & 61.5 & 55.9 & 73.1 & 62.9 \\ IntervalMZ-5.20B-Chat & 60.7 & 47.0 & 75.1 & 61.2 \\ XuanYunaz-7.0B-Chat & 56.8 & 55.4 & 72.1 & 59.8 \\ InternallMZ-20B-Chat & 55.6 & 48.9 & 71.3 & 57.7 \\ XuanYunaz-7.0B-Chat & 53.5 & 55.8 & 70.1 & 57.4 \\ GLMA+9B-Chat & 54.5 & 56.4 & 59.6 & 55.9 \\ CFGFPT-2R & 53.6 & 41.2 & 68.0 & 54.4 \\ VII-5.9B-Chat & 56.8 & 43.8 & 46.1 & 52.1 \\ Baichuan2-13B-Chat & 42.7 & 54.7 & 48.1 & 46.1 \\ ChartGX/GAS-6B & 37.3 & 52.9 & 51.2 & 43.1 \\ DiscF-railLM & 34.7 & 46.7 & 26.2 & 35.1 \\ FinGTPV3.1 & 24.4 & 30.9 & 23.5 & 25.4 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Average five-shot scores across three evaluated categories and one final weighted average: FAK (Financial Academic Knowledge), FIX (Financial Industry Knowledge), FTSK (Financial Security Knowledge), and WA (Weighted Average)



\begin{table}
\begin{tabular}{l l c c c c c} \hline \hline
**Model** & **Size** & **Finance** & **Economy** & **Accounting** & **Certificate** & **Average** \\ \hline Claude 3.5-sonnet & unknown & **73.7** & **83.1** & **69.6** & **70.9** & **73.9** \\ GPT-4o & unknown & 72.6 & 78.8 & 66.3 & 69.8 & 71.5 \\ Qwen2.5-72B-Instruct & 72B & 65.8 & 71.5 & 72.4 & 69.7 & 69.7 \\ Gemini-1.5-pro & unknown & 68.5 & 75.1 & 61.1 & 71.4 & 68.3 \\ Qwen2.5-7B-Instruct & 7B & 58.2 & 63.8 & 64.5 & 65.6 & 62.7 \\ GPT-4o-mini & unknown & 65.2 & 67.7 & 55.2 & 63.5 & 62.4 \\ Gemini-1.5-flash & unknown & 60.0 & 67.7 & 60.4 & 61.9 & 62.1 \\ Yi1.5-34B-Chat & 34B & 54.7 & 65.2 & 54.6 & 63.5 & 59.5 \\ XuanYuan3-70B-Chat & 70B & 52.9 & 57.5 & 55.4 & 55.8 & 55.2 \\ Yi1.5-9B-Chat & 9B & 51.6 & 58.1 & 52.3 & 60.7 & 55.0 \\ InternLM2.5-20B-ChaT & 20B & 59.8 & 61.6 & 64.9 & 62.1 & 54.7 \\ InternLM2-20B-Chat & 20B & 52.9 & 55.8 & 54.6 & 55.3 & 54.7 \\ GLM4-9B-Chat & 9B & 54.1 & 53.0 & 56.0 & 55.3 & 54.7 \\ CFGPT2-7B & 7B & 51.6 & 55.8 & 56.0 & 52.2 & 53.9 \\ XuanYuan2-70B-Chat & 70B & 53.6 & 51.1 & 53.2 & 52.2 & 52.8 \\ Baichuan2-13B-Chat & 13B & 38.8 & 40.0 & 40.0 & 47.3 & 41.1 \\ DISC-FinLLM & 13B & 45.6 & 40.5 & 32.7 & 37.3 & 39.1 \\ ChatGLM3-6B & 6B & 42.8 & 36.5 & 35.9 & 40.0 & 38.9 \\ FinGTPv3.1 & 6B & 24.6 & 23.5 & 23.2 & 29.1 & 25.3 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Evaluation Results (zero-shot) for Finance Academic Knowledge(Average Accuracy(%))

\begin{table}
\begin{tabular}{l l c c c c c} \hline \hline
**Model** & **Size** & **Finance** & **Economy** & **Accounting** & **Certificate** & **Average** \\ \hline Claude 3.5-sonnet & unknown & **73.3** & **83.6** & **70.7** & **72.0** & **74.4** \\ GPT-4o & unknown & **73.3** & 79.3 & 66.6 & 70.9 & 72.1 \\ Qwen2.5-72B-Instruct & 72B & 68.5 & 71.9 & 71.6 & 72.8 & 71.0 \\ Gemini-1.5-pro & unknown & 69.3 & 74.1 & 61.1 & 70.9 & 68.2 \\ Qwen2.5-7B-Instruct & 7B & 59.9 & 64.4 & 66.6 & 67.2 & 61.9 \\ GPT-4o-mini & unknown & 64.4 & 66.6 & 52.6 & 61.6 & 61.9 \\ Yi1.5-34B-Chat & 34B & 59.9 & 62.2 & 61.4 & 59.8 & 61.2 \\ Gemini-1.5-flash & unknown & 61.5 & 66.7 & 58.1 & 59.8 & 61.2 \\ InternLM2.5-20B-Chat & 20B & 58.9 & 60.7 & 63.4 & 60.0 & 60.7 \\ Yi1.5-9B-Chat & 9B & 54.2 & 57.4 & 56.7 & 58.7 & 56.8 \\ XuanYuan3-70B-Chat & 70B & 53.5 & 57.9 & 56.4 & 56.7 & 56.8 \\ InternLM2-20B-Chat & 20B & 54.2 & 56.8 & 56.4 & 55.5 & 55.6 \\ GLM4-9B-Chat & 9B & 54.8 & 52.8 & 56.4 & 53.2 & 54.3 \\ CFGPT2-7B & 7B & 50.3 & 56.4 & 55.4 & 52.2 & 53.5 \\ Xuan Yuan2-70B-Chat & 70B & 54.4 & 51.4 & 52.9 & 55.0 & 53.5 \\ Baichuan2-13B-Chat & 13B & 40.8 & 45.9 & 40.8 & 46.4 & 42.7 \\ ChatGLM3-6B & 6B & 39.8 & 36.5 & 35.9 & 40.0 & 38.1 \\ DISC-Fin LLM & 13B & 33.5 & 38.7 & 31.5 & 36.4 & 35.0 \\ FinGPtv3.1 & 6B & 26.9 & 25.3 & 21.8 & 23.5 & 24.4 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Evaluation Results (five-shot) for Finance Academic Knowledge(Average Accuracy(%))



\begin{table}
\begin{tabular}{l l c c c c c} \hline \hline
**Model** & **Size** & **Finance** & **Economy** & **Accounting** & **Certificate** & **Average** \\ \hline Qwen2.5-72B-Instruct & 72B & **74.8** & 78.8 & **78.5** & **76.5** & **76.5** \\ Claude 3.5-sonnet & unknown & 73.3 & **83.6** & 70.7 & **76.5** & 74.4 \\ GPT-4o & unknown & 73.3 & 79.3 & 66.6 & 70.9 & 72.1 \\ Qwen2.5-7B-Instruct & 7B & 62.6 & 62.7 & 75.9 & 73.5 & 69.7 \\ Gemini-1.5-pro & unknown & 69.3 & 74.1 & 61.1 & 70.9 & 68.9 \\ InternLM2.5-20B-Chat & 20B & 67.0 & 69.3 & 68.5 & 64.0 & 67.3 \\ GLM4-9B-Chat & 9B & 63.0 & 68.3 & 66.7 & 68.3 & 66.6 \\ Yi1.5-34B-Chat & 34B & 61.3 & 63.4 & 63.4 & 66.7 & 65.0 \\ Yi1.5-9B-Chat & 9B & 59.6 & 62.1 & 63.6 & 65.6 & 62.7 \\ GPT-4o-mini & unknown & 64.4 & 66.6 & 52.6 & 66.5 & 61.9 \\ Gemini-1.5-flash & unknown & 61.5 & 66.7 & 58.1 & 59.8 & 61.2 \\ InternLM2-20B-Chat & 20B & 60.7 & 63.3 & 56.7 & 60.8 & 60.3 \\ CFGPT2-7B & 7B & 60.4 & 52.4 & 57.9 & 55.3 & 56.5 \\ XuanYuan3-70B-Chat & 70B & 61.2 & 63.8 & 49.5 & 54.6 & 57.2 \\ XuanYuan2-70B-Chat & 70B & 60.7 & 71.4 & 52.2 & 55.0 & 58.3 \\ Baichuan2-13B-Chat & 13B & 50.7 & 47.1 & 45.6 & 47.4 & 47.7 \\ DISC-FinLLM & 13B & 49.3 & 45.9 & 41.4 & 46.4 & 45.7 \\ ChatGLM3-6B & 6B & 46.3 & 43.4 & 45.6 & 41.8 & 44.3 \\ FinGPTv3.1 & 6B & 31.3 & 28.4 & 30.6 & 26.8 & 29.3 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Evaluation Results (zero-shot CoT) for Finance Academic Knowledge(Average Accuracy(%))

\begin{table}
\begin{tabular}{l l r r r r r} \hline \hline
**Model** & **Size** & **Finance** & **Economy** & **Accounting** & **Certificate** & **Average** \\ \hline Qwen2.5-72B-Instruct & 72B & **74.8** & 75.7 & 73.7 & **79.9** & **75.7** \\ Claude 3.5-sonnet & unknown & 74.4 & **80.0** & **74.9** & 69.2 & 73.7 \\ GPT-4o & unknown & 71.9 & 78.3 & 71.1 & 74.6 & 73.5 \\ Qwen2.5-7B-Instruct & 7B & 71.1 & 67.7 & 65.6 & 74.1 & 69.6 \\ Xuan Yuan3-70B-Chat & 70B & 65.6 & 67.7 & 65.2 & 64.4 & 65.7 \\ Yi1.5-34B-Chat & 34B & 64.2 & 65.5 & 64.3 & 62.1 & 64.0 \\ Xuan Yuan2-70B-Chat & 70B & 63.3 & 65.2 & 62.2 & 61.3 & 63.7 \\ GLM4-9B-Chat & 9B & 61.5 & 64.6 & 63.3 & 63.3 & 63.2 \\ CFGPT2-7B & 7B & 63.3 & 60.4 & 63.1 & 59.5 & 61.6 \\ Yin1.5-9B-Chat & 9B & 56.3 & 57.3 & 60.3 & 58.9 & 58.2 \\ InternLM2.5-20B-Chat & 20B & 58.1 & 57.1 & 61.4 & 59.6 & 59.0 \\ InternLM2-20B-Chat & 20B & 58.1 & 60.3 & 61.9 & 59.8 & 60.0 \\ Gemini-1.5-flash & unknown & 61.5 & 67.2 & 54.8 & 59.3 & 59.7 \\ Gemini-1.5-pro & unknown & 59.6 & 67.2 & 54.8 & 59.3 & 59.7 \\ GPT-4o-mini & unknown & 60.3 & 67.2 & 51.9 & 53.4 & 57.8 \\ Baichuan2-13B-Chat & 13B & 48.1 & 48.1 & 48.1 & 48.4 & 48.4 \\ ChatGLM3-6B & 6B & 43.1 & 49.2 & 38.9 & 45.5 & 44.2 \\ DISC-Fin LLM & 13B & 43.0 & 41.3 & 38.9 & 45.5 & 42.2 \\ FinGPTv3.1 & 6B & 26.1 & 31.1 & 21.9 & 27.4 & 26.6 \\ \hline \hline \end{tabular}
\end{table}
Table 11: Evaluation Results (five-shot CoT) for Finance Academic Knowledge(Average Accuracy(%))



\begin{table}
\begin{tabular}{l l c r r r r r r} \hline \hline
**Model** & **Size** & **FTC** & **FSA** & **RE** & **FEE** & **NEE** & **CEE** & **Average** \\ \hline GPT-4o-mini & unknown & 53.2 & 93.3 & 83.3 & 78.2 & 91.1 & **69.3** & **78.1** \\ Gemini-1.5-flash & unknown & 53.6 & 91.6 & **86.7** & **80.1** & 85.6 & 65.0 & 77.1 \\ GPT-4o-mini & unknown & 51.8 & 92.5 & 80.0 & 77.3 & 88.9 & 69.1 & 76.6 \\ Gemini-1.5-pro & unknown & 53.4 & 91.7 & 80.0 & 75.6 & 88.9 & 64.5 & 75.7 \\ Claude 3.5-sonnet & unknown & 55.4 & 86.7 & 80 & 76.3 & 87.8 & 65.1 & 75.2 \\ GLM4-9B-Chat & 9B & 37.1 & 87.8 & 76.7 & 34.5 & 86.7 & 65.6 & 64.7 \\ Qwen2.5-72B-Instruct & 72B & 37.9 & 86.7 & 80.0 & 32.3 & 84.4 & 65.6 & 64.5 \\ Baichuan2-13B-Chat & 13B & 23.9 & **94.5** & 70.0 & 55.2 & **95.6** & 42.8 & 63.7 \\ InternLM2.5-20B-Chat & 20B & 43.7 & 92.5 & 76.7 & 21.0 & 84.5 & 62.6 & 63.5 \\ XuanYuan3-70B-Chat & 70B & 39.3 & 90.0 & 76.7 & 34.9 & 95.6 & 40.7 & 62.9 \\ InterLM2-20B-Chat & 20B & 37.1 & 87.8 & 76.7 & 34.5 & 92.2 & 30.4 & 59.8 \\ CFGPT2-7B & 7B & **55.6** & 67.5 & 70.0 & 38.1 & 93.3 & 30.4 & 59.2 \\ Yi1.5-34B-Chat & 34B & 35.9 & 93.3 & 73.2 & 34.5 & 86.7 & 22.3 & 57.7 \\ ChatGLM3-6B & 6B & 35.9 & 67.5 & 76.7 & 31.4 & 86.7 & 42.8 & 56.8 \\ DISC-FinLLM & 13B & 18.1 & 90.0 & 66.7 & 59.1 & 70.0 & 36.6 & 56.7 \\ Qwen2.5-7B-Instrct & 7B & 37.1 & 87.8 & 76.7 & 31.4 & 90.0 & 15.0 & 56.3 \\ XuanYuan2-70B-Chat & 70B & 33.6 & 66.6 & 83.3 & 31.2 & 78.9 & 25.4 & 53.2 \\ Yi1.5-9B-Chat & 9B & 34.6 & 56.7 & 76.7 & 27.2 & 92.2 & 20.2 & 51.3 \\ FinGPTV3.1 & 6B & 24.6 & 56.6 & 63.2 & 21.2 & 74.4 & 10.2 & 41.7 \\ \hline \hline \end{tabular}
\end{table}
Table 13: Evaluation Results (five-shot) for Finance Industry Knowledge (Average Similarity(%)). These are objective short-answer questions including FTC: Financial Text Classification, FSA: Financial Sentiment Analysis, RE: Relation Extraction, FEE: Financial Event Extraction, NEE: Negative Entity Extraction, CEE: Causal Event Extraction

\begin{table}
\begin{tabular}{l l l l l l l l l} \hline \hline
**Model** & **Size** & **FTC** & **FSA** & **RE** & **FEE** & **NEE** & **CEE** & **Average** \\ \hline GPT-4o & unknown & 53.2 & 93.3 & 83.3 & 78.2 & 91.1 & **69.3** & **78.1** \\ Gemini-1.5-flash & unknown & 53.6 & 91.6 & **86.7** & **80.1** & 85.6 & 65.0 & 77.1 \\ GPT-4o-mini & unknown & 51.8 & 92.5 & 80.0 & 77.3 & 88.9 & 69.1 & 76.6 \\ Gemini-1.5-pro & unknown & 53.4 & 91.7 & 80.0 & 75.6 & 88.9 & 64.5 & 75.7 \\ Claude 3.5-sonnet & unknown & 55.4 & 86.7 & 80 & 76.3 & 87.8 & 65.1 & 75.2 \\ GLM4-9B-Chat & 9B & 37.1 & 87.8 & 76.7 & 34.5 & 86.7 & 65.6 & 64.7 \\ Quren2.5-72B-Instruct & 72B & 37.9 & 86.7 & 80.0 & 32.3 & 84.4 & 65.6 & 64.5 \\ Baichuan2-13B-Chat & 13B & 23.9 & **94.5** & 70.0 & 55.2 & **95.6** & 42.8 & 63.7 \\ InternLM2.5-20B-Chat & 20B & 43.7 & 92.5 & 76.7 & 21.0 & 84.5 & 62.6 & 63.5 \\ XuanYuan3-70B-Chat & 70B & 39.3 & 90.0 & 76.7 & 34.9 & 95.6 & 40.7 & 62.9 \\ InnerLM2-20B-Chat & 20B & 37.1 & 87.8 & 76.7 & 34.5 & 92.2 & 30.4 & 59.8 \\ CFGPT2-7B & 7B & **55.6** & 67.5 & 70.0 & 38.1 & 93.3 & 30.4 & 59.2 \\ Yi1.5-34B-Chat & 34B & 35.9 & 93.3 & 73.2 & 34.5 & 86.7 & 22.3 & 57.7 \\ ChatGLM3-6B & 6B & 35.9 & 67.5 & 76.7 & 31.4 & 86.7 & 42.8 & 56.8 \\ DISC-FinLLM & 13B & 18.1 & 90.0 & 66.7 & 59.1 & 70.0 & 36.6 & 56.7 \\ Qwen2.5-7B-Instruct & 7B & 37.1 & 87.8 & 76.7 & 31.4 & 90.0 & 15.0 & 56.3 \\ XuanYuan2-70B-Chat & 70B & 33.6 & 66.6 & 83.3 & 31.2 & 78.9 & 25.4 & 53.2 \\ Yi1.5-9B-Chat & 9B & 34.6 & 56.7 & 76.7 & 27.2 & 92.2 & 20.2 & 51.3 \\ FinGPTv3.1 & 6B & 24.6 & 56.6 & 63.2 & 21.2 & 74.4 & 10.2 & 41.7 \\ \hline \hline \end{tabular}
\end{table}
Table 12: Evaluation Results (zero-shot) for Finance Industry Knowledge(Average Similarity(%)). These are objective short-answer question including FTC: Financial Text Classification, FSA: Financial Sentiment Analysis, RE: Relation Extraction, FEE: Financial Event Extraction, NEE: Negative Entity Extraction, CEE: Causal Event Extraction



\begin{table}
\begin{tabular}{l l c r r r r} \hline \hline
**Model** & **Size** & **FTS** & **FCP** & **MSR** & **IA** & **Average** \\ \hline Queen2.5-72B-Instruct & 72B & 31.0 & 80.0 & 22.2 & **24.0** & **39.3** \\ Claude 3.5-sonnet & unknown & 25.9 & **83.3** & 22.2 & 23.9 & 38.8 \\ Gemini-1.5-pro & unknown & 29.5 & 76.7 & 22.0 & 23.0 & 37.8 \\ GPT-4o-mini & unknown & 28.6 & 76.7 & 22.0 & 23.7 & 37.8 \\ InternLM2.5-2OB-Chat & 20B & 27.9 & 76.7 & 22.1 & 23.9 & 37.7 \\ Yi1.5-34B-Chat & 34B & 28.8 & 76.7 & **22.7** & 21.5 & 37.4 \\ Gemini-1.5-flash & unknown & 30.1 & 73.3 & 22.2 & 23.4 & 37.3 \\ XuanYuan2-70B-Chat & 70B & 28.5 & 73.3 & 22.2 & 23.3 & 36.8 \\ CFGPT2-7B & 7B & **34.8** & 66.7 & 22.0 & 23.6 & 36.8 \\ Qwen2.5-7B-Instruct & 7B & 26.4 & 73.3 & 22.1 & 23.9 & 36.4 \\ ChatGLM3-6B & 6B & 29.0 & 71.3 & 22.1 & 23.3 & 36.4 \\ InternLM2-20B-Chat & 20B & 28.8 & 70.0 & 22.1 & 23.5 & 36.1 \\ GPT-4o & unknown & 28.4 & 70.0 & 22.0 & 23.7 & 36.0 \\ GLM4-9B-Chat & 9B & 31.1 & 66.7 & 21.7 & 23.6 & 35.8 \\ XuanYuan3-70B-Chat & 70B & 27.1 & 70.0 & 22.1 & 23.5 & 35.7 \\ Yi1.5-9B-Chat & 9B & 32.6 & 60.0 & 22.6 & 23.4 & 34.7 \\ DISC-FinLLM & 13B & 24.5 & 50.0 & 22.4 & 23.9 & 30.2 \\ Baichuan2-13B-Chat & 13B & 27.2 & 46.7 & 22.0 & 23.8 & 29.9 \\ FinGPTv3.1 & 6B & 22.5 & 43.2 & 22.0 & 22.5 & 27.6 \\ \hline \hline \end{tabular}
\end{table}
Table 14: Evaluation Results (zero-shot) for Finance Industry Knowledge (Average Similarity(%)). These are subjective open-ended question including FTS: Financial Text Summarization, FCP: Financial Customer Portrait, MSR: Marketing Scripts Recommendation, IA: Investment Advice.

\begin{table}
\begin{tabular}{l l l l l l l} \hline \hline
**Model** & **Size** & **FTS** & **FCP** & **MSR** & **IA** & **Average** \\ \hline XuanYuan3-70B-Chat & 70B & 35.2 & 83.3 & 22.4 & 23.8 & **41.2** \\ Claude 3.5-sonnet & unknown & 35.0 & 83.3 & 22.2 & 23.7 & 41.1 \\ Qwen2.5-7B-Instruct & 7B & 26.0 & **86.7** & **22.5** & **24.0** & 39.8 \\ Quen2.5-72B-Instruct & 72B & 31.6 & 80.0 & **22.5** & **24.0** & 39.5 \\ XuanYuan2-70B-Chat & 70B & 31.6 & 80.0 & 22.4 & 23.8 & 39.5 \\ Yi1.5-34B-Chat & 34B & **36.3** & 73.3 & 22.4 & 23.8 & 39.0 \\ GLM4-9B-Chat & 9B & 35.0 & 73.3 & 22.4 & 23.8 & 38.6 \\ Cemini-1.5-pro & unknown & 29.5 & 76.7 & 22.0 & 23.0 & 37.8 \\ GPT-4o-mini & unknown & 28.6 & 76.7 & 22.0 & 23.7 & 37.8 \\ Gemini-1.5-flash & unknown & 30.1 & 73.3 & 22.2 & 23.4 & 37.3 \\ Baichuan2-13B-Chat & 13B & 35.9 & 66.7 & 22.2 & 23.5 & 37.1 \\ ChatGLM3-6B & 6B & 25.8 & 73.3 & 22.3 & 23.8 & 36.3 \\ GPT-4o & unknown & 28.4 & 70.0 & 22.0 & 23.7 & 36.0 \\ InternLM2.5-20B-Chat & 20B & 31.8 & 56.7 & 22.3 & **24.0** & 33.7 \\ InternLM2-20B-Chat & 20B & 31.6 & 56.7 & 22.4 & 23.8 & 33.6 \\ Yi1.5-9B-Chat & 9B & 11.1 & 76.7 & 22.4 & 23.8 & 33.5 \\ DISC-FinLLM & 13B & 33.3 & 46.7 & **22.5** & 23.8 & 31.6 \\ CFGPT2-7B & 7B & 30.5 & 36.7 & 22.3 & **24.0** & 28.4 \\ FinGPTv3.1 & 6B & 15.6 & 48.5 & 22.0 & 22.5 & 27.2 \\ \hline \hline \end{tabular}
\end{table}
Table 15: Evaluation Results (five-shot) for Finance Industry Knowledge (Average Similarity(%)). These are subjective open-ended question including FTS: Financial Text Summarization, FCP: Financial Customer Portrait, MSR: Marketing Scripts Recommendation, IA: Investment Advice.





\begin{table}
\begin{tabular}{l l c c c c c c} \hline \hline
**Model** & **Size** & **FTC** & **RE** & **FEE** & **NEE** & **CEE** & **Average** \\ \hline GPT-4o & unknown & **53.2** & **86.7** & 76.4 & 88.4 & **68.0** & **74.5** \\ GPT-4o-mini & unknown & 52.4 & 83.3 & 74.5 & 90.0 & 62.0 & 72.4 \\ Gemini-1.5-pro & unknown & 48.6 & 80.0 & 78.1 & 86.7 & 57.1 & 70.1 \\ Claude 3.5-sonnet & unknown & 51.7 & 80.0 & 72.2 & 90.0 & 54.8 & 69.7 \\ GLM4-9B-Chat & 9B & 42.0 & **86.7** & 58.6 & 96.7 & 57.9 & 68.4 \\ Geminii-1.5-flash & unknown & 52.3 & 83.3 & **78.3** & 81.7 & 45.7 & 68.3 \\ XuanYuan3-70B-Chat & 7OB & 31.7 & 80.0 & 67.8 & 96.7 & 58.7 & 67.0 \\ Yi1.5-9B-Chat & 9B & 40.4 & 66.7 & 71.5 & **100.0** & 55.3 & 66.8 \\ Baichuan2-13B-Chat & 13B & 22.2 & 80.0 & 74.5 & 96.7 & 59.2 & 66.5 \\ Yil1.5-34B-Chat & 34B & 37.0 & 73.3 & 66.4 & 96.7 & 54.6 & 65.6 \\ InternLM2.5-20B-Chat & 20B & 45.0 & 80.0 & 53.6 & 93.3 & 53.5 & 65.1 \\ Qwen2.5-72B-Instruct & 72B & 40.4 & 76.7 & 56.8 & 90.0 & 58.7 & 64.5 \\ XuanYuan2-70B-Chat & 7OB & 26.7 & 76.7 & 66.4 & 90.0 & 54.6 & 62.9 \\ InternLM2-20B-Chat & 20B & 37.0 & 73.3 & 58.6 & 90.0 & 54.6 & 62.7 \\ ChatGLM3-6B & 6B & 35.0 & 71.3 & 56.6 & 88.0 & 52.6 & 60.7 \\ Qwen2.5-7B-Instruct & 7B & 26.7 & 73.3 & 57.9 & 90.0 & 51.5 & 59.9 \\ CFGPT2-7B & 7B & 51.5 & 73.3 & 37.4 & 86.7 & 30.2 & 55.8 \\ FinGPIV3.1 & 6B & 14.7 & 59.3 & 41.6 & 76.0 & 39.5 & 46.2 \\ DISC-FinLLM & 13B & 26.7 & 63.3 & 37.4 & 76.7 & 19.9 & 44.8 \\ \hline \hline \end{tabular}
\end{table}
Table 16: Evaluation Results (zero-shot CoT) for Finance Industry Knowledge (Average Similarity(%)). These are objective short-answer questions including FTC: Financial Text Classification, FSA: Financial Sentiment Analysis, RE: Relation Extraction, FEE: Financial Event Extraction, NEE: Negative Entity Extraction, CEE: Causal Event Extraction.

\begin{table}
\begin{tabular}{l l r r r r r r} \hline \hline
**Model** & **Size** & **FTC** & **RE** & **FEE** & **NEE** & **CCE** & **Average** \\ \hline GLM4-9B-Chat & 9B & **61.2** & **83.3** & 60.1 & **96.7** & 65.4 & **73.3** \\ Qwen2.5-72B-Instruct & 72B & 46.3 & **83.3** & 80.1 & 86.7 & 62.9 & 71.9 \\ InternLM2.5-20B-Chat & 20B & 40.0 & 80.0 & 79.8 & 86.7 & **66.4** & 70.6 \\ Gemini-1.5-flash & unknown & 52.4 & 70.0 & 75.5 & 89.6 & 65.6 & 70.6 \\ GPT-4o-mini & unknown & 52.9 & 73.3 & 76.3 & 88.3 & 60.2 & 70.2 \\ GPT-4o & unknown & 56.7 & 66.7 & 79.0 & 87.4 & 60.9 & 70.1 \\ Yi1.5-9B-Chat & 9B & 50.0 & **83.3** & 76.8 & 73.3 & 62.6 & 69.2 \\ Claude 3.5-sonnet & unknown & 47.7 & 76.7 & 71.0 & 87.4 & 62.9 & 69.1 \\ Yi1.5-34B-Chat & 34B & 47.4 & 73.3 & **83.1** & 73.3 & 61.9 & 67.8 \\ XuanYuan3-70B-Chat & 7OB & 36.4 & **83.3** & 78.2 & 86.7 & 50.3 & 67.0 \\ InternLM2-20B-Chat & 20B & 36.4 & 80.0 & 76.8 & 80.0 & 58.7 & 66.4 \\ Qwen2.5-7B-Instruct & 7B & 28.9 & 80.0 & 73.2 & **96.7** & 51.3 & 66.0 \\ XuanYuan2-70B-Chat & 7OB & 36.4 & 76.7 & 78.2 & 73.3 & 58.7 & 64.7 \\ gemini-1.5-pro & unknown & 43.8 & 66.7 & 76.0 & 78.0 & 57.1 & 64.3 \\ ChatGLM3-6B & 6B & 33.4 & 77.0 & 73.8 & 77.0 & 55.7 & 63.4 \\ CFGPT2-7B & 7B & 36.4 & 70.0 & 38.9 & 80.0 & 38.7 & 52.8 \\ Baichuan2-13B-Chat & 13B & 22.2 & 66.7 & 26.3 & 73.3 & 44.8 & 46.7 \\ DISC-FinLLM & 13B & 28.9 & 70.0 & 42.3 & 67.3 & 17.3 & 45.2 \\ FinGPIV3.1 & 6B & 16.2 & 60.7 & 20.3 & 67.3 & 32.7 & 39.4 \\ \hline \hline \end{tabular}
\end{table}
Table 17: Evaluation Results (five-shot CoT) for Finance Industry Knowledge (Average Similarity(%)). These are objective short-answer questions including FTC: Financial Text Classification, FSA: Financial Sentiment Analysis, RE: Relation Extraction, FEE: Financial Event Extraction, NEE: Negative Entity Extraction, CEE: Causal Event Extraction.





\begin{table}
\begin{tabular}{l l c c c c} \hline \hline
**Model** & **Size** & **FTS** & **FCP** & **IA** & **Average** \\ \hline Claude 3.5-sonnet & unknown & 28.3 & **83.3** & 21.8 & **44.5** \\ Yi1.5-9B-Chat & 9B & **35.8** & 73.3 & 22.2 & 43.8 \\ Qwen2.5-72B-Instruct & 72B & 34.2 & 73.3 & 23.0 & 43.5 \\ InternLM2.5-20B-Chat & 20B & 29.8 & 76.7 & 22.8 & 43.1 \\ Geminii-1.5-pro & unknown & 24.4 & 80.0 & 22.9 & 42.4 \\ GPT-4o & unknown & 27.8 & 73.3 & 22.2 & 41.1 \\ XuanYuan3-70B-Chat & 70B & 25.6 & 73.3 & 23.0 & 40.6 \\ GLM4-9B-Chat & 9B & 35.1 & 63.3 & 22.7 & 40.4 \\ GPT-4o-mini & unknown & 27.8 & 70.0 & 22.8 & 40.2 \\ XuanYuan2-70B-Chat & 70B & 24.3 & 73.3 & 22.2 & 39.9 \\ Yil.5-34B-Chat & 34B & 32.2 & 63.3 & 22.6 & 39.4 \\ Geminii-1.5-flash & unknown & 28.3 & 66.7 & 22.7 & 39.2 \\ Qwen2.5-7B-Instruct & 7B & 27.8 & 63.3 & 22.6 & 37.9 \\ ChatGLM3-6B & 6B & 24.3 & 63.3 & 22.2 & 36.6 \\ CFGPT2-7B & 7B & 33.6 & 53.3 & 22.6 & 36.5 \\ Baichuan2-13B-Chat & 13B & 30.5 & 43.3 & 22.8 & 32.2 \\ DISC-FinLLM & 13B & 26.5 & 45.6 & **23.3** & 31.8 \\ FinGPTv3.1 & 6B & 22.5 & 35.9 & 22.1 & 26.8 \\ \hline \hline \end{tabular}
\end{table}
Table 19: Evaluation Results (five-shot CoT) for Finance Industry Knowledge (Average Similarity(%)). These are subjective open-ended question including FTS: Financial Text Summarization, FCP: Financial Customer Portrait, IA: Investment Advice.

\begin{table}
\begin{tabular}{l l c c c c} \hline \hline
**Model** & **Size** & **FTS** & **FCP** & **IA** & **Average** \\ \hline Qusen2.5-72B-Instruct & 72B & 29.0 & **83.3** & 22.8 & **45.0** \\ InternLM2.5-20B-Chat & 20B & 28.2 & **83.3** & 23.2 & 44.9 \\ GPT-4o-mini & unknown & 30.4 & 80.0 & 22.3 & 44.2 \\ Claude 3.5-sonnet & unknown & 29.4 & 76.7 & 22.2 & 42.8 \\ Geminii-1.5-pro & unknown & 31.7 & 73.3 & 22.2 & 42.4 \\ GPT-4o & unknown & 29.3 & 73.3 & 23.0 & 41.9 \\ Yi1.5-9B-Chat & 9B & 29.0 & 73.3 & 22.5 & 41.6 \\ Qwen2.5-7B-Instruct & 7B & 24.3 & 76.7 & 23.0 & 41.3 \\ GLM4-9B-Chat & 9B & 30.7 & 70.0 & 22.2 & 41.0 \\ InternLM2-20B-Chat & 20B & 28.7 & 70.0 & 22.5 & 40.4 \\ XuanYuan3-70B-Chat & 70B & 28.2 & 70.0 & 22.4 & 40.2 \\ Geminii-1.5-flash & unknown & 31.4 & 66.7 & 21.8 & 40.0 \\ XuanYuan2-70B-Chat & 70B & 24.3 & 70.0 & 23.0 & 39.1 \\ ChatGLM3-6B & 6B & 24.2 & 70.0 & 22.1 & 38.8 \\ CFGPT2-7B & 7B & **34.7** & 50.0 & 23.1 & 35.9 \\ Yi1.5-34B-Chat & 34B & 28.7 & 56.7 & 22.1 & 35.8 \\ DISC-FinLLM & 13B & 26.9 & 53.3 & **23.3** & 34.5 \\ Baichuan2-13B-Chat & 13B & 20.1 & 53.3 & 23.2 & 32.2 \\ FinGPTv3.1 & 6B & 18.2 & 44.2 & 22.2 & 28.2 \\ \hline \hline \end{tabular}
\end{table}
Table 18: Evaluation Results (zero-shot CoT) for Finance Industry Knowledge (Average Similarity(%)). These are subjective open-ended question including FTS: Financial Text Summarization, FCP: Financial Customer Portrait, IA: Investment Advice.





\begin{table}
\begin{tabular}{l l c c c c c c c c c c c c} \hline \hline
**Model** & **Size** & **App** & **Cryp** & **MA** & **MS** & **NS** & **Pent** & **Rove** & **Soft** & **Syst** & **Vul** & **WS** & **Average** \\ \hline GPT-4o & unknown & 77.8 & 70.4 & 77.8 & **92.6** & 70.4 & **96.3** & 85.2 & 81.5 & 85.2 & **81.5** & 81.5 & **81.8** \\ Qwen2.5-72B-Instruct & 72B & 77.8 & **85.2** & **81.5** & 81.5 & 77.8 & 78.8 & 81.5 & 77.8 & **92.6** & **81.5** & 85.2 & 81.8 \\ GPT-4o-mini & unknown & 74.1 & 70.4 & 74.1 & 85.2 & 77.8 & 88.9 & 77.8 & 81.5 & 81.5 & 77.8 & 81.5 & 79.1 \\ Claude 3.5-sonnet & unknown & 70.4 & 66.7 & 70.4 & 81.5 & **81.5** & 85.2 & 85.2 & 81.5 & 70.4 & 85.2 & 78.1 \\ Gemini-1.5-pro & unknown & 81.5 & 55.6 & 77.8 & 81.5 & 70.4 & 88.9 & 77.8 & **85.2** & 85.2 & 70.4 & 81.5 & 77.8 \\ Gemini-1.5-flash & unknown & 77.8 & 59.3 & 77.8 & 85.2 & **81.5** & 88.9 & 85.2 & 70.4 & 81.5 & 70.4 & 74.1 & 77.5 \\ Yi1.5-34B-Chat & 34B & 70.4 & 77.8 & 74.1 & 74.1 & 70.4 & 78.8 & 78.8 & 74.1 & 88.9 & 74.1 & 74.1 & 76.0 \\ XuanYuan3-7OB-Chat & 70B & 77.8 & 77.8 & 66.7 & 76.7 & 74.1 & 92.6 & 70.4 & 63.0 & 88.9 & 70.4 & 70.4 & 74.4 \\ IntemLM2.5-2OB-Chat & 20B & **85.2** & 77.8 & 55.6 & 77.8 & 59.3 & 74.1 & 74.1 & 63.0 & 81.5 & 77.8 & **88.9** & 74.1 \\ GLM4-99-ChaT & 9B & 66.7 & 77.8 & 55.6 & 70.4 & 74.1 & 77.8 & **88.9** & 66.7 & 81.5 & 66.7 & 77.8 & 73.1 \\ InterImLM2-2OB-Chat & 20B & 74.1 & 77.8 & 63.0 & 70.4 & 70.4 & 77.8 & 74.1 & 66.7 & 81.5 & 70.4 & 77.8 & 73.1 \\ Owen2.5-7FB-Instruct & 7B & 74.1 & 81.5 & 63.0 & 74.1 & 63.0 & 74.1 & 66.7 & 70.4 & 74.1 & 66.7 & 81.5 & 71.7 \\ Yi1.5-9B-Chat & 9B & 63.0 & 77.8 & 70.4 & 70.4 & 74.1 & 66.7 & 85.2 & 66.7 & 77.8 & 55.6 & 77.8 & 71.4 \\ XuanYuan2-7OB-Chat & 70B & 74.1 & 74.1 & 55.6 & 44.4 & 59.3 & 66.7 & 74.1 & 59.3 & 88.9 & 74.1 & 77.8 & 68.0 \\ CFGPT2-7B & 7B & 66.7 & 70.4 & 59.2 & 59.2 & 66.7 & 77.8 & 60.4 & 55.6 & 70.4 & 59.3 & 70.4 & 65.1 \\ Baichuan2-13B-Chat & 13B & 66.7 & 66.7 & 40.7 & 51.9 & 63.0 & 63.0 & 70.4 & 55.6 & 70.4 & 63.0 & 66.7 & 61.6 \\ ChatGLM3-6B & 6B & 44.4 & 55.6 & 40.7 & 29.6 & 55.6 & 51.9 & 59.3 & 33.3 & 59.3 & 40.7 & 59.3 & 48.2 \\ DISC-FinLLM & 13B & 25.9 & 22.2 & 18.5 & 11.1 & 18.5 & 25.9 & 37.0 & 37.0 & 29.6 & 29.6 & 22.2 & 25.2 \\ FinGTPv3.1 & 6B & 24.4 & 19.5 & 14.8 & 7.7 & 17.1 & 21.4 & 36.4 & 30.9 & 27.9 & 29.1 & 19.8 & 22.7 \\ \hline \end{tabular}
\end{table}
Table 21: Evaluation Results (five-shot) for Finance Security Knowledge (Average Accuracy(%)). App: Application security, Crypp: Cryptographic protection, MA: Malware analysis, MS: Memory security, NS: Network security, Pent: Pentest, Reeve: Reverse engineering, Soft: Software security, Syst: System security, Vul: Vulnerability detection, WS: Web security

\begin{table}
\begin{tabular}{l l l l l l l l l l l l l l} \hline \hline
**Model** & **Size** & **App** & **Cryp** & **MA** & **MS** & **NS** & **Pent** & **Rove** & **Soft** & **Syst** & **Vul** & **WS** & **Average** \\ \hline GPT-4o & unknown & 74.1 & 70.4 & 74.1 & **92.6** & 81.5 & **96.3** & **81.5** & 88.9 & 85.2 & 74.1 & 81.5 & **81.8** \\ GPT-4o-mini & unknown & 74.1 & 77.8 & 77.8 & 88.9 & 70.4 & 88.9 & **81.5** & **92.6** & 77.8 & **81.5** & 81.5 & 81.2 \\ Qwen2.5-72B-Instruct & 72B & 74.1 & **81.5** & **85.2** & 74.1 & 70.4 & 85.2 & **81.5** & 77.8 & **92.6** & 74.1 & 85.2 & 80.2 \\ Claude 3.5-sonnet & unknown & 66.7 & 70.4 & 70.4 & 88.9 & 77.8 & 92.6 & **81.5** & 85.2 & 81.5 & 74.1 & **88.9** & 79.8 \\ Gemini-1.5-flash & unknown & 77.8 & 55.6 & **85.2** & 88.9 & **85.2** & **96.3** & 74.1 & 77.8 & 81.5 & 74.1 & 74.1 & 79.1 \\ Gemini-1.5-pro & unknown & 81.5 & 63.0 & 74.1 & 81.5 & 66.7 & 85.2 & **81.5** & 88.9 & 81.5 & 66.7 & 74.1 & 76.8 \\ InternLM2.5-2OB-Chat & 20B & **85.2** & 74.1 & 70.4 & 70.4 & 74.1 & 70.4 & 70.1 & 74.1 & 81.5 & 77.8 & 77.8 & 75.1 \\ Yi1.5-34B-Chat & 34B & 59.3 & 77.8 & 70.4 & 74.1 & 74.1 & 81.5 & 77.8 & 70.4 & 81.5 & 63.0 & 74.1 & 73.1 \\ XuanYuan3-7OB-Chat & 70B & 74.1 & **81.5** & 59.3 & 59.3 & 74.1 & 85.2 & **81.5** & 55.6 & 88.9 & 59.3 & 74.1 & 72.1 \\ Open2.5-7FB-Instruct & 7B & 66.7 & 66.7 & 51.9 & 74.1 & 63.0 & 77.8 & 81.4 & 70.3 & 81.4 & 77.8 & 74.1 & 71.4 \\ IntermLM2-20B-Chat & 20B & 70.3 & 78.2 & 55.6 & 66.7 & 74.1 & 81.5 & 79.5 & 55.6 & 81.5 & 66.7 & 74.1 & 71.3 \\ XuanYuan2-7OB-Chat & 70B & 65.2 & 78.2 & 55.5 & 57.6 & 74.1 & 82.5 & 79.5 & 54.6 & 84.2 & 65.2 & 74.1



\begin{table}
\begin{tabular}{l l c r r r r r r r r r r r} \hline \hline
**Model** & **Size** & **App** & **Cryp** & **MA** & **MS** & **NS** & **Pent** & **Reev** & **Soft** & **Syst** & **Vul** & **WS** & **Average** \\ \hline GPT-4o & unknown & 74.1 & 70.4 & 70.4 & **92.6** & **81.5** & **96.3** & 81.5 & **88.9** & 85.2 & 70.4 & 81.5 & **81.2** \\ Gemini-1.5-flash & unknown & 77.8 & 59.3 & **85.2** & 88.9 & **81.5** & 92.6 & 77.8 & 81.5 & 85.2 & 66.7 & 70.4 & 78.8 \\ GPT-4o-mini & unknown & 74.1 & 74.1 & 66.7 & 85.2 & 74.1 & 88.9 & 77.8 & **88.9** & 74.1 & **81.5** & 77.8 & 78.5 \\ Qwen2.5-72B-Instruct & 72B & 74.1 & 74.1 & 77.8 & 81.5 & 77.8 & 66.7 & **85.2** & 81.5 & **92.6** & 66.7 & **85.2** & 78.5 \\ Claude 3.5-sonnet & unknown & 66.7 & 59.3 & 63.0 & 85.2 & 74.1 & 92.6 & **85.2** & **88.9** & 81.5 & 74.1 & 81.5 & 77.4 \\ Gemini-1.5-pro & unknown & 66.7 & 59.3 & 74.1 & 85.2 & 70.4 & 92.6 & 77.8 & 81.5 & 81.5 & 66.7 & 77.8 & 75.8 \\ GLM4-9B-Chat & 9B & 63.0 & 63.0 & 66.7 & 63.0 & 59.3 & 74.1 & 81.5 & 63.0 & 77.8 & 63.0 & 70.4 & 67.7 \\ InterLMZ.5-20B-Chat & 20B & **85.2** & 59.3 & 66.7 & 70.4 & 55.6 & 51.9 & 51.9 & 59.3 & 70.4 & 63.0 & 74.1 & 64.3 \\ XuanYuan3-70B-Chat & 70B & 77.8 & 51.9 & 70.4 & 55.6 & 63.0 & 55.6 & 81.5 & 51.9 & 70.4 & 55.6 & 63.0 & 63.3 \\ Yi1.5-34B-Chat & 34B & 66.7 & 59.3 & 63.0 & 66.7 & 44.4 & 59.3 & 66.7 & 74.1 & 55.6 & 77.8 & 63.0 \\ Yuan Yuan2.5-7B-Chat & 70B & 76.2 & 53.4 & 54.9 & 52.5 & 64.2 & 53.2 & 81.5 & 64.1 & 62.1 & 53.2 & 63.0 & 61.7 \\ IntermLMZ-20B-Chat & 20B & 66.7 & 53.4 & 59.3 & 55.6 & 59.3 & 55.6 & 81.5 & 55.6 & 66.7 & 53.2 & 63.0 & 60.9 \\ Yi1.5-9B-Chat & 9B & 55.6 & **77.8** & 59.3 & 55.6 & 55.6 & 44.4 & 51.9 & 55.6 & 63.0 & 51.9 & 85.2 & 59.6 \\ CFGPT2-7B & 7B & 70.4 & 48.1 & 63.0 & 55.6 & 59.3 & 74.1 & 55.6 & 51.9 & 66.7 & 44.4 & 59.3 & 58.9 \\ Baichuan2-13B-Chat & 13B & 55.6 & 51.9 & 40.7 & 37.0 & 48.2 & 48.2 & 55.6 & 37.0 & 55.6 & 40.7 & 55.6 & 47.8 \\ ChatGLM3-6B & 6B & 40.7 & 55.6 & 51.9 & 37.0 & 48.1 & 55.6 & 44.4 & 40.7 & 55.6 & 40.7 & 51.8 & 47.5 \\ Qwen2.5-7B-Instruct & 7B & 40.7 & 48.2 & 37.0 & 48.2 & 22.2 & 63.0 & 48.2 & 37.0 & 33.3 & 48.1 & 58.1 & 44.0 \\ DISC-FinLLM & 13B & 33.3 & 25.9 & 29.6 & 29.6 & 29.6 & 44.4 & 25.9 & 29.6 & 44.4 & 18.5 & 29.6 & 30.9 \\ FinGPTy3.1 & 6B & 27.1 & 22.4 & 22.6 & 25.4 & 22.0 & 38.2 & 22.4 & 26.9 & 43.1 & 18.4 & 26.9 & 26.9 \\ \hline \end{tabular}
\end{table}
Table 22: Evaluation Results (zero-shot CoT) for Finance Security Knowledge (Average Accuracy(%)). App: Application security, Cryp: Cryptographic protection, MA: Malware analysis, MS: Memory security, NS: Network security, Pent: Pentest, Reve: Reverse engineering, Soft: Software security, Syst: System security, Vul: Vulnerability detection, WS: Web security

\begin{table}
\begin{tabular}{l l l l l l l l l l l l l} \hline \hline
**Model** & **Size** & **App** & **Cryp** & **MA** & **MS** & **NS** & **Pent** & **Reev** & **Soft** & **Syst** & **Val** & **WS** & **Average** \\ \hline GPT-4o & unknown & 77.8 & 70.4 & **77.8** & **92.6** & 70.4 & **96.3** & **85.2** & 81.5 & 85.2 & **81.5** & 81.5 & **81.8** \\ GPT-4o-mini & unknown & 77.8 & 74.1 & 74.1 & 88.9 & 74.1 & 88.9 & 77.8 & **85.2** & 74.1 & 77.8 & 81.5 & 79.5 \\ Claude 3.5-sonnet & unknown & 59.3 & 77.8 & 66.7 & 85.2 & 77.8 & 88.9 & 81.5 & 81.5 & 81.5 & 77.8 & 81.5 & 78.1 \\ Gemini-1.5-pro & unknown & **81.5** & 55.6 & 77.8 & 81.5 & 70.4 & 88.9 & 77.8 & 85.2 & 85.2 & 70.4 & 77.8 & 77.4 \\ Gemini-1.5-flash & unknown & 81.5 & 55.6 & 70.4 & 85.2 & **85.2** & 88.9 & 81.5 & 70.4 & 85.2 & 74.1 & 70.4 & 77.1 \\ InternLMM2.5-20B-Chat & 20B & 74.1 & **81.5** & 55.6 & 81.5 & 70.4 & 74.1 & 77.8 & 55.6 & 81.5 & 74.1 & 77.8 & 73.1 \\ Qwen2.5-72B-Instruct & 72B & 63.0 & 77.8 & 66.7 & 63.0 & 74.1 & 70.4 & 77.8 & 63.0 & **92.6** & 66.7 & 85.2 & 72.8 \\ GLM4-9B-Chat & 9B & 74.1 & 63.0 & 63.0 & 66.7 & 59.4 & 74.1 & 77.8 & 63.0 & 70.4 & 74.1 & **88.9** & 70.4 \\ XuanYuan3-70B-Chat & 70B & 77.8 & 59.3 & 63.0 & 51.9 & 66.7 & 59.3 & 81.5 & 51.9 & 81.5 & 59.3 & 74.1 & 66.0 \\ Xi1.5-9B-Chat & 9B & 59.3 & 77.8 & 70.4 & 74.1 & 55.6 & 51.9 & 66.7 & 59.3 & 74.1 & 63.0 & 70.4 & 65.7 \\ XuanYuan2.70B-Chat & 70B & 76.2 & 59.3 & 50.2 & 47.6 & 65.2 & 59.3 & 81.5 & 64.1 & 69.3 & 50.2 & 74.1 & 63.4 \\ InterLMM2-20B-Chat & 20B & 66.7 & 59.3 & 63.0 & 51.9 & 59.4 & 59.3 & 77.8 & 59.3 & 70.4 & 59.3 & 70.4 & 63.3



\begin{table}
\begin{tabular}{l l c r r r r r r r} \hline \hline
**Model** & **Size** & **COT** & **RAG** & **FT** & **MC** & **MD** & **API-I** & **API-R** & **Average** \\ \hline Claude 3.5-sonnet & unknown & **69.6** & **91.8** & **80.0** & **71.2** & **74.4** & **84.6** & **83.5** & **79.3** \\ GPT-4o & unknown & 68.8 & 83.4 & **80.0** & 63.0 & 69.8 & 76.5 & 75.6 & 73.9 \\ GPT-4o-mini & unknown & 63.0 & 82.6 & **80.0** & 63.0 & 68.9 & 77.4 & 75.2 & 72.9 \\ Gemini-1.5-pro & unknown & 63.0 & 82.2 & **80.0** & 63.0 & 72.0 & 73.1 & 76.3 & 72.8 \\ Gemini-1.5-flash & unknown & 50.4 & 85.2 & **80.0** & 62.2 & 71.1 & 72.2 & 75.2 & 70.9 \\ Qwen2.5-72B-Instruct & 72B & 62.2 & 91.1 & 74.3 & 31.1 & 69.8 & 75.7 & 74.4 & 68.4 \\ Qwen2.5-7B-Instruct & 7B & 44.4 & 86.7 & 77.8 & 63.0 & 66.3 & 59.6 & 68.9 & 66.7 \\ Yi1.5-34B-Chat & 34B & 50.2 & 69.3 & 79.3 & 65.2 & 66.2 & 61.2 & 70.4 & 66.0 \\ XuanYuan3-70B-Chat & 70B & 48.9 & 80.7 & 77.1 & 31.5 & 69.3 & 65.4 & 74.1 & 63.9 \\ InternLM2.5-20B-Chat & 20B & 52.6 & 85.9 & **80.0** & 31.5 & 66.7 & 56.9 & 68.1 & 63.1 \\ XuanYuan2-70B-Chat & 70B & 44.4 & 81.5 & 79.3 & 31.5 & 67.8 & 58.0 & 69.6 & 61.7 \\ Yi1.5-9B-Chat & 9B & 45.2 & 66.0 & 69.6 & 63.7 & 65.7 & 50.2 & 67.0 & 61.1 \\ InternLMM2-20B-Chat & 20B & 40.2 & 75.7 & 72.1 & 58.0 & 61.3 & 54.6 & 64.6 & 60.9 \\ GLM4-9B-Chat & 9B & 48.1 & 64.4 & 75.7 & 31.1 & 66.5 & 64.1 & 71.5 & 60.2 \\ Baichuan2-13B-Chat & 13B & 31.9 & 67.4 & **80.0** & 31.5 & 66.1 & 46.5 & 66.3 & 55.7 \\ CFGPT2-7B & 7B & 28.2 & 63.0 & 64.4 & 63.8 & 60.6 & 24.3 & 52.2 & 50.9 \\ ChatGLM3-6B & 6B & 25.2 & 71.1 & 70.0 & 31.5 & 59.1 & 33.3 & 57.2 & 49.6 \\ DISC-FinLLM & 13B & 20.7 & 61.5 & 62.9 & 31.1 & 43.1 & 21.3 & 52.0 & 41.8 \\ FinGPTv3.1 & 6B & 4.6 & 48.2 & 42.8 & 31.1 & 36.4 & 25.3 & 30.0 & 31.2 \\ \hline \hline \end{tabular}
\end{table}
Table 24: Evaluation Result (zero-shot) for Finance Agent(Similarity(%)). COT: Chain of Thought, RAG: Retrieval Augmented Generation, FT: Financial tasks, MC: Multi-turn conversation, MD: Multi-document question and answer, API-I: API invocation, API-R: API retrieval

\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & **1** & **2** & **3** & **4** & **5** & **6** & **7** \\ \hline
1 & **0.66** & -0.24 & -0.08 & 0.15 & 0.01 & -0.24 & 0.18 \\
2 & -0.07 & **0.84** & -0.07 & -0.19 & 0.15 & 0.24 & 0.04 \\
3 & -0.01 & -0.04 & **0.68** & 0.14 & 0.14 & 0.15 & 0.25 \\
4 & 0.30 & 0.02 & 0.08 & **0.85** & -0.27 & 0.12 & 0.07 \\
5 & -0.13 & 0.10 & 0.30 & -0.19 & **0.85** & 0.04 & 0.31 \\
6 & -0.07 & 0.09 & 0.00 & 0.02 & -0.70 & **0.63** & -0.32 \\
7 & -0.08 & 0.34 & 0.05 & 0.15 & 0.02 & 0.42 & **0.55** \\ \hline \hline \end{tabular}
\end{table}
Table 25: The Spearman correlation coefficient matrix results between human evaluation and GPT evaluation for Gemini 1.5-Pro in seven financial agent tasks, with 20% of all response results randomly sampled for testing in each task



\begin{table}
\begin{tabular}{l l c c c c c c c c} \hline \hline
**Model** & **Size** & **COT** & **RAG** & **FT** & **MC** & **MD** & **API-I** & **API-R** & **Average** \\ \hline Claude 3.5-sonnet & unknown & 77.0 & 85.2 & **81.5** & 72.6 & **76.7** & **89.6** & 72.2 & **79.3** \\ GPT-4o & unknown & **79.2** & 86.6 & 80.7 & 73.2 & 70.7 & 81.5 & 73.3 & 77.9 \\ GPT-4o-mini & unknown & 71.8 & **87.4** & 80.7 & **74.0** & 73.0 & 76.3 & **73.7** & 76.7 \\ Gemini-1.5-pro & unknown & 71.8 & 81.4 & 80.0 & 72.6 & **76.7** & 81.1 & **73.7** & 76.8 \\ Gemini-1.5-flash & unknown & 62.2 & 80.7 & 80.0 & 68.8 & 74.1 & 79.3 & 64.8 & 72.9 \\ \hline \hline \end{tabular}
\end{table}
Table 26: Agent Task Evaluation Result (zero-shot) Judged by Claude 3.5-Bonnet

\begin{table}
\begin{tabular}{l l c c c c c c c} \hline \hline
**Model** & **Size** & **SOT** & **RAG** & **FT** & **MC** & **MD** & **API-I** & **API-R** & **Average** \\ \hline Claude 3.5-sonnet & unknown & **73.3** & 64.4 & **80.0** & **92.6** & **86.7** & **92.6** & 87.4 & 82.4 \\ GPT-4o & unknown & **73.3** & **74.8** & 79.2 & 91.8 & 82.2 & 90.4 & **90.0** & 83.1 \\ GPT-4o-mini & unknown & 71.0 & **74.8** & **80.0** & **92.6** & 83.3 & 89.6 & 88.9 & 82.9 \\ Gemini-1.5-pro & unknown & **73.3** & 73.3 & **80.0** & **92.6** & 88.1 & 89.4 & 89.3 & **83.7** \\ Gemini-1.5-flash & unknown & 60.7 & 72.6 & **80.0** & 91.8 & 82.6 & 90.2 & 86.5 & 80.6 \\ \hline \hline \end{tabular}
\end{table}
Table 27: Agent Task Evaluation Result (zero-shot) Judged by Gemini-1.5-Pro

\begin{table}
\begin{tabular}{l l c c c c c} \hline \hline
**Model** & **Size** & **GPT-4o** & **Claude-3.5-Sonnet** & **Gemini-1.5-Pro** & **Average** \\ \hline Claude 3.5-sonnet & unknown & **79.3** & **79.3** & 82.4 & **80.3** \\ GPT-4o & unknown & 73.9 & 77.9 & 83.1 & 78.3 \\ GPT-4o-mini & unknown & 72.9 & 76.7 & 82.9 & 77.5 \\ Gemini-1.5-pro & unknown & 72.8 & 76.8 & **83.7** & 77.8 \\ Gemini-1.5-flash & unknown & 70.9 & 72.9 & 80.6 & 74.8 \\ \hline \hline \end{tabular}
\end{table}
Table 28: Agent Task Score Overall Comparision