Benchmarking Large Language Models on CFLUE - A Chinese Financial Language Understanding Evaluation Dataset (2024)

Abstract: In light of recent breakthroughs in large language models (LLMs) that have revolutionized natural language processing (NLP), there is an urgent need for new benchmarks to keep pace with the fast development of LLMs. In this paper, we propose CFLUE, the Chinese Financial Language Understanding Evaluation benchmark, designed to assess the capability of LLMs across various dimensions. Specifically, CFLUE provides datasets tailored for both knowledge assessment and application assessment. In knowledge assessment, it consists of 38K+ multiple-choice questions with associated solution explanations. These questions serve dual purposes: answer prediction and question reasoning. In application assessment, CFLUE features 16K+ test instances across distinct groups of NLP tasks such as text classification, machine translation, relation extraction, reading comprehension, and text generation. Upon CFLUE, we conduct a thorough evaluation of representative LLMs. The results reveal that only Qwen-72B, GPT-4, and GPT4-turbo achieve an accuracy exceeding 60% in answer prediction for knowledge assessment, suggesting that there is still substantial room for improvement in current LLMs. In application assessment, while GPT-4 and GPT-4turbo rank as the top two performers on average, their significant advantage over opensource LLMs is noticeably diminished, given that Qwen-72B achieves the best performance in 2 out of 5 tasks. The datasets and scripts associated with CFLUE are openly accessible at https://github.com/aliyun/cflue.

heading: 1 Introduction
text: Recently, the remarkable capabilities exhibited by large language models (LLMs) have brought significant advancements and revolutionized natural language processing (NLP). In response to the emergence of LLMs, new benchmarks have been intro-
*Corresponding Author
duced to assess a varied range of abilities exhibited by these models. Notable examples include MMLU (Hendrycks et al., 2021), HELM (Liang et al., 2023), BIG-bench (Srivastava et al., 2023), and GLUE (Wang et al., 2019b), which are widely employed to evaluate LLMs’ capabilities in English. Additionally, benchmarks such as CLEU (Xu et al., 2020), CMMLU (Li et al., 2023), and CEval (Huang et al., 2023) are widely used to evaluate their capabilities in Chinese.
However, assessing LLMs in the financial domain, especially within the Chinese context, presents considerable challenges due to limitations in existing Chinese financial evaluation datasets. Notably, datasets like FinanceIQ (Duxiaoman-DI, 2022) and FinEval (Zhang et al., 2023) have restrictions in terms of size and diversity, hindering a comprehensive evaluation of LLM performance. While FinanceIQ and FinEval focus solely on multiple-choice tasks, there is a lack of datasets allowing a thorough examination of LLM capabilities in text generation. Additionally, existing shared tasks, such as those in CCKS (Tianchi, 2019, 2020, 2021, 2022), predominantly concentrate on event extraction tasks, limiting the objective and quantitative measurement of LLM performance.
Inspired by the FLUE benchmark (Shah et al., 2022), which encompasses a comprehensive set of datasets across five financial domain tasks in English, this paper introduces a novel dataset named CFLUE (Chinese Financial Language Understanding Evaluation). CFLUE addresses the aforementioned challenges by providing a benchmark for evaluating LLM performance through various NLP tasks, categorized into knowledge assessment and application assessment. In the knowledge assessment section, CFLUE contains 38K+ questions from 15 types of financial qualification mock exams of varying difficulty levels and subjects. These questions are in the multiple-choice question format to enable standardized and objective evalua-
5673
tions. Additionally, each question is accompanied by a solution explanation, facilitating open-ended or chain-of-thought assessments of LLMs’ reasoning capabilities. In the application assessment section, CFLUE contains 16K+ test instances covering five groups of classical NLP tasks: text classification, machine translation, relation extraction, reading comprehension, and text generation. These instances are sourced from existing shared tasks or are annotated by professionals using real data sources.
Based on CFLUE, we assess the effectiveness of several representative LLMs across general and financial domains. This includes an examination of three OpenAI LLMs, nine open-source LLMs in the general domain, and three open-source LLMs in the financial domain. From the experimental results, we have the following key findings:
• The findings are not consistent in terms of tasks in knowledge assessment and application assessment. In knowledge assessment, GPT-4 and GPT4-turbo (OpenAI, 2023) demonstrate significant superiority over other LLMs, achieving over 60% accuracy in answer prediction. However, this suggests substantial room for improvement in current LLMs. In application assessment, although GPT-4 and GPT-4-turbo remain top performers on average, and in specific tasks, they may even fall behind certain open-source LLMs. This could be attributed to the specialized design of these open-source LLMs, which are tailored for Chinese data.
• Existing financial domain LLMs, such as FinGPT (Yang et al., 2023), DISC-FinLLM (Chen et al., 2023), and Tongyi-Finance (TongyiFinance, 2023), exhibit poor zero-shot performance in both knowledge and application assessment tasks, indicating limited coverage of financial knowledge and significant room for improvement.
• Open-source LLMs benefit significantly from supervised fine-tuning. For instance, ChatGLM36B (Zeng et al., 2022), Qwen-7B (Bai et al., 2023), and Baichuan2-7B (Baichuan, 2023) achieve comparable or superior performance to ChatGPT (OpenAI, 2022) in both answer prediction and reasoning tasks, despite having only 4% of ChatGPT’s parameters.
In summary, this paper provides valuable insights into the performance of LLMs in Chinese
financial contexts from multiple perspectives. Our findings suggest that there still exists much room for improvement even for the current best performers. We hope that CFLUE could guide the developers to understand the abilities of their models from multiple dimensions and facilitate the growth of foundation models in Chinese financial domain.

heading: 2 Related Work
text: Financial Evaluation Datasets. A summary of recent benchmarks in the financial domain is presented in Table 1. In English, FINQA by Chen et al. (2021) introduces a dataset comprising 8,281 question-answering pairs, emphasizing numerical reasoning processes. Both TAT-QA by Zhu et al. (2021) build a large-scale QA dataset containing both Tabular And Textual data from real financial reports. BizBench by Koncel-Kedziorski et al. (2023) consists of 8 quantitative reasoning tasks of question-answering (QA) for structured and unstructured financial data via program synthesis. FINANCEBENCH by Islam et al. (2023) consists of 10,231 questions about publicly traded companies, with corresponding answers and evidence strings. FLUE by Shah et al. (2022) utilizes datasets from existing literature to establish comprehensive benchmark across five tasks in financial domain, including sentiment analysis classification from Financial PhraseBank dataset (Malo et al., 2014) and sentiment analysis regression from FiQA 2018 Task-1 Maia et al. (2018),1 named entity recognition (NER) from NER data on loan agreement (Alvarado et al., 2015), question answering from FiQA 2018 Shared Task-2 Maia et al. (2018), news headline classification from gold commodity news (Sinha and Khandait, 2020), and structure boundary detection from FinSBD-3 Shared Task (FinSBD3, 2021).
In Chinese, the CCKS series since 2019 CCKS has released various datasets for event extraction tasks (Tianchi, 2019, 2020, 2021, 2022). Additionally, both FinanceIQ and FinEval offer thousands of multiple-choice question answering pairs that could be used as evaluation suites for LLMs (DuxiaomanDI, 2022; Zhang et al., 2023). However, compared to English, Chinese datasets tend to focus on either event extraction (Tianchi, 2022) or multiple-choice question-answering (Duxiaoman-DI, 2022; Zhang et al., 2023; Yang et al., 2023), demonstrating limited task diversity. In contrast, our paper introduces
1https://sites.google.com/view/fiqa
CFLUE (Chinese Financial Language Understanding Evaluation), featuring a set of heterogeneous benchmark tasks for a more comprehensive evaluation. Additionally, beyond evaluation suites, there are financial datasets like SmoothNLP2, IREE (Ren et al., 2022), suitable for training or fine-tuning models in the finance domain. Other Benchmark Datasets. The development of LMs (Devlin et al., 2019; Radford et al., 2019) has witnessed heterogeneous benchmarks to probe their diverse abilities. In English, traditional benchmarks focus on single tasks such as natural language understanding (Wang et al., 2019b,a), reading comprehension (Rajpurkar et al., 2018; Dua et al., 2019), and reasoning (Zellers et al., 2019; Sakaguchi et al., 2021). Recently, researchers have introduced broader benchmarks like TruthfulQA (Lin et al., 2022), evaluating truthfulness across 38 categories spanning various domains. MMLU (Hendrycks et al., 2021) offers a multitask evaluation covering 57 tasks across different fields. Additionally, BIG-bench (Srivastava et al., 2023) and HELM (Liang et al., 2023) benchmarks cover as much to 204 and 42 tasks, respectively.
In Chinese, CLEU (Xu et al., 2020) is the first large-scale Chinese benchmark that covers tasks such as classification and reading comprehension. AIGEval (Zhong et al., 2023) evaluates LLM capabilities in human-centric exams, including college entrance exams, law school admission tests, math competitions, and lawyer qualification tests.
2https://github.com/smoothnlp/ FinancialDatasets
Both CMMLU (Li et al., 2023) and CEval (Huang et al., 2023) are comprehensive Chinese benchmarks spanning multiple levels and diverse subjects. CMExam (Liu et al., 2023) focus on the Chinese medical domain, providing multiple-choice questions from the Chinese National Medical Licensing Examination.

heading: 3 CFLUE: Chinese Financial Language Understanding Evaluation
text: While various LLMs may exhibit similar performance on tasks within the general domain, it is often the domain-specific tasks that set them apart. The primary objective of CFLUE is to assess the financial domain knowledge capabilities of models from multiple dimensions. As illustrated in Figure 1, CFLUE evaluates the proficiency of LLMs from both knowledge and application perspective. The statistics of CFLUE are presented in Table 2.

heading: 3.1 Knowledge Assessment
text: This assessment consists of questions presented in a multiple-choice format, sourced from real-world, challenging mock exams accessible through public channels. The primary origins include 15 types of qualification exams covering diverse difficulty levels and subjects. The mock exams are exclusively sourced from PDF or Microsoft Word documents available on the Internet, avoiding direct extraction from plain text webpages. Using tools like pdfplumber3 and PaddleOCR4, we have collected a
3https://github.com/jsvine/pdfplumber 4https://github.com/PaddlePaddle/PaddleOCR
total of 166,681 multiple-choice questions. We then exclude questions that relied on nontext information, such as those containing HTML tags, images, tables, and questions with keywords 图/image and表/table. Additionally, we remove duplicate questions from the dataset. In an additional step to minimize the risk of data contamination, we employ GPT-4 to rephrase the questions and shuffle the multiple choices, we randomly shuffle the order of all options and use GPT-4 to rewrite the questions, enhancing the diversity of the dataset.
After preprocessing, we retain 38,636 questions. Among them, 60% of the questions have up to six options with only one correct answer, 10% are true/false questions, and the remaining questions may have multiple correct options. Each question in the dataset has a unique ID and is accompanied by detailed explanations provided by professionals to facilitate a thorough evaluation of the language
model’s logical reasoning abilities. Figure 2 in Appendix gives an example of annotated multichoice question.

heading: 3.2 Application Assessment
text: This assessment consists of five sets of real-word financial NLP tasks ( Section 3.2.1∼ 3.2.5). Then we discuss our strategy for quality control in Section 3.2.6.

heading: 3.2.1 Text Classification
text: It covers the following five subtasks.
• Dialogue intent classification. We randomly sample 500 instances from Banking dataset (Casanueva et al., 2020)5 which consists of 13,083 instances across 77 intends. The sampled 500 instances are then translated into Chinese by GPT-4, followed by manual verification.
5https://github.com/PolyAI-LDN/ task-specific-datasets
• ESG (environment, social and governance) classification and ESG sentiment classification. It contains 180 instances belonging to 14 classes for ESG classification and 3 classes for ESG sentiment classification. All instances are sourced from news portals such as证券时报/Securities Times, 每日经济新闻/Daily Economic News, and上海证券报/Shanghai Securities News, etc. Two senior doctoral students carry out the annotations following the CSMAR guideline available in https://data.csmar.com/.
• Financial Event Classification. We randomly select 1,000 instances from CCKS 2020,6 distributed across 27 classes, with each class having a similar number of instances.
• Financial Industry Classification. It contains 1,000 instances belonging to 68 industries. All the instances are news from EastMoney website7, each labeled with its corresponding industry.
• Financial Meeting Content Classification. It contains 452 instances distributed among three classes. The ASR toolkit8 is first utilized to convert audio files from brokerage firms’ meetings into text. Annotators then segment each meeting text and categorize segments as either财务 情况/Financial Situation,未来展望/Future Outlook, or经营情况/Operational Status. Segments not falling into these classes are excluded.

heading: 3.2.2 Machine Translation
text: We gather Chinese-English bilingual economic news reports from https://www.kekenet.com/ course/13650/. A total of 1,500 sentence pairs, each containing more than 30 words on each side, are selected. These sentence pairs serve as the dataset for both Chinese-to-English and English-toChinese translation.

heading: 3.2.3 Relation Extraction
text: It covers the following four subtasks.
• Feature extraction in Sentiment Analysis. We collect 500 stock comments from a financial company, and annotators mark the subject of the comments along with their sentiment, covering positive, neutral, and negative.
6https://tianchi.aliyun.com/dataset/111209 7https://www.eastmoney.com/ 8https://github.com/alibaba-damo-academy/
FunASR
• Financial Event Causality Extraction. We randomly select 1,000 instances from CCKS 2021 task6-2.9
• Financial Event Entity Extraction. We randomly select 1,000 instances from CCKS 2019.10
• Financial Event Extraction. We randomly select 1,000 instances from DuEE-fin (Han et al., 2022).11

heading: 3.2.4 Reading Comprehension
text: Initially, we set up a document repository comprising approximately 90,000 items, which includes news articles, research reports, and annual reports. Subsequently, we generate a list of questions by drawing from the common queries encountered in real customer scenarios, involving input from over 20 financial analysts. For each question, we employ a standard extraction framework12 (utilizing es+m3e embedding13) to establish a RAG (Retrieval-Augmented Generation) process, resulting in the most relevant chunk referred to as the financial text. Following this, for each (financial text, question) pair, annotators evaluate and determine if the question can be answered based on the financial text. If it is feasible, analysts write down the answer based on the financial text. Conversely, if not, analysts mark the answer as根据提供的背 景信息，无法回答该问题/According to the provided information, I cannot answer that question. After removing similar questions, we retain 2,710 triples of (financial text, question, answer), with 245 questions (9%) that cannot be answered based on the financial text.

heading: 3.2.5 Text generation
text: It covers the following five subtasks.
• Dialogue Summarization. We randomly select 1000 instances from CSDS (Lin et al., 2021).14
CSDS offers both overall summaries and roleoriented summaries. We utilize the overall summary for dialogue summarization.
• Meeting Summarization. We employ the identical audio files utilized in Financial Meeting Content Classification. Subsequently, we invite three
9https://www.biendata.xyz/competition/ccks_ 2021_task6_2/
10https://www.biendata.xyz/competition/ccks_ 2019_4
11https://www.luge.ai/#/luge/dataDetail?id=7 12https://www.langchain.com/ 13https://huggingface.co/moka-ai/m3e-base 14https://github.com/xiaolinAndy/CSDS
annotators to annotate the summary for each meeting.
• News Headline Generation. We use the same news articles as those utilized in Financial Industry Classification. Note that each news article is associated with a corresponding headline.
• Research Report Headline Generation. We retrieve research reports from the EastMoney website. Likewise, each research report is associated with a corresponding headline.
• Term Interpretation. We extract financial terminologies and their explanations from ChinaValue website.15

heading: 3.2.6 Quality Control
text: The annotated datasets are sourced from our diverse projects, with 47 experienced annotators engaged for CFLUE. Ranging from senior doctoral students to finance professionals like treasury analysts and MBAs, they undergo training and access guidance documents. Additional senior analysts for each project oversee their work, offering feedback.
• For both ESG Classification and ESG Sentiment Classification, two senior doctoral students are recruited. Over several weeks, tasks were gradually assigned as annotators gained proficiency. Weekly, each annotator tackled around 20-25 articles, with five shared between them. Discrepancies in annotations were resolved through discussion, and remaining articles were revisited afterward.
• For both Financial Meeting Content Classification and Meeting Summarization, 20 analysts are recruited to segment meeting content and then categorize and summarize each segment. Using ASR output, annotators initially segment the text based on the topic.16 Only segments with well-defined boundaries are chosen for classification and summarization. Before annotation, comprehensive training is provided, and discussions involving 10 ASR texts ensure consensus. The project spans several weeks due to the timeconsuming nature of both text segmentation and summarization. 15http://www.chinavalue.net/Wiki 16The segmentation is performed using the automatic segmentation result from the toolkit https://www.modelscope.cn/models/iic/nlp_bert_ document-segmentation_chinese-base/summary
• For Feature extraction in Sentiment Analysis, five analysts label the subject of the comments and their corresponding sentiment. Before annotation, they undergo training and discuss 20 comments until a high level of agreement is achieved. From the total 1,115 instances collected, 500 are randomly chosen based on length.
• For reading comprehension, 20 analysts gather queries and mark answers from related financial text. Before annotation, they undergo training, and discuss 10 pairs of (financial text, question) until a high level of agreement is achieved. Triples are discarded if the financial texts are not relevant to their corresponding questions. The project spans three months, with each annotator averaging 300 triples. Out of 6,000 triples, 2,710 are retained by removing similar questions.

heading: 3.3 Data Statistics
text: Table 2 shows the statistics of all CFLUE tasks after preprocessing. In the knowledge assessment’s multiple-choice question answering, questions are randomly divided into training, validation, and test sets at a ratio of 8:1:1, with the validation set designated for hyperparameter tuning.17 As for tasks in the application assessment, they exclusively comprise test questions, ranging from 2.7K to 4.0K. For detailed statistics on these datasets, please refer to Table 6 and Figure 3 in Appendix A.

heading: 4 Experimentation

heading: 4.1 Models, Settings, Prompts, and Metrics
text: Models. For a comprehensive assessment of the state of LLMs in the Chinese language context, we categorize the benchmarked LLMs into two groups: LLMs of the general domain and LLMs of the financial domain, as shown in Table 3.
• LLMs of the general domain: This group consists of models trained on extensive volume of general-purpose corpora. Included in this group are three OpenAI LLMs –GPT-4-turbo, GPT-4 (OpenAI, 2023), and ChatGPT (GPT-3.5turbo) (OpenAI, 2022) –along with five families of nine open-source LLMs, LLaMA2-7B/17We acknowledge that, in the context of few-shot and chain-of-thought evaluation, the questions in the knowledge assessment could be partitioned into dev, valid, and test sets, such as at a ratio of 1:1:8. However, in this paper, we opt not to adhere to this division due to budget constraints on OpenAI API calls.
70B (Touvron et al., 2023), Vicuna v1.5-7B (Chiang et al., 2023), ChatGLM3-6B (Zeng et al., 2022), Qwen-7B/-14B/-72B (Bai et al., 2023), and Baichuan2-7B/-13B (Baichuan, 2023).
• LLMs in the financial domain: This group comprises several representative LLMs specifically tailored for the financial domain. FinGPT6B (Yang et al., 2023) (fingpt-mt_chatglm26b_lora) is initialized with ChatGLM2-6B parameters and fine-tuned using LoRA (Hu et al., 2022) on datasets covering multiple tasks, including financial sentiment analysis, relation extraction, headline classification, and named entity recognition. DISC-FinLLM-13B (Chen et al., 2023) is based upon Baichuan-Chat-13B, with all parameters fine-tuned on Chinese financial corpora. Similarly, Tongyi-Finance-14B (TongyiFinance, 2023) is based on Qwen-14B and fine-tuned on Chinese financial corpora as well.
Among the aforementioned LLMs, GPT-4 turbo, GPT-4, ChatGPT, LLaMA2, and Vicuna v1.5 are English-oriented, while the remaining models are tailored for Chinese.
Supervised Fine-Tuning Settings. For knowledge assessment, we conduct additional fine-tuning of the open-source models using the training dataset. Specifically, we employ LoRA (Hu et al., 2022) to fine-tune LLaMA2-7B, Vicuna v1.57B, ChatGLM3-6B, Qwen-7B, and Baichuan2-7B, with the rank set to 8, alpha to 32, and dropout to 0.1. During the fine-tuning, a single NVIDIA A100/80G GPU is utilized. The batch size is set to 16 with a gradient accumulation step of 8.
Prompts. Our evaluation of LLMs is conducted in a zero-shot setting.18 In the knowledge assess-
18This paper focuses on the zero-shot setting, given the well-defined nature of all tasks in our assessment. We leave the exploration of the few-shot setting in our future work.
ment, we follow the approach of Liu et al. (2023), wherein we perform answer prediction and reasoning simultaneously by instructing LLMs to generate both the answer and a solution explanation. Specifically, we use three similar prompts, one for each question type. In the application assessment, we use a single prompt for each subtask, maintaining consistency by employing similar prompts for subtasks within each group. We present a few prompt examples in Appendix C.
Metrics. In the knowledge assessment, we measure the performance of answer prediction using accuracy and weighed F1 score. Additionally, for the open-ended reasoning we report BLEU (Papineni et al., 2002) and ROUGE (Lin and Hovy, 2002) scores. In the application assessment, we report accuracy for text classification, BLEU and COMET (Rei et al., 2020) for machine translation, F1 score for relation extraction, ROUGE for reading comprehension and text generation.

heading: 4.2 Results of Knowledge Assessment
text: Table 4 compares the performance of general domain LLMs and financial domain LLMs in the contexts of answer prediction and reasoning.19 Regarding the answer prediction task, we have the following observations:
• GPT-4-turbo and GPT-4 exhibit superior performance compared to ChatGPT and other LLMs except Qwen-72B. These two models achieve a notable 60% accuracy and 0.60 in F1, highlighting the challenges posed by CFLUE. Consistent with observations in other Chinese benchmarks (Huang et al., 2023; Liu et al., 2023), GPT4 substantially outperform ChatGPT with a performance gap of 17% difference in accuracy. 19Table 7 and Table 8 in Appendix B offer a comprehensive breakdown of performance, detailing results for each subject and question type.
• Among the nine open-source general domain LLMs with sizes ranging from 6B to 72B parameters, Qwen-72B attains the best performance, followed by Qwen-14B and Qwen-7B which even surpass ChatGPT. The performance gap between the three Qwen models also suggests that increasing LLM size significantly benefits downstream tasks. ChatGLM3-6B and Baichuan213B achieve similar performance, slightly underperforming ChatGPT. Beichuan2-7B achieves 36% accuracy and 0.35 in F1, followed by Vicuna v1.5-7B, LLaMA2-70B and LLaMA2-7B.
• Among the three LLMs designed for the financial domain, Tongyi-Finance-14B attains the highest performance. Despite being built upon Qwen14B, Tongyi Finance-14B unexpectedly lags behind with a 10% lower accuracy. In comparison to general domain LLMs, financial domain LLMs struggle to achieve satisfactory performance. This finding aligns with that of Liu et al. (2023). Similarly, financial domain LLMs may suffer from limited corpus diversity, hindering the acquisition of broad financial knowledge needed for CFLUE questions.
• Fine-tuning general domain LLMs with LoRA on the CFLUE dataset significantly boosts their performance. For example, the accuracy of Qwen7B increases to 49.84% from 45.70%. Even with only 4% of the parameters used in ChatGPT, Qwen-7B-CFLUE, Baichuan2-7B-CFLUE, and
ChatGLM3-6B-CFLUE all surpass ChatGPT.
In the reasoning task, the performance trend differs from the answer prediction task. Qwen-72B excels in BLEU and ROUGE metrics, outperforming other models. In contrast, ChatGPT surpasses GPT-4 and GPT-4-turbo, while ChatGLM3-6B and Qwen-7B match or surpass GPT-4. However, models like LLaMA2-7B, Baichuan2-7B, and DISCFinLLM-13B exhibit subpar performance, potentially due to generating concise explanations not aligning with predicted answers. A close examination of the generated explanations indicates that, in many instances, these explanations do not align with the predicted answers. Through fine-tuning, all models show an improvement in generating more coherent explanations, achieving ∼20 BLEU4 score and ∼40 ROUGE-L score.

heading: 4.3 Results of Application Assessment
text: Overall Performance. Table 5 provides an overview of the performance across the five application assessment tasks. GPT-4 emerges as the top performer, followed closely by GPT-4-turbo and ChatGPT. The performance gap among these three OpenAI LLMs is minimal. Among the opensource LLMs, Qwen-72B leads, followed by Qwen14B, Qwen-7B, Baichuan2-13B, ChatGLM3-6B, Baichuan2-7B, Vicuna v1.5-7B, LlaMA2-70B, and LlaMA2-7B. Similar to the knowledge assessment results, the three financial LLMs lag behind their
general LLMs with similar parameter sizes.
Results of Text Classification. LLMs display varied text classification performance, strongly linked to model size. For example, OpenAI LLMs outperform open-source models by a significant margin, with a performance gap exceeding 20% in accuracy. Additionally, a substantial improvement from 26.07% to 51.06% is achieved by merely increasing the parameter size from 7B to 72B for Qwen. The performance gap for 6/7B LLMs (except LLaMA2-7B) is comparatively small.
Results of Machine Translation. Firstly, Qwen7B/14B/72B performance is comparable to OpenAI LLMs in machine translation. Secondly, the performance trend for BLEU and COMET is not consistently aligned. For instance, Qwen-14B and Tongyi-Finance-14B achieve the higher BLEU scores in Chinese→English translation, but their COMET scores are significantly lower than those of OpenAI LLMs. DISC-FinLLM-13B attains a low BLEU score but a reasonable COMET score due to the translation involving numerous sourceside words.
Results of Relation Extraction. Similar to text classification, LLMs display varied performance, with OpenAI models significantly outperforming Qwen-72B by ∼ 4% in F1. Unlike text classification, there is no observed improvement by simply increasing the parameter size, as ChatGPT slightly underperforms GPT-4/GPT-4-turbo, while Qwen7B performs similarly to Qwen-14B.
Results of Reading Comprehension. With the exception of LLaMA2-7B/-70B and FinGPT-6B,
the performance gap among other LLMs is minimal, ranging from 42.56 to 47.43 in ROUGEL. In contrast to other tasks, ChatGPT slightly outperforms GPT-4/GPT-4-turbo, while Qwen-7B performs similarly to Qwen-14B/-72B. We report BERTScore (Zhang et al., 2020) in Table 10 of Appendix B.
Results of Text Generation. OpenAI LLMs do not exhibit superiority over open-source LLMs, especially Qwen-14B, which achieves the best performance. Similar to reading comprehension, the performance gap among all LLMs, except LLaMA27B/-70B, Baichuan2-7B, and FinGPT-6B, is relatively small (22.64 to 30.52). Similarly, we report BERTScore in Table 10 of Appendix B.
For more detailed results per subtask in text classification, relation extraction, and text generation, please refer to Table 11, 12, and 13 in Appendex B.

heading: 5 Conclusion
text: In summary, this paper have presented CFLUE, comprehensive Chinese datasets in the financial domain designed for evaluating the performance of LLMs across diverse NLP tasks. With over 38K multiple-choice questions and 16K instances across generation tasks, CFLUE offers a robust benchmark for assessing LLM capabilities. The detailed results presented for several LLMs on CFLUE provide valuable insights into their performance. We hope that CFLUE will facilitate future advancements in LLMs, particularly in enhancing their proficiency of Chinese NLP tasks within the financial domain.

heading: Acknowledgments
text: The authors would like to thank the anonymous reviewers for their constructive feedback.
Ethics
CFLUE is a composite dataset sourced from various origins. Questions and corresponding solution explanations in the knowledge assessment section are derived from online mock exams. Instances in the application assessment are a combination of data from existing shared task datasets and information from select financial firms, annotated by our professional annotators.20 We extend full credit to the original authors of each dataset incorporated into our CFLUE benchmark, and we have secured permissions for the inclusion of each dataset in CFLUE. The data obtained from our financial firm partners is also covered by our copyright. It is important to note that all datasets within CFLUE carry low ethical risks, with stringent measures in place to ensure the absence of any sensitive or personally identifiable information.
Limitations
Our work has several limitations. Firstly, we primarily use BLEU, COMET, ROUGE and BERTScore metrics to evaluate the performance of solution reasoning, machine translation, reading comprehension, and text generation. While informative, these metrics may not provide a comprehensive assessment of LLM outputs. Secondly, we employ identical or similar prompts for each group of subtasks in the application assessment, potentially hindering performance compared to prompts specifically tailored for each subtask. Lastly, our focus on the zero-shot setting when preparing prompts may limit LLMs from generating outputs as effectively as in a few-shot setting.

heading: A Detailed Statistics of CFLUE
text: Table 6 lists the subjects in the knowledge assessment, as well as the number of questions included in each subject. Figure 3 shows the subtasks in the application assessment, as well as the number of instances included in each subtask.

heading: B Detailed Performance of CFLUE
text: For knowledge assessment, Table 7 and Table 8 show the performance of answer prediction in accuracy per subject and per question type, respectively. Figure 4 shows the performance curves concerning input length. It shows that LLMs generally show higher accuracy with input lengths between 60 and 90. Interesting, their performance varies when the input is shorter with less then 30 tokens.
It is important to highlight that there has been extensive research on quantitative reasoning, particularly in the context of program synthesis (KoncelKedziorski et al., 2023; Islam et al., 2023). Our set of multiple-choice questions specifically contains quantitative reasoning questions. We utilize heuristic rules, followed by human validation, to identify quantitative reasoning questions within our test set. Consequently, we identify 334 quantitative reasoning questions, and their performance is detailed in Table 9. The results indicate that the performance on quantitative reasoning questions is significantly lower than the overall performance, suggesting the challenges that LLMs face in answering such questions.
For application assessment, Table 11, 12, and 13
show the performance per subtasks in text classification, relation extraction, and text generation, respectively.
Figure 5 compares the performance of models in application assessment tasks.

heading: C Prompt Examples
text: Figure 6 to 11 illustrate the evaluation prompts used in our experimentation.
Knowledge
TC
MT
RE
RC
TG
0 10 20 30 40 50 60 70
Model GPT-4-turbo GPT-4 ChatGPT LLaMA2-7B LLaMA2-70B Vicuna v1.5-7B ChatGLM3-6B Baichuan2-7B Baichuan2-13B Qwen-7B Qwen-14B Qwen-72B FinGPT V3-6B DISC-FinLLM-13B Tongyi-Finance-14B
Loading [MathJax]/extensions/MathMenu.jsFigure 5: Comparison among LLMs in subtasks of knowledge and application assessment.

heading: Future Work
text: We hope that CFLUE will facilitate future advancements in LLMs, particularly in enhancing their proficiency of Chinese NLP tasks within the financial domain