SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qianqian Xie
Wuhan, Hubei, China
xieq@whu.edu.cn
Weiguang Han
han.wei.guang@whu.edu.cn
Xiao Zhang
Sun Yat-Sen University
Shenzhen, Guangdong, China
zhangx767@mail2.sysu.edu.cn
Yanzhao Lai
Chengdu, Sichuan, China
laiyanzhao@swjtu.edu.cn
Min Peng
pengm@whu.edu.cn
Alejandro Lopez-Lira
alejandro.lopez-lira@warrington.ufl.edu
Jimin Huang
ChanceFocus AMC.
jimin@chancefocus.com
Although large language models (LLMs) have shown great performance in natural language processing (NLP) in the financial domain, there are no publicly available financially tailored LLMs, instruction tuning datasets, and evaluation benchmarks, which is critical for continually pushing forward the open-source development of financial artificial intelligence (AI). This paper introduces PIXIU, a comprehensive framework including the first financial LLM based on fine-tuning LLaMA with instruction data, the first instruction data with 128K data samples to support the fine-tuning, and an evaluation benchmark with 8 tasks and 15 datasets. We first construct the large-scale multi-task instruction data considering a variety of financial tasks, financial document types, and financial data modalities. We then propose a financial LLM called FinMA by fine-tuning LLaMA with the constructed dataset to be able to follow instructions for various financial tasks. To support the evaluation of financial LLMs, we propose a standardized benchmark that covers a set of critical financial tasks, including six financial NLP tasks and two financial prediction tasks. With this benchmark, we conduct a detailed analysis of FinMA and several existing LLMs, uncovering their strengths and weaknesses in handling critical financial tasks. The model, datasets, benchmark, and experimental results are open-sourced<sup>1</sup> to facilitate future research in financial AI.
<https://github.com/chancefocus/PIXIU
Financial technology (FinTech) has been continually advanced by the development of natural language processing (NLP) and machine learning (ML) techniques, unlocking diversity capabilities from predicting stock price movements to advanced financial analytics (Araci, 2019; Han et al., 2023; Xie et al., 2023; Lopez-Lira and Tang, 2023; Li et al., 2023). Specifically, the most recent large language models (LLMs) (Brown et al.,  $2020)^{2}$  have exhibited remarkable abilities in natural language understanding (NU) and performing various tasks by following natural language instructions without training data. Despite these successes, the highly technical nature of financial texts requires domain-specific LLMs to understand complex financial language and concepts effectively. Such efforts include existing financial pre-trained language models (PLMs) such as finBERT (Araci, 2019), FinBERT (Yang et al., 2020) and FLANG (Shah et al., 2022). However, those models are considered small since their parameter size is below one billion, limiting their generalization ability. Recently, a proprietary financial LLM called BloombergGPT (Wu et al., 2023) with 50 billion parameters has been proposed by pre-training a Bloom-style LLM (Scao et al., 2022) on large-scale financial data.
Despite these efforts Liu et al. (2023), there remain several issues, as shown in Table 1. Firstly, BloombergGPT and its training data are not openly released. Currently, there are no open-sourced financial LLMs, which can hinder development in the research community. Secondly, previous financial PLMs and the latest BloombergGPT are not fine-tuned for following natural language instructions (also known as instruction tuning), which is critical for improving the zero-shot ability on dealing with downstream financial tasks (Wei et al., 2021; Ouyang et al., 2022). Thirdly, there are also no financial instruction data for supporting the instruction tuning of LLMs and evaluation benchmarks for comprehensively assessing and comparing the abilities of LLMs for financial tasks. We are thus motivated to consider the following research questions: 1) how can we develop efficient and openly available LLMs tailored for finance? 2) how can we build large-scale and high-quality financial instruction data? 3) how can we build the holistic financial evaluation benchmark for assessing financial LLMs?
Table 1: The comparison of pre-trained language models and large language models for finance. "Instruct" means whether the model can follow instructions. "NLP" and "Fin" mean if the model is evaluated with financial NLP tasks and financial prediction tasks.
Model	Backbone	Size	Open Source	Instruct	Language	Evaluation	Release Date
Model	Data	NLP	Fin
finBERT (Araci, 2019)	BERT	110M	✓	✓	✗	English	✓	✗	08/27/19
FinBERT (Yang et al., 2020)	BERT	110M	✓	✗	✗	English	✓	✗	06/15/20
Mengzi-fin (Zhang et al., 2021)	RoBERTa	103M	✓	✗	✗	Chinese	✓	✗	10/13/21
FLANG (Shah et al., 2022)	ELECTRA	110M	✓	✓	✗	English	✓	✗	10/31/22
BBT-FinT5 (Lu et al., 2023)	T5	220M	✓	✓	✗	Chinese	✓	✗	02/18/23
BloombergGPT (Wu et al., 2023)	BLOOM	50B	✗	✗	✗	English	✓	✗	03/30/23
FinMA	LLaMA	7/30B	✓	✓	✓	English	✓	✓	06/01/23
To deal with these research questions, we propose PIXIU (貌)，a comprehensive framework that includes the first open sourced fine-tuned financial LLM, FinMA, which is based on fine-tuning LLaMA (Touvron et al., 2023) with multi-task and multi-modal instruction data. Fig 1 presents an overview of multi-task and multi-modal instruction tuning of FinMA for diverse financial tasks. PIXIU also contains the first instruction data with 128K data samples to support the fine-tuning and a holistic evaluation benchmark with six financial NLP tasks and two financial prediction tasks. It has the following distinguishing features:
- Open resources. We have openly released the financial LLM, instruction tuning data, and datasets included in the evaluation benchmark, and implementation, to encourage open research and transparency in the research field.
- Multi-task. PXIU includes multi-task instruction tuning data covering a diverse set of financial tasks, including six financial NLP tasks and two financial prediction tasks. The
$^{2}$ https://openai.com/blog/chatgpt
<sup>3</sup>PIXIU (貌貅)https://en.wikipedia.org/wiki/Pixiu is a mythical creature in Chinese folklore. It has the head of a dragon and the body of a lion and is believed to be an auspicious creature attracting money and good fortune.
multi-task instruction tuning has been proven to be critical for improving the model's generalization ability (Sanh et al., 2022; Longpre et al., 2023) to new tasks.
- Multi-modality. Our instruction tuning data consists of multi-modality financial data such as tables in financial reports and historical stock prices as time-series data for the stock-movement prediction tasks beyond texts. Moreover, they encompass diverse types of financial texts, including reports, news articles, tweets, and regulatory filings.
- Diversity. Compared with the evaluation tasks used in BloombergGPT and existing FLUE benchmark (Shah et al., 2022), which mainly cover financial NLP tasks, our evaluation benchmark includes financial prediction tasks such as stock movement prediction and credit scoring. It requires the model to fully exploit both natural texts and time-series data to extract essential information for accurate prediction. Compared with financial NLP tasks, the financial prediction task is more aligned with real-world scenarios and more challenging.
To build the multi-task and multi-modal instruction data, we collect open-released training data from diverse tasks, including financial sentiment analysis, news headline classification, named entity recognition (NER), question answering, text summarization, stock movement prediction, credit scoring and hawkish-doish classification, and propose the diverse task-specific instructions written by domain experts for each task. We create a large-scale financial instruction tuning data (FIT) by assembling the task-specific instructions with data samples from each task. We thus propose the domain-specific LLM FinMA by conducting the multi-task instruction tuning on LLaMA with the building dataset. To evaluate our model and other LLMs holistically, we build the Financial Language Understanding And PRediction Evaluation Benchmark (FLARE) covering 6 financial NLP tasks with 10 datasets, and 2 financial prediction task with 5 datasets.
Based on FLARE, we evaluate the performance of our model, BloombergGPT, and advanced LLMs in the general domain, such as ChatGPT $^4$  and GPT-4 (OpenAI, 2023). Experimental results show that: 1) FinMA significantly outperforms LLMs, including BloombergGPT, ChatGPT, and GPT-4 on most tasks in FLARE, including financial sentiment analysis, news headline classification, and stock movement prediction. This demonstrates the importance of tailoring the LLMs specifically for the financial domain. 2) Despite promising results on most tasks, FinMA underperforms BloombergGPT, ChatGPT, and GPT-4 on the question answering, which assesses the quantitative reasoning ability of LLMs. Our analysis finds that this is caused by the limitation of LLaMA on quantitative reasoning and mathematics. FinMA also shows limited performance on NER tasks although it outperforms BloombergGPT, which is also due to the drawbacks of LLaMA. 3) Compared with NLP tasks, all LLMs, including FinMA, ChatGPT and GPT-4, still present limited performance on stock movement prediction, indicating room for further improvement. 4) FinMA fine-tuned with both NLP and financial prediction tasks, presents the best performance on one of the stock prediction datasets, indicating the potential of task-specific instruction tuning of LLMs on financial prediction tasks.
Our contributions can be summarized as follows: 1) We introduce FIT, the first multi-task and multi-modal instruction tuning data in the financial domain, covering 5 tasks and 9 datasets with 128,640 (128K) data samples. 2) We introduce FLARE, the first evaluation benchmark with both financial natural language understanding and prediction tasks. 3) We introduce FinMA, the first openly released and instruction-following financial large language model, which achieves SOTA on 6 financial NLP tasks and 2 financial prediction tasks. 4) We compare FinMA and existing LLMs on FLARE. The results demonstrate the superiority of FinMA, the key limitations of LLMs for finance, and future directions to advance LLMs for finance.
Financial Language Models Many PLMs for the financial domain have been proposed by continual pre-training PLMs with large-scale financial texts. Araci (2019) proposed the first financial PLM called finBERT that pre-trained BERT (Devlin et al., 2019) with open released financial corpus such as TRC2-financial and Financial Phrase Bank (Malo et al., 2014). finBERT outperforms neural network methods such as LSTM in financial sentiment classification tasks. Yang et al. (2020) further proposed FinBERT by pre-training BERT with a 4.9 billion tokens financial communication
<sup>5</sup>https://trec.nist.gov/data/reuters/reuters.html
corpus, which outperforms BERT on three financial sentiment classification datasets. Shah et al. (2022) proposed FLANG, a financial PLM with BERT and ELECTRA (Clark et al., 2020) as the backbone. Besides English, financial PLMs in other languages, such as Chinese, were also proposed, such as Mengzi-fin (Zhang et al., 2021) and BBT-FinT5 (Lu et al., 2023). Latest, Wu et al. (2023) proposed BloombergGPT, the first financial large language model with 50 billion parameters, that is pre-trained with mixed datasets from the general and financial domain. However, neither the model nor pre-trained domain datasets are not released. The model is also not instruction-following like other LLMs such as ChatGPT and GPT-4.
Financial Evaluation Benchmark Shah et al. (2022) proposed the first heterogeneous evaluation benchmark FLUE with 5 financial NLP tasks, including financial sentiment analysis (Malo et al., 2014), news headline classification (Sinha and Khandait, 2021), named entity recognition (Alvarado et al., 2015), structure boundary detection  $^{6}$  and question answering (Maia et al., 2018). Lu et al. (2023) proposed the first Chinese financial evaluation benchmark BBT-CFLEB  $^{7}$  with financial news classification, summarization, relation extraction, question answering, and negative news determination task, as well as sentiment classification task of financial social media texts. However, these benchmarks only consider financial NLP tasks and don't include financial prediction tasks, such as stock movement prediction, that are critical for evaluating the model's performance applied to real-world scenarios.
Open Sourced Large Language Models Recent studies have made efforts on democratic AI, where the representative work is LLaMA (Touvron et al., 2023) from Meta AI, an open-source LLM with parameters ranging from 7B and 13B to 65B. LLaMA-13B has comparable and even better performance than GPT-3 (Brown et al., 2020) with 175B parameters on common sense reasoning tasks. Following efforts have been proposed to improve LLaMA for instruction following like ChatGPT, by instruction tuning. Such as Taori et al. (2023) proposed Alpaca by fine-tuning LLaMA-7B with 52K instruction-following samples generated with the self-instruct method (Wang et al., 2022). Chiang et al. (2023) proposed Vicuna-13B by fine-tuning LLaMA-13B with 70K conversation data from ShareGPT  $^{8}$ . It can generate better answers to user's questions compared with Alpaca. However, there are no open-sourced LLMs and instruction-tuning data focused on the financial domain.
3 FIT: Financial Instruction Tuning Dataset
In this section, we introduce our financial instruction tuning dataset FIT, including the background of raw data, tasks in FIT, and the construction process based on raw data. Different from existing financial datasets, FIT is the first instruction-tuning dataset for finance LLMs and includes financial prediction tasks except for financial NLP tasks, which is fundamental for real-world financial applications.
3.1 Raw Data
Derived from real-world finance scenarios, we build our financial instruction tuning dataset FIT based on the open-sourced data of various financial NLP and prediction tasks. Compared with the self-instruct method (Wang et al., 2022) commonly used by existing LLMs such as Alpaca, we choose to build instruction tuning datasets from open-sourced datasets due to the following reasons: 1) the open-sourced datasets are usually annotated by domain experts, showing high quality, 2) it has very low cost and has no limitation on commercial use unlike datasets constructed from ChatGPT or GPT-4, 3) these open-sourced datasets cover a variety of text types such as news, reports and tweets, as well as multi-modalities including time series data, tables, and texts. The details<sup>9</sup> of the raw data and instruction data are shown in Table 2.
Financial Sentiment Analysis. Financial sentiment analysis task has long been a critical task in the financial domain (Araci, 2019; Yang et al., 2020), aiming to analyze the sentiment information of the input financial texts. Following existing benchmark FLUE (Shah et al., 2022), we use two datasets: the Financial Phrase Bank (FPB) dataset (Malo et al., 2014) and FiQA-SA (Maia et al., 2018). FPB includes English sentences from financial news and their sentiment label of positive, negative, or
<sup>6</sup>https://sites.google.com/nlg.csie.ntu.edu.tw/binweb2021/shared-task-finsbd-3
<sup>7</sup>https://bbt.ssymmetry.com/evaluation.html
<sup>8</sup>https://sharegpt.com
For further details of the data split and pre-processing, please refer to Appendix.
Table 2: The details of the raw data and instruction data.
Data	Task	Raw	Instruction	Data Types	Modalities	License
FPB	sentiment analysis	4,845	48,450	news	text	CC BY-SA 3.0
FiQA-SA	sentiment analysis	1,173	11,730	news headlines,tweets	text	Public
Headlines	news headline classification	11,412	11,412	news headlines	text	CC BY-SA 3.0
FOMC	hawkish-doish classification	496	496	FOMC transcripts	text	CC BY-NC 4.0
NER	named entity recognition	609	6,090	financial agreements	text	CC BY-SA 3.0
FiNER-ORD	named entity recognition	1,080	1,080	news articles	text	CC BY-SA 3.0
FinQA	question answering	8,281	8,281	earnings reports	texttable	MIT License
ConvFinQA	question answering	3,458	3,458	earnings reports	texttable	MIT License
ECTSum	text summarization	495	495	earning call transcripts	text	Public
EDTSum	text summarization	2,000	2,000	news articles	text	Public
BigData22	stock movement prediction	7,168	7,168	tweets,historical prices	text,time series	Public
ACL18	stock movement prediction	27,080	27,080	tweets,historical prices	text,time series	MIT License
CIKM18	stock movement prediction	4,971	4,971	tweets,historical prices	text,time series	Public
German	credit scoring	1,000	1,000	credit records	table	CC BY 4.0
Australia	credit scoring	690	690	credit records	table	CC BY 4.0
neutral annotated by domain experts. FiQA-SA is another widely adopted dataset, which aims to predict the sentiment of English financial news and microblog posts on a scale of  $[-1,1]$ , where 1 means the most positive.
News Headline Classification. The news headline classification task aims to analyze other information, such as price movement in financial texts. We use the Gold news headline dataset (Sinha and Khandait, 2021) consisting of news headlines from 2000 to 2019 about "gold" and their corresponding 9 tags: "price or not", "price up", "price down", "price stable", "past price", "future price", "past general", "future general", "asset comparison". The task is to conduct the binary classification for each tag of each data sample.
Hawkish-dovish Classification. The Hawkish-Dovish classification aims to classify sentences from monetary policy texts into a 'hawkish' or 'dovish' stance, unlike standard sentiment analysis. The key to this task lies in understanding the nuanced language of financial texts and being able to identify the economic implications conveyed through these 'hawkish' or 'dovish' signals. We use the FOMC dataset Shah et al. (2023a), including sentences extracted from the Federal Open Market Committee (FOMC) meetings, where each sentence is manually annotated as either 'hawkish' or 'dovish.
Named Entity Recognition. Named Entity Recognition (NER) task is to detect critical financial entities such as persons, organizations, and locations, which can be used to build financial knowledge graphs. We use two datasets: NER (Alvarado et al., 2015) and FiNER-ORD Shah et al. (2023b). includes sentences from public financial agreements through U.S. Security and Exchange Commission (SEC) filings, while FiNER-ORD consists of sentences from news articles. LOCATION (LOC), ORGANISATION (ORG) and PERSON (PER) entities of sentences from both datasets are manually annotated.
Question Answering. Question answering is the task of automatically answer a financial question based on the provided information. We use two datasets: FinQA (Chen et al., 2021) and ConvFinQA (Chen et al., 2022). FinQA consists of question-answering pairs annotated by experts and their corresponding earnings reports (including unstructured documents and tables) from S&P 500 companies. ConvFinQA is an expansion on FinQA that has conversations with the multi-turn question and answering over earnings reports.
Text Summarization. Text summarization aims to condense the long unstructured financial texts into the short summaries that capture crucial information and maintain factual consistency with the original long texts. We utilize two datasets: ECTSum Mukherjee et al. (2022) for extractive summarization and EDTSum Zhou et al. (2021) for abstractive summarization. ECTSum includes 2,425 long earnings call transcripts (ECT) and corresponding bullet-point summarization written by domain experts. EDTSum consists of financial news articles and corresponding titles as summaries.
Stock Movement Prediction. As one of the fundamental financial tasks, stock movement prediction has great potential value in real applications such as investment strategies. Following previous work (Soun et al., 2022), we frame the task as a binary classification problem, which is to predict the binary stock price movement given historical stock prices and tweets. If price movement is higher than  $0.55\%$ , it will be assigned to positive samples (1), or negative samples (-1) if it is lower than  $-0.5\%$ . We adopt three commonly-used datasets: BigData22 (Soun et al., 2022), ACL18 (Xu and Cohen, 2018), and CIKM18 (Wu et al., 2018).
Figure 1: An overview of multi-task and multi-modal instruction tuning of FinMA for diverse financial tasks.
Credit Scoring. Credit scoring is a crucial task in financial services, aiming to classify consumers described by a set of attributes as either good or bad credit risks. We employ two datasets: German Credit Data (German) Hofmann (1994) and Australian Credit Data (Australian) Quinlan (1987). German contain 1000 instances of customers, each represented by 20 attributes, including status of existing checking account, credit history, etc., and the corresponding label of being a good or bad credit risk. Australian contains 690 instances with 14 attributes and the corresponding credit risk label. The task is to predict whether they are a good or bad credit risk based on these attributes.
3.2 Instruction construction
Base on the raw datasets, we further construct our financial instruction datasets, whose statistics are presented in Table 2. We ask domain experts to write 10 diverse instructions for all datasets except the ConvFinQA, where we only use one instruction. Since ConvFinQA is a multi-turn conversational question-answering dataset, which has diverse questions as instructions in nature. For BigData22, ACL18, CIKM18, we use the same instruction set, since they have the same data types of input data and task formulation<sup>10</sup>. Based on these prompts, we convert raw datasets from these tasks into instruction-tuning samples, by gathering human-designed instructions, and input texts along with responses of each dataset. For FPB, FiQA-SA, Headlines, NER, FinER-ORD, ECTSum, EDTSum, German, Australian, FOMC, BigData22, ACL18, and CIKM18 datasets, we build instruction tuning samples with the following template:
Instruction: [task prompt] Text: [input text] Response: [output]
[task prompt] is the prompt designed for each data, [input text] is the input financial data from each data, e.g. the historical prices and tweets for stock movement prediction datasets, [output] is the corresponding output for input text, e.g. sentiment label of input text from ["Positive", "Negative", "Neutral"] in FiQA-SA dataset. For FPB, FiQA-SA, and NER, we employ all 10 instructions for each sample, while we randomly sample one instruction for each sample in remaining datasets.
For FinQA and ConvFinQA, we use the following template:
Instruction: [task prompt] Context: [input context] Question: [input question] Response: [answer]
[input context] is the input contextual information for each data sample. For example, the input context can be filled with the text and table from the filling files for FinQA. ConvFinQA has multi-turn conversations with questions and answering. We transform each turn of the conversation for each data sample into one instruction via the template, which will append previous questions and answer in the [input context].
10The instruction examples are presented in Appendix
4 FinMA: Financial Large Language Model