PIXIU: A Comprehensive Benchmark, Instruction Dataset and Large Language Model for Finance
Authors: Qianqian Xie
Weiguang Han
Xiao Zhang
Yanzhao Lai
Min Peng
Alejandro Lopez-Lira
Jimin Huang
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Financial Natural Language Processing
Large Language Models
Fine-tuning
Benchmarking

Abstract: Although large language models (LLMs) have shown great performance in natural language processing (NLP) in the financial domain, there are no publicly available financially tailored LLMs, instruction tuning datasets, and evaluation benchmarks, which is critical for continually pushing forward the open-source development of financial artificial intelligence (AI). This paper introduces PIXIU, a comprehensive framework including the first financial LLM based on fine-tuning LLaMA with instruction data, the first instruction data with 128K data samples to support the fine-tuning, and an evaluation benchmark with 8 tasks and 15 datasets. We first construct the large-scale multi-task instruction data considering a variety of financial tasks, financial document types, and financial data modalities. We then propose a financial LLM called FinMA by fine-tuning LLaMA with the constructed dataset to be able to follow instructions for various financial tasks. To support the evaluation of financial LLMs, we propose a standardized benchmark that covers a set of critical financial tasks, including six financial NLP tasks and two financial prediction tasks. With this benchmark, we conduct a detailed analysis of FinMA and several existing LLMs, uncovering their strengths and weaknesses in handling critical financial tasks. The model, datasets, benchmark, and experimental results are open-sourced to facilitate future research in financial AI.

PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance
Qianqian Xie
Weiguang Han
School of Computer Science
School of Computer Science
Wuhan University Wuhan, Hubei, China xieq@whu. edu. en
Wuhan University Wuhan, Hubei, China
han. wei. www.gggunngwwuueco edu. cn
Xiao Zhang
Yanzhao Lai
Sun Yat-Sen University
School of Economics and Management Southwest Jiaotong University Chengdu, Sichuan, China laiyanzhao@swjtur tu. edu, cn
Shenzhen, Guangdong, China 2hangx767 sysu. edu.
Min Peng
Alejandro Lopez-Lira University of Florida
School of Computer Science
Wuhan University Wuhan, Hubei, China pengm@whu edu. cn
alejandro wtw.aadro...eeee-rrraarrrrraeee edu
Jimin Huang
ChanceFocus AMC.
Shanghai. China imin@chancefoccu.ccmm
Abstract
Although large language models (LLMs) have shown great performance in natural language processing (NLP) in the financial domain, there no publicly available financially tailored LLMs, instruction tuning datasets, and evaluation benchmarks, which is critical for continually pushing forward the open-source development of financial artificial intelligence (A1). This paper introduces PIXIU, a comprehensive framework including the first financial L.MM based fine-tuning LLaMM with instruction data, the first instruction data with 128K data samples to support the fine-tuning, and evaluation benchmark with tasks and 15 datasets. We first construct the large-scale multi-task instruction data considering variety of financial tasks, financial document types, and financial data modalities. We then propose financial LLM called FinMA by fine-tuning LLaMA with the constructed dataset to be able follow instructions for various financial tasks. To support the evaluation of financial LLMs, we propose a standardized benchmark that covers a set of critical financial tasks, including six financial NLP tasks and two financial prediction tasks. With this benchmark, we conduct detailed analysis of FinMA and several existing L.MMs, uncovering their strengths and weaknesses handling critical financial tasks. The model, datasets, benchmark, and experimental results are open-sourced to facilitate future research financial AI.
https///gihhbl.cc/////ooooooottp
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Introduction
Financial technology (FinTTch) has been continually advanced by the development of natural language processing (NLP) and machine learning (ML) techniques, unlocking diversity capabilities from predicting stock price movements to advanced financial analytics (Araci) 2019; Han et al. 2023
Xie et al. 2023 Lopez-Lira and Tang, 2023 Li et al. 2023). Specifically, the most recent large
language models (LMM)) [Brown et al. 2020 have exhibited remarkable abilities in natural language
understanding (NLU) and performing various tasks by following natural language instructions without training data. Despite these successes, the highly technical nature of financial texts requires domain- specific LLMs to understand complex financial language and concepts effectively. Such efforts include existing financial pre-trained language models (PLMs) such as finBERT (Araci) (019)) FinBERT YYngg et al. 2020) and FLANG (Shah et al. 2022). However, those models are considered small since their parameter size is below one billion, limiting their generalization ability. Recently, proprietary financial LLM called BloombergGPP (Wu et al. (023) with 50 billion parameters has been proposed by pre-training B Bloom-style LLM (Scao et al. (022) on large-scale financial data.
Despite these efforts Liu al. 2023) there remain several issues, as shown in Table T Firstly, BloombergGPT and its training data are not openly released. Currently, there are no open-sourced financial LLMs, which can hinder development in the research community. Secondly, previous financial PLMs and the latest BloombergGPT are not fine-tuned for following natural language instructions (also known as instruction tuning), which is critical for improving the zero-shot ability on dealing with downstream financial tasks (Wei et al. [022]] Ouyang et al. 2022). Thirdly, there are also no financial instruction data for supporting the instruction tuning of LLMs and evaluation benchmarks for comprehensively assessing and comparing the abilities of LLMs for financial tasks.
We are thus motivated to consider the following research questions: 1) how can we develop efficient and openly available LLMs tailored for finance? 2) how can we build large-scale and high-quality financial instruction data? 3) how can we build the holistic financial evaluation benchmark for assessing financial LLMs?
Table 1:
The comparison of pre-trained language models and large language models for finance. "Instruct" means whether the model can follow instructions. "NLP" and "Fin" mean if the model is evaluated with financial NLP tasks and financial prediction tasks.
Model
Backhone
Size
Model
Open Source
Data
Instruet
Langunge
Evaluation
NLP
Fin
Release Date
BoRERT Araci 2019
BERT
Faglish
V
X
08/27/19
FinBERT Sancer a 2020
BERT
110M
x
X
English
V
X
06/15/20
Mengri-fin Zheap al X021
RoBERTo
103M
Chinese
10/13/21
FLANG Shah
ELECTRA
X
English
1031/22
La al
2023
TS
220M
Chinese
02/18/23
Bloumberg al
2023
BLOOM
X
Faglish
03/30/23
FinMA
LLMMA
7/30B
English
06/01/23
To deal with these research questions, we propose PIXIU & $*
a comprehensive framework that includes the first open sourced fine-tuned financial LLM, FinMA, which is based on fine-tuning instruction data. Fig | presents an overview of multi-task and multi-modal instruction tuning of FinMA for diverse financial tasks. PIXIU also contains the first instruction data with 128K data samples to support the fine-tuning and holistic evaluation benchmark with six financial NLP tasks and two financial prediction tasks. It has the following distinguishing features:
LLaMA (Tovrron al. 2023) with multi-task and
Open resources. We have openly released the financial LLM, instruction tuning data, and datasets included in the evaluation benchmark, and implementation, to encourage open research and transparency in the research field.
Multi-task. PIXIU includes multi-task instruction tuning data covering a diverse set of financial tasks, including six financial NLP tasks and two financial prediction tasks. The
"PIXIU (&S 98 wtt.:/ttps/// // en. wikipedia wtg/wwki////oooooooooo...... Pixxiu is mythical creature in Chinese folklore. has the head of a dragon and the body of a lion and is believed to be an auspicious creature attracting money and good fortune.

---

multi-task instruction tuning has been proven to be critical for improving the model's
generalization ability (Sanh al. 2022 Longpre et al. 2023) new tasks.
Multi-modality. Our instruction tuning data consists of multi-modality financial data such as tables in financial reports and historical stock prices as time-series data for the stock-movement prediction tasks beyond texts. Moreover, they encompass diverse types of financial texts, including reports, news articles, tweets, and regulatory filings.
Diversity. Compared with the evaluation tasks used in BloombergGPT and existing FLUE benchmark (Shah et al. 2022), which mainly cover financial NLP tasks, our evaluation benchmark includes financial prediction tasks such stock movement prediction and credit scoring. It requires the model to fully exploit both natural texts and time-series data to extract essential information for accurate prediction. Compared with financial NLP tasks, the financial prediction task is more aligned with real-world scenarios and more challenging.
To build the multi-task and multi- instruction data, we collect open-released training data from diverse tasks, including financial sentiment analysis, news headline classification, named entity recognition (NER), question answering, text summarization, stock movement prediction, credit scoring and hawkish-dovish classification, and propose the diverse task-specific instructions written by domain experts for each task. We create a large-scale financial instruction tuning data (FIT) by assembling the task-specific instructions with data samples from each task. We thus propose the domain-specific LLM Fin by conducting the multi-task instruction tuning on LLaMA with the building dataset. To evaluate our model and other LLMs holistically, we build the Financial Language Understanding And PRediction Evaluation Benchmark (FLARE) covering 6 financial NLP tasks with 10 datasets, and 2 financial prediction task with 5 datasets.
Based on FLARE, we evaluate the performance of our model, BloombergGPPT, and advanced LLMs in the general domain, such as CharGP7"ann and GPT-4 (OpenAl 2023). Experimental results show that: 1) FinMA significantly outperforms L.MMs, including BloombergGPP, ChatGPT, and GPT-4 on most tasks in FLARE, including financial sentiment analysis, news headline classification, and stock movement prediction. This demonstrates the importance of tailoring the LLMs specifically for the financial domain. 2) Despite promising results most tasks, FinMA underperforms BIo.mbergGPT. ChatGPT, and GPT-4 on the question answering, which assesses the quantitative reasoning ability of LLMs. Our analysis finds that this is caused by the limitation of LLaMA quantitative reasoning and mathematics. FinMA also shows limited performance on NER tasks although it outperforms BloomerbergGPT. which is also due to the drawbacks of LLaMA. 3) Compared with NLP tasks, all LLMs, including FinMA, ChatGPT and GPT-4, still present limited performance on stock movement prediction, indicating room for further improvement. 4) FinMA fine-tuned with both NLP and financial prediction tasks, presents the best performance on one of the stock prediction datasets, indicating the potential of task-specific instruction tuning of LLMs on financial prediction tasks.
Our contributions can be summarized as follows: 1) We introduce FIT, the first multi-task and multi-modal instruction tuning data in the financial domain, covering 5 tasks and datasets with 128,640 (128K) data samples. 2) We introduce FLARE, the first evaluation benchmark with both financial natural language understanding and prediction tasks. 3) We introduce FinMA, the first openly released and catructtionfollooiinn financial large language model, which achieves SOTA on financial NLP tasks and 2 financial prediction tasks. 4) We compare FinMA and existing LLMs FLARE. The results demonstrate the superiority of FinMA, the key limitations of LLMs for finance, and future directions to advance L.MMs for finance.
2 Related Work
Financial Language Models Many PLMs for the financial domain have been proposed by continual
pre-training PLMS with large-scale financial texts. Araci (2019) proposed the first financial PLM called finBERT that pre-trained BERT (Devlin et al. 2019) with open released financial corpus pre such as ( and Financial Phrase Bank Malo et al. 2014)) finBERT outperforms neural network methods such as LSTM in financial sentiment classifictiion tasks. Yang et al. (2020)
further proposed FinBERT by pre-training BERT with a 4.9 billion tokens financial communication
httpp://ppenai..cm///l///haaaapp
.ttll
3

---

corpus, which outperforms BERT on three financial sentiment classification datasets. Shah et al. (2022) proposed FLANG, financial PLM with BERT and ELECTRA (Clark et al., 2020) as the backbone. Besides English, financial PLMs in other languages, such as Chinese, were also proposed, such as Mengzi-fin (Zhang et al. 202T) and BBT-Fin'3s (Lu et al. 2023). Latest, Wu et al. (2023) proposed BloombergGiPT, the first financial large language model with 50 billion parameters, that is pre-trained with mixed datasets from the general and financial domain. However, neither the model nor pre-trained domain datasets are not released. The model is also not instruction-followe other LLMS such as ChatGPT and GPT-4.
like
Financial Evaluation Benchmark (Shah et al. (0022) proposed the first heterogeneous evaluation benchmark FLUE with financial NLP tasks, including financial sentiment analysis (Malo et al.) 2014), news headline classification (Sinha and Khandait 202I), named entity recognition (Alvarado
2015). structure boundary detection and question answering (Maia et al. 2018). Lu et al. (2023) proposed the first Chinese financial evaluation benchmark BBT-CFLEB with financial news classification, summarization, relation extraction, question answering, and negative news determination task, as well as sentiment classification task of financial social media texts. However, these benchmarks only consider financial NLP tasks and don't 't include financial prediction tasks, such as stock movement prediction, that are critical for evaluating the model's performance applied to real-world scenarios.
al.
Open Sourced Large Language Models Recent studies have made efforts on democratic AI, where the representative work is LLaMA (Touvron et al., (023) from Meta AI, an open- LLM with parameters ranging from 7B and T3B to 65B. LLaMA-13B has comparable and even better performance than GPT-3 (Brown al. 2020) with 175B parameters on common sense reasoning tasks, Following efforts have been proposed to improve LLaMA for instruction following like ChatGPT, by instruction tuning. Such as Taori et al. (2023) proposed Alpaca by fine-tuning LLaMA-7B with 52K instruction-following samples generated with the self-instruct method (Wang et al. 2022). Chiang ct al. (2023) proposed Vicuna- 13B by fine-tuning LLaMA-13B with 70K conversation data from GPT It can generate better answers to user's questions compared with Alpaca. However, there are no upen-sourced LLMs and instruction tuniin data focused on the financial domain.
3 FIT: Financial Instruction Tuning Dataset
In this section, we introduce our financial instruction tuning dataset FIT. including the background of raw data, tasks in FIT, and the construction process based on raw data. Different from existing financial datasets, FIT is the first instruction- ttning dataset for finance LLMs and includes financial prediction tasks except for financial NLP tasks, which is fundamental for real-world financial applications.
3.1 Raw Data
Derived from real-world finance scenarios, build our financial instruction tuning dataset FIT based on the open-sourced data of various financial NLP and prediction tasks. Compared with the self-instruct method (Wang al. 2022) commonly used by existing LLMs such as Alpaca, we choose to build instruction tuning datasets from open-sourced datasets due to the following reasons: 1) the open-sourced datasets are usually annotated by domain experts, showing high quality, 2) it has very low cost and has no limitation on commercial use unlike datasets constructed from ChatGPT or GPT-4, 3) these open-sourced datasets cover a variety of text types such as news, reports and tweets, as well as multi-modalle
including time series data, tables, and texts. The detail of the raw data and instruction data are shown in T
Financial Sentiment Analysis. Financial sentiment analysis task has long been a critical task in the financial domain (Araci) 2019 Yang al. (2020), aiming to analyze the sentiment information of the input financial texts. Following existing benchmark FLUE (Shah et al.) 2022). we use two datasets: the Financial Phrase Bank (FPB) dataset (Malo et al. 2014) and --QASA MMaia al. 2018). FPB
includes English sentences from financial news and their sentiment label of positive, negative, or
http:://sitesssggglee.eo
google. cow com//// csie. ntt.nntuene edu, wtt..ttt///iiiee///////areeedede com/ www.m///ewaaaiinnnoommmmm html
https://bbt www.sstryyyyo.oom httpp:://bb..ssyymmetyyycco
hattpa://shareppt. com com
For further details of the data split and pre- prepprocsssing, please refer Appendix.

---

Table 2: The details of the raw data and instruction data.
Duta
Tosk
Raw
Instruction
Data Types
Modalities
License
sentiment analysus
48,450
CUBY-SA Puhlic
FIQA-SA
snntmment andlysii
1,175
11,730
news headllines, tweets
text
Headlines
news headline clossitcation
11,412
11,422
news headllines
tEXT
CC BY-SA 3.0
HDMC
haw dovash clasedication
HMMC transciits
CC CUBBYNCC 4,0
named entity recognition
6,090
fanniiall agreements
CC RY-SA 3.
FINER-ORD
named entity recognition
1,080
1,050
news articles
text
CC BY-SA 3.0
FinQA
question answering
8,281
8,251
earnings reports
text.tible
MIT License
CasvFiiQA
question answering
carnings roports
test.tible
Licesse
BCT
text summotization
495
carning call transcipts
Public
FDTTSam
text snmmsiization
2,000
2,000
news articles twoets, historical prices
text
Public
BigDana22
stock movement prediction
7,168
7,368
text,time series
Public
AELIS
stock movemeut prediction
27,050
27,080
histerical poices twoets historical prioes
CIKM1&
stock movement peedictioo
4,971
4,971
tix.fime series
Public
Gemman
cradit scoring
1,000
1,000
credit records
table
CC
4.0
Australia
credit wooring
690
690
enedii necords
table
CC BY 4.0
neutral annotated by domain experts. FIQA-SA is another widely adopted dataset, which aims to predict the sentiment of English financial news and microblog posts on a scale of [-1,1], where 1 means the most positive.
News Headline Classification. The news headline classification task aims to analyze other informa- tion, such price movement in financial texts. We use the Gold news headline dataset (Sinha and Khandait (021) consisting of news headlines from 2000 to 2019 about "gold" and their corresponding tags: "price or not", "price up". "price down". "price stable", "past price". "future price", "past general", "future general", "asset comparison". The task is to conduct the binary classification for each tag of each data sample.
Hawkish-dovish Classification. The Hawkish-Dovin classification aims to classify sentences from monetary policy texts into a 'hawkish' or 'dovish' stance, unlike standard sentiment analysis. The key to this task lies in understanding the nuanced language of financial texts and being able to identify the economic implications conveyed through these 'hawkish' or 'dovish' signals. We use the FOMC dataset (Shah et al. (2023a) including sentences extracted from the Federal Open Market Committee (FOMC) meetings, where each sentence is manually annotated as either 'hawkish' or 'dovish.
Named Entity Recognition. Named Entity Recognition (NER) task is to detect critical financial entities such persons, organizations, and locations, which can be used to build financial knowledge graphs. We use two datasets: NER (Alvarado et al., 2015) and FINER-ORI Shah et al. (2023b) includes sentences from public financial agreements through U.S. Security and Exchange Commission (SEC) filings, while FiNER-ORD consists of sentences from news articles. LOCATION (LOC), ORGANISATION (ORG) and PERSON (PER) entities of sentences from both datasets are manually annotated.
Question Answering. Question answering is the task of automatically answer financial question based on the provided information. We use two datascts: FinQA (Chen et al.) 202T) and Con- vFinQA (Chen et al. 2022). FinQA consists of question-answerte their corresponding earnings reports (including unstructured documents and tables) from S&P 500 companies. ConvFinQA is an expansion FinQA that has conversations with the multi-turn question and answering over earnings reports.
pairs annotated by experts and
Text Summarization. Text summarization aims to condense the long unstructured financial texts into the short summaries that capture crucial information and maintain factual consistency with the original long texts. We utilize two datasets: ECTSum Mukherjee et al. (2022) for extractive summarization and EDTSum Zhou et al. (202I) for abstractive summarization. ECTSum includes 2,425 long carnings call transcripts (ECT) and corresponding bullet-point summarization written by domain experts. EDTSum consists of financial news articles and corresponding titles as summaries.
Stock Movement Prediction. As one of the fundamental financial tasks, stock movement prediction has great potential value in real applications such as investment strategies. Following previous work (Soun et al. 2022), we frame the task as binary classification problem, which is to predict the binary stock price movement given historical stock prices and tweets. price movement is higher than 0.55%, it will be assigned to positive samples (1), or negative samples (-1) if it is lower than
-0.5%. We adopt three commonly-used datasets: BigData22 (Soun al., 2022). ACL.18 (Xu and
Cohen 2018), and CIKMI8 (Wu et al. 2018).
5

---

NER
ventun parton (TRRT eccurnaaion sadlen Conurs
legundl asser Sciar 020 Opennt
for
eouldbee
CNG
Cln li FCR
Callfrria LOG bark ORB
LOSN peo IiFy pcceprannt larvnry belwver ELLCOL VALLET eusus procu ollue lockee du aniid
Cloo, LOC Califonia LOC
Willarm
FPB
FPB
Amuiura speineett cfiee akis went cahyrletien thee feaecial mode aeseer
wonde mould ba evalled an ausscve
Positive
li ITGC 102 Royal Bank Conland
BigDala22
Big
Riae
Gudkeng sesiii the
caluugee uiia aindaesure wow.e///eeeeeeee 3009/1 A..1 2.48224/24-25 Ansver
FinMA
Figure I: An overview of multi-task and multi-modal instruction tuning of FinMA for diverse financial tasks.
Credit Scoring. Credit scoring is a crucial task in financial services, aiming to classify consumers deseribed by set of attributes as either good or bad credit risks. We employ two datasets: German Credit Data (German) Hofmann (1994) and Australian Credit Data (Australian) Quinlan (1987). German contain 1000 instances of customers, each represented by 20 attributes, including status of existing checking account, credit history, etc., and the corresponding label of being a good or bad credit risk. Australian contains 690 instances with 14 attributes and the corresponding credit risk label. The task is to predict whether they are a good or bad credit risk based on these attributes.
3.2 Instruction construction
Base on the raw datasets, we further construct our financial instruction datasets, whose statistics are presented in Table 2
We ask domain experts write 10 diverse instructions for datasets except the ConvFinQA, where only one instruction. Since ConvFinQA is multi-turn conversational guestion-answering dataset, which has diverse questions as instructions in nature. For BigData22, ACL18, CIKM18, use the same instruction set, since they have the same data types of input data and task formulation Based on these prompts, we convert raw datasets from these tasks into instruction-tumi -tttinnntttnnnggggn samples. by gathering human-designed instructions. and input texts along with responses of each dataset. For FPB, FiQA-SA, Headlines, NER, FiNER-ORD, ECTSum, EDTSum, German, Australian, FOMC, BigData22, ACL18, and CIKM18 datasets, build instruction tuning samples with the following template:
Instruction: [task prompt]
Text: [input text]
Response:
[output]
[task prompt] is the prompt designed for each data, [input text] is the input financial data from each data, e.g. the historical prices and tweets for stock movement prediction datasets, [output] is the corresponding output for input text, c.g. sentiment label of input text from ["Positive", "Negative", "Neutral"" in FiQA-SA dataset. For FPB, FiQA-SA, and NER, employ all 10 instructions for each sample, while we randomly sample one instruction for each sample in remaining datasets.
For FinQA and ConvFinQA, we use the following template:
Instruction: [task prompt] Context:
[input context]
Question: [input question]
Response: [an-
swer]
[input context] is the input contextual information for each data sample. For example, the input context can be filled with the text and table from the filling files for FinQA. ConvFinQA has multi-turn conversations with questions and answering. We transform each turn of the conversation for each data sample into one instruction via the template, which will append previous questions and answer the [input context].
The instruction examples presented in Appendix

---

4 FinMA: Financial Large Language Model
We further build FinMA by fine-tuning LLaMA (ouvrro et al. 2023) with FIT. We train four models: FinMA-7B and FinMA-30B by fine-tuning LLaMA 7B and 30B checkpoint with instruction tuning data covering NLP tasks, FinMA-7B -B-rrade by fine-tuning LLaMA 7B checkpoint with instruction tuning data covering forecasting tasks, and FinMA-7B-fuu by fine-tuning LLaMA 7B with full instruction tuning data. We fine-tune tune LLaMA-7B and LLaMA- -----B--rrree -tade with 15 epochs and LLaAM- 7B-full with 3 epochs based on Adam optimizer (Loshchilov and Hutter 2017). The batch size is set to 32, the initial learning rate is 8e-6, and the weight decay is 1e-5. We also set warmup steps to 5% of all training steps. The maximum length of input texts is 2048. The FinMA-7B is fine-tuned on 8 A100 40GB GPUs. As for the FinMA-30B model, we fine-tune LLaMA-30B with 20 epochs, which is also based on the AdamW optimizer. The batch size set to 24, the initial learning rate is 8c-6, the weight decay is lc-5, and warmup steps to 5% of all training steps. The maximum length of input texts is 2048. Different from FinMA-7B, it can only be distributed fine-tuned on 128 A100 40GB GPUs.
5 FLARE: Financial Evaluation Benchmark
Based on FIT, we design our financial natural language understanding and prediction evaluation benchmark (FLARE). We randomll select validation sets from FIT to select the best model checkpoint, and test sets for evaluation. Compared with the existing benchmark FLUE (Sanh ct al. 2022), FLARE covers financial prediction tasks in addition to NLP task [I] We believe it is vital to include financial prediction tasks such as stock movement prediction, to comprehensively evaluate the performance of LLMs on the practical applications of the financial domain. We show the data statistics of validation, and test set for each dataset in Table Following previous methods et. (2023): Shah et al. (2022).
Table 3: The details of our evaluation datasets. To compare the performance with BloombergGPT whose test data is not openly released, we keep the same numbers and data distributions of our test datasets with that of BloOmbergGPT. To further evaluate the emergent and generalization ability of the LLMs, we only adopt the test data of FOMC. FINER-ORD, ECTSum, EDTSum, German, and Australian for evaluation on the FLARE. For ConvFinQA, we take each turn of the conversations as the instruction, whose number would be different from the number of conversations.
Data
Task
Valid
Test
Evaluation
FPB Malore
2014
sentiment analysis sentiment analysis
775
970
Accuracy
FiQA-SA Milla arat 2018)
188
235
Fl
Headlines Sinba and Kliandai
2021
news headline clussification
1.141
2,283
Avg F1
NER (Alvarado
2015)
named entity recognition named entity recognition
103
980
Entity Fl
FINER=ORD Shahh al
2023h
1080
Entity Fl
FinQA 2 A024
question answering
883
1,147
EM Accuracy
(f el 2022
question answering
2,210
1,490
EM Accuracy
BigData22 (Sotin al
stock movement prediction
798
1,470
Accuracy, MCC
ACL18 Xu lil Cmire
2013
stock movement prediction
2,560
3,720
Accurcy, MCC
CIKMIS Wa 0l. 2718
stock movement prediction
431
1,140
Accuracy, MCC
FCTSum Makherjee
2022
text summarization
495
ROUGE, BERTSoore, BARTSanee
EDTSum Zhou et 01 2022
text summarization
2000
ROUGE, BERTScore, BoRTScore
German Hormann (159))
credit scoring
MCC
Australian | Vunnia
(1%%
credit scoring
690
FI. MCC
FOMC Shih et
hawkish-davish classificattinn
Accurney
we evaluate the performance of the sentiment classification task on FPB and FIQA-SA datasets, with the accuracy (ACC) and weighted F1 Score (F1). The performance of the news headline classification task is evaluated with the weighted averages of F1 score over all nine categories (Avg F1). For the performance of the NER task, we evaluate with the entity- Fl score (Entity F1). The performance on the question- -anteirng task is evaluated with exact match accuracy (EM Acc). For summarization tasks such as ECTSum and EDTSum, we assess the relevance and factuality of generated summaries with the ground truth using metrics such as ROUGE score Lin (200)), BERTSSore ZZang et al. (20T9), and BARTScore Yuan et al. (202T). ). As for the financial prediction task, following previous methods (Xu and Cohen 2018 g Xie et al. 2023), we evaluate the performance with the accuracy
Following BloombergGPT. we don't don't include the structure boundary detection task included in FLUE because it is hard to be converted into the instruction following task.
7

---

(ACC) and the Matthews correlation coefficient (MCC) for stock price movement prediction, and the Fl score with MCC for credit scoring. Although Macro-Fl is more fair for the unbalanced dataset, in this paper we adopt the same metrics following previous methods to make fair comparison.
6 Experiments on FLARE
The proposed FIT and FLARE allow to train, select the model, and evaluate the performance of LLMs financial understanding and predictions. In this section, we investigate how powerful the FIT-fine-tuncs FinMA and other LLMs are on FLARE. We compare FinMA with following LLMs: 1) BloombergGPT (Wu et al. 2023). The only large language model with 50B parameters pre-trained with the financial texts. 2) GPT-4 OOeeAA 2023). A powerful instruction following large language model with around IT parameters proposed by OpenAL. 3) ChatGPT. A instruction following large language model with 175B parameters from OpenAL. 4) Vicuna-7B (Zhang et al.
2022). An instruction following large language model by fine-tuning LLaMA-7B.
Following previous methods (Wu al., 2023 Li et al. 2023), we report the 20-shot performance of BloombergGPPT and the 5-shot performance of other baseline methods on the FIN dataset. We report the 5-shot performance of BloombergGPT on FPB and FiQA-SA. We report the 5-shot performance of all baselines the Headlines dataset. For the remaining results, we report the zero-shot performance. The results of some baselines are based on human evaluations, since LLMs without fine-tuning will fail to generate answers pre-defined in the given instruction. All results of FinMA are conducted on zero-shot and can be automatically evaluated.
Table 4:
The zero-shot and few-shot performance of different LLMs on the FLARE benchmark. Results of BloombergGP'T. ChatGPT, and GPT4 on FPB, FIQASA, Headlines, NER, FinQA, and ConvFinQA are referenced from the paper (Li et al. 2023). The results of BloombergGPP are referenced from the original paper Wu et al. (2023). Test datasets were built to have the same data distribution with that of BloombergGPP and the performance of FinMA was directly compared with BloombergGPT following the previous method (Li et al. 2023). All results via our evaluations the average of three runs. " represents the result that is currently unable to yield due to model size or availability, and "g" represents the result from the previous paper.
Chat GPT
GPT
Bloom berg GPT
Vicuna
FinMA
FinMA 7B- trade
FinMA 7B- full
FinMA
Dataset
Metries
Fl
0.78*
0,78
0,51
0.29
0.94
0.03
0.94
0.88
FPB
Acc
0.78*
0.76*
0,26
0.94
0.12
0.94
0.87
FIQA-SA
0.00
0.80
0.75
0.32
0.85
0.16
0.82
0.87
Headlines
AvgFT
0,77
0
0.82
0.60
0.97
0.28
0.97
0.97
NER
EntityFi
0.77
0,83*
0.61
0.12
0.59
0.00
0.64
0.62
FINER ORD
EmiitFFi
0.28
0.00
0.00
0.00
0.00
FIBQA
EmAre
0.58
0.63
0.00
0.06
0.00
0.04
0.11
ConvFinQA
EmAce
0.60
0,76
0.43
0.00
0.25
0.00
0.20
0.40
BigData22
Acc
0.55
0.54
0.44
0.45
0.45
0.31
0.47
MCC
40.25
0.03
40.05
0.02
0.00
0,02
0.04
ACL18
Ace
0.50
0,52
0.50
0.49
0.49
0.51
0.49
MCC
0.005
-0.01
0,03
0.03
0.00
CIKMI8
Acc
0.55
0.57
0.44
0.43
0.43
0.50
0.43
MCC
0.01
0.02
4:03
0.02
0.003
0.88
0.03
Rouge-1
0.17
0.05
0.13
0.17
EDTSUM
Ronge 2
0.08
0.09
0.10
0.04
0.02
0.06
0.08
Konge-N
0.13
0.08
0.05
0.10
0.14
BertSoore
0.66
0.67
0.61
0.56
031
0,38
0.54
BartScore
..64
-.62
4.13
1.12
6.91
3.71
-5.24
Rouge:1
0.00
0.00
0.00
0.00
0.00
ECTSUM
Ronge: 2
1.000
0.00
0.00
0.000
0.00
0.00
0.00
Kongo-N
0.00
0.00
0.00
0.00
0.00
BertSoore
0.00
0.00
0.00
0.00
0.00
0.00
0.00
BartScore
-5.18
3.18
5.18
-5.18
-5.88
-5.18
-3.88
0.20
0.17
0.52
0.17
0.53
German
MCC
0.10
0.02
0.000
0.000
40077
0.00
-007
Australian
0.41
0.74
0.26
0.41
0.26
0.41
0.46
MCC
0.000
0.47
0.00
0.00
0.00
0.00)
0001
FI
0.64
0.71
0.19
0.49
0.10
0.49
0.43
ROMC
0.6
0.47
0.25
0.46
8

---

6.1 Results
Overall Performance. For financial NLP tasks, as shown in Table our fine-tuned model FinMA significantly outperform other LLMs on FPB, FIQA-SA and Headlines datasets, showing the impor- tance of domain specific instruction tuning on improving the performance of LLMs in the specific domain. For example, FinMA-30B outperforms GPT-4 by 10% FI score, and BloombergGPT by 37% Fl score on the FPB dataset. On the NER dataset, FinMA-7B also outperforms BloombergGPP and other L.Ms,, and achieve competitive results compared with ChatGPT and GPT-4. For FinQA and ConvFinQA which requires complex numeric reasoning, there a large gap between the performance of GPT and FinMA. As reported in existing studies (Tovrron et al., 2023 Lewkowycz et al. 2022), LLaMA includes no mathematical datasets for pre-training, resulted in poor performance on the mathematical benchmark datasets such as GSM8K (Cobbe et al. 2021). Our results are also consistent with previous studies which reveals that LLaMA with larger parameters present better performance on mathematical benchmark datasets. The performance of FinMA-30B is significantly better than that of FinMA-7B on FinQA and ConvFinQA. This finding indicate the importance of numeric reasoning for financial question answering, which could be the potential direction for advancing L.LMs in the finance area. Despite the strengths exhibited by our methods on known tasks, they underperform relative GPT-4 on unseen tasks such as FINER-ORE ORD, EDTSUM, ECTSUM, and FOMC. This performance gap suggests that a more diverse set of domain-specifir tasks is needed for effective fine-tuning. Specifically, in FINER-ORD and ECTSUM, we employ a complex prompt design that requires the model to generate label sequences directly. The results indicate that our fine-tuned models consistently fail to produce outputs in the desired formats. While models like ChatGPT and GPT-4 demonstrate some capability for token labeling in FINER-ORD, they also struggle to generate sentence labels for ECTSUM, particularly when faced with longer contextual information. In contrast to existing financial benchmarks, FLARE offers more comprchensive suite of both generation and classification tasks, thereby providing a fuller assessment of Large Language Models' capabilities in the realm of financial NLP.
For financial prediction tasks, all LLMs including FinMA, ChatGPT and GPT-4 struggle in stock movement prediction as previous methods [3] While FinMA- has been fine-tuned specifically for the task of stock movement prediction, the observed performance gains in this area are marginal at best. After fine-tuned with both NLP and financial prediction tasks, FinMA-7B- significantly better performance on ACL18 dataset compared with ChatGPT and GPT-4. However, still presents almost zero MCC on the other two datasets like ChatGPT and GPT-4. Similarly, all tested methods, with the exception of GPT-4, exhibited either zero or negative Matthews Correlation Coefficient (MCC) values on the credit scoring task. This highlights their limitations in accurately forecasting individual default risks. While GPT-4 shows a marked improvement in performance on the Australian dataset, it fails to register positive MCC on the German dataset. Such limitations can be attributed the challenges associated with the tabular data input and the highly imbalanced label distribution inherent to credit scoring tasks. This indicates the complexity and challenging of the financial prediction tasks FLARE. Compared with existing financial benchmarks that focusing on NLP tasks, FLARE provides exciting opportunities for the improvement of LLMss on the fundamental of financial academic studies and applications. It also demonstrates the importance of multiple task learning in the financial domain for LLMs, which can provide essential domain knowledge and skills to handle complex applications in this area.
can achieve a
Further Analysis. We further analyze the influences of model size, and instruction tuning data on the performance of LLMS on different tasks. FinMA-30B has no significantly better performance than FinMA-7B on most NLP tasks and the stock movement prediction task. Apparently, the quality of the instructions rather than the model size is critical for the performance of these tasks. For the complex question-answering tasks such as ConvFinQA, as shown in Table 4. the larger LLaMA model generally has better performance. Particularly, Vicuna-7B based on LLaMA-7B has the worst performance, which are also consistent with previous findings (Cobbe et al. (021 that LLaMA with larger parameters presents better performance on mathematical benchmark datasets. In contrast, for generation tasks such as abstractive summarization (EDTSUM), Vicuna-7B presents the best performance while fine-tuned models shows a decreased performance on almost all metrics. This may indicate that fine-tuning with only classification tasks could lead to better classification performance but also compromise the generation ability. For financial prediction tasks and NLP tasks that are not
For the performance of other L.MMs, please see Appendix. For the performance of traditional methods, please see Appendix.
9

---

included in the fine-tuning instruction dataset of FinMA-7B and FinMA-30P 30B, i.e, FOMC, FINER- ORD, Australian, and ACLI8, our models present limited improvement. While demonstrates some degree of emerge ability, the performance gap compared with GPT-4 indicates the needs for further optimization. However, FinMA-7B-full fine-tuned with both NL.P and prediction datasets, has shown significantly better performance on financial prediction datasets, and comparable performance on NLP tasks with FinMA-7B and GPT-4. This indicates the potential of LLMs to be further adapted and applied directly on financial prediction tasks via pre-training and fine-tuning on diversed domain datasets.
Limitations
Despite the positive contributions of this study, we recognize the following limitations: 1) Model and Training Constraints: We only present FinMA models up to 30B. Due to computational constraints, FinMA-30B has not been fine-tuned on the full dataset. 2) Complex Task Performance: FinMA, due to the limitation of the backbone model LLaMA, struggles with tasks requiring quan- titative reasoning, such as financial question answering, and the difficult financial prediction task. Reesuree Constraints and Generalizability: The development of FinMA, FIT, and FLARE is influenced by available resources and handcrafted instructions, potentially affecting model diversity and generalizability. The maximum input size of FinMA is also limited by the maximum input texts that can be handled by the backbone model LLaMA. Pooonnilaal Negative Impacts: While our study primarily focuses on the positive aspects and advancements of financial language understanding models, it is important to acknowledge the potential negative impacts associated with their use, such as the spread of financial misinformation or unethical market influence. We recommend using our method for academic research only
Conclusion
In this work, we presented PIXIU, encompassing the first open-sourced financial large language model FinMA, the instruction tuning dataset FIT, and the evaluation benchmark FLARE. Through extensive evaluation, we demonstrated the effectiveness of FinMA across various financial tasks, showing the potential of domain-speciff instruction tuning of large language models in the financial domain. However, challenges such improving performance on complex tasks and addressing resource constraints remain. Our open-source contribution aims to facilitate further research and innovation in financial language understanding, prediction, and LLMs, toward more useful and safe LLMs in the field of finance.
Acknowledgement
The PIXIU project mainly supported by ChanceFocus AMC and partially supported by the National Key Research and Development Program of China (No.2021ZD01 13304) and the General Program of Natural Science Foundation of China (NSFC) (Grant No.62072346). Additional funding was provided by the Key R&D Project of Hubei Province (Grant No.2021BBA099, 1BBBB229)))) and by the Joint& & Laboratory on Credit Technology.
References
Julio Cesar Salinas Alvarado, Karin Verspoor, and Timothy Baldwin. 2015. Domain adaption of named entity recognition to support credit risk assessment, In Proceedings of the Australasian Language Technology Association Workshop 2015. 84-90.
Dogu Araci. 2019. Finbert: Financial sentiment analysis with pre-trained language models. arXiv
preprint ar0iv: 10063 (2019).
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot leamerr.. Advances in neural information processing systems 33 (2020), 1877-1901.
For detailed ethical and legal statement concerning this work, please Appendix.
10

---

Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R Routledge. al. 2021. FinQA: A Dataset of Numerical Reasoning over Financial Data. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing. 3697-3711.
Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. 2022. Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering. arXiv preprint arXiv: 10.03449 (2022).
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric Xing. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* * ChatGPT Quality.
https
//lmsys. wttp:o/gg.boooo-o...//rggg 03- 30- vicuna/
Kevin Clark, Minh-Than Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre- training text encoders discriminators rather than generators. arXiv preprint arXiv:200./0555 (2020).
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek. Jacob Hilton, Reichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14111 (2021).
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies. Volume I (Long and Short Papers). 4171-4186.
Weiguang Han, Boyi Zhang, Qianqian Xie, Min Peng, Yanzhao Lai, and Jimin Huang. 2023. Select and Trade: Towards Unified Pair Trading with Hierarchical Reinforcement Learning. arXiv preprint arXiv:2301.10724 (2023).
Hans Hofmann. 1994. Statlog (German Credit Data). UCI Machine Learning Repository. DOI: https///bb..gg/1.224322CCCCCC
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo et al. 2022. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206. 220. (2022).
Xianzhi Li, Xiaodan Zhu, Zhiqiang Ma, Xiaomo Liu, and Sameena Shah. 2023. Are ChatGPT and GPT-4 General- Solvers for Financial Text Analytics? An Examination on Several Typical Tasks. arXiv preprint arXiv:2305.055 05862 (2023).
Chin-Yeu Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization
branches out. 81
Xiao- Xiao-Yang Liu, Guoxuan Wang. Hongyang Yang, and Daochen Zha. 2023. FinGPT: Democratiz- ing Internet-scale Data for Financial Large Language Models. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, al. 2023. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301. 13688 (2023).
Alejandro Lopez-Lira and Yuehua Tang. 2023. Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models. arXiv preprint arXiv:2304.07619 (2023).
Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv: (2017).
Dakuan Lu, Jiaqing Liang, Yipei Xu, Qianyu He, Yipeng Geng. Mengkun Han, Yingsi Xin, Hengkui Wu, and Yanghua Xiao. 2023. BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark, arXiv preprint arXiv:2302.094. (2023).
11

---

Macedo Maia. Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. Www' 18 open challenge: financial opinion mining and question answering. In Companion proceedings of the the web conference 2018. 1941-1942.
Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius, and Pyry Takala. 2014. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology 65, 4 (2014), 782-796.
Rajdeep Mukherjee, Abhinav Bohra, Akash Banerjee, Soumya Sharma, Manjunath Hegde, Afreen Shaikh, Shivani Shrivastava, Koustuv Dasgupta, Niloy Ganguly, Saptarshi Ghosh, et al. 2022. ECTSum: A New Benchmark Dataset For Bullet Point Summarization of Long Earnings Call Transcripts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 10893-10906.
Jingwei Ni, Zhijing Jin, Qian Wang. Mrinmaya Sachan, and Markus Leippold. 2023. When Does Aggregating Multiple Skills with Multi-Task Learning Work? A Case Study in Financial NLP. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). Association for Computational Linguistics, Toronto, Canada, 7465-7488.
https://doo httpas://dooooog/////// 10. 18653/v1/2022 acl-long. 412
OpenAl. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL]
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (2022), 27730-2277 27744..
Ross Quinlan. 1987. Statlog (Australian Credit Approval). UCI Machine Learning Repository. DOI: https///bboorg//022444/////////
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2022. Multitask Prompted Training Enables Zero-Shot Task Generalization. In ICLR 2022-Temth International Conference on Learning Representations.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arX1v:2211.051100 (2022).
Agam Shah and Sudheer Chava. 2023. Zero is Not Hero Yet: Benchmarking Zero-Shot Performance of LLMs for Financial Tasks. ArXiv abs/2305. 16633 (2023).
org/ harg/CorpusID.... 258941519
Agam Shah, Suvan Paturi, and Sudheer Chava. 2023a. Trillion Dollar Words: A New Financial Dataset, Task & Market Analysis. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (ACL).
Agam Shah, Ruchit Vithani, Abhinav Gullapalli, and Sudheer Chava. 2023b. Finer: Financial named entity recognition dataset and weak-supervision model. arXiv preprint arXiv:2302.11157 (2023).
Raj Sanjay Shah, Kunal Chawla, Dheeraj Eidnani, Agam Shah, Wendi Du, Sudheer Chava, Natraj Raman, Charese Smiley, Jiaao Chen, and Diyi Yang. 2022. WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domain. arXiv preprint arXiv:2211.00083 (2022).
Ankur Sinha and Tanmay Khandait. 2021. Impact of news on the commodity market: Dataset and results. Advances in Information and Communication: Proceedings of the 2021 Future of Information and Communication Conference (FICC), Volume 2. Springer, 589-601.
Yejun Soun, Jaemin Yoo, Minyong Cho, LihyeongJJonn and U Kang. 2022. Accurate Stock Movement Prediction with Self-supervised Learning from Sparse Noisy Tweets. In 2022 IEEE International Conference Big Data (Big Data). IEEE, 1691-1700.
12

---

Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An instructionfoolloiing LLaMA model. https: btttps://githbb com/tatsu- 1ab/ sttpf/tdfordddooom alpaca
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397) (2023).
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-Instruct: Aligning Language Model with Self Generated Instructions. arXiv preprint arXiv:2212. (2022).
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint ar 006522 (2021).
Huizhe Wu, Wei Zhang, Weiwei Shen, and Jun Wang. 2018. Hybrid deep sequential modeling for social text-driven stock prediction. In Proceedings of the 27th ACM international conference information and knowledge management. 1627-1630.
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prab- hanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303. 17564 (2023).
Qianqian Xie, Weiguang Han, Yanzhao Lai, Min Peng, and Jimin Huang. 2023. The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challenges. arXiv preprint arXiv:2304.053: 05355 (2023).
Yumo Xu and Shay B Cohen. 2018. Stock movement prediction from tweets and historical prices. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). 1970-1979.
Yi Yang, Mark Christopher Siy Uy, and Allen Huang. 2020. Finbert: A pretrained language model for financial communications. arXiv preprint arXiv:2006.0800 (2020).
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: generation. Advances in Neural Information Processing Systems 34 (2021). 27263-27277.
Evaluating generated text as text
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint wxXiv:2005.0106 (2022).
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Weinberger, and Yoav Artzi. 2019. BERTScorr:: Evaluating Text Generation with BERT. In International Conference on Learning Representations.
Zhuosheng Zhang, Hanqing Zhang, Keming Chen, Yuhang Guo, Jingyun Hua, Yulong Wang, and Ming Zhou. 2021. Mengzi: Towards lightweight yet ingenious pre-trained models for chinese. arXiv preprint arXiv:2110.0669 (2021).
Zhihan Zhou, Liqian Ma, and Han Liu. 2021. Trade the Event: Corporate Events Detection for News-Based Event-Driven Trading. In Findings of the Association for Computational Linguistics: ACI-IJCNLP 2021. 21 14-2124.
13

---

A Instructions
Table presents all the prompts for each dataset the FLARE benchmark and FIT instruction dataset.
Table 5: The example prompt for each dataset. FiQA-SA has two types of text, including news headlines and tweets. We will fill the detailed text type into [category] for each data sample. For stock movement prediction data such as BigData22, we will fill {tid} and (point) with the detailed stock name and time from each data sample.
Data
Prompt
"Analyze the sentiment of this statement extracted from a financial news article. Provide your answer either negative, positive neutral. For instance, "The company's 's stocks plummeted following the scandal." would be classified as negative."
FPB
"What is the sentiment of the following financial [category]:
FQQA-S
Pocitive, Negative, or Neutma?"
"Consider whether the headline mentions the price of gold.
Headlines
there Price Not in the gold commodity market indicated news headline?
Please answer Yes or No."
"In sentences extracted from financial agreements U.S. flligs,
identity the named eniities that represent person ('PER', ) organization ('ORG"), or location ("LOOC"). ) The required answer format is: "entity name, entity type'. For instance, in 'Elon Musk CEO of SpaceX, announced the launch from Cape Canaveral,' the entities would be: 'Elon Musk, PER: SpaceX, ORG; Cape Canaveral, LOC""
NER
"In the list of tokens, identify 'Person', 'Location', and OOganisationn and label each accordingly. B.LOC,
the entity spans multiple tokens, use prefix B. and I.PER, I-LOC, or L-ORG for the subsequent tokens of that entity. The beginning of each separate entity should always be labeled with B. PER, B B-LOC, or ORG prefix. If the taken does not fit inlo any the ihree named categories, token and its corresponding label, soparated by a colon. Do not combine tokens on your own. The format for each should 'token:label.. Text all because you failed to prepare Answer:"
for the first loken,
FNNER-ORD
not named entity, label
'O', Fach line should contain
FinQA
"Given the linancial data and expert analysis, please unswer this question:""
"In the context of this series of interconnected finance related queries and the additional information provided by pretext, table data, and post text from a company's financial filings, please provide response to the final question. This may require extracting information
ConvFinQA
from the context and performing mathematical calculations. Please take into account the information provided the preceding questions and their answers when formulating your response:"
tweets to guess whether the closing price (tid) surge or decline (point). Please declare with either Rise or Fall."
Contemplate
BigDara22
"Given the following asticle, please produce list of O and each separated by indicate which sentences should be included in the linal summuryy. The article's sentences have been split by " Please mark each sentence should
ECTSum
with if should
luded the summary
"You are given a text that consists of multiple sentences. Your task is to perform abstructive summarization on this text. Use
EDTSum
your understanding of content express main ideas crucial dotails shorter, coherent, natural sounding text."
"Assess credinwortinees of a customer using following table attribates for financial starus. Respond with either "godd' or 'bad'. And the table attributes incloding categorical attribuses and 7 numerical attributes are follows:"
German
release below. Classiyy it HAWKISH advocates a tightening of monetary policy. DOVISII if suggests an easing of monetary policy, or NEUTRAL if the stance is unbiased. Your response should return only HAWKISH, DOVISH, or NEUTRAL.
"Examine excerpt from central
FOMC
B Traditional Methods for Stock Movement Forecasting
In the context of stock movement prediction, traditional models, as summarized in Table have long been employed but face significant challenges in achieving consistently high levels of accuracy. This underscores the inherent difficulty of the task at hand. In contrast, Large Language Models (LLMs) introduce a level of adaptability by learning from multiple tasks, although they also have limitations such as numeric understanding and reasoning. The task's difficulty is thus a common challenge for both traditional models and LLMs.
14

---

Table 6: Movement prediction performance of non-LLM models vs FinMA, measured with the accuracy (ACC) and the Matthews correlation coefficient (MCC). The best of non-LLM models is in red and the best of all is in bold.
BIGDATA22
ACL18
CIKM18
Method
ACC
MCC
ACC
MCC
ACC
MCC
Logistic regression (LR)
0.53
0.02
0.52
0.04
0.53
-0.04
Random forest (RF)
0.47
-0.11
0.52
0.03
0.54
0.01
LSTM
0,51
0.01
0.53
0.06
0.53
0.02
Attention LSTM (ALSTM)
0.49
-0.03
0.52
0.04
0.53
-0.01
Adv-ALSTM
0.50
0.01
0.53
0.07
0.54
0.02
DTML
0.52
0.07
0.58
0.18
0.54
-0.00
XGBoost
0,52
-0.04
0.49
-0.02
0.58
0.07
XGBRefressor
0.46
-0.13
0,50
-0.01
0.53
-0.03
ALSTM-W
0.48
-0.01
0.53
0.08
0.54
0.03
ALSTM-D
0.49
0.01
0.53
0.07
0.50
-0.04
StockNet
0.53
-0.00
0.54
-0.03
0.52
-0.02
SLOT
0.55
0.10
0.59
0.21
0.56
0.09
FinMA-7B
0.48
0.04
0.50
0.00
0.56
-0.02
FinMA-30B
0.47
0.04
0.49
0.00
0.43
-0.05
FinMA-7B-full
0.49
0.01
0.56
0.10
0.53
-0.03
C Performance of BERT based Large Language Models
Table 7 presents previously reported performance of pre-trained language models (PLMs) including FinBERT and FLANG-BERY across three selected tasks under the FLARE benchmark. Although both models demonstrate impressive results in specific performance metrics, it's crucial to note that they have been pre-trained with large-scale financial texts and utilize task-specific headers for different tasks, which is the main reason for their better performance. However, compared with LLMs, these PLMs can't 't be adaptable to unseen tasks without supervised fine-tuning, thus having the bottleneck of adapting to multi-task environment and zero-shot learning. For LLMs, our results suggest their performance still underperforms these finely tuned PLMs on some tasks, but offer greater flexibility, adaptability, and zero-shot ability. They have the advantage of learning directly from prompts and being more flcxible in handling wide array of tasks, even without the need for labeled training data. The underperformance of LLMs compared with PLMs also highlights the need for Jomain-speciffe pre-training of LLMs in the future, to further improve their performance in the specific domain.
These observations align with recent studies, such as Shah and Chava (202)), which also observed that shoo-shot LLMs like ChatGPT offer respectable performance across financial tasks without the need for labeled data. The recent work by Ni et al. (2023), also highlights the challenges of fine-tuned PLMs adapting unseen and multiple tasks.
Table Results of BERT-based GincoderDecoodee Large Language Models (LLMs) across multiple tasks reported in previous papers.
FPB
Headline
NER
Method
Accuracy
AvgF1
F1
FinBERT
0.872
0.968
0.8
FLANG-BER
0.912
0.972
0.83
15

---

D Performance of Previous Large Language Models
Table 8 provides detailed comparison of zero-shot and few-shot performance metrics for various Large Language Models (LLMs) GPT NeoX 20B, OPT 66B, and BLOOM- across multiple datasets. These metries, obtained from a previous paper, serve as valuable baseline for understanding the capabilities of these models in both zero-shot and few-shot scenarios under the FLARE benchmark.
Table 8:
The zero-shot and few-shot performance of different LLMs on the FLARE benchmark. Results are referenced from previous paper.
OPT
Dataset
Metries
NeoX
BLOOM
20B
F1
0.45*
0.49*
0.50*
FPB
Ace
FIQA-SA
FI
0.51
0.52*
0.53*
Headlines
AvgF1
0.73
0.79*
0.77
NER
EntiyFi
0.61
0.57
0.56
FinQA
EmAce
COnFFINQA
EmAcc
0
0.30
0.36
BigDama22
Acc
041
MCC
Aec
0.35
MCC
Acc
CIKM18
MCC
-0.12
Rouge-1
Rouge-2
0.01
Rouge-N
0.02
BariSeore
-572
Rouge-1
ECTSUM
Rouge-2
DXU
Rouge-N
0.00
BariScore
5.18
German
F1
0.17
MCC
0.02
FT
0.00
Australion
FOMC
FI
0.37
Ace
0.27
16


Reviews:
[{"review_id": "Apd7IvBXsC", "reviewer": "Reviewer_kysA", "summary": "", "questions": "", "limitations": "The main cons is that the dataset is an assembly of already existing datasets.This limits the novelty of the contribution.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The proposed dataset is the first instruction tuning corpus covering the financial domain. Unlike the prior works on the field, both the model and the data are publicly available. \nThe dataset covers multiple financial NLP tasks and one financial prediction task.\nThe model released by the author is the first LLM fine-tuned with instruction data and it achieves superior performance in most tasks with respect to other popular non-fine-tuned models. I believe that the public availability of the dataset is the main contribution of the paper, which is strengthen by the related fine-tuned model and evaluation benchmark.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "i6V81uJEBm", "reviewer": "Reviewer_v2GB", "summary": "", "questions": "", "limitations": "1. As mentioned before, as a responsible AI model, the prediction of the stock market should be accompanied by risk warnings.\n\n2. The evaluation can be carefully designed. The current evaluation design of FinLLM is too simple to be effective. The factuality and faithfulness can not be measured by simply conducting evaluations on FinNLP and stock prediction tasks.\n\n3. Methodology: Fine-tuning LLaMA with downstream instructions is not an exciting method in the era of LLM. ", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "Topic: In the era of LLM, some researchers argue that deploying the generalization ability of LLMs into downstream domains is more important than the model architecture design. This work lies on the topic of FinLLM, which is a suitable testbed.\n\nResource: The authors released their datasets on Github, covering 4 financial NLP tasks with 6 datasets, and 1 financial prediction task with 3 datasets. Although most of the datasets are originally contributed by others. \n\nEvaluation: The authors are aware of the importance of evaluation towards LLM. As LLMs continue to play a vital role in both research and daily use, their evaluation becomes increasingly critical, not only at the task level but also at the society level for a better understanding of their potential risks, especially in Finance, which is a high-stake domain.\n\n\n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "FmEPDEhxzz", "reviewer": "Reviewer_TmzR", "summary": "", "questions": "", "limitations": "The authors have adequately addressed the limitations and negative societal impact of this work. ", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. This work presents a comprehensive framework including the first financial LLM based on fine-tuning LLaMA with instruction data, the first instruction data with 136K data samples to support the fine-tuning, and an evaluation benchmark with 5 tasks and 9 datasets.  \n2. Extensive evaluation results demonstrate the effectiveness of FinMA across various financial tasks, showing the potential of domain -specific instruction tuning of large language models in the financial domain. This open-source contribution is expected to facilitate further research and innovation in financial LLM in the field of finance. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "bLBJaK2YOE", "reviewer": "Reviewer_7TUR", "summary": "", "questions": "", "limitations": "Please refer to the section \"Opportunities For Improvement.\"", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- This paper comprehensively evaluates the existing general LLMs, like ChatGPT and GPT-4, and the existing finance domain LLMs, like BloombergGPT, including 9 datasets across 5 tasks.\n\n- This paper released a set of instruction-tuning datasets curated based on public financial NLP datasets, which offers a way for future research to build instruction-tuned LLMs for finance.\n\n- This paper open-sourced the instruction-tuned LLM models based on LLAMA-7B and 13B.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Initially, the reviewers had many concerns and questions about this paper. The authors addressed most of these concerns and questions in their responses and the revised version, leading to an improvement in the overall scores from the reviewers. However, there are still remaining concerns regarding the scientific values of fine-tuning LLaMA on a downstream domain and the performance gap between the decoder-only LLM and other models of the selected tasks. Hopefully, the comments and suggestions from reviewers can help the authors further improve this paper.
Decision: Accept (Poster)