SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We further build FinMA by fine-tuning LLaMA (Touvron et al., 2023) with FIT. We train four models: FinMA-7B and FinMA-30B by fine-tuning LLaMA 7B and 30B checkpoint with instruction tuning data covering NLP tasks, FinMA-7B-trade by fine-tuning LLaMA 7B checkpoint with instruction tuning data covering forecasting tasks, and FinMA-7B-full by fine-tuning LLaMA 7B with full instruction tuning data. We fine-tune LLaMA-7B and LLaMA-7B-trade with 15 epochs and LLaAM-7B-full with 3 epochs based on AdamW optimizer (Loshchilov and Hutter, 2017). The batch size is set to 32, the initial learning rate is 8e-6, and the weight decay is 1e-5. We also set warmup steps to  $5\%$  of all training steps. The maximum length of input texts is 2048. The FinMA-7B is fine-tuned on 8 A100 40GB GPUs. As for the FinMA-30B model, we fine-tune LLaMA-30B with 20 epochs, which is also based on the AdamW optimizer. The batch size is set to 24, the initial learning rate is 8e-6, the weight decay is 1e-5, and warmup steps to  $5\%$  of all training steps. The maximum length of input texts is 2048. Different from FinMA-7B, it can only be distributed fine-tuned on 128 A100 40GB GPUs.
5 FLARE: Financial Evaluation Benchmark
Based on FIT, we design our financial natural language understanding and prediction evaluation benchmark (FLARE). We randomly select validation sets from FIT to select the best model checkpoint, and test sets for evaluation. Compared with the existing benchmark FLUE (Sanh et al., 2022), FLARE covers financial prediction tasks in addition to NLP tasks<sup>11</sup>. We believe it is vital to include financial prediction tasks such as stock movement prediction, to comprehensively evaluate the performance of LLMs on the practical applications of the financial domain. We show the data statistics of validation, and test set for each dataset in Table 3. Following previous methods Li et al. (2023); Shah et al. (2022),
Table 3: The details of our evaluation datasets. To compare the performance with BloombergGPT whose test data is not openly released, we keep the same numbers and data distributions of our test datasets with that of BloombergGPT. To further evaluate the emergent and generalization ability of the LLMs, we only adopt the test data of FOMC, FINER-ORD, ECTSum, EDTSum, German, and Australian for evaluation on the FLARE. For ConvFinQA, we take each turn of the conversations as the instruction, whose number would be different from the number of conversations.
Data	Task	Valid	Test	Evaluation
FPB (Malo et al., 2014)	sentiment analysis	775	970	F1, Accuracy
FiQA-SA (Maia et al., 2018)	sentiment analysis	188	235	F1
Headlines (Sinha and Khandait, 2021)	news headline classification	1,141	2,283	Avg F1
NER (Alvarado et al., 2015)	named entity recognition	103	980	Entity F1
FiNER-ORD (Shah et al., 2023b)	named entity recognition	-	1080	Entity F1
FinQA (Chen et al., 2021)	question answering	883	1,147	EM Accuracy
ConvFinQA (Chen et al., 2022)	question answering	2,210	1,490	EM Accuracy
BigData22 (Soun et al., 2022)	stock movement prediction	798	1,470	Accuracy, MCC
ACL18 (Xu and Cohen, 2018)	stock movement prediction	2,560	3,720	Accuracy, MCC
CIKM18 (Wu et al., 2018)	stock movement prediction	431	1,140	Accuracy, MCC
ECTSum Mukherjee et al. (2022)	text summarization	-	495	ROUGE, BERTScore, BARTScore
EDTSum Zhou et al. (2021)	text summarization	-	2000	ROUGE, BERTScore, BARTScore
German Hofmann (1994)	credit scoring	-	1000	F1, MCC
Australian Quinlan (1987)	credit scoring	-	690	F1, MCC
FOMC Shah et al. (2023a)	hawkish-dovish classification	-	496	F1, Accuracy
we evaluate the performance of the sentiment classification task on FPB and FiQA-SA datasets, with the accuracy (ACC) and weighted F1 Score (F1). The performance of the news headline classification task is evaluated with the weighted averages of F1 score over all nine categories (Avg F1). For the performance of the NER task, we evaluate with the entity-level F1 score (Entity F1). The performance on the question-answering task is evaluated with exact match accuracy (EM Acc). For summarization tasks such as ECTSum and EDTSum, we assess the relevance and factuality of generated summaries with the ground truth using metrics such as ROUGE score Lin (2004), BERTScore Zhang et al. (2019), and BARTScore Yuan et al. (2021). As for the financial prediction task, following previous methods (Xu and Cohen, 2018; Xie et al., 2023), we evaluate the performance with the accuracy
11Following BloombergGPT, we don't include the structure boundary detection task included in FLUE because it is hard to be converted into the instruction following task.
(ACC) and the Matthews correlation coefficient (MCC) for stock price movement prediction, and the F1 score with MCC for credit scoring. Although Macro-F1 is more fair for the unbalanced dataset, in this paper we adopt the same metrics following previous methods to make a fair comparison.
6 Experiments on FLARE
The proposed FIT and FLARE allow to train, select the model, and evaluate the performance of LLMs on financial understanding and predictions. In this section, we investigate how powerful the FIT-fine-tuned FinMA and other LLMs are on FLARE. We compare FinMA with following LLMs: 1) BloombergGPT (Wu et al., 2023). The only large language model with 50B parameters pre-trained with the financial texts. 2) GPT-4 (OpenAI, 2023). A powerful instruction following large language model with around 1T parameters proposed by OpenAI. 3) ChatGPT. A instruction following large language model with 175B parameters from OpenAI. 4) Vicuna-7B (Zhang et al., 2022). An instruction following large language model by fine-tuning LLaMA-7B.
Following previous methods (Wu et al., 2023; Li et al., 2023), we report the 20-shot performance of BloombergGPT and the 5-shot performance of other baseline methods on the FIN dataset. We report the 5-shot performance of BloombergGPT on FPB and FiQA-SA. We report the 5-shot performance of all baselines on the Headlines dataset. For the remaining results, we report the zero-shot performance. The results of some baselines are based on human evaluations, since LLMs without fine-tuning will fail to generate answers pre-defined in the given instruction. All results of FinMA are conducted on zero-shot and can be automatically evaluated.
Table 4: The zero-shot and few-shot performance of different LLMs on the FLARE benchmark. Results of BloombergGPT, ChatGPT, and GPT4 on FPB, FiQASA, Headlines, NER, FinQA, and ConvFinQA are referenced from the paper (Li et al., 2023). The results of BloombergGPT are referenced from the original paper Wu et al. (2023). Test datasets were built to have the same data distribution with that of BloombergGPT and the performance of FinMA was directly compared with BloombergGPT following the previous method (Li et al., 2023). All results via our evaluations are the average of three runs. “-” represents the result that is currently unable to yield due to model size or availability, and “*” represents the result from the previous paper.
Dataset	Metrics	ChatGPT	GPT4	BloombergGPT	Vicuna7B	FinMA7B	FinMA7B-trade	FinMA7B-full	FinMA30B
FPB	F1	0.78*	0.78*	0.51*	0.29	0.94	0.03	0.94	0.88
Acc	0.78*	0.76*	-	0.26	0.94	0.12	0.94	0.87
FiQA-SA	F1	0.60	0.80	0.75*	0.32	0.85	0.16	0.82	0.87
Headlines	AvgF1	0.77*	0.86*	0.82*	0.60	0.97	0.28	0.97	0.97
NER	EntityF1	0.77*	0.83*	0.61*	0.12	0.59	0.00	0.64	0.62
FINER-ORD	EntityF1	0.28	0.77	-	0.00	0.00	0.00	0.00	0.00
FinQA	EmAcc	0.58*	0.63*	-	0.00	0.06	0.00	0.04	0.11
ConvFinQA	EmAcc	0.60*	0.76*	0.43*	0.00	0.25	0.00	0.20	0.40
BigData22	Acc	0.53	0.54	-	0.44	0.45	0.45	0.51	0.47
MCC	-0.025	0.03	-	-0.05	0.02	0.00	0.02	0.04
ACL18	Acc	0.50	0.52	-	0.50	0.49	0.49	0.51	0.49
MCC	0.005	0.02	-	0.02	-0.01	0.03	0.03	0.00
CIKM18	Acc	0.55	0.57	-	0.44	0.43	0.43	0.50	0.43
MCC	0.01	0.02	-	-0.03	-0.02	-0.003	0.08	-0.05
EDTSUM	Rouge-1	0.17	0.2	-	0.22	0.09	0.05	0.13	0.17
Rouge-2	0.08	0.09	-	0.10	0.04	0.02	0.06	0.08
Rouge-N	0.13	0.15	-	0.17	0.08	0.05	0.10	0.14
BertScore	0.66	0.67	-	0.61	0.56	0.51	0.38	0.54
BartScore	-3.64	-3.62	-	-4.13	-6.12	-6.91	-5.71	-5.24
ECTSUM	Rouge-1	0.00	0.00	-	0.00	0.00	0.00	0.00	0.00
Rouge-2	0.00	0.00	-	0.00	0.00	0.00	0.00	0.00
Rouge-N	0.00	0.00	-	0.00	0.00	0.00	0.00	0.00
BertScore	0.00	0.00	-	0.00	0.00	0.00	0.00	0.00
BartScore	-5.18	-5.18	-	-5.18	-5.18	-5.18	-5.18	-5.18
German	F1	0.20	0.55	-	0.52	0.17	0.52	0.17	0.53
MCC	-0.10	-0.02	-	0.00	0.00	-0.07	0.00	-0.07
Australian	F1	0.41	0.74	-	0.26	0.41	0.26	0.41	0.46
MCC	0.00	0.47	-	0.00	0.00	0.00	0.00	-0.01
FOMC	F1	0.64	0.71	-	0.19	0.49	0.10	0.49	0.43
Acc	0.6	0.69	-	0.28	0.47	0.25	0.46	0.53
Overall Performance. For financial NLP tasks, as shown in Table  $4^{12}$ , our fine-tuned model FinMA significantly outperform other LLMs on FPB, FiQA-SA and Headlines datasets, showing the importance of domain specific instruction tuning on improving the performance of LLMs in the specific domain. For example, FinMA-30B outperforms GPT-4 by  $10\%$  F1 score, and BloombergGPT by  $37\%$  F1 score on the FPB dataset. On the NER dataset, FinMA-7B also outperforms BloombergGPT and other LLMs, and achieve competitive results compared with ChatGPT and GPT-4. For FinQA and ConvFinQA which requires complex numeric reasoning, there is a large gap between the performance of GPT and FinMA. As reported in existing studies (Touvron et al., 2023; Lewkowycz et al., 2022), LLaMA includes no mathematical datasets for pre-training, resulted in poor performance on the mathematical benchmark datasets such as GSM8K (Cobbe et al., 2021). Our results are also consistent with previous studies which reveals that LLaMA with larger parameters present better performance on mathematical benchmark datasets. The performance of FinMA-30B is significantly better than that of FinMA-7B on FinQA and ConvFinQA. This finding indicate the importance of numeric reasoning for financial question answering, which could be the potential direction for advancing LLMs in the finance area. Despite the strengths exhibited by our methods on known tasks, they underperform relative to GPT-4 on unseen tasks such as FINER-ORD, ECTSUM, ECTSUM, and FOMC. This performance gap suggests that a more diverse set of domain-specific tasks is needed for effective fine-tuning. Specifically, in FINER-ORD and ECTSUM, we employ a complex prompt design that requires the model to generate label sequences directly. The results indicate that our fine-tuned models consistently fail to produce outputs in the desired formats. While models like ChatGPT and GPT-4 demonstrate some capability for token labeling in FINER-ORD, they also struggle to generate sentence labels for ECTSUM, particularly when faced with longer contextual information. In contrast to existing financial benchmarks, FLARE offers a more comprehensive suite of both generation and classification tasks, thereby providing a fuller assessment of Large Language Models' capabilities in the realm of financial NLP.
For financial prediction tasks, all LLMs including FinMA, ChatGPT and GPT-4 struggle in stock movement prediction as previous methods $^{13}$ . While FinMA-7B-trade has been fine-tuned specifically for the task of stock movement prediction, the observed performance gains in this area are marginal at best. After fine-tuned with both NLP and financial prediction tasks, FinMA-7B-full can achieve a significantly better performance on ACL18 dataset compared with ChatGPT and GPT-4. However, it still presents almost zero MCC on the other two datasets like ChatGPT and GPT-4. Similarly, all tested methods, with the exception of GPT-4, exhibited either zero or negative Matthews Correlation Coefficient (MCC) values on the credit scoring task. This highlights their limitations in accurately forecasting individual default risks. While GPT-4 shows a marked improvement in performance on the Australian dataset, it fails to register a positive MCC on the German dataset. Such limitations can be attributed to the challenges associated with the tabular data input and the highly imbalanced label distribution inherent to credit scoring tasks. This indicates the complexity and challenging of the financial prediction tasks in FLARE. Compared with existing financial benchmarks that focusing on NLP tasks, FLARE provides exciting opportunities for the improvement of LLMs on the fundamental of financial academic studies and applications. It also demonstrates the importance of multiple task learning in the financial domain for LLMs, which can provide essential domain knowledge and skills to handle complex applications in this area.
Further Analysis. We further analyze the influences of model size, and instruction tuning data on the performance of LLMs on different tasks. FinMA-30B has no significantly better performance than FinMA-7B on most NLP tasks and the stock movement prediction task. Apparently, the quality of the instructions rather than the model size is critical for the performance of these tasks. For the complex question-answering tasks such as ConvFinQA, as shown in Table 4, the larger LLaMA model generally has better performance. Particularly, Vicuna-7B based on LLaMA-7B has the worst performance, which are also consistent with previous findings (Cobbe et al., 2021) that LLaMA with larger parameters presents better performance on mathematical benchmark datasets. In contrast, for generation tasks such as abstractive summarization (EDTSUM), Vicuna-7B presents the best performance while fine-tuned models shows a decreased performance on almost all metrics. This may indicate that fine-tuning with only classification tasks could lead to better classification performance but also compromise the generation ability. For financial prediction tasks and NLP tasks that are not
12For the performance of other LLMs, please see Appendix.
13 For the performance of traditional methods, please see Appendix.
included in the fine-tuning instruction dataset of FinMA-7B and FinMA-30B, i.e., FOMC, FINER-ORD, Australian, and ACL18, our models present limited improvement. While it demonstrates some degree of emerge ability, the performance gap compared with GPT-4 indicates the needs for further optimization. However, FinMA-7B-full fine-tuned with both NLP and prediction datasets, has shown significantly better performance on financial prediction datasets, and comparable performance on NLP tasks with FinMA-7B and GPT-4. This indicates the potential of LLMs to be further adapted and applied directly on financial prediction tasks via pre-training and fine-tuning on diversified domain datasets.
Despite the positive contributions of this study, we recognize the following limitations: 1) Model and Training Constraints: We only present FinMA models up to 30B. Due to computational constraints, FinMA-30B has not been fine-tuned on the full dataset. 2) Complex Task Performance: FinMA, due to the limitation of the backbone model LLaMA, struggles with tasks requiring quantitative reasoning, such as financial question answering, and the difficult financial prediction task. 3) Resource Constraints and Generalizability: The development of FinMA, FIT, and FLARE is influenced by available resources and handcrafted instructions, potentially affecting model diversity and generalizability. The maximum input size of FinMA is also limited by the maximum input texts that can be handled by the backbone model LLaMA. 4) Potential Negative Impacts: While our study primarily focuses on the positive aspects and advancements of financial language understanding models, it is important to acknowledge the potential negative impacts associated with their use, such as the spread of financial misinformation or unethical market influence. We recommend using our method for academic research only. $^{14}$
In this work, we presented PIXIU, encompassing the first open-sourced financial large language model FinMA, the instruction tuning dataset FIT, and the evaluation benchmark FLARE. Through extensive evaluation, we demonstrated the effectiveness of FinMA across various financial tasks, showing the potential of domain-specific instruction tuning of large language models in the financial domain. However, challenges such as improving performance on complex tasks and addressing resource constraints remain. Our open-source contribution aims to facilitate further research and innovation in financial language understanding, prediction, and LLMs, toward more useful and safe LLMs in the field of finance.
The PIXIU project is mainly supported by ChanceFocus AMC and partially supported by the National Key Research and Development Program of China (No.2021ZD0113304) and the General Program of Natural Science Foundation of China (NSFC) (Grant No.62072346). Additional funding was provided by the Key R&D Project of Hubei Province (Grant No.2021BBA099, No.2021BBA029) and by the Joint & Laboratory on Credit Technology.
Julio Cesar Salinas Alvarado, Karin Verspoor, and Timothy Baldwin. 2015. Domain adaptation of named entity recognition to support credit risk assessment. In Proceedings of the Australasian Language Technology Association Workshop 2015. 84-90.
Dogu Araci. 2019. Finbert: Financial sentiment analysis with pre-trained language models. arXiv preprint arXiv:1908.10063 (2019).
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877-1901.
<sup>14</sup>For a detailed ethical and legal statement concerning this work, please see Appendix.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023).
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-Instruct: Aligning Language Model with Self Generated Instructions. arXiv preprint arXiv:2212.10560 (2022).
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 (2021).
Huizhe Wu, Wei Zhang, Weiwei Shen, and Jun Wang. 2018. Hybrid deep sequential modeling for social text-driven stock prediction. In Proceedings of the 27th ACM international conference on information and knowledge management. 1627-1630.
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564 (2023).
Qianqian Xie, Weiguang Han, Yanzhao Lai, Min Peng, and Jimin Huang. 2023. The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challenges. arXiv preprint arXiv:2304.05351 (2023).
Yumo Xu and Shay B Cohen. 2018. Stock movement prediction from tweets and historical prices. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1970-1979.
Yi Yang, Mark Christopher Siy Uy, and Allen Huang. 2020. Finbert: A pretrained language model for financial communications. arXiv preprint arXiv:2006.08097 (2020).
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems 34 (2021), 27263-27277.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 (2022).
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations.
Zhuosheng Zhang, Hanqing Zhang, Keming Chen, Yuhang Guo, Jingyun Hua, Yulong Wang, and Ming Zhou. 2021. Mengzi: Towards lightweight yet ingenious pre-trained models for chinese. arXiv preprint arXiv:2110.06696 (2021).
Zhihan Zhou, Liqian Ma, and Han Liu. 2021. Trade the Event: Corporate Events Detection for News-Based Event-Driven Trading. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP* 2021. 2114–2124.
A Instructions
Table 5 presents all the prompts for each dataset in the FLARE benchmark and FIT instruction dataset.
B Traditional Methods for Stock Movement Forecasting
In the context of stock movement prediction, traditional models, as summarized in Table 6, have long been employed but face significant challenges in achieving consistently high levels of accuracy. This underscores the inherent difficulty of the task at hand. In contrast, Large Language Models (LLMs) introduce a level of adaptability by learning from multiple tasks, although they also have limitations such as numeric understanding and reasoning. The task's difficulty is thus a common challenge for both traditional models and LLMs.
Table 6: Movement prediction performance of non-LLM models vs FinMA, measured with the accuracy (ACC) and the Matthews correlation coefficient (MCC). The best of non-LLM models is in red and the best of all is in bold.
Method	BIGDATA22	ACL18	CIKM18
ACC	MCC	ACC	MCC	ACC	MCC
Logistic regression (LR)	0.53	0.02	0.52	0.04	0.53	-0.04
Random forest (RF)	0.47	-0.11	0.52	0.03	0.54	0.01
LSTM	0.51	0.01	0.53	0.06	0.53	0.02
Attention LSTM (ALSTM)	0.49	-0.03	0.52	0.04	0.53	-0.01
Adv-ALSTM	0.50	0.01	0.53	0.07	0.54	0.02
DTML	0.52	0.07	0.58	0.18	0.54	-0.00
XGBoost	0.52	-0.04	0.49	-0.02	0.58	0.07
XGBReffressor	0.46	-0.13	0.50	-0.01	0.53	-0.03
ALSTM-W	0.48	-0.01	0.53	0.08	0.54	0.03
ALSTM-D	0.49	0.01	0.53	0.07	0.50	-0.04
StockNet	0.53	-0.00	0.54	-0.03	0.52	-0.02
SLOT	0.55	0.10	0.59	0.21	0.56	0.09
FinMA-7B	0.48	0.04	0.50	0.00	0.56	-0.02
FinMA-30B	0.47	0.04	0.49	0.00	0.43	-0.05
FinMA-7B-full	0.49	0.01	0.56	0.10	0.53	-0.03
C Performance of BERT based Large Language Models
Table 7 presents previously reported performance of pre-trained language models (PLMs) including FinBERT and FLANG-BERT across three selected tasks under the FLARE benchmark. Although both models demonstrate impressive results in specific performance metrics, it's crucial to note that they have been pre-trained with large-scale financial texts and utilize task-specific headers for different tasks, which is the main reason for their better performance. However, compared with LLMs, these PLMs can't be adaptable to unseen tasks without supervised fine-tuning, thus having the bottleneck of adapting to a multi-task environment and zero-shot learning. For LLMs, our results suggest their performance still underperforms these finely tuned PLMs on some tasks, but offer greater flexibility, adaptability, and zero-shot ability. They have the advantage of learning directly from prompts and being more flexible in handling a wide array of tasks, even without the need for labeled training data. The underperformance of LLMs compared with PLMs also highlights the need for domain-specific pre-training of LLMs in the future, to further improve their performance in the specific domain.
These observations align with recent studies, such as Shah and Chava (2023), which also observed that zero-shot LLMs like ChatGPT offer respectable performance across financial tasks without the need for labeled data. The recent work by Ni et al. (2023), also highlights the challenges of fine-tuned PLMs adapting to unseen and multiple tasks.
Table 7: Results of BERT-based Encoder-Decoder Large Language Models (LLMs) across multiple tasks reported in previous papers.
Method	FPB	Headline	NER
Accuracy	AvgF1	F1
FinBERT	0.872	0.968	0.8
FLANG-BERT	0.912	0.972	0.83
D Performance of Previous Large Language Models