Is ChatGPT a Financial Expert?  Evaluating Language Models on Financial Natural Language Processing
Authors: Yue Guo
Zian Xu
Yi Yang
Venue: EMNLP 2023 Findings (2023)
Keywords: Financial Natural Language Processing
Large Language Models
ChatGPT

Abstract: The emergence of Large Language Models (LLMs), such as ChatGPT, has revolutionized general natural language preprocessing (NLP) tasks. However, their expertise in the financial domain lacks a comprehensive evaluation. To assess the ability of LLMs to solve financial NLP tasks, we present FinLMEval, a framework for Financial  Language Model Evaluation, comprising nine datasets designed to evaluate the performance of language models.
This study compares the performance of fine-tuned auto-encoding language models (BERT, RoBERTa, FinBERT) and the LLM ChatGPT. Our findings reveal that while ChatGPT demonstrates notable performance across most financial tasks, it generally lags behind the fine-tuned expert models, especially when dealing with proprietary datasets. We hope this study builds foundation evaluation benchmarks for continuing efforts to build more advanced LLMs in the financial domain.

Is ChatGPT a Financial Expert? Evaluating Language Models on Financial Natural Language Processing
Yue Guo Zian Xu Yi Yang
The Hong Kong University of Science and Technology
yguoare@onnect.ust ust. zxubz@connect .st. hk imyiyang@ust.he hk
Abstract
workers. Therefore, how much essential knowledge and skills are acquired during the learning process remains uncertain. As a result, a comprehensive investigation is necessary to assess its performance on financial NLP tasks.
The emergence of Large Language Models (LLMs), such as ChatGPT, has revolutionized general natural language preprocessing (NLP) tasks. However, their expertise in the finan- cial domain lacks comprehensive evalua- tion. To assess the ability of LLMs solve financial NLP tasks, we present FinLMEval, framework for Financial Language Model Evaluation, comprising nine datasets designed evaluate the performance language mod- els. This study compares the performance of encoder-only language models and the decoder- only language models. Our findings reveal that while some decoder-only LLMs demonstrate notable performance across most financial tasks via zero-shot prompting, they generally lag be- hind the fine-tuned expert models, especially when dealing with proprietary datasets. We hope this study provides foundation evaluations for continuing efforts to build more advanced LLMs the financial domain.
To fill this research gap, we are motivated evaluate language models on financial tasks com- prehensively. For doing so, we propose a frame- work for Financial Language Model Evaluation (FinLMEval).. We collected nine datasets on fi- nancial tasks, five from public datasets evaluated before. However, for those public datasets, it is possible that their training process or provided by the model users as online feedback. To eliminate this issue, We used four proprietary datasets on different financial tasks: financial sentiment classification (FinSent), environmental, social, and corporate governance classification (ESG), hawward-lookig statements classification (FLS), and question -nnwwernngnn clas- sification (QA) for evaluation.
are leaked during the
1 Introduction
In the evaluation benchmark, we evaluate the encoder-only language models with supervised fine-tuning, with representatives of BERT (Devlin al., 2019), RoBERTa (Liu et al., 2019), FinBERT (Yang et al., 2020) and FLANG (Shah et al., 2022). We then compare the encoder-only models with the decoder-only models, with representatives of Chat- GPT (Ouyang et al., 2022), GPT-4 (OpenAI, 2023), PIXIU (Xic et al.. 2023), LLAMA2-7B (Touvron et al., 2023b) and Kioomberg-GPP (Wu et al., 2023) by zero-shot prompting. Besides, we evaluate the efficacy of in-context learning of ChatGPT with different in-context sample selection strategies. Experiment results show that (1) the fine-tuned task-specific encoder-only model generally per- forms better than decoder-only models the fi- nancial tasks, even if decoder-only models have much larger model size and have gone through more pre-training and instruction tuning or RLHF; (2) when the supervised data is insufficient, the
Recent progress in natural language processing (NLP) demonstrates that large language models (LLMs), like ChatGPT, achieve impressive results on various general domain NLP tasks. Those LLMs are generally trained by first conducting self-supervised training the unlabeled text (Rad- ford al., 2019; Brown al., 2020; Touvron al., 2023a) and then conducting instruction tun- ing (Wang et al., 2023; Taori et al., 2023) or rein- forcement learning from human feedback (RLHF) (Ouyang et al., 2022) to let them perform tasks following human instructions.
Financial NLP, in contrast, demands specialized knowledge and specific reasoning skills to tackle tasks within the financial domain. However, for general language models like ChatGPT, their self- supervised training is performed on the text from various domains, and the reinforcement learning feedback they receive generated by non-expert

---

FinLMEval Framework
Evaluating Encoder-only Models
Evaluating Decoder-oni Models
Fine-tune the parameters of the encoder-only models with the training sets output the correct label.
Get response with Prompt:
Perform financial sentiment classification. Classify the following sentence into "neutral'' positive', or 'negative' class. Only provide label output. (placehalder)
pesitive
Taus
The sentence: Energy- Related investments 277%
are some marplec josompic sentente (oompien
BERT
increase quarterly earnings The label:
the lubal jube nf The wenenon Feeegy
telated movammenta reported quarto oa carnings
Zero-shot Learning
Energy-Rellatee investments reported 277% Increase quarterly earnings.
Expected output:
In-context Learning
Financial Tasks
Sentiment
Forward-lookin statements
Environmental, social, and governance
QA
News Headlines
NER
Monetary policy
Figure 1: The framework of financial language model evaluation (FinLMEval).
zero-shot decoder-only models have more advan- tages than fine-tuned encoder-only models; (3) the performance gap between fine-tuned encoder-only models and zero-shot decoder-only models is more significant on private datasets than the publicly available datasets; (4) in-context learning only effective under certain cireumstances.
perform better in financial tasks. Recent work such as FLUE (Shah al., 2022) has been introduced to benchmark those language models in the finance domain. However, the capability of more advanced LLMs, like ChatGPT and GPT-4, has not been benchmarked, especially on proprietary datasets. In this work, in addition to the public tasks used in FLUE, we newly include four proprietary tasks in FinLMEval and conduct comprehensive evalua- tions for those financial language models.
To summarize, we propose an evaluation frame- work for financial language models. Compared to previous benchmarks in the financial domain like FLUE (Shah et al., 2022), our evaluation includes four new datasets and involves more advanced LLMs like ChatGPT. We show that even the most advanced LLMss still fall behind the fine-tuned ex- pert models. We hope this study contributes to the continuing efforts to build more advanced LLMs in the financial domain.
Methods
We compare two types of models in FivLMEval: the Transformers encoder-only models that require fine-tuning on the labeled dataset, and decoder-only models that prompted with zero-shot or few- shot in-context instructions. Figure provides outline of evaluation methods of Livll.
2
Related Works
3.1 Encoder-only Models
The utilization of language models financial NLP is a thriving research area. While some general do- main language models, like BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), GPT (Brown et al., 2020; OpenAI, 2023) and LLAMA (Tou- vron et al., 2023a,b) have been applied to finan- cial NLP tasks, financial domain models like Fin- BERT (Araci, 2019; Yang al., 2020; Huang et al.. 2023), FLANG (Shah al., 2022), PIXIU (Xie al., 2023), InvestLM (Yang al., 2023) and BloombergGPT (Wu et al., 2023) are specifically designed to contain domain expertise and generally
Our experiments explore the performance of vari- ous notable encoder-only models: BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), FinBERT (Yang et al., 2020) and FLANG (Shah et al., 2022). BERT and RoBERTa are pre-trained on general domain corpora, while FinBERT and FLANG are pre-trained on substantial financial domain cor- pus. We fine-tune the language models on specific tasks. Following the fine-tuning process, inference can be performed on the fine-tuned models for spe- cific applications.

---

train
# test
source
description
FinSent
8996
1000
Finuncial sentiment classification dataset from analyst reports.
FPB
2453
1000
(Malo al., 2014)
Sentiment classification dataset from financial news.
FIQA SA
973
200
(FiQA)
Aspect-based financial sentiment unalysis.
ESG
3000
1000
Environmental, social, and corporate governance classification dataset. forkand-looking statements classification dataset from corporute reports. Classification on the validity aresstonnanswwringg pairs.
FLS
2600
1000
QA
868
200
Headlines
9570
1000
(Sinha and Khandait, 2020)
Mulitple tasks classification datasct from news headlines.
NER
14041
1000
(AIvarado et al., 2015)
Named entity recognition on financial agreements.
FOMC
1831
450
(Shah al., 2023)
Hawkish-dovish monetary policy classification from FOMC documents.
Table 1: The summarization of nine datasets in FivL. FPB, FIQA SA, Headlines, NER and FOMC are from public datasets, and FinSent, ESG, FLS and QA are newly collected and not released before.
3.2 Decoder-only Models
4 Datasets
Our evaluation relies on nine datasets designed to evaluate the financial expertise of the models from diverse perspectives. Table overviews the number of training and testing samples and the source information for each dataset. Below, we provide an introduction to each of the nine datasets.
We also evaluate the performance of various pop- ular decoder-only language models: ChatGPT (Ouyang al., 2022), GPT-4 (OpenAI, 2023), PIXIU (Xie et al., 2023), LLAMA2-7B (Touvron et al., 2023b) and Bloomberg-GPT (Wu et al., 2023). ChatGPT and GPT-4, developed by Ope- nAI, are two advanced LLMs that showcase ex- ceptional language understanding and generation abilities. The models are pre-trained on a wide array of textual data and reinforced by human feed- back. PIXIU is a financial LLM based on fine- tuning LLAMA (Touvron et al., 2023a) with in- struction data. LLAMA2 a popular open-sourced LLM pre-trained on extensive online data, and BloombergGPT is an LLM for finance trained on a wide range of financial data. As the model size of the evaluated decoder-only models is extremely large, they usually do not require fine-tuning the whole model downstream tasks. Instead, the decoder-only models provide answers via zero-shot and fhw- in-context prompting.
FinSent is a newly collected sentiment classifica- tion dataset containing 10,000 manually annotated sentences from analyst reports of S&P 500 firms. FPB Sentiment Classification (Malo al., 2014) classic sentiment dataset of sentences from financial news. The datasct consists of 4840 sentences divided by the agreement rate of 5-8 an- notators. We use the subset of 75% agreement. FiQA SA (FiQA) is aspect- based financial sen- timent analysis dataset. Following the "Sentences for QA-M" method in (Sun al., 2019), for each (sentence, target, aspect) pair, we transform the sentence into the form "what do you think the (aspect) of (target]? (sentence)" for classification. ESG evaluates organization's considerations environmental, social, and corporate gover- nance. We collected 2,000 manually annotated sentences from firms' ESG reports and annual re- ports.
We conduct zero-shot prompting for all decoder- only models. We manually write the prompts for every task. An example of prompts for the senti- ment classification task is provided in Figure 1, and the manual prompts for other tasks are provided in Appendix A. Furthermore, to evaluate whether few-shot in-context learning can improve the model performance, we also conduct in-context learning experiments on ChatGPT. We use two strategies to select the in-context examples for few-shot in- context learning: random and similar. The former strategy refers random selection, and the lat- ter selects the most similar sentence regarding the query sentence. All in-context examples are se- lected from the training set, and one example is provided from each label class.
FLS, the forward-looking statements, are belief' and opinions about firm's future events or re- sults. FLS dataset, aiming classify whether sentence contains forward-looking statements, con- tains 3,500 manually annotated sentences from the Management Discussion and Analysis section of annual reports of Russell 3000 firms.
QA contains question-answerinng answering pairs extracted from earnings conference call transcripts. The goal of the dataset is to identify whether the answer is valid to the question.
Headlines (Sinha and Khandait, 2020) a dataset for the commodity market that analyzes

---

Encoder-only Models
Decoder-only Models
Datasets
FLANG-
LLAMA2-
Bloomberg-
BERT
RoBERTa
FinBERT
ChatGPT
GPT-4
PIXIU
BERT
7B
GPT
FinSent
0.841
0.871
0.851
0.849
0.782
0.809
0.800
0.243
FPB
0.914
0.934
0.912
0.881
0.869
0.905
0.965
0.339
0.511
FIQA SA
0.750
0.875
0.805
0.695
0.898
0.920
0.930
0,480
0.751
ESG
0,931
0.956
0.958
0.925
0.477
0.626
0,509
0.209
FLS
0,875
0,862
0.882
0.861
0,652
0.565
0.275
0,365
QA
0.865
0.825
0.825
0.785
0.695
0.775
0.680
0.625
Hcadlines-PDUU
0.937
0.947
0.956
0.940
0.889
0.878
0.842
0,411
Headlines-PDo
0.978
0.979
0.981
0.978
0.936
0.947
0.702
0.053
Hcadlines-PDD
0.954
0.961
0.960
0.956
0.896
0.900
0.763
0,382
Headlines-Pi
0.974
0.964
0.976
0.977
0,225
0.105
0.753
0.966
Hea
0,996
0,993
0.997
0,806
0.838
0.902
0,346
Headlines-FI
0.976
0.964
0.976
0.974
0.711
0.780
0.981
0,048
Headlines-PS
0.905
0.918
0.924
0.906
0.630
0.811
0.776
0.546
NER
0.980
0.981
0,964
0.978
0.748
0.707
0.749
0.714
0.608
FOMC
0.587
0.611
0.602
0.602
0.633
0.729
0.522
0.349
Average 0.897
0.909
0.905
0.907
0.723
0.753
0.739
0,405
Table 2: The results of fine-tuned encoder-only models and zero-shot decoder-only models in 9 financial datasets. The results, except the NER dataset, are measured micro-F1 score. NER is measured in accuracy. Although some zero-shot decoder-only models achieve considerate results in most cases, the fine-tuned encoder-only models usually perform better than decoder-only models.
news headlines across multiple dimensions. The tasks include the classifications of Price Direction Up (PDU), Price Direction Constant (PDC), Price Direction Down (PDD), Asset Comparison(AC), Past Information (PI), Future Information (FI), and Price Sentiment (PS).
fine-tune the model for three epochs. Other hyper- parameters remain the default in Trainer.
Decoder-only models setups. In the zero-shot setting, for ChatGPT and GPT-4, We use the "gpt- 3.5-turbo" and "gpt-4" model API from OpenAI, respectively. We set the temperature and top_p as 1, and other hyperparameters default by Ope- nAI API. The ChatGPT results are retrieved from the May 2023 version, and the GPT-4 results are retrieved in August 2023. For PIXIU and LLAMA2, we use the "ChanceFocus/fmma-777 nlp" and "meta-llama/llama-22
NER (Alvarado et al., 2015) a named entity recognition dataset financial agreements.
FOMC (Shah et al., 2023) aims to classify the stance for the FOMC documents into the tightening or the easing of the monetary policy.
Among the datasets, FinSent, ESG, FLS, and QA are newly collected proprietary datasets.
models from Huggingface. The model responses are generated greedily. All prompts we used in the zero-shot setting are shown in Appendix A. Besides, as the BloPmbergGPT (Wu et al., 2023) is not publicly available, we directly adopt the results from the original paper.
5 Experiments
This section introduces the experiment setups and reports the evaluation results.
5.1 Model Setups
For in-context learning, we conduct two strate- gies for in-context sample selection: random and similar. We select one example from each label with equal probability weighting for random sam- ple selection. For similar sample selection, we get the sentence embeddings by SentenceTramm Transformer (Reimers and Gurevych, 2019) "all-MiniLM--L" v2" model and use cosine similarity as the measure of similarity. Then, we select the sentences with the highest similarity with the query sentence as the
Encoder-only models setups. We use the BERT (hase,uncased), RoBERTa (base), FinBERT (pre-
train), and FLANG-BERT from Huggingface!, and the model fine-tuning is implemented via Trainer? 2
For all tasks, we fix the learning rate as 2 x 10-5, weight decay as 0.01, and the batch size as 48. We randomly select 10% examples from the training set the validation set for model selection and
wttpaaa..o...ctcaaaaerrreeeeeeee mai_cllasses/trainer
Phttps: httpp::/wwwwbbrrt..r. sbert.net/

---

ChatGPT
prietary datasets than publicly available ones. For example, the FinSent, FPB, and FiQA SA datasets are comparable and about financial sen- timent classification. However, zero-shot LLMs perform the worst on the proprietary dataset Fin- Sent. The performance gaps between fine-tuned models and zero-shot LLMs are also more signifi- cant other proprietary datasets (ESG, FLS, and QA) than the public dataset.
Datasets
zero
ic-ran
ie-sim
FinSent
0.782
0.761
0.761
FPB
0.869
0.832
0.844
FIQA SA
0.898
0.891
0.891
ESG
0.477
0.726
0.800
FLS
0.652
0.673
0.636
QA
0.695
0.660
0.675
Headlines- PDU
0.889
0.839
0.765
Table 3 compares the zero-shot and in-context few-shot learning of ChatGPT. ChatGPT, the zero-shot and few-shot performances are com- parable in most cases. When zero-shot prompting is ineffective, adding demonstrations can improve ChatGPT's performance by clarifying the task, the results of ESG and Hcadlines-PP PI tasks show. Demonstrations are ineffective for easy and well- defined tasks, such sentiment classifications and Headlines (PDU, PDC, PDD, AC, and FT), as the zero-shot prompts clearly instruct ChatGPT.
Headlines-PPi PPC
0.936
0.323
0,413
Hcadlines-PDD
0.896
0,816
0.788
Headlines-Pl
0.225
0.768
0.844
Headlines-AC
0.806
0.576
0.597
Headlines-Fl
0.711
0.606
0.592
Hcadlines-PS
0.630
0.690
0,729
NER 0.748
0.784
0.793
FOMC
0.633
0.672
0.650
Average
0.723
0.708
0.719
Table 3:
The results ChatGPT zero-shot and in- context few-shot learning. Zero, ic-ran, and iim repre- sent zero-shot learning, in-context learning with random sample selection, and in-context learning with similar sample selection. The zero-shot and few-shot perfor- mances are comparable in most cases.
Conclusions
We present FinL.Eval, an evaluation framework for financial language models. FinLMEval com- prises nine datasets from the financial domain, and we conduct the evaluations various popular lan- guage models. Our results show that fine-tuning expert encoder-only models generally perform bet- than the decoder-only LLMs on the financial NLP tasks, and adding in-context demonstrations barely improves the results. Our findings sug- gest that there remains room for enhancement for more advanced LLMs in the financial NLP field. Our study provides foundation evaluations for con- tinued progress in developing more sophisticated LLMs within the financial sector.
in-context examples. The prompts for in in-conexxt learning directly extended from the correspond- ing zero-shot prompts, with the template shown in Figure 1.
5.2 Main Results
Table 2 compares the results of the fine-tuned encoder-only models and zero-shot decoder-oni onll models in financial datasets. We have the follow- ing findings:
In 6 out of 9 datasets, fine-tuned encoder- only models can perform better than decoder- only models. The decoder-only models, especially those that have experienced RLHF or instruction- tuning, demonstrate considerable performance zero-shot settings the financial NLP tasks. How- ever, their performance generally falls behind the fine-tuned language models, implying that these large language models still have the potential to im- prove their financial expertise. On the other hand, fine-tuned models are less effective when the training examples are insufficient (FiQA SA) or imbalanced (FOMC).
Limitations
This paper has several limitations improve in future research. First, our evaluation is limited to some notable language models, while other ad- vanced LLMs may exhibit different performances from our reported models. Also, as the LLMs keep evolving and improving over time, the future ver- sions of the evaluated models can have different performance from the reported results. Second, Livll only focuses financial classification tasks, and the analysis of the generation ability of the L.Mss still needs be included. Future work can be done toward developing evaluation bench- marks on generation tasks the financial domain.
The performance gaps between fine-tuned models and zero-shot LLMs are larger on pro-

---

References
Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instruc- tions with human feedback. In NeurIPS.
Julio Cesar Salinas Alvarado, Karin Verspoor, and Timo- thy Baldwin. 2015. Domain adaption of named entity recognition to support credit risk assessment. Pro- ceedings of the Australasian Language Technology Association Workshop, ALTA 2015, Parramatta, Aus- tralia, December 8 9, 2015. pages 84-90. ACL.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Dogu Araci. 2019. Finbert: Financial sentiment analysis with pre-trained language models. CoRR, abs/1908. 10063.
Nils Reimers and Iryna Gurevych. 2019. Sentence-ber Sentence embeddings using siamese bert nert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranay Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Cerbert-Voss,, Gretchen Krueger. Tom Henighan. Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess. Jack Clark, Christopher Bemnrr, Sam McCandlish, Alec Radford, Nly Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Agam Shah, Suvan Paturi, and Sudheer Chava, 2023, Trillion dollar words: A new financial dataset, task & market analysis. In Proceedings of the 61 Annual Meeting of the Association for Computational Lin- guistics ( (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 6664-6679, Associa- tion for Computational Linguistics.
Raj Shah, Kunal Chawla, Dheeraj Eidnani, Agam Shah, Wendi Du Sudheer Chava, Natraj Raman, Charese Smiley, Jiaao Chen, and Diyi Yang. 2022. When FLUE meets FLANG: Benchmarks and large pre- trained language model for financial domain. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 2322- 2335, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-HLT 2019. Minneupolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pages 4171-4186 Association for Computational Linguistics.
Ankur Sinha and Tanmay Khandait. 2020. Impact of news on the commodity market: Dataset and results. CoRR, abs/2009.046 042002.
Chi Sun, Luyao Huang, and Xipeng Qiu. 2019. Uti- lizing BERT for aspect-based sentiment analysis via constructing auxiliary sentence. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HITT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 380-385. Associa- tion for Computational Linguistics.
FiQA. Financial question answering. https://sites. google. com//vie/fiqa
Allen H Huang, Hui Wang, and Yi Yang. 2023. Finbert: large language model for extracting information from financial text. Contemporary Accounting Re- search, 841.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CORR, abs/1900.11692.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction followwin 1lama model. https:/ // gi thub. wtt./coo//attae wato://ttttullab/ttt stw.aa/sssaooordaaaaaaaa
Pekka Malo, Ankur Sinha, Pekka J. Korhonen, Jyrki Wallenius, and Pyry Takala. 2014. Good debt or bad debt: Detecting semantic orientations in economic texts. J. Assoc. Inf. Sci. Technol., 6544)7882-796
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. CoRR, Abs/2302.139711
OpenAl. 2023. abs/2303.08774.
GPT-4 technical report. CoRR,
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti

---

Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton- Ferrer. Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Di- ana Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov, Pushkar Mishra. Igor Moly- bog. Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang, Ross Tay- lor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Ilyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Ro- driguez. Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b. Llama Open foundation and fine-tuned chat models. CoRR, abs/2307, 0.9288.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceed- ings of the 61 Annual Meeting of the Association for Computational Linguistics Volume Long Pa- pers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 71484-13508 Association for Computational Linguistics.
Shijie Wu, Ozan Insoy, Steven Lu, Vadim Dabravol- ski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David S. Rosenberg, and Gideon Mann. 2023. Bloomberggpt: A large language model for finance. CoRR, abs/2303.17564.
Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. 2023. PIXIU: A large language model, in- struction data and evaluation benchmark for finance. CoRR, (044436.
Yi Yang. Yixuan Tang, and Kar Yan Tam. 2023. In- vestlm: A large language model for investment using financial domain instruction tuning, arXiv preprint arXiv:2309. 13064.
Yi Yang, Mark Christopher Siy Uy, and Allen Huang. 2020. Finbert: pretrained language model for financial communications. CoRR, abs/2006.08097.


Reviews:
[{"review_id": "WaYM7NqlOh", "reviewer": "Reviewer_Fsxu", "summary": "", "questions": "Q: While there were access restrictions in place for GPT-4 previously, there remains a question as to why the authors did not assess the performance of other Language Models. Solely conducting tests on ChatGPT could potentially compromise the accuracy and authenticity of the experimental findings.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Fsxu:\n\nThank you for your review of our work. We truly appreciate the time you took to evaluate our work. Your insights and feedback are invaluable to us.\n \nWe would like to give some explanations on the raised points:\n\n1. Response to reason 1 to reject:\n\nThanks for raising the suggestion that our comparison is not enough. To have a more comprehensive evaluation of other advanced generative LLMs, we include the experimental results on decoder-only models ChatGPT (August 3 Version), GPT-4, PIXIU (a newly released financial LLM), LLAMA, LLAMA2, and the encoder-only financial language models FLANG-BERT and FLANG-ELECTRA. The results are shown as below. \n\n|               | Previous best results | FLANG-BERT | FLANG-ELECTRA | ChatGPT, August | GPT-4 |      PIXIU      | LLAMA-7B | LLAMA2-7B |\n|---------------|:---------------------:|:----------:|:-------------:|:---------------:|:-----:|:---------------:|:--------:|:---------:|\n| FinSent       |    **0.871(RoBERTa)**    |    0.849   |     0.853     |      0.751      | 0.809 | 0.800   |   0.528  |   0.243   |\n| FPB           |     0.934(RoBERTa)    |    0.881   |     0.784     |      0.755      | 0.905 |      **0.965**      |   0.119  |   0.339   |\n| FiQA SA       |  0.898(ChatGPT, May)  |    0.695   |     0.695     |      0.769      | 0.920 |      **0.930**      |   0.310  |   0.480   |\n| ESG           |     **0.958(FinBERT)**    |    0.925   |     0.914     |      0.643      | 0.626 |      0.509      |   0.190  |   0.209   |\n| FLS           |     **0.882(FinBERT)**    |    0.861   |     0.864     |      0.566      | 0.565 |      0.275      |   0.343  |   0.365   |\n| QA            |      **0.865(BERT)**     |    0.785   |     0.820     |      0.705      | 0.775 |      0.680      |   0.720  |   0.625   |\n| Headlines-PDU |     **0.956(FinBERT)**    |    0.940   |     0.948     |      0.848      | 0.878 |      0.842      |   0.405  |   0.411   |\n| Headlines-PDC |     **0.981(FinBERT)**   |    0.978   |     0.980     |      0.786      | 0.947 |      0.702      |   0.053  |   0.053   |\n| Headlines-PDD |    **0.961(RoBERTa)**   |    0.956   |     0.954     |      0.866      | 0.900 |      0.763      |   0.384  |   0.382   |\n| Headlines-PI  |     0.976(FinBERT)    |   **0.977**   |     0.963     |      0.195      | 0.105 |      0.753      |   0.966  |   0.966   |\n| Headlines-AC  |     **0.997(FinBERT)**   |    0.995   |     0.815     |      0.763      | 0.838 |      0.902      |   0.318  |   0.346   |\n| Headlines-FI  |  0.976(BERT,FinBERT)  |    0.974   |     0.977     |      0.795      | 0.780 |     **0.981**    |   0.083  |   0.048   |\n| Headlines-PS  |    **0.924(FinBERT)**    |    0.906   |     0.918     |      0.773      | 0.811 |      0.776      |   0.460  |   0.546   |\n| NER           |     **0.981(RoBERTa)**    |    0.978   |     0.978     |      0.654      | 0.707 |      0.749      |   0.815  |   0.714   |\n| FOMC          |   0.633(ChatGPT,May)  |    0.602   |     0.480     |      0.596      | **0.729** |      0.522      |   0.251  |   0.349   |\n| Average       |    **0.909(RoBERTa)**   |    0.907   |     0.890     |      0.698      | 0.753 |      0.739      |   0.396  |   0.405   |\n\nAfter comparing more language models, the main conclusions in the paper still hold, i.e.,\n\n(1)\tThe fine-tuned expert models are usually better than the general generative LMs. After fine-tuning the generative LMs on the financial datasets (PIXIU model), their performance is improved.\n\n(2)\tThe performance gaps between fine-tuned expert models and the general generative models are more significant on private datasets than the publicly available datasets.\n\n2. Response to reason 2 to reject\n\nWe apologize for not providing a comprehensive literature review due to the space limit of the short paper.  Some previous works developed language models for the finance domain. FinBERT (Araci, 2019) further pre-trains BERT on the financial domain corpus, while  FinBERT (Yang et al., 2020) collects a large-scale financial communication corpus to pre-train a financial domain-specific BERT model. Also, FinBERT (Liu et al.,2020) uses multi-task pre-training to simultaneously train the model on general corpora and financial domain corpora. Besides, FLANG (Shah et al., 2022) apply better masking using financial keywords and phrases with span boundary and in-filing objectives. Recently, several large generative financial language models have been developed. BloombergGPT (Wu et al., 2023) is a 50-billion parameter language model trained on a wide range of financial data. PIXIU (Xie et al., 2023) further fine-tunes LLaMA with financial instruction data. While many financial language models are proposed, their comprehensive evaluation needs to be addressed. FLUE (Shah et al., 2022) introduced to benchmark those language models in the finance domain. However, it mainly focuses on the encoder-only models, and the performance of the generative decoder-only model is not evaluated. Adding four extra proprietary financial datasets, we conduct a comprehensive empirical analysis and comparison of the performance of various financial language models.\n\n3. Response to reason 3 to reject:\n\nWe introduce the experimental setup in section 5.1. We prompt ChatGPT via the API provided by OpenAI, and the prompts for zero-shot learning on different datasets are provided in Appendix A. For in-context learning, \" The sentence: {}\" in prompts is replaced by \" Here are some examples: The sentence: {sentence 1} The label: {label 1} … The sentence: {sentence n} The label: {label n} The sentence: {} The label:\", as shown in figure 1.\n\n4. Response to question 1 for the authors\n\nSee response 1.\n\nIf you have any further questions or suggestions for collaboration, please feel free to reach out. We are always open to fruitful discussions and potential partnerships.\n\nOnce again, thank you for your support and feedback. We look forward to sharing more of our work with you in the future.\n\nBest regards,\n\nAuthors\n\n\nReferences:\n\nDogu Araci. 2019. Finbert: Financial sentiment analysis with pre-trained language models. CoRR,abs/1908.10063.\n\nZhuang Liu, Degen Huang, Kaiyu Huang, Zhuang Li, and Jun Zhao. 2020. Finbert: A pre-trained financial language representation model for financial text mining. In IJCAI, pages 5–10.\n\nRaj Shah, Kunal Chawla, Dheeraj Eidnani, Agam Shah, Wendi Du, Sudheer Chava, Natraj Raman, Charese Smiley, Jiaao Chen, and Diyi Yang. 2022. When FLUE meets FLANG: Benchmarks and large pre-trained language model for financial domain. In EMNLP pages 2322–2335.\n\nShijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kam-badur, David Rosenberg, and Gideon Mann. 2023. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564\n\nQianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. 2023. PIXIU: A large language model, instruction data and evaluation benchmark for finance. CoRR, abs/2306.05443.\n\nYi Yang, Mark Christopher Siy Uy, and Allen Huang.2020. Finbert: A pretrained language model for financial communications. CoRR, abs/2006.08097.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper explore the proficiency of ChatGPT in the financial domain, and introduce a tool called \"FinLMEval\" for evaluating ChatGPT on financial tasks, and get some interesting conclusion as following:\n\n1. While ChatGPT performs well across most financial tasks, it generally lags behind the fine-tuned expert models, especially on proprietary datasets.\n2. The fine-tuned task-specific models generally outperform ChatGPT, except when the supervised data is insufficient or imbalanced.\n\nAdditional, the author proposes FinLMEval, a comprehensive evaluation framework for assessing the performance of language models on financial natural language processing (NLP) tasks. FinLMEval comprises nine datasets specifically designed to evaluate the expertise of language models in the financial domain.", "reasons_to_reject": "1. The authors focuses solely on comparing ChatGPT with fine-tuned auto-encoding models and does not include a comparison with other advanced LLMs, such as GPT-4, LLAMA and Alpaca. So I think the contributions are not enough. \n2. The related work section briefly mentions previous language models used in financial NLP tasks but does not provide a comprehensive overview of the existing literature. \n3. The authors briefly mentions the evaluation strategies used for ChatGPT (zero-shot and few-shot in-context learning) but does not provide sufficient details about the experimental setup and methodology.", "questions_to_authors": "Q: While there were access restrictions in place for GPT-4 previously, there remains a question as to why the authors did not assess the performance of other Language Models. Solely conducting tests on ChatGPT could potentially compromise the accuracy and authenticity of the experimental findings.", "paper_topic_and_main_contributions": "This paper explore the proficiency of ChatGPT in the financial domain, and introduce a tool called \"FinLMEval\" for evaluating ChatGPT on financial tasks. \n\nThe main contributions of this paper as following:\n1. The author proposes FinLMEval, a comprehensive evaluation framework for assessing the performance of language models on financial natural language processing (NLP) tasks. FinLMEval comprises nine datasets specifically designed to evaluate the expertise of language models in the financial domain.\n2. The authors compare the performance of fine-tuned auto-encoding language models (BERT, RoBERTa, FinBERT) and the large language model (LLM) ChatGPT on financial NLP tasks, and reveal that while ChatGPT performs well across most financial tasks, it generally lags behind the fine-tuned expert models, especially on proprietary datasets.\n3. The authors identifies factors that affect the performance of language models on financial NLP tasks. It shows that fine-tuned task-specific models generally outperform ChatGPT, except when the supervised data is insufficient or imbalanced.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "89vavPuYiB", "reviewer": "Reviewer_nbqJ", "summary": "", "questions": "Question A: would the collected dataset be public? It would be helpful for the community to reproduce and develop new models.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer nbqJ:\n\nThank you for your review of our work. We truly appreciate the time you took to evaluate our work. Your insights and feedback are invaluable to us.\n \nWe would like to give some explanations on the raised points:\n \n1. Response to reason 1 to reject:\n\nWe apologize for the misleading taxonomy. Our paper is intended to evaluate the BERT-like encoder-only models as fine-tuned expert models and GPT-like decoder-only models as general language models. We will modify the paper by using the terms encoder- and decoder-only models more professionally.\n\n2. Response to reason 2 to reject:\n\nThanks for raising the suggestion that some baselines still need to be included. To have a more comprehensive evaluation of other advanced LMs, we include the experimental results on decoder-only models ChatGPT (August 3 Version), GPT-4, PIXIU (a newly released financial LLM), LLAMA, LLAMA2, and the encoder-only financial language models FLANG-BERT and FLANG-ELECTRA. The results are shown below.\n\n|               | Previous best results | FLANG-BERT | FLANG-ELECTRA | ChatGPT, August | GPT-4 |      PIXIU      | LLAMA-7B | LLAMA2-7B |\n|---------------|:---------------------:|:----------:|:-------------:|:---------------:|:-----:|:---------------:|:--------:|:---------:|\n| FinSent       |    **0.871(RoBERTa)**    |    0.849   |     0.853     |      0.751      | 0.809 | 0.800   |   0.528  |   0.243   |\n| FPB           |     0.934(RoBERTa)    |    0.881   |     0.784     |      0.755      | 0.905 |      **0.965**      |   0.119  |   0.339   |\n| FiQA SA       |  0.898(ChatGPT, May)  |    0.695   |     0.695     |      0.769      | 0.920 |      **0.930**      |   0.310  |   0.480   |\n| ESG           |     **0.958(FinBERT)**    |    0.925   |     0.914     |      0.643      | 0.626 |      0.509      |   0.190  |   0.209   |\n| FLS           |     **0.882(FinBERT)**    |    0.861   |     0.864     |      0.566      | 0.565 |      0.275      |   0.343  |   0.365   |\n| QA            |      **0.865(BERT)**     |    0.785   |     0.820     |      0.705      | 0.775 |      0.680      |   0.720  |   0.625   |\n| Headlines-PDU |     **0.956(FinBERT)**    |    0.940   |     0.948     |      0.848      | 0.878 |      0.842      |   0.405  |   0.411   |\n| Headlines-PDC |     **0.981(FinBERT)**   |    0.978   |     0.980     |      0.786      | 0.947 |      0.702      |   0.053  |   0.053   |\n| Headlines-PDD |    **0.961(RoBERTa)**   |    0.956   |     0.954     |      0.866      | 0.900 |      0.763      |   0.384  |   0.382   |\n| Headlines-PI  |     0.976(FinBERT)    |   **0.977**   |     0.963     |      0.195      | 0.105 |      0.753      |   0.966  |   0.966   |\n| Headlines-AC  |     **0.997(FinBERT)**   |    0.995   |     0.815     |      0.763      | 0.838 |      0.902      |   0.318  |   0.346   |\n| Headlines-FI  |  0.976(BERT,FinBERT)  |    0.974   |     0.977     |      0.795      | 0.780 |     **0.981**    |   0.083  |   0.048   |\n| Headlines-PS  |    **0.924(FinBERT)**    |    0.906   |     0.918     |      0.773      | 0.811 |      0.776      |   0.460  |   0.546   |\n| NER           |     **0.981(RoBERTa)**    |    0.978   |     0.978     |      0.654      | 0.707 |      0.749      |   0.815  |   0.714   |\n| FOMC          |   0.633(ChatGPT,May)  |    0.602   |     0.480     |      0.596      | **0.729** |      0.522      |   0.251  |   0.349   |\n| Average       |    **0.909(RoBERTa)**   |    0.907   |     0.890     |      0.698      | 0.753 |      0.739      |   0.396  |   0.405   |\n\nAfter comparing more language models, the main conclusions in the paper still hold, i.e.,\n(1)   The fine-tuned expert models are usually better than the general generative LMs. Their performance is improved after fine-tuning the generative LMs on the financial datasets (PIXIU model).\n(2)   The performance gaps between fine-tuned expert models and the general generative models are more significant on private than publicly available datasets.\n \n3. Response to the question for the authors\n\nThe release of the data is restricted by copyright concerns of the financial reports. Upon acceptance, we will build a financial leaderboard to evaluate the open LMs without explicitly releasing the data. \n\nIf you have any further questions or suggestions for collaboration, please feel free to reach out. We are always open to fruitful discussions and potential partnerships.\n \nOnce again, thank you for your support and feedback. We look forward to sharing more of our work with you in the future.\n \nBest regards,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Filling the gap that the financial understanding ability of ChatGPT is under-explored.\n\n- Comprehensive experiment setting for the GPT-like generative models: \n\n  - Design prompts to implement zero-shot and in-context learning protocols on ChatGPT to evaluate.\n\n  - Collect extra datasets on 4 tasks to verify the LLM ability on private corpora.", "reasons_to_reject": "- Misleading Taxonomy: \"auto-encoding\" models, i.e. encoder-only models, should never be treated as a parallel concept of \"LLM\". The authors misuse the terminology encoder-only, decoder-only, and encoder-decoder architectures here. Such categorisation can be found in Wang, Hongyu, et al. Foundation Transformers. arXiv:2210.06423, arXiv, 19 Oct. 2022. arXiv.org, https://doi.org/10.48550/arXiv.2210.06423. \n- Missing Baselines:\n  - Following the taxonomy mentioned above, the authors should provide the evaluation results from other generative pre-trained LM baselines (eg. T5, BART) to see whether difference in the results are brought by the distinguished architecture. If authors also want to use models with similar parameters and training data sizes to ChatGPT , they can alternatively provide results from GLM or FLAN-T5.\n  - Since this work is an incremental version of FLUE benchmark (shah et al 2022), the baseline models (FLANG-BERT, FLANG-ELECTREA) should be included for comparison.", "questions_to_authors": "Question A: would the collected dataset be public? It would be helpful for the community to reproduce and develop new models.", "paper_topic_and_main_contributions": "This paper proposes a new evaluation framework for financial natural language processing, which mainly aims to evaluate how much domain knowledge is learned by the recently emerged LLMs (mostly ChatGPT). Designed prompts corresponding to different tasks and new originally annotated dataset are provided.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "line#39: => are performed on text from various domains\n\nTwo “consequently” are used consecutively in line#42 and line#44. Do the author meant to emphasise a chain-of-thought?\n\nline#47 \"because of this\": \"this\" is vague in the context."}}, {"review_id": "aYMKXwXjhP", "reviewer": "Reviewer_VqpE", "summary": "", "questions": "Questions:\n1.\tThere are tough financial NLP tasks, such as financial technical report generation, season financial report generation, market analysis by financial news analysis. These are true financial NLP tasks, and simple sentiment analysis or NER are far from enough for real-world financial NLP tasks using LLMs. Wish to see more evaluation on these directions. Also, if the authors have any feedbacks and ideas in these directions, willing to learn as well.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer VqpE:\n\nThank you for your review of our work. We truly appreciate the time you took to evaluate our work. Your insights and feedback are invaluable to us.\n\nWe would like to give some explanations on the raised points:\n\n1. Response to reason to reject:\n\nAlthough the method in our paper is experimental, our work provides an empirical study of how well the language models perform on financial tasks. We address the research gap that, though the GPT-like general domain LLMs show unprecedented ability on general NLP tasks, their ability on financial tasks remains unclear. Our work aims to conduct an empirical study to fill this gap.\n\nOur contributions are summarized as (1) We draw interesting conclusions from the empirical study. For example, we find that the fine-tuned expert models are usually better than the general generative LMs, especially on the private datasets; when the labeled data is insufficient or imbalanced, general generative LMs can perform better. Our empirical study can guide future researchers and domain practitioners to choose suitable LMs when facing related problems. (1) We provide a new tool and new financial evaluation benchmark. Four datasets (FinSent, FLS, ESG, QA) are newly included in our evaluation tools.\n\n2. Response to question for the authors:\n\nWe agree that most of our tasks are based on classifications, which can only evaluate the LMs from one perspective; additional generative tasks on the financial domain are needed.\nNevertheless, compared to the previous FLUE benchmarks (Shah et al., 2022), we include four new document classification datasets, which require domain expert knowledge to understand the financial concepts and judge the financial reports, which is an improvement from the previous datasets.\n\nFuture works can focus on evaluating financial LMs on generative tasks. Potential directions could be: (1) construct datasets on financial generation tasks, e.g., financial report generation/summarization, financial news analysis, financial knowledge question-answering, etc., and (2) evaluation tools to align LMs’ responses with financial experts’ preferences.\nIf you have any further questions or suggestions for collaboration, please feel free to reach out. We are always open to fruitful discussions and potential partnerships.\n\nOnce again, thank you for your support and feedback. We look forward to sharing more of our work with you in the future.\n\nBest regards,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strong:\n1.\tFinLMEval helps evaluating financial LLMs in a relatively fair way;", "reasons_to_reject": "Weak:\n1.\tThis paper is more fit as a demonstration paper, and the major contribution can hardly be concluded (existing datasets’ collection, evaluation, and comparison are more experimental).", "questions_to_authors": "Questions:\n1.\tThere are tough financial NLP tasks, such as financial technical report generation, season financial report generation, market analysis by financial news analysis. These are true financial NLP tasks, and simple sentiment analysis or NER are far from enough for real-world financial NLP tasks using LLMs. Wish to see more evaluation on these directions. Also, if the authors have any feedbacks and ideas in these directions, willing to learn as well.", "paper_topic_and_main_contributions": "This paper presents FinLMEval, an evaluation framework for financial language models (LLMs). FinLMEval comprises 9 datasets in financial domain. The authors also conducted the evaluations on fine-tuning auto-encoding models and zero/few-shot prompting ChatGPT. Current results show that the fine-tuning auto-encoding models generally perform better than ChatGPT.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents FinLMEval, an evaluation framework for assessing the performance of language models on financial natural language processing tasks. The authors conduct experiments to compare the performance of ChatGPT with fine-tuned auto-encoding models like BERT, RoBERTa, and FinBERT. The paper addresses a gap in the literature by focusing on the financial understanding abilities of large language models like ChatGPT. Reviewers mention that the paper could have been more comprehensive by including other large language models for comparison. There are also questions about the soundness of the methodology, especially regarding the specific NLP tasks chosen for evaluation. In light of the identified merits and shortcomings, the collective judgment leans toward accepting the paper for the "Findings" section. The manuscript makes a sound yet incremental contribution to the field, and its limitations could be addressed in future work.
Decision: Accept-Findings