Exploring the Impact of Corpus Diversity on Financial Pretrained Language Models
Authors: Jaeyoung Choe
Keonwoong Noh
Nayeon Kim
Seyun Ahn
Woohwan Jung
Venue: EMNLP 2023 Findings (2023)
Keywords: Financial NLP
Pre-trained Language Model
Generalization
TL;DR: Enhancing the generalization performance of financial PLMs through corpus diversity.

Abstract: Over the past few years, various domain-specific pretrained language models (PLMs) have been proposed and have outperformed general-domain PLMs in specialized areas such as biomedical, scientific, and clinical domains. In addition, financial PLMs have been studied because of the high economic impact of financial data analysis. However, we found that financial PLMs were not pretrained on sufficiently diverse financial data. This lack of diverse training data leads to a subpar generalization performance, resulting in general-purpose PLMs, including BERT, often outperforming financial PLMs on many downstream tasks. To address this issue, we collected a broad range of financial corpus and trained the Financial Language Model (FiLM) on these diverse datasets. Our experimental results confirm that FiLM outperforms not only existing financial PLMs but also general domain PLMs. Furthermore, we provide empirical evidence that this improvement can be achieved even for unseen corpus groups.

Exploring the Impact of Corpus Diversity on Financial Pretrained Language Models
Jaeyoung Choe'; Keonwoong Noh?, Nayeon Kim'; Seyun Ahn', Woohwan Jung!
DDepartment of Applied Artificial Intelligence, Hanyang University
Division of Computer Science, Hanyang University
3 Ramply Inc. {cjy9100, rohgw011, na2na8, tpdbs0907,
ac kr
Abstract
et al., 2019) outperformed the financial PLMs on financial tasks.
Over the past few years, various domain- specific pretrained language models (PLMs) have been proposed and have outperformed general-domain PLMs specialized areas such biomedical, scientific, and clinical domains. In addition, financial PLMs have been stud- ied because the high economic impact of financial data analysis. However, we found that financial PLMs were not pretrained on sufficiently diverse financial data. This lack of diverse training data leads to subpar gen- eralization performance, resulting in general- purpose PLMs, including BERT, often outper- forming financial PLMs on many downstream tasks. To address this issue, we collected broad range of financial corpus and trained the Financial Language Model (FiLM) on these diverse datasets. Our experimental results con- firm that FiLM outperforms not only existing financial PLMs but also general domain PLMM. Furthermore, we provide empirical evidence that this improvement can be achieved even for unseen corpus groups
To investigate the reasons for the low general- ization performance, we categorize the financial corpus into five groups and examine their use in existing financial PLMs. This reveals that most ex- existing PLMs are pretrained on a corpus consisting of only two of these corpus groups. This indicates that the pretraining data for financial PLMs lacks diversity, which might cause their low generaliza- tion performance across various financial tasks.
Motivated by this observation, we collect a broad range of corpus and analyze the impact of corpus diversity on the performance of language models. Our investigations demonstrate that as a corpus be- comes more diverse, the model's performance im- proves. Incorporating diverse corpus, even with fewer tokens, yields better generalization perfor- mance than relying numerous tokens from non- diverse corpus. Furthermore, when using diverse corpora, the model exhibits robust generalization performance on unseen tasks.
1 Introduction
We train our Financial Language Model (FiLM) corpus with diverse documents and evaluate on six financial tasks, including recently intro- duced works (Chen al., 2021; Loukas al., 2022; Shah et al., 2023). Our experimental results show that FiLM outperforms not only existing financial PLMs but also general domain PLMs on most finan- cial tasks. In addition, we achieve an improvement of performance while reducing the number of to- kens trained and energy consumption. To the best of our knowledge, FiLM is the first model to sur- pass RoBERTa in financial domain. We make our model and code publicly available on GitHub and Huggingface hub² for continuous advancement financial domain.
Pretrained Language Models (PLMs) have been successfully employed in various natural language processing tasks. This trend has been extended to further pretraining of PLMs on domain-specific corpus in various fields such biomedical (Lee al., 2020), scientific (Beltagy et al., 2019), and clinical (Alsentzer al., 2019) domains.
In financial domain, Araci (2019); Yang et al. (2020); Liu et al. (2021); Loukas et al. (2022) pro- posed domain-specifie PLMs based on BERT (De- vlin al., 2019) model. However, each study tested a small number of tasks, resulting in inadequate validation. We conduct extensive experiments to show that the generalization performance of exist- ing financial PLMs is unsatisfactory. Even general domain PLMs such as BERT and RoBERTa (Liu
http::///ggtuub..mm//dee---pppe Pnttps://huggingface hotps////nnaaaa.ooooo///////LL
Major in Bio Artificial Intelligence

---

Group
Name
Description
Financial PLMS
# TTokkens
TRC2
Pinancial news stories from Reuters
1, IV
227.3M
Investing com com
Stock, options, commodity etc. News article
130.8M
News
Nttmess
Economic articles from the New York Times
75M
EIA
Commodity related news articles from EIA
1.1M
SEC filings
Annual reports( IO-K) and quarterly reports I0-Q)
307.IM
Earnings Call
Earnings conference call transcripts
i1,
1.6B
ArXiv
A collection of abstracts of economic research papers
42.1M
Papers
AIHUB
A collection of Korean economics research papers
5.8M
Investopedia
Economic glossary
5.3M
MISC
FinWEB
Finance, loans, and insurance related articles
2.8M
total 10 datasets
2.4
Table 1: Summary of the financial datasets used for pre-training the models. The 'Financial PLMs column represents the financial PL.Ss that have utilized each respective dataset: i. Araci (2019), ii. Yang et al. (2020), Liu et al. (2021), iv. Shah et al. (2022), v. Loukas et al. (2022)
ence calls obtained from Seeking Alpha.
Papers: This group contains the abstracts of
research papers in the field of economics. It includes other datasets the finan- cial domain.
MISC:
For more detailed information the corpus and groups, please refer to Appendix A.1. Figure la shows the embeddings of sentences in the pretrain- ing corpus. The color of each point represents the group to which the corresponding sentence belongs. This indicates that sentences within the same group have similar embeddings, thereby forming clusters. For example, sentences in the financial news group are primarily distributed in the right and lower ar- eas, whereas SEC filings are concentrated on the top side.
500 Fligg
Hoclhe
(a) Pretraining datasets
(b) Financial tasks
Figure Sentence embeddings visualization for both
corpus groups and financial tasks.
2 Proposed method
First, the datasets used to train the model are in- troduced. Second, we describe the method for pre- processing the datasets. Finally, describe FiLM training procedure.
None of the corpus groups was distributed over the entire space. From this observation, we could conclude that it is essential to use data from all groups for pretraining to enable the language model to learn diverse features.
2.1 Pretraining datasets
However, existing studies do not use all these groups to further train language models. In addi- tion, most of them (Yang al., 2020; Araci, 2019; Loukas et al., 2022) use only two groups datasets.
For further pretraining the language models, we collected a financial corpus from 10 sources. There are various financial documents with different char- acteristics. For instance, financial reports have higher prevalence of numerical information than other texts such as the abstracts of research in fi- nance (Loukas al., 2022).
Consequently, these models might not achieve high performance across the entire financial domain. To attain robust performance over the entire financial domain, we employ all groups of datasets for pre- training.
As shown in Table 1, we categorize the pretrain- ing datasets into the five groups as follows:
News: The datasets are sourced from financial news articles.
2.2 Preprocessing
The preprocessing for the pretraining involves two steps: cleaning and deduplication. In the clean- ing step, remove HTML tags using Beautiful- Soup² and unnecessary special characters such
SEC filings: This dataset comprises financial reports (10-K, 10-Q) submitted to the U.S. Securities and Exchange Commission (SEC). Earnings call: This dataset comprises the in- formation and transcripts of earnings confer-
www. www.crrrmmmmmmmmm cow.sof sof www.eeBeauti fulSop/

---

newline characters. Deduplication could improve the efficiency of pretraining and the accuracy of PLMs (Lee et al.. 2022). Thus, we remove du- plicate sentences from the datasets during prepro- cessing. For deduplication, we remove all but one sentence from each group of duplicate sentences in the corpus. The contents of the remaining doc- uments were preserved if a sentence was removed during deduplication. After deduplication, the to- ken count decreased from 3.3B 2.4B, and the size of the training dataset reduced from 19.5GB to 14GB.
is based on profit and loss reports of S&P 500 companies.
FOMC (Shah al., 2023): This aims to clas- sify text generated by the Federal Open Mar- ket Committee (FOMC) in order to assess its impact on the financial markets. This dataset classifies the policy stance as either "Hawk- ish" or "Dovish." Data collection for this task was conducted up until October 15, 2022.
Figure b shows the embeddings of the sentences sampled from the financial tasks. The colors of each point distinguish the six tasks and the gray points represent the sentences from the pretraining data. FINER and FinQA are located on the top side because both tasks use SEC filings to create the dataset. Meanwhile, the Headline task is located further away from the other tasks due to its focus on the gold commodity. In addition, the sentences in FPB, NER, and FOMC form their own clusters, separate from the other datasets.
2.3 Training procedure for FiLM
To train FiLM, we further pretrain RoBERTa on financial corpus presented in Section $2.1. Follow- ing RoBERTa, we use the masked language model for the pretraining task. In addition, we use the same tokenization method and model input format in RoBERTa. We further pretrain model for only one epoch because the performance saturates after one epoch. We use a batch size of 16 and a learning rate of le-5. We use the Adam optimizer and do not set any specific scheduler or warm-up steps. Please refer to Table 5 for the hyperparame- ter settings.
As observed, each financial NLP task has unique aims and distinctive textual features. This implies that the performance of language model has be evaluated across a broad range of tasks. For more detailed information about financial tasks, please refer to Appendix B.
3 Financial tasks for evaluation
Experiments
We evaluate the performance of financial PLMs on the following six financial NLP tasks:
4.1 Experiment setup
We compared the FiLM model with existing fi- nancial domain PLMs: FinBERT-A (Araci, 2019). FinBERT-Y (Yang et al., 2020), FLANG-BERT & TTLANG-RoEERYS (Shah et al., 2022), and SEC- BERT (Loukas et al., 2022). Furthermore, we fine- tune general domain PLMs, BERT (Devlin et al.. 2019) and RoBERTa (Liu et al., 2019), to establish the baselines. For detailed information on each model, please refer to Appendix C.
FPB (Malo et al., 2014): This sentiment clas- sification task involves three categories: posi- tive, negative, and neutral. The dataset com- prises 4,840 sentences from financial news articles.
NER (Alvarado et al., 2015): The goal of this financial named entity recognition task is to identify four types of named entities (PER,
LOC, ORG, and MISC) within financial con- tracts reported to the SEC.
For all experiments, except for FINER and FinQA, the results are computed based on the aver- age score across three seed values. For the standard deviation obtained through all experiments, please refer to Table 8.
Headline (Sinha and Khandait, 2021): The ob-
jective of this task is to classify the impact of news articles pertaining to gold commodities based on the headline of the articles.
All the models are trained on an RTX3090 GPU. For the detailed settings for fine-tuning, please refer to Appendix B. The datasct and hyperparameters used to further pretrain FiLM are provided in Ap- pendix A. Using this setup, the training of FiLM on the entire dataset can be completed within 24 hours, resulting in a high-performance language model while maintaining a low training cost.
FiNER (Loukas et al., 2022) This is numeric entity recognition task. The dataset comprises XBRL-tagged financial reports from publicly traded companies.
FinQA (Chen et al., 2021): This is financial question- -anhooringn task for evaluating numer- ical reasoning and understanding. The dataset

---

Headline F-1
FinQA
Model
(Financial Corpus)
Accuracy
Prog Ace
Exe
General Domain PLMs
BERT (Devlin al., 2019)
83.30
81.73
75.09
89.54
79.40
51.09
53.10
63.81
(Liu et al., 2019)
78.81
91.29
56.76
Existing Financial Domain
FinBERT-A (Araci, 2019)
237M
85.25
82.45
77.93
90.48
81.49
50.04
64.50
FinBERT-Y (Yang 2020)
4.9
83.68
82.52
70.40
90.83
81.08
38.79
40.54
64,30
FLANG-BERT (Shah al.. 2022)
NA
84.76
83.12
75.58
91.06
81.52
49.17
51.43
64.93
FLANG RoBERTa (Shah et al., 2022)
NA
83.86
82.19
71.36
90.46
80.77
30.68
32.17
68.02
SEC-BERT (Loukas et al., 2022)
3.1B
84.37
82.18
78.74
90.52
82,35
53.18
55.45
65.07
Proposed models
FILM
86.25
84.48
79.78
91.79
82.02
58.85
G1.38
69.60
(5,5B)
86.14
84.11
78.82
91.74
$9.37
61.64
Table 2: Main results. FiLM (5.5B) is the model trained an additional SEC filings dataset comprising 3. IB tokens (Appendix A). each task, the best score marked in bold, while the second best score underlined. Shah et al. (2022) did not report the number of tokens but utilized approximately 2.78 million documents.
4.2 Main results
NER
FiNER
FinQA
Groups
F-1
F-1
Prog Acc
Table reports the results of evaluating the per- formance of PLMs on financial tasks. Despite further training on financial corpus, existing fi- nancial PLMs often perform worse than gen- eral domain PLMs certain tasks. For exam- plc, RoBERTa outperforms all existing financial PLMs on FPB, FinQA, and FOMC. addition, BERT outperforms FinBERT-A, FinBERT-Y, and FLANG-BERT on FinQA tasks. This implies that financial PLMs exhibit a deficiency in their gener- alization capabilities, limiting their effectiveness across diverse spectrum of documents within the financial domain. However, model, which in- corporates diverse range of financial corpus for pretraining, has been equipped with robust gener- alization capabilities, enabling
FiLM [Qurs](2.4B)
5
79.78
82.02
58.85
w/o SEC filings(2.1B)
4
76.51
81.94
57.54
only SEC filings(311B) I
75.51
81.91
57.45
only SEC filings(0.3B)
75.30
81.64
57.10
Table 3: Comparison of model performance NER.
FiNER, FinQA under different pre-training data set- tings.
finding suggests that a more diverse corpus leads to enhanced model performance.
0.855
Averoge score
0.845
perform well across various NLP tasks in the financial domain. Although our model is trained on fewer tokens (2.4B) than SEC-BERT (3. IB) and FinBERT-Y (4.9B), it outperforms existing PLMs most fi- nancial tasks.
ZERO
0.840
Number of corpus groups
Figure 2: Average Fl scores measured on four financial tasks, with varying the number of corpus groups for pretraining.
4.3 Impacts of the diversity in the pretraining dataset
4.4 Performance extrapolation
We investigate the variation in the performance of the language model based on the number of cor- pus groups used. To achieve this, we generate all combinations of corpus groups and pretrain sep- arate RoBERTa on each combination. Then, we evaluate each model by caleulating the average Fl score across four downstream tasks: FPB, NER, Headline, and FiNER. Figure presents the aver- age Fl scores when the number of corpus groups varied. This indicates that as the number of groups increases, the model's performance improves. This
4.4.1 SEC filings dataset
We present empirical evidence that performance improvements can be achieved even for unseen cor- pus groups. To validate this, we select downstream tasks derived from SEC filings: NER, FiNER, and FinQA. Table 3 shows the performance of our model trained on different pretraining corpus. The model pretrained on the four corpus groups without SEC filings outperform the model further trained on SEC filings only. Furthermore, despite the use of a larger number (3.1B) of tokens derived

---

FinQA
Model
Tokens
# Groups
GPU(power)
GPU Time
Total Energy Consumption
Prog Acc
FiLM
2.48
5
RTX309000.36RW
23h
8.3kWh
79.78
82.02
58.85
-only SEC filings
3.IB
1
RTX309010 36k
32b
11.2kWh
75.51
81.91
57.45
FinBERT-Y
4.9B
2
Testa P1000.25kW)
192h
48.0kWh
70.40
81.08
38.79
Table 4: Total energy consumptions for training FiLM and FinBERT-Y. The GPU time of FinBERT-Y is from Yang et al. (2020).
from SEC filings for further pretraining, this model does not exceed the performance of the model trained using fewer (B. tokens from four dif- ferent corpus groups This demonstrates that incor- porating diverse corpora can improve performance for unseen tasks, which can be more important than data volume. Furthermore, we anticipate that our model, denoted FiLM, may exhibit robust perfor- mance across wide range of financial downstream tasks derived from unseen financial corpora.
2023): Energy (Wh) GPU power (W) GPU time (h). Consequently, compared FinBERT-Y, FiLM exhibits an 82% reduction in total energy consumption while achieving an average perfor- mance of 10% gain.
Conclusion
We show that financial PLMs struggle with various financial tasks due to constrained pretraining data diversity. To address this problem, train FiLM model using a broad spectrum of financial data. We empirically show that FiLM outperforms other financial PLMs across six financial down- stream tasks. Specifically, FiLM works robustly on unseen tasks and also attains superior perfor- mance on macroeconomies- more, our experimental results indicate training diverse corpora reduces energy consumption, sulting in environmental benefits. Our study under- scores the significance leveraging diverse data to train
4.4.2 Macroeconomic perspective
The FOMC task is a macroeconomics- based task designed to predict changes monetary policy stance (Shah et al., 2023). This dataset comprises sentences extracted from FOMC speeches, meet- ing minutes, and press conferences. None of these sentences are included in pretraining dataset used for FiLM. To substantiate that FiLM performs ro- bustly unseen tasks, evaluate models the FOMC task. Table represents that existing finan- cial PLMs underperform RoBERTa, as pointed out Shah et al. (2023). Mcanwhile, our FiLM model outperforms RoBERTa. This highlights that FiLM is the first model to surpass RoBERTa in financial domain. For the results of all experiments of the FOMC task and their detailed explanation, please refer Table and Appendix D, respectively.
task. Further-
language models.
Limitations
Financial documents often attribute substantial sig- nificance to numerical data, more so than other types of documents. Therefore, we acknowledge the necessity of an efficient technique for process- ing numerical tokens, particularly for financial PLMs. However, when we tested several meth- ods proposed in previous studies, we did not ob- serve any significant improvement. Techniques for handling numeric tokens are excluded from our study; however, we highlight this as a valuable area for future investigations. Finally, our FiLM is an encoder-based model that is not suitable for gener- ative tasks, such financial news summarization.
4.5 Cost-efficiency
Recently, the environmental impacts of large lan- guage models have become critical issue due their significant energy consumption and car- bon emissions during the training model pro- cess (Strubell et al., 2019; Scao et al., 2022; Zeng et al., 2022; Touvron et al., 2023). Since further training is required for developing domain-speciin PLMMs additional resources are consumed. We pro- vide empirical evidence that when ensuring corpus diversity, energy consumption is decreased while enhancing performance. Table 4 shows the total en- ergy consumption for training financial PLM. We compute the electric energy required for training a model, using the following formula (Touvron et al.,
A Acknowledgments
This work was supported by Institute of In- formation communications Technology Plan- ning & Evaluation(IITP) grant funded by the Ko- rea goverM (No. RS- 2022-002661068

---

Development of Lightweight Multimodal Anti- Phishing Models and Split-Lcaming Techniques for Privacy-Preserving Anti-Phishing). (No.RS- 2022-00155885, Artificial Intelligence Conver- gence Innovation Human Resources Development (Hanyang University ERICA)), and (2018-0-00192, the National Program for Excellence in SW). This work was supported by the National Re- search Foundation of Korea(NRFF grant funded by the Korea gMVerimen(MMSTT) (No. NRF- 2022R AOP2RGIATON 10135545.. Finally, we thank the re- viewers for their detailed feedback, which helped to improve the quality of this paper.
Katherine Lee. Daphne Ippolito. Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Buni Burch, and Nicholas Carlini. 2022. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 8424-8445, Dublin, Ireland. Association for Computational Linguistics.
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint 19t..1197..12
Zhuang Liu, Degen Huang, Kaiyu Huang. Zhuang Li, and Jun Zhao. 2021. Finbert: pre-trained finan- cial language representation model for financial text mining. In Proceedings of the twenty-ninth interna- tional conference on international joint conferences on artificial intelligence, pages 4513-4519,
References
Emily Alsentzer, John R Murphy, Willie Boag, Wei- Hung Weng. Di Jin, Tristan Naumann, WA Redmond, and Matthew BA McDermott. 2019. Publicly avail- able clinical bert embeddings. NAACL HLT 2019, page 72.
Lefteris Loukas, Manos Fergadiotis, lon Androutsopou- los, and Prodromos Malakasiotis. 2021. EDGAR- CORPUS: Billions of tokens make the world go round. Proceedings of the Third Workshop on Economics and Natural Language Processing, pages 13-18. Punta Cana, Dominican Republic. Associa- tion for Computational Linguistics.
Julio Cesar Salinas Alvarado, Karin Verspoor, and Timo- thy Baldwin. 2015. Domain adaption of named entity recognition to support credit risk assessment. In Pro- ceedings of the Australasian Language Technology Association Workshop 2015, pages 84-90.
Lefteris Loukas, Manos Fergadiotis, This Chalkidis, Eirini Spyropoulou, Prodromos Malakasiotis, lon Androutsopoulos, and Georgios Paliouras. 2022. FINER: Financial numeric entity recognition for xbrl tagging. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4419-4431.
Dogu Araci. 2019. Finbert: Financial sentiment analy- sis with tre-trained language models. arXiv preprint urXiv:7908 10063.
Tz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lunguage P Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IUL -JCNNPP, pages 3615-3620.
Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wal- lenius, and Pyry Takala. 2014. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65(4))782-776.
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, et al. 2021. Finqa: A dataset of numerical reasoning over financial data. arXiv preprint 210.002109.00122
Leland McInnes, John Healy, and James Melville. 2018. Umap: Uniform manifold approximation and pro- jection for dimension reduction. arXiv preprint 10Xiv:180..03426.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLL pages 3982 3992, Hong Kong, China. Association for Com- putational Linguistics.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171 4186.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ilie. Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Mathias Gallé, et al. 2022. Bloom: A 176b- parameter acen-access multilingual language model. arXiv preprint arX
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: pre-trained biomedical language representation model for biomedical text mining. Bioinformatiic, (6(4)):1334-222

---

Agam Shah, Suvan Paturi, and Sudheer Chava, 2023. Trillion dollar words: A new financial dataset, task market analysis. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Pupers), pages 6664-6679, Toronto, Canada. Association for Computational Lin- guistics.
Investing.com com": Investing.com is financial platform and news website that offers a wide range of information including stocks, fu- tures, options, and commodities. The his- torical dataset is a compilation of news data sourced from Investing.com between 2009 and 2020.02.
Raj Sanjay Shah, Kunal Chawla, Dheeraj Eidnani, Agam Shah, Wendi Du, Sudheer Chava, Natraj Ra- man, Charese Smiley, Jiaao Chen, and Diyi Yang. 2022. When flue meets flang: Benchmarks and large pre-trained language model for financial do- main. arXiv preprint arXiv:2211 1,00000..
NYtimes 5: The New York Times is a well- known newspaper worldwide. The NYTimes Dataset comprises finance-related articles col- lected from the NYTimes website. The data collection period spanned from 2005 to 2021.
Ankur Sinha and Tanmay Khandait. 2021. Impact of
EIA: The U.S. Energy Information Admin- istration (EIA) gathers, analyzes, and shares unbiased energy information to help create better policies, ensure efficient markets, and provide a public understanding of the impact of energy on the economy and environment.
news on the commodity market: Dataset and results. In Advances in Information and Communication: Proceedings of the 2021 Future of Information and Communication Conference (FICC), Volume 2, pages 609-601. Springer.
Emma Strubell, Ananya Ganesh, and Andrew McCal- lum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages
We collected news data only from the infor- mation provided by cia.gov.
SEC filings The SEC receives annual re- ports (10-K) and quarterly reports (10-Q) from public US companies. These reports con- tains information about company's 's business and financial performance. We downloaded the reports from SEC Edgar between 2020 and 2021. For the experiment(Section 4.4), additional data from 1993 to 2019 were in- cluded (Loukas et al., 2021).
Florence, Italy. Asso- ciation for Computational Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Anne Lachaux. Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint a7X7
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kam- badur, David Rosenberg, and Gideon Mann. 2023. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303. 176..
Earnings call: Earnings conference calls are important in delivering corporate information. Secking Alpha? provides access to carnings conference call transcripts, and we collected the data.
Yi Yang, Mark Christopher Siy Uy. and Allen Huang. 2020. Finbert: A pretrained language model for financial communications. arXiv preprint arXiv:2006.08097.
Arxivă: This dataset is a collection of ab- stracts from economics research sourced from arxiv.org
AIHUB: This dataset comprises a collection of Korean academic papers obtained and trans- lated by professional translators. Furthermore, professors specializing in translation studies reviewed the translated corpus. We used sub- set of the corpus that focused on economics- related data. This research used datasets from 'The Open AI Dataset Project (AI-Hub, South
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang. Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm- 130b: An open bilingual pre-trained model. arXiv preprint 1rX0v:2210.02114.
A Pretraining information
A.1 Datasets
TRC2:
TThe Thomson Reuters Text Re- search Collection (TRC2) corpus comprises 1,800,370 news stories published from 2008 to 2009. For more information the data and acquisition, please refer to h/tpec nist. /ettg///aat//atttee/rrretesss html
hhttps://ww/ kaggle. comm//daaasettt/ggeddddrr. us equiiiessseeeeee data
Phttps://www/ www.:wyytmmee..mmmmm htm.es.com/seetioo/bbusssssssss economy
hhttps://ww.sec....
" /httpss//seekkngaalhaaa con/
http:///aaavv..oosssrrceeeeo

---

Korea)', All data information can be accessed through 'AI-Hub' www...wwwul aihub. or kr).
C Existing financial PLMs
We provide a summary of the financial PLMs used in previous research and organize the huggingface hub model used for the experiments. In the list below, items marked with "v" indicate models that have undergone further training, while those with- out the mark are models trained from scratch.
FinWEB": The dataset a website that pro- vides economic knowledge and information on finance, loans, and products. We collected data by crawling websites.
Investopedial The dataset is a financial web- site that functions as an economic dictionary, similar to Wikipedia, and provides definitions of cconomic terms. We collected all the cco- nomic terms available on the website.
FinBERT-A (Araci, 2019) is the initial fi- nancial PLM. It is trained using the TRC2 dataset. FinBERT-A focuses on the FPB task and demonstrates superior performance compared with the original BERT. https:
A.2 Hyperparameters for pretraining
Table 5 shows the hyperparameters used further pretrain FiLM.
FinBERT-Y (Yang et al., 2020) used more than three datasets compared with FinBERT- A and validated its performance across three additional tasks. In addition, in- stead of using the traditional BERT tok- enizer and pretraining for the financial ap- proach, FinBERT-Y generates and applies a financial vocabulary, resulting in improved performance. https: http:/:://hggingaaaeee co/ yiyanghkust/finbeett-com tone
Hyperparameters
FILM
Initializaton
RoBERTa base
Learning rate
lc-5
Batch size
16
Max epochs
Max length
512
Weight decay
Warmup steps
Table Pretraining hyperparameters setting.
B Fine-tuning methodologies
FinBERT-L (Liu et al., 2021) collected a gen- eral domain corpus (3.3B tokens) and fi- nancial corpus (12.7B tokens) to train the fi- nancial BERT model. Unlike the traditional further pretraining approach, FinBERT-L. em- ploys multitask self-supervised pretraining during the training process. However, because the proposed model not publicly available, a comparative experiment could not be con- ducted.
We introduce the methods applied for finetuning PLMs on financial tasks well as the experimen- tal settings used. All tasks followed the methods and datasets proposed in each study. The headline task was performed using a sentiment classification method proposed by the authors" Table 6 lists the descriptions, dataset sizes, and metrics for each task. We followed the finetuning method of BERT. Furthermore, classification tasks (FPB, Headline, FOMC) were finetuned using sequence classifica- tion, which involved passing the [CLS] token rep- resentation for processing. The entity recognition (NER, FiNER) task follows token classification finetuning using word representations. FinQA fol- lows the question answering system introduced by Chen et al. (2021). Table 7 provides the parameters used to finetune each task are provided. FiNER and FinQA 13 can be accessed and reviewed on the official GitHub repository provided.
SEC-BERT (Loukas al., 2022) the first introduce the FINER task. The BERT model was pretrained exclusively using the SEC filings. This study emphasized con- structing vocabulary solely from the finan- cial reports dataset, with specific focus on numerical tokens found in financial reports. In addition, SEC-BERT proposes a method for substituting numeric tokens in the con- text of FINER. https: /ttgggggggaace...c/////// co/ nlpaueb/sec-bers b-
"https://www. finweb. com/
"https:///w investopedia. com/
FLANG-BERT & FLANG-ROBERTA (Shah al., 2022) is the first to create a bench- mark dataset for financial tasks by aggregat- ing a diverse range of tasks. addition,
"https://www http://wwwwwggl...ee// com/datasets znngr
montiienn-aalllss--nnnooo ty-
/nnttps://githut httpttuubb.com/nnppuue///ieeee Phttps: t/gittub/// wttptttub..coo/ccyssss///ii

---

Dataset Size
Name
Task
Metric
Train
Valid
Test
FPB
Sentiment classification
3,391
726
726
Accuracy F-1
NER
Named entity recognition
932
232
302
F-1
Hcadline
News headlines classification
7,989
1,141
2,282
F-1
FINER
Numeric entity recognition
900,384
112,494
108,378
F-1
FinQA
Question answering
6,251
883
1,147
Accuracy(Prog & Exc)
FOMC
Sentiment classification
1,588
396
496
F-1 (Combined-S)
Table 6: Summary financial tasks and their main aspects.
Hyperparam
FPB
NER
Headline
FOMC
FiNER
FinQA
Learning rate
[Te-3, Ic-4, Ic-5, 2o-5, 3e-5, 4c-5, 5c-5]
lc-5
2c-5
Batch size
{16, 32, 64)
{8, 16)
{64, 128)
{8, 16, 32)
16
16
Max epochs
20
100
100
100
30
300
Max length
(64, 128)
512
64
256
200
512
Table 7: Hyperparameter settings for each downstream task.
this study investigated and applied pretraining methods optimized for economies during the finetuning process. https://huggingfef Co/SALT-NLP/FLANG-
FiLM demonstrates superior performance across all types with the exception of "PC". When com- pared to AaBERTa-basc there is an improvement of +1.6 in performance in the "Combined". No- tably, there are substantial increases in performance metrics for "MM-S" and "SP", with "MM-S" show- ing 3.98% improvement and "SP" 4.69% im- provement.
Results of the FOMC task
In this section, provide supplementary expla- nations for the results presented in Table 9, The FOMC task is composed of datasets that are cate- gorized into three types:
E Comparison with financial LLM
Wu al. (2023) introduced the BloombergGPT, language model with 50 billion parameters trained on large-scale financial corpus. Ta- ble presents the results of the BloombergGPP and encoder-based PLMs on FPB and NER. For the BloombergGPT, we report the numbers pro- vided in this study (Wu et al., 2023). Note that BloombergGPT was evaluated by conduct- ing 5-shot learning the FPB and 20-shot learn- ing the NER. Remarkably, FiLM, BERT, and RoBERTa outperformed BloombergGPT, which had many more parameters and required higher cost. This demonstrates the limitations of using large language models for in-context learning in financial domain.
Meeting Minutes (MM): These are reports de- rived from the FOMC's eight annually sched- uled meetings.
Press Conference Transcripts (PC): These in- clude prepared remarks as well as the Q&A sessions between the Federal Reserve Chair and the press.
Speeches (SP): This category comprises any speeches delivered by officials of the Federal Reserve.
Additionally, there a "Combined" category that merges all three aforementioned types.
Texts with "-S" indicator signify that they have been split. This is because the FOMC often em- ploys neutral sentences to maintain market stability and minimize excessive reactions. To address this issue, the authors of this study employed rule- based approach to split sentences that exhibit neutral stance. All scores were obtained by strictly following the settings provided in the GitHub link in the Shah et al. (2023). Numbers in parentheses indicate the standard deviation.
F Visualization of financial domain datasets
To examine the distribution of sentences from fi- nancial domain datasets in the embedding space in Figure 1, sampled 10,000 sentences from each pretraining dataset and 500 sentences from each downstream task. Then, we generate embed- ding representations for sampled sentences using the approach proposed Reimers and Gurevych
httpss: //gi /tub.bttt.tttuteee wtw gt intechlttthbb// fomc- hawkishwhhhh... sh- dovish

---

FPB
NER
Headline
FOMC
Model
Std(ace)
Std(F-1) I)
Std(F-1)
Std(F-1)
Std(F-1)
BERT
1.302
0.682
1.479
1.852
1.49
RoBERTa
0.908
0.734
0.961
1.385
FinBERT-A
1.540
0.897
1.087
1.102
FinBERT-Y
1.014
1.734
0.927
1.482
FLANG-BFEE BERT
0.202
1.197
0.951
0.429
1.661
FLANG-RoBERTA
0.835
0.914
5.002
1.233
0,509
BECBBRRT
0.986
0.676
1.642
0.637
2.778
FiLM
0.605
0.702
1,528
1.387
1.802
FiLM (5.5B)
0.724
0.854
1.446
1.507
1.58
Table The standard deviation of model performance for FPB, NER, Headline, and FOMC.
Model
MM
MM-S
PC
PC-S
SP
SP-S
Combined
Combined-S
57.82
63.42
45.33
53.26
61.93
61.92
62.60
63.81
BERT
(5.182)
(2.705)
(9.604)
(3.686)
(3.072)
(0.843)
(1.154)
(1.49)
66,96
66.19
54.08
54.55
65.08
66.99
69.04
69.16
RoBERTA
(4.619)
(4.048)
(0.944)
(9.701)
(2.036)
(1.66)
(0.784)
(1.385)
59,98
65.01
45.04
51.33
65.59
62.82
63.93
64.50
FinBERT-A
(3.786)
(2,654)
(5.62)
(5,262)
(2.341)
(3,419)
(2.625)
(1.102)
58.39
60.24
46.36
52.45
63.40
61.87
59.41
64.30
FinBERT-Y
(3.351)
(1,093)
(8.636)
(16.101)
(0,599)
(1,155
(3.081)
(1.482)
61.54
66.71
51.74
47.03
62.35
62.57
63.39
64.93
BLANG-BERY
(5.214)
(1.601)
(8.891)
(7.254)
(2.973)
(0.874)
(1.261)
(1.661)
62.30
67.18
49.44
65.18
64.82
68.02
FLANG-ROBERTH
(2.813)
(1.711)
(2.595)
(2.265)
(1.754)
(1.553)
(2.853)
(0.509)
64.46
67.64
53.67
44.51
67.10
65.11
68.10
65.06
SEC-BERT
(8.053)
(3,455)
(2,238)
(14,825)
(2,015)
(3.114)
(0,525)
(2.778)
67.54
70.17
52.27
54.09
69.77
68.66
70.64
69.60
FiLM (2.4B)
(4.062)
(1.682)
(3.322)
(2.2)
(2,118)
(1.901)
(1.077)
(1,802)
Table 9: Results of the FOMC task.
Model
Params
FPB F-1
NER F-1
G Quantitative analysis for the corpus grouping
BloombergGPP
50B
51.07
60.82
BERT
110M
81.73
75.09
To confirm whether the corpus groups we divided have distinct characteristics, we visualized sentence embeddings for each group in Figure 1a. Further- more, in this section, we aim to establish the unique features of each group through quantitative analy- sis. We calculate the mean distances between sen- tence embeddings within corpus group (Table 11) and across corpus groups (Table 12). Inter-group distances are generally larger than intra-group dis- tances. Thus, sentences within the same group have certain similarities. This result supports our claim that creating training data using diverse cor- pus groups is more effective approach. This is because the distinct characteristics of each group contribute to more comprehensive and varied learning experience.
RoBERTa
110M
83.93
78.81
Fil.M (2.4B tokens)
110M
84.48
79.79
Table 10: Results on FPB and NER for BloombergGPT, BERT, RoBERTa, and FiL.M
Intra-group distance
News
3.529
SEC filings
1,819
Earnings call
2.557
Papers
2.568
MISC
3,378
Table 11: Mean distances among sentence embeddings
within a corpus group.
H Effectiveness of the deduplication
We compared the results before and after apply- ing the preprocessing step to the proposed method (Section 2.2). Specifically, we compare the model's 's performance trained on the entire corpus before pre-
(2019). To visualize sentence embeddings, we reduce the dimensionality of embeddings using UMAP (McInnes et al., 2018).

---

News
SEC hlings
Earnings call
Papers
MISC
News
5.062
3.574
3,542
4.933
SEC filings
5.062
3.832
4.763
3.910
Inter-group distance
Earnings call
3.754
3,832
2.917
3.518
Papers
3.542
4.763
2.917
4.008
MISC
4.933
3.910
3,518
4.008
Table 12: Mean distances among sentence embeddings across corpus groups.
Model
Tokens (Financial Corpus)
FPB
NER
Headline
FINER
FİnQA
FOMC
Accuracy
F-1
F-1
F-1
F-1
Prog Acc
Exe Acc
F-1
Belore
3.3B
86.70
83.38
79.16
89.74
81,95
58.50
60.68
67.89
[Ater]
2.48
86.25
84.48
79.78
91.79
82.02
58.85
61,38
69.60
Table 13: Compare the performance of before and after preprocessing models.
processing, referred as the 'Before' model, with the performance of FiLM model trained using the proposed method after preprocessing. Table 13 presents the results comparison. In addition, we compare the number duplicate sentences before and after preprocessing for each dataset, as shown in Table 14.
Top1000 Unigram Overlap(%)
FPB
46
37
42
-70
NER
31
47
31
27
39
-60
Headliee
22
21
28
-50
76
39
40
Original sentences
Duplicate sentences
Duplicate ratio
Name SEC filings
FinDA
76
41
52
13,969.168
5,565,936
39.84
-30
Earnings call
77,864.753
1,789,634
2.30
FOMC
45
35
48
43
42
TRC2
7.784.537
1,297,580
16.67
NYtimes
3,424,448
69.926
2.04
SEC hings
Papers
MISC
Papers
EIA 1,777.151
8,589
0.48
Eariings
45,496
1,966
4.32
Investing.com
220,890
0.19
FinWEB
144,456
390
0.27
AIHUB 278,054
0.00
Figure 3: Vocabulary overlap ratio between pretraining
Investopedia
238,344
0.00
and downstream task datasets.
Total
105,747,297
8,734,450
8,26
Table 14: Comparison original sentences and dupli- sentences.
closest corpus groups for each downstream task in the embedding space. markers represent sen- tence embeddings in corpus groups, while X mark- indicate sentence embeddings in downstream task. The mean distance of sentence embeddings between each corpus group and downstream task are calculated, and the two nearest groups are se- lected based on these distances. We discover that there are slight differences between corpus groups identified as similar in the embedding space and those identified as similar based on the vocabulary overlap ratio. For instance, noted that the News corpus group demonstrated similarity to FPB in vo- cabulary overlap, but the SEC-filings corpus group showed similarity to FPB within the embedding space. This indicates that multiple factors should be considered comprehensively when assessing the similarity between the datasets.
I Similarity between corpus groups and downstream tasks
In this section, conduct qualitative analysis using corpus embeddings to measure the similar- ity between corpus groups and tasks. The Fig- ure la embedding map illustrates that even within the same Financial Domain Dataset, each corpus has distinct characteristics, leading to a wide dis- tribution in the embedding space. Furthermore, to confirm similar corpus groups for each down- stream task in financial domain, caleulate both the vocabulary overlap ratio and distances in the embedding space. Figure 3 shows the vocabulary ratio between the corpus groups and downstream tasks. The top 1,000 most frequent unigrams are used to calculate the ratio. Figure 4 shows the two

---

NER
News
MISC
filings
SEC filings
FPB
NER
Headline
FINER
News
SEC filings
Papers
MISC
Headline
FINER
FinQA
FOMC
SEC filings
News
MISC
Papers
FinQA
FDMC
Figure 4: Two nearest corpus groups to each downstream dataset in embedding space.


Reviews:
[{"review_id": "Q42GWlRDdZ", "reviewer": "Reviewer_8cJY", "summary": "", "questions": "Please refer to the weaknesses section.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough and detailed comments. We have carefully addressed all the points you raised. \n\n> The analysis lacks essential details. The experimental setup's clarity, like the variance and mean of fine-tuning performance across tasks, and the rationale behind \"# of tokens NA\" for FLANG-BERT, remains incomplete.\n> \n\nOur experimental setup was primarily based on the FLUE benchmark (Shah, Raj, et al. 2022)[1]. However, for tasks that were not included in the FLUE benchmark, such as FiNER and FinQA, we adhered to the settings in the respective papers and their GitHub repositories.\n\nRegarding the section labeled \"# of tokens NA,\" we marked it as NA in Table 1 of our paper because the corpus used in Shah, Raj, et al. (2022) records the number of documents instead of the number of tokens. However, as you have pointed out, we will replace NA with a \"# of documents\" notation.\n\nWe will clarify the experimental setups in the revised paper.\n\n> For the FLANG-<model> family, a RoBERTa version of the model also exists. Comparing a RoBERTa-based FLANG with FiLM would provide a more equitable comparison.\n> \n\nThanks for the valuable feedback to improve our experiments. Following the reviewer’s suggestion, we conducted experiments with the FLANG-RoBERTa model. The proposed model FiLM outperforms FLANG-RoBERTa in all settings. The result demonstrates the effectiveness of our method.\n\nWe will include the experimental results of FLANG-RoBERTa in our revised paper.\n\n| Model | FPB |  | NER | Headline | FiNER | FinQA |  |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| Metric | Accuracy | F-1 | F-1 | F-1 | F-1 | Prog Acc | Exe Acc |\n| FLANG-BERT | 86.18 | 83.85 | 76.35 | 93.20 | 81.52 | 49.17 | 51.43 |\n| FLANG-RoBERTa | 85.78 | 83.14 | 71.21 | 93.41 | 80.77 | 30.68 | 32.17 |\n| FiLM | 87.17  | 85.32  | 81.63  | 93.69  | 82.02  | 58.85 | 61.38 |\n| FiLM(5.5B) | 88.06 | 84.68 | 80.67 | 93.86 | 82.39 | 59.37 | 61.64 |\n\n> The paper concludes that enhanced corpus pre-training diversity leads to improved results across tasks, a widely accepted notion in the NLP community, substantiated by numerous research papers. Therefore, the presented results lack novelty unless I've overlooked a crucial aspect of the paper.\n> \n\nWhile it is widely accepted that using a diverse corpus is important for improving generalization performance, most domain-specific studies focused on collecting a large corpus and overlooked the importance of corpus diversity. Below, We’d like to emphasize the unique and important aspects of our paper:\n\n1. **Cost-efficiency**\n    \n    Compared to FinBERT-Y, our FiLM model exhibits an 88% decrease in GPU time, while achieving an average performance increase of 10%. Our study empirically demonstrates that harnessing diverse corpus groups enhances model performance while reducing the number of tokens, GPU time, and energy consumption.\n    \n    | Model | # Tokens | # Corpus Groups | GPU (power) | GPU Time | Total Energy Consumption  | NER(F1) | FiNER(F1) | FinQA(F1) |\n    | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n    | FiLM | 2.4B | 5 | RTX3090 (0.36kW) | 23h | 8.3kWh | 81.63 | 82.02 | 58.85 |\n    | FinBERT-Y | 4.9B | 2 | Tesla P100 (0.25kW) | 192h | 48.0kWh | 72.38 | 81.08 | 38.79 |\n2. **Beyond data volume: corpus diversity as a significance factor for generalization performance**\n    \n    Even with 1 billion fewer tokens trained, the model trained on the remaining 4 corpus groups, excluding SEC-filings, exhibits superior performance compared to the model exclusively trained on SEC filings, when evaluating them on NER, FiNER, and FinQA tasks derived from SEC-filings. (Table 3) This result emphasizes that corpus diversity plays a significant role in the generalization performance of the model, beyond the data volume.\n    \n    | Model | # Tokens | # Corpus Groups | GPU (power) | GPU Time | Total Energy Consumption  | NER(F1) | FiNER(F1) | FinQA(F1) |\n    | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n    | FiLM w/o SEC Filings | 2.1B | 4 | RTX3090 (0.36kW) | 20h | 7.2kWh | 80.55 | 81.94 | 57.54 |\n    | FiLM only SEC Filings | 3.1B | 1 | RTX3090 (0.36kW) | 32h | 19.8kWh | 79.76 | 81.91 | 57.45 |\n3. **The first PLM to surpass RoBERTa in various financial tasks**\n    \n    As pointed out in a recent work Shah et el. (2023)[2],  there has been no financial PLM that outperforms RoBERTa in various financial tasks. Our FiLM model is the first model to surpass RoBERTa in the financial domain. To substantiate this claim, we additionally evaluate our FiLM model on FOMC task(Shah et al., 2023 [2]). The result shows superior performance of our model compared to existing financial PLMs. We will report these results in the camera-ready version of our paper.\n    \n    | Model | FOMC (F1-score) |\n    | --- | --- |\n    | FinBERT-A | 0.6325 |\n    | FLANG-BERT | 0.6307 |\n    | FLANG-RoBERTa | 0.6482 |\n    | RoBERTa-base | 0.6755 |\n    | FiLM | 0.7064 |\n\nIn summary, the importance of corpus diversity has been neglected in previous PLM works. Our study highlights that leveraging diverse corpus groups not only boosts model performance but also significantly (more than 80%) cuts down on GPU time, and energy consumption.\n\n> The paper lacks a qualitative error analysis demonstrating why FiLM outperforms other models on the issues mentioned in the introduction.\n> \n\nOur qualitative analysis is conveyed through Figure 1, which focuses on Corpus Embeddings. The embedding map illustrates that even within the same Financial Domain Dataset, each corpus has distinct characteristics, leading to a wide distribution in the embedding space. For a deeper understanding of the relationship between the tasks and the corpora, we recommend referring to Figure 4 in Appendix Section H.\nFurthermore, as the reviewer suggested, we will enhance qualitative error analysis. Specifically, we are going to conduct a detailed analysis of instances where FiLM made errors in various tasks, examining both the model's limitations and potential areas for improvement. Additionally, we will analyze cases where FiLM correctly answered questions that other models got wrong. Through this, we aim to clearly elucidate the unique strengths of the FiLM model.\n\n[1] Shah, Raj, et al. \"When FLUE Meets FLANG: Benchmarks and Large Pretrained Language Model for Financial Domain.\" *Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing*. 2022.\n\n[2] Shah, Agam, Suvan Paturi, and Sudheer Chava. \"Trillion Dollar Words: A New Financial Dataset, Task & Market Analysis.\" *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL)*. 2023. [https://arxiv.org/abs/2305.07972](https://arxiv.org/abs/2305.07972)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Foundational models pre-trained for specific domains often fall short in performance compared to their open-domain counterparts. Identifying the issues and potential solutions to address this challenge is crucial. The authors of this paper have centered their focus on this significant problem and constructed an analysis around it.", "reasons_to_reject": "1. The analysis lacks essential details. The experimental setup's clarity, like the variance and mean of fine-tuning performance across tasks, and the rationale behind \"# of tokens NA\" for FLANG-BERT, remains incomplete.\n\n2. For the FLANG-<model> family, a RoBERTa version of the model also exists. Comparing a RoBERTa-based FLANG with FiLM would provide a more equitable comparison.\n\n3. The paper concludes that enhanced corpus pre-training diversity leads to improved results across tasks, a widely accepted notion in the NLP community, substantiated by numerous research papers. Therefore, the presented results lack novelty unless I've overlooked a crucial aspect of the paper. \n\n4. The paper lacks a qualitative error analysis demonstrating why FiLM outperforms other models on the issues mentioned in the introduction.", "questions_to_authors": "Please refer to the weaknesses section.", "paper_topic_and_main_contributions": "The paper investigates the impact of diversity in the pre-training data for foundational models trained for financial domain. The authors show that more diversity leads to better overall performance across a variety of tasks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I believe a proofreading would improve the paper's overall presentation.\n1. There are a lot of details which are not quite important for the main section of the paper, such as, the discussion around the corpus's embeddings.\n2. In the abstract, acronym FiLM is used while it has not been introduced before. \n3. Taking an example of the first sentence of the conclusion - it could be worded better and concise as \"We show that financial PLMs struggle with diverse financial tasks due to constrained pretraining data diversity\". Similarly, there are a lot of improvements that can be made throughout the paper."}}, {"review_id": "UkYPqy9ThC", "reviewer": "Reviewer_dzK5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and valuable feedback!\n\n> It is unclear wether the other specialized (financial) LM have been trained from scratch or domain adaptions of general purpose models as is the case of FiML.\n> \n\nFinBERT-Y and SEC-BERT are trained from scratch, while FinBERT-A and FLANG-BERT are domain-adaptive PLMs as is the case of our FiLM models.\n\nWe will include this information in our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This study contributes significantly to the field by introducing and making a new financial language model available to the public. Moreover the authors characterize the training dataset providing insights regarding the importance of input diversity for efficient pre-training and better model generalizability. The paper is very clear and well written. ", "reasons_to_reject": "It is unclear wether the other specialized (financial) LM have been trained from scratch or domain adaptions of general purpose models as is the case of FiML. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this study the authors develop FiML, a RoBERTA model adapted to the financial domain through additional pre-training of specialized corpus, and test its performance in comparison to other general purpose and specialized models on a diverse set of tasks. \nThe main contribution of the paper include the development and characterization of domain-specific training dataset, as well as the development of a publicly available financial-domain model, FiML, which outperforms baseline models on several downstream tasks. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RDEbzuURsB", "reviewer": "Reviewer_7icf", "summary": "", "questions": "A. Was there any analysis done on general out-of-domain tasks WiC, GLUE or MNLI?\n\nB. Can you add the state-of-the-art models evaluations for each of the tasks in table 2?\n\nC. What is the FiLM (5.5B) model mentioned in table 2, which showcases best performance in almost all of the tasks? I could not find reference to that specific training data which contains 5.5B tokens? ", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and valuable feedback!\n\n> The paper mainly focuses on domain-specific training/evaluation and fails to give reason as to why this is better than a pre-existing generalized LLM\n> \n\nThe underperformance of general-domain LLM arises from the distinct textual characteristics between general-domain and financial corpora. For instance, financial reports prominently include numerical information(Section 2.1.). To substantiate this claim, we compute the mean distances among sentence embeddings within and across domains. Consequently, the inter-domain distance are larger than the intra-domain distances. This difference in distance indicates dissimilarity between general-domain and financial corpora. Nevertheless, existing financial PLMs exhibit subpar performance on many financial tasks, even when compared to general domain PLMs, due to the deficiency of corpus diversity. However, our FiLM models outperform not only financial PLMs but also surpass general-domain PLMs in financial tasks.\n\n|  | Intra-domain |  | Inter-domain |\n| --- | --- | --- | --- |\n|  | Financial | General | Financial-General |\n| Avg distance between sentences | 3.7205 | 4.2048 | 6.2576 |\n\n> Though the work presented has stronger contributions to the financial community, the work fails to give a novel contribution to the NLP community. The work presents the improvements of PLM trained on more diverse data, which is common knowledge.\n>\n\nWhile it is widely accepted that using a diverse corpus is important for improving generalization performance, most domain-specific studies focused on collecting a large corpus and overlooked the importance of corpus diversity. Below, We’d like to emphasize the unique and important aspects of our paper:\n1. **Cost-efficiency**\n    \n    Compared to FinBERT-Y, our FiLM model exhibits an 88% decrease in GPU time, while achieving an average performance increase of 10%. Our study empirically demonstrates that harnessing diverse corpus groups enhances model performance while reducing the number of tokens, GPU time, and energy consumption.\n    \n    | Model | # Tokens | # Corpus Groups | GPU (power) | GPU Time | Total Energy Consumption  | NER(F1) | FiNER(F1) | FinQA(F1) |\n    | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n    | FiLM | 2.4B | 5 | RTX3090 (0.36kW) | 23h | 8.3kWh | 81.63 | 82.02 | 58.85 |\n    | FinBERT-Y | 4.9B | 2 | Tesla P100 (0.25kW) | 192h | 48.0kWh | 72.38 | 81.08 | 38.79 |\n2. **Beyond data volume: corpus diversity as a significance factor for generalization performance**\n    \n    Even with 1 billion fewer tokens trained, the model trained on the remaining 4 corpus groups, excluding SEC-filings, exhibits superior performance compared to the model exclusively trained on SEC filings, when evaluating them on NER, FiNER, and FinQA tasks derived from SEC-filings. (Table 3) This result emphasizes that corpus diversity plays a significant role in the generalization performance of the model, beyond the data volume.\n    \n    | Model | # Tokens | # Corpus Groups | GPU (power) | GPU Time | Total Energy Consumption  | NER(F1) | FiNER(F1) | FinQA(F1) |\n    | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n    | FiLM w/o SEC Filings | 2.1B | 4 | RTX3090 (0.36kW) | 20h | 7.2kWh | 80.55 | 81.94 | 57.54 |\n    | FiLM only SEC Filings | 3.1B | 1 | RTX3090 (0.36kW) | 32h | 19.8kWh | 79.76 | 81.91 | 57.45 |\n\nIn summary, the importance of corpus diversity has been neglected in previous PLM works. Our study highlights that leveraging diverse corpus groups not only boosts model performance but also significantly (more than 80%) cuts down on GPU time, and energy consumption.\n\n> Can you add the state-of-the-art models evaluations for each of the tasks in table 2?\n> \n\nSEC-BERT is the state-of-the-art model for NER, FiNER and FinQA tasks, while FLANG-BERT exhibits superior performance in other financial tasks. In conclusion, our FiLM model outperforms the-state-of-the-art model in each financial task (Table 2).\n\n> What is the FiLM (5.5B) model mentioned in table 2, which showcases the best performance in almost all of the tasks? I could not find reference to that specific training data which contains 5.5B tokens?\n>\nThank you for identifying an oversight in our paper. \n\nTo obtain the 5.5B token corpus, we use additional SEC filings data containing 3.1B tokens. Note that the additional data is also used in the experiment in Section 4.4. We will specify it in our revised paper.\n\n> Was there any analysis done on general out-of-domain tasks WiC, GLUE or MNLI?\n> \n\nSince our study focuses on domain-specific PLMs, the model evaluations have been concentrated on financial tasks. We are preparing the experiments on general domain tasks, however, we cannot include the result due to time constraints. We will report the performance of models on out-of-domain tasks in the camera-ready version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The work presents strong evidence in favor of inclusion of a more diverse dataset in model training \n\n* The work showcases better performance than in-domain LLM over domain-specific tasks", "reasons_to_reject": "* The paper mainly focuses on domain-specific training/evaluation and fails to give reason as to why this is better than a pre-existing generalized LLM\n\n* Even though the work showcases strong performance on in-domain tasks, the paper fails to report the performance comparison/effect on general out-of-domain tasks\n\n* Though the work presented has stronger contributions to the financial community, the work fails to give a novel contribution to the NLP community. The work presents the improvements of PLM trained on a more diverse data, which is a common knowledge. ", "questions_to_authors": "A. Was there any analysis done on general out-of-domain tasks WiC, GLUE or MNLI?\n\nB. Can you add the state-of-the-art models evaluations for each of the tasks in table 2?\n\nC. What is the FiLM (5.5B) model mentioned in table 2, which showcases best performance in almost all of the tasks? I could not find reference to that specific training data which contains 5.5B tokens? ", "paper_topic_and_main_contributions": "The paper discusses a financial language model called FiLM, abbreviated for Financial Language Model. The main research question for this work revolves around the premise that the financial literature data can be split into 5 major groups and most of the Pretrained Language Models (PLMs) further trained on financial data have only 2 out of 5 groups in their training corpus, as showed by multiple examples in the paper. \n\nThe main research question framed is whether a model trained on a more diverse training data would perform better then the existing PLMs. The author thoroughly explore the research question by curating a dataset which contains all of the 5 different data groups and further training a masked language model (RoBERTa) on a pre-processed version of the data. \n\nThey compare their FiLM model across 5 different tasks against 4 different existing PLMs and 2 baselines finetuned for each specific task. The results show that the FiLM model outperforms each of the 6 (4 PLMs + 2 baselines) models in all of the tasks. They also present an analysis of the effect of number of groups on the evaluation across an average F1 score across 4 different tasks. \n\nThe main contributions of the paper are:\n\n* Analysis of different pre-existing financial PLMs and the shortcomings in terms of data diversity in their datasets.\n\n* A new dataset curated which has a higher diversity across 5 different data groups for financial data\n\n* FilLM models trained on this curated data which outperforms even BloombergGPT which is exponentially bigger and trained on a much larger data corpus\n\n* An analysis of effect of number of groups included in the training dataset on different evaluation metrics. ", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dgEjOoopex", "reviewer": "Reviewer_ptSf", "summary": "", "questions": "- Most of the existing finance pretrained language models are either pretrained from scratch using BERT model architecture or pretrained after initializing from the BERT model. Why did the authors choose to initialize the model from the RoBERTa model instead of the BERT model? RoBERTa model is pretrained on 160GB of data, while the BERT model is pretrained on just 16B of data. So,initializing from the RoBERTa model will give an additional advantage to the FiLM model compared to the existing models, which are initialized from the BERT model. \n- In Table 2, the authors presented the results for FiLM model pretrained using 2.4B and 5.5B tokens. It is never mentioned in the paper, how 5.5B token corpus is obtained. \n- I suggest the authors to include experiments using FLANG-RoBERTa to have a more fair comparison. Here is the link for the model - https://huggingface.co/SALT-NLP/FLANG-Roberta\n\n", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and valuable feedback!\n\n> Q1. Most of the existing finance pretrained language models are either pretrained from scratch using BERT model architecture or pretrained after initializing from the BERT model. Why did the authors choose to initialize the model from the RoBERTa model instead of the BERT model? RoBERTa model is pretrained on 160GB of data, while the BERT model is pretrained on just 16B of data. So,initializing from the RoBERTa model will give an additional advantage to the FiLM model compared to the existing models, which are initialized from the BERT model.\n> \n\n> Q3. I suggest the authors to include experiments using FLANG-RoBERTa to have a more fair comparison. Here is the link for the model - https://huggingface.co/SALT-NLP/FLANG-Roberta\n> \n\nThanks for the valuable feedback to improve our experiments. Following the reviewer’s suggestion, we conducted experiments with the FLANG-RoBERTa model. The proposed model FiLM outperform FLANG-RoBERTa in all settings. The result demonstrate the effectiveness our method.\n\nWe will include the experimental results of FLANG-RoBERTa in our revised paper.\n\n| Model | FPB |  | NER | Headline | FiNER | FinQA |  |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| Metric | Accuracy | F-1 | F-1 | F-1 | F-1 | Prog Acc | Exe Acc |\n| FLANG-BERT | 86.18 | 83.85 | 76.35 | 93.20 | 81.52 | 49.17 | 51.43 |\n| FLANG-RoBERTa | 85.78 | 83.14 | 71.21 | 93.41 | 80.77 | 30.68 | 32.17 |\n| FiLM | 87.17  | 85.32  | 81.63  | 93.69  | 82.02  | 58.85 | 61.38 |\n| FiLM(5.5B) | 88.06 | 84.68 | 80.67 | 93.86 | 82.39 | 59.37 | 61.64 |\n\n> Q2. In Table 2, the authors presented the results for FiLM model pretrained using 2.4B and 5.5B tokens. It is never mentioned in the paper, how 5.5B token corpus is obtained.\n> \n\nThank you for identifying an oversight in our paper. \n\nTo obtain the 5.5B token corpus, we use additional SEC filings data containing 3.1B tokens. Note that the additional data is also used in the experiment in Section 4.4. We will specify it in our revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-written, easy to understand, and the findings are well supported by the experiment results.\n- The paper highlights the importance of having more diversified finance data in the pretraining corpus to get better models.\n- The FiLM model pretrained using just 2.4B tokens from diverse sources outperformed existing models pretrained using a less diversified corpus having more than 3B tokens.\n", "reasons_to_reject": "- The performance of FiLM is compared with existing models, which are either pretrained from scratch using BERT architecture or pretrained after initializing from the BERT model. Here the main concern is the initialization of FiLM from the RoBERTa model instead of the BERT model. ", "questions_to_authors": "- Most of the existing finance pretrained language models are either pretrained from scratch using BERT model architecture or pretrained after initializing from the BERT model. Why did the authors choose to initialize the model from the RoBERTa model instead of the BERT model? RoBERTa model is pretrained on 160GB of data, while the BERT model is pretrained on just 16B of data. So,initializing from the RoBERTa model will give an additional advantage to the FiLM model compared to the existing models, which are initialized from the BERT model. \n- In Table 2, the authors presented the results for FiLM model pretrained using 2.4B and 5.5B tokens. It is never mentioned in the paper, how 5.5B token corpus is obtained. \n- I suggest the authors to include experiments using FLANG-RoBERTa to have a more fair comparison. Here is the link for the model - https://huggingface.co/SALT-NLP/FLANG-Roberta\n\n", "paper_topic_and_main_contributions": "The authors observed that the existing pretrained language models in the finance domain are pretrained on less diverse finance data, and hence the authors investigated whether pretraining on diverse finance data results in better models. The authors pretrained the FiLM model (initialized from the RoBERTa model) using finance texts gathered from diverse sources and showed that FiLM outperformed existing models. Because of the diversity in the data, the FiLM model pretrained using just 2.4B tokens  outperformed the existing models pretrained using more than 3B tokens. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: A new LLM is trained on a diverse set of financial datasets and obtains improved performance on target financial related NLP tasks. Reviewers overall appreciated the improved empirical results on financial datasets and the release of  a new larger LLM on this domain. However, there was less excitement due to the very focused scope (only financial domain) and the lack of novelty in the method or conclusions (helpfulness of diversity is already known).

In terms of soundess, overall most reviewers found the evaluation solid. However, it can definitely be improved. For instance, figure 1 is unclear (sides a and b look identical only with/without colors?... and overall this 2D visualization is more suitable as an interpretability tool but is not a proof to motivate a method on top). Also, reporting confidence intervals would be better (e.g. with bootsrapping). Reporting out of domain results would be helpful. Table 2 is not explained (I assume underline is for second best?)
Decision: Accept-Findings