Evaluating Parameter-Efficient Finetuning Approaches for Pre-trained Models on the Financial Domain
Authors: Isabella Olariu
Cedric Lothritz
Jacques Klein
Tegawendé F. Bissyandé
Siwen Guo
Shohreh Haddadan
Venue: EMNLP 2023 Findings (2023)
Keywords: NLP
fine-tuning
parameter efficiency
financial domain

Abstract: Large-scale language models with millions, billions, or trillions of trainable parameters are becoming increasingly popular. However, they risk becoming rapidly over-parameterized and the adaptation cost of fully fine-tuning them increases significantly. Storing them becomes progressively impractical as it requires keeping a separate copy of all the fine-tuned weights for each task. By freezing all pre-trained weights during fine-tuning, parameter-efficient tuning approaches have become an appealing alternative to traditional fine-tuning. The performance of these approaches has been evaluated on common NLP tasks of the GLUE benchmark and shown to match full fine-tuning performance, however, their impact is less researched in domain-specific fields such as finance. This work compares the performance of a set of financial BERT-like models to their fully fine-tuned counterparts by leveraging different parameter-efficient tuning methods. We see that results are comparable to traditional fine-tuning while gaining in time and resource efficiency.

Evaluating Parameter-Efficiet Finetuning Approaches for Pre-trained Models on the Financial Domain
Isabella Olariu Cedric Lothritz', Jaarrit Klein', Tegawendé F. Bissyandé", Siwen G Shohreh Haddadan
o Zortify S.A. 19, Rue du Laboratoire, L-1911 Gare Luxembourg
*University of Luxembourg 6, Rue Coudenhove-Kalergi, L-1359 Luxembourg (isabella, siwen, shohrehj@zortiiyy. com jacques. klein, tegawende Setpawende.bssyaddeeeaaae. bissyandee@uni. lu
Abstract
Yang et al., 2020; Liu et al., 2021), FLANG-BERT (Shah al., 2022) or BloombergGPT (Wu et al., 2023). Despite their immense potential, these pre- trained models require fine-tuning to adapt their parameters to specific tasks, which becomes pro- hibitively expensive the larger the models are.
Large-scale language models with millions, bil- lions, trillions of trainable parameters are becoming increasingly popular. However, they risk becoming rapidly over-parameterized and the adaptation cost of fully fine-tuning them increases significantly. Storing them becomes progressively impractical as it requires keeping a separate copy of all the fine-tuned weights for each task. By freezing all pre-trained weights during fine-tuning, parameter-efficient tuning approaches have become an appealing alter- native to traditional fine-tuning. The perfor- mance of these approaches has been evaluated on common NLP tasks of the GLUE bench- mark and shown match full fine-tuning per- formance, however, their impact is less re- searched in domain-specific fields such as fi- nance. This work compares the performance of a set of financial BERT-like models to their fully fine-tuned counterparts by leveraging dif- ferent parameter-efficient tuning methods. We see that results are comparable to traditional fine-tuning while gaining in time and resource efficiency.
Recently, more purameter-ccffcient alternatives full fine-tuning such as Low-Rank Adaptation (LoRA; Hu et al., 2021), Prefix-Tuning (Li and Liang, 2021) or Adapters (Houlsby et al., 2019; Pfciffer et al., 2020, 2021) have been proposed to mitigate the vast adaptation cost of language models while maintaining full fine-tuning perfor- mance (Rücklé et al., 2021). These relatively new approaches share the same underlying principle of freezing the pre-trained weights of the base model, while fine-tuning only of newly introduced parameters, resulting in a substantial reduction the total number of trainable parameters. While parameter-efficien tuning methods have been eval- uated on generic benchmarks such as GLUE (Wang et al., 2018 8) and shown to reach results that are com- parable to full fine-tuning, they are less researched in domain speciffc fields such finance.
1 Introduction
In this paper, we evaluate the efficacy of different parameter-efficceet fine-tuning approaches applied to BERT-like models the financial domain. Our contributions are twofold: (a) we compare the per- formance of parameter-efficien tuning approaches on range of financial tasks against their fully fine- tuned counterparts, and (b) show both their model size and training time benefits.
Natural Language Processing (NLP) and the in- creasing popularity of large language models have redefined how computers understand and interpret human language. In particular, the financial do- main, fundamentally intertwined with language and communica communication, benefits from NLP techniques to process complex financial documents. This do- main deals with a vast array of unstructured textual data coming from sources such as news articles, fi- nancial reports, or investor opinions, each offering potentially valuable insights for processes such as risk assessment decision-making. To this end, numerous efforts have been made tailor large lan- guage models such BERT (Devlin et al., 2019) or GPT-3 (Brown et al., 2020) to the financial domain, resulting in models such as FinBERT(Arace 2019;
2 Experimental Setup
In this section, list our research questions, de- scribe the experimental framework we set up address these questions and briefly explain the parameter-efficient fine-tuning approaches that we apply.

---

2.1 Research Questions
et al., 2019). During fine-tuning, only the weights associated with the adapter layers and the predic- tion head are trained. Although there exist numer- ous variations of adapter- based fine-tur tuning, we use the approach of Houlsby et al. (2019) for this study.
We investigate the following research questions: RQI: How do parameter-efficient fine-tuning ap- proaches perform on financial tasks?
RQ2: What are the advantages or disadvantages in terms of model size and time efficiency?
2.4 Downstream Tasks
2.2 Baselines
To evaluate the performance of parameter-efficient tuning techniques, we fine-tune our pre-trained models using adapters and LoRA four tasks from the Financial Language Understanding Evalu- ation (FLUE) benchmark (Shah et al., 2022).
We examine three baseline models for comparison purposes: BERT-base as our "generic" baseline, as well as FinBERT and FLANG-BERT as baselines specialized for financial documents.
BERT-base: BERT-base (Devlin al., 2019) is a bidirectional Transformer model with 12 layers, 768 hidden units per layer, 12 attention heads, and a total of 110 million parameters.
Sentiment Classification. We use the Finan- cial Phrasebank dataset (FPB; Malo et al. (2014)) to recognize sentiment from financial news. The dataset consists of 1845 sentences categorized into positive, negative, and neutral labels.
FinBERT: FinBERT (Yang et al., 2020) is a domain-specific variant of the BERT model for the financial field. It was pre-trained on text derived from corporate reports, earnings call transcripts, and analyst reports, and uses a domain- fi- nancial vocabulary. There are several FinBERT models (Araci, 2019; Liu et al., 2021), we use the same variant as Shah et al. (2022) for this study, FLANG-BERT: FLANG-BERT is part of a suite of recently released financial language mod- cls (Shah et al., 2022). It is specialized through further training BERT within the financial domain and employing finance-specifie pre-training ob- jectives for enhanced understanding of domain- specific terms.
Sentiment Regression. We perform aspect- aspect-based sentiment analysis using the publicly available Fi- nancial Question Answering (FiQA) dataset (Maia et al., 2018), containing 1173 samples from head- lines, news statements, and microblogs. Sentiment scores range from -1 -1 1.
News Headline Classification. sensitive data in financial news headlines, such as price fluctuations, we use the Gold News Headlines dataset (Sinha and Khandait, 2020). It comprises 11 412 news headlines and nine corresponding tags such as Future Price, Past News, or Asset Compar- ison. The aim is to perform binary classification for all nine labels.
To detect time-
2.3 Parameter-efficient Fine-tuning
Named Entity Recognition. Named Entity Recognition (NER) aims to identify key financial entities to enhance the understanding of the rela- tionship between them. The dataset for this task consists of 1466 samples (Salinas Alvarado et al., 20155) and four entity types: PERSON (PER), LOCAA- TION (LOC), ORGANISATION (ORG), and MISCEL- LANEOUS (MISC).
Techniques
Low-Rank Adaptation. LoRA is a technique that inserts trainable rank decomposition matrices into each layer of a Transformer model. During fine-tuning, it freezes all the model's weights ex- cept those of the newly added low-rank matrices, a process that significantly reduces the number of trainable parameters and minimizes the memory footprint during model training (Hu al., 2021). Adapter-tuning. Adapter-tuning adds a set of new parameters to a Transformer in the form of new layers. The design of adapters varies as dif- possi-
2.5 Fine-tuning Parameters
We conduct hyperparameter searches using BERT- base for each task to determine the best values for the learning rate and batch size. The number of epochs varies across each fine-tuning approach, it ranges from 1 for full fine-tuning, for LoRA, and to 20 for Adapterи. The learn- ing rate chosen from (1e-7, 1e-6, le-5, 2e-5, 3e-5, 5c-5, Ic-4, 1c-3) and the batch size from {8, 16, 32). For LoRA, we further search for the
ferent configurations and placements ble (Stickland and Murray, 2019; Pfeiffer al., 2020). The adapter layer typically consists of down-projection, an up-projection, and a non-linear activation function, usually inserted after the multi- head attention or the feed-forward layer (Houlsby
100

---

rank!, alpha value², and dropout rate in {1,8, 16). {1,8, 16, 32), and {0.01, 0.05, 0.1] respectively. We re-use the same parameters for FinBERT and FLANG-BERT as they share the same base archi- tecture. See Appendix A. for details about the resulting hyperparameters used for each task.
The advantage of one method over the other, is not definitive, the effectiveness of each ap- proach in terms of performance varies depending on the task. Although, it can be seen that Adapter does not underperform the baseline for each model variant on all of the tasks except for NER. Over- all, these results suggest that parameter-efficient approaches are a suitable alternative to full fine- tuning and do not significantly hurt performance on financial tasks.
3 Experimental Results
In this section, we present the results from our ex- periments across four downstream tasks and answer the research questions introduced in Section 2. For each task, we fine-tune the pre-trained models over five runs and report the average performance as the final metric. Results are shown in Table I.
3.2
RQ2: What are the advantages disadvantages in terms of model size and time efficiency?
One clear advantage of LoRA and Adapter,, is the significant reduction in model size, for each task only the newly added parameters stored in the model file. Table 2 shows the variations in size and time taken for fine-tuning on Headline Classifica- tion. 3 Overall, we can see that the LoRA approach is the most parameter-efficieet variant with at most 0.3% of trainable parameters for the Headline Clas- sification task. We can see the same trend for the other tasks in Appendix A.4 where LoRA requires as little as 0.04% of updatable parameters for FiQA SR and and FPB.
3.1 RQ1: How do parameter-efficient fine-tuning approaches perform on financial tasks?
Table shows the results across four tasks of all models fine-tuned in a traditional way as well as fine-tuned with LoRA and the Houlsby adapter (Adapter)) by Houlsby et al. (2019). BERT-base, FinBERT, and FLANG-BERT represent the perfor- mance results obtained by Shah et al. (2022) on the FLUE benchmark. We reproduced these results, but because numbers differ slightly from the original ones, also report them in Table 1 and mark them by "(repr.)". We compare LoRA and Adapter against our reproductions. As shown in Table 1, applying LoRA and Ad leads to results that are comparable to the fully fine-tuned models for some of the tasks.
In terms of time efficiency, applying LoRA and Adapter reduced the training time for the largest datasct used for Headline Classification. For other tasks, the gain might not be as evident as shown in Appendix A.4.
4 Discussion
For FPB SC, BERT-base Adapter outper- forms BERT-base + LoRA and performs as well as BERT-base. BERT-base + Adapter also performs better than BERT-baas and BERT-base LoRA on FIQA SR, and outperforms BERT-base LoRA on Headline while reaching the same performance as fully fine-tuned BERT-base.
Our experiments show that performance on finan- cial tasks is not necessarily compromised with parameter-efficient tuning approaches. We see that even on the smaller FiQA and NER datasets for Fin- BERT, these approaches achieve almost the same performance as full fine-tuning. Due the small set of trainable parameters, the models might be less at risk of overfitting and catastrophic forget- ting (He et al., 2021).
FinBERT + LoRA outperforms the adapter- tuned and fully fine-tuned versions of the baseline on FPB SC and FiQA SR. respectively. FLANG- BERT+ Adapteræ beats FLANG-BERT LoRA on all tasks except on NER, and outperforms the fully fine-tuned
Regarding the performance between the two parameter-efffcient methods, we argue that Adapterи performs better than LoRA. It almost always maintains full fine-tuning performance, and tends to outperform LoRA more frequently. We believe this be the case because, compared LoRA. Adapter keeps a bigger share of trainable parameters. In addition, LoRA needs a much larger
BLAANGBERT on FiQA SR.
None of the parameter-efficcint approaches out- perform the baselines on NER, but FinBERT with both LoRA and Adapter closcly reaches that base- line performance.
DDmenssion of the low-rank matrices *Scaling factor for the weight matrices
"See Appendix A.4 for the remaining tasks.

---

Model
FPB SC
FIQA SR
Headline
NER
Metric
Accuracy
MSE
Mean F-1
F-1
BERT-base
0.86
0.07
0.97
0.79
BERT-base (repr.)
0.86
0.12
0.97
0.81
BERT-base + LoRA
0.83
0.16
0.95
0.75
BERT-base + AdapterH
0.86
0.10
0.97
0.77
FinBERT
0.87
0.07
0.97
0.80
FinBERT (repr.)
0.86
0.09
0.97
0.77
FinBERT + LoRA
0.89
0.08
0.94
0.76
FinBERT + Adapterи
0.86
0.09
0.97
0.76
FLANG- BANG-BERT
0.91
0.05
0.97
0.83
FLANG- BLANG-BERT (repr.)
0.86
0.08
0.97
0.81
FLANG-BERT LoRA
0.81
0.17
0.95
0.78
FLANG-BERT Adapterи
0.86
0.07
0.97
0.76
Table 1:
Summary of results on downstream tasks. Mean F-1 is the average F-1 score taken across all nine labels of the News Headline dataset. Breakdown of results for each label can found in Appendix A.3. Numbers marked bold highlight the best-per best-performing fine-tuning approach model.
Model
Size
Trainable Parameters
Fine-tuning Time
BERT-base
418.0 MB
109 780 228
(100%)
187 sec
BERT-basc + LoRA
1.2 MB
297 988
(0.3%)
152 sec
BERT-base + AdapterH
3.5 MB
186 658
(1.4%)
122
FinBERT
419.0 MB
119 796
(100%)
307 sec
FinBERT + LORA
1.2 MB
297 988
(0.3%)
147 sec
FinBERT + Adaptern
3.5 MB
1486 658
(1.4%)
131 sec
FLANG-BERT
418.0 MB
109 780 228
(100%)
305 sec
FLANG-BERT LoRA
1.2 MB
297 988
(0.3%)
145 sec
FLANG-BERT Adapteræ
3.5 MB
486 658
(1.4%)
117
Table 2: Comparison of model size, number of trainable parameters and time for Headline Classification.
number of epochs to be able to reach a comparable performance to full fine-tuning (Hu et al., 2021; Chen et al., 2022).
from scratch or through continual in-domain pre- training and compare the domain-specific models to general-domaii baseline models (Peng et al.. 2021; Yang et al., 2020; Shah et al., 2022), but do not employ parameter-efficient techniques. Large language models for finance et al., 2023; Xie al., 2023) but have either access restrictions do not take advantage of parameter- efficient tuning approaches. To the best of our knowledge, the first financial language model, fine- tuned with LoRA in combination with reinforce- ment learning, was released (Yang et al., 2023).
From practical point of view, the gain in stor- age space is apparent and allows to keep and load numerous fine-tuned models for abundance of tasks locally. From time-saving perspective, the datasets used in this study are relatively small so that even the full fine-tuning requires only a few minutes in a lot of cases. Our experiments indi- cate that larger gain in time efficiency is observed for parameter- efffcieent techniques when applied to larger models and bigger datasets.
the rise (Wu
Conclusion
5 Related Work
In this paper, evaluated the effectiveness of LoRA and adapter fine-tuning applied tasks in the financial domain. We showed that in financial setting, we can reach full fine-tuning performance with smaller model sizes which allows for casier storage and sharing of fine-tuned models among
Numerous studies have been released on the effec- tiveness of parameter-efficient approaches on the GLUE benchmark (Pfeiffer et al., 2020; Rücklé et al., 2021; He et al., 1). In the financial do- main, a number studies build language models

---

practitioners.
Acknowledgements
We would like aspire future research to evalu- ate more parameter-
This research was funded in part by the Luxem- bourgish National Fund (FNR), grant reference The simulations were performed on the Luxembourg national super- computer MeluXina. We gratefully acknowledge the Lux Provide teams for their expert support.
strategies and extend them to other domain-specific fields. Based on our results and with the increasing number of larger language models in the financial sector (Wu et al., 2023; Lu et al.. 2023), we recommend the imple- mentation of more parameter-efficient fine-tuning practices to strive towards reducing their carbon footprint and environmental impact.
References
Dogu Araci. 2019, Finbert:
Financial sentiment analy- sis with pre-trained language models.
7 Limitations
Tom Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei, 2020, Language models are few-shot learners. In Ad- vances Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Similar to the majority of experimental research, this study may encounter potential limitations and validity threats.
First, the outcomes of our experiments might depend on the volume of data used for the different downstream tasks. We did not experiment with distinct dataset sizes for the FPB SA and News Headline Classification tasks, where more training samples are available, to see the impact on perfor- mance if we reduced the number of samples.
Second, while there are other parameter-efie -fffcceeeee fine-tuning methods, our study is restricted to the use of LoRA and Houlsby Adapter. We did not assess the performance of other techniques to show their potential on financial tasks. In addition, did not experiment with different adapter hidden sizes that might have an impact on performance.
Guanzheng Chen, Fangyu Liu, Zaiqiao Meng, and Shangsong Liang. 2022. Revisiting parameter- efficient tuning: Are we really there yet? In Proceed- ings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 2612-2626, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Lastly, our hyperparameters differ from the ones employed in the baseline models. We conducted a hyperparameter scarch for each task to obtain the best configurations, however, for some tasks the number of epochs used for the fine-tuning varies a lot between the fully fine-tuned models and their efficiently fine-tuned counterparts. Hence, some results could be attributed confounding variables that we did not manage to regulate.
Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding. Liying Cheng. Jiawei Low, Lidong Bing, and Luo Si. 2021. On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing [Volume 1: Long Papers), pages Online. Association for Computational Linguisties.
8 Ethical Considerations
For this study, we fine-tuned our models using dif- ferent openly available financial text corpora in- cluded in the FLUE benchmark (Shah et al., 2022). Despite the financial nature of the datasets. none of them carry high ethical risks as the respective authors ensure that they do not reveal any sensitive data, nor contain any information that could allow to identify personal data.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski. Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfrr learning for NLP. Proceedings of the 36th International Conference

---

on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya
Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Tryna Gurevych. 2020. AdapterHub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Nat- urul Language Processing: System Demonstrations. pages 46-54, Online. Association for Computational Linguistics.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models.
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeifeer, Nils Reimers, and Iryna Gurevych. 2021. AdapterDrop: On the efficiency of adapters in transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7930-7946, Online and Punta Cana. Dominican Republic. Association for Computational Linguistics.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pages 4582- 4597, Online. Association for Computational Lin- guistics.
Julio Cesar Salinas Alvarado, Karin Verspoor, and Tim- othy Baldwin. 2015. Domain adaption of named entity recognition to support credit risk assessment. In Proceedings of the Australasian Language Tech- nology Association Workshop 2015, pages 84-90, Parramatta, Australia.
Zhuang Liu, Degen Huang, Kaiyu Huang, Zhuang Li, and Jun Zhao. 2021. Finbert: A pre-trained finan- cial language representation model for financial text mining. In Proceedings of the Twenty-Ning Inter- national Joint Conference on Artificial Intelligence IJCAT'0.
Raj Shah, Kunal Chawla, Dheeraj Eidnani. Agam Shah, Wendi Du Sudheer Chava, Natraj Raman, Charese Smiley, Jiaao Chen, and Diyi Yang. 2022. When FLUE meets FLANG: Benchmarks and large pre- trained language model for financial domain. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 2322- 2335, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Dakuan Lu, Jiaqing Liang, Yipei Xu, Qi He, Yipeng Geng, Mengkun Han, Ying Xin, Hengkui Wu, and Yanghua Xiao. 2023. Bbt-fin: Comprehensive con- struction of chinese financial domain pre-trained language model, corpus and benehmark. ArXiv, abs/2302.09432.
Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018 Www' open challenge: Financial opinion mining and question answering. In Companion Proceedings of the The Web Conference 2018, WWW '18. page 1941-1942, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
Ankur Sinha and Tanmay Khandait. 2020. Impact of news on the commodity market: Dataset and results.
Asa Cooper Stickland and lain Murray, 2019, Bert and pals: Projected attention layers for efficient adapta- tion multi-task learning. In International Confer- ence on Machine Learning.
P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and Takala. 2014. Good debt bad debt: Detecting se- mantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018, GLUE: A multi-task benchmark and analysis platform for nat- ural language understanding, In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Com- putational Linguistics.
Bo Peng. Emmanuele Chersoni, Yu- Yin Hsu, and Chu- Ren Huang. 2021. Is domain adaptation worth your investment? comparing BERT and FinBERT fi- nancial tasks. In Proceedings of the Third Workshop on Economics and Natural Language Processing, pages 37-44, Punta Cana, Dominican Republic. As- sociation for Computational Linguistics.
Shijic Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kam- badur, David Rosenberg, and Gideon Mann. 2023. Bloomberggpt: A large language model for finance.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. AdapterFusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 487-503, Online. Association for Computational Lin- guistics.
Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. 2023. Pixiu: A large language model, in- struction data and evaluation benehmark for finance.
Hongyang Yang, Xiao- Yang Liu, and Christina Dan Wang. 2023. Fingpt: Open-source financial large language models.

---

Yi Yang, Mark Christopher Siy UY and Allen Huang. 2020. Finbert: A pretrained language model for financial communications.

---

A Appendix
A.1 Hyperparameter Search Results
Hyperparameters
#epochs
bs
Ir
lora alpha
lora dropout
FT FPB
32
2c-5
FT FiQA
le-4
FT Headline
8
le-2
FT NER
5
16
le-4
LoRA FPB
80
16
2e-5
32
0.01
LoRA FiQA
80
16
2c-5
1
1
0.01
LoRA Headline
16
5e-5
8
16
0.05
LoRA NER
16
le-3
16
16
0.05
Adapters FPB
16
le-4
Adapters FIQA
16
le-4
Adapterи Headline
16
le-4
Adapter# NER
20
16
le-4
Table 3: Hyperparameters used for full fine-tuning (FT),
LoRA, and Adapter, on the downstream tasks.
A.2 Description of Datasets
Datasets
Train
Dev
Test
Classes
FPB (Malo et al., 2014)
3488
388
969
3
FiQA SA (Maia al., 2018)
822
117
234
Headline (Sinha and Khandait, 2020)
7989
1141
2282
2
NER (Salinas Alvarado et al., 2015)
932
232
302
5
Table Descriptive Statistics of FLUE datasets.
A.3
Breakdown of results for News Headline Classification
Label
Bert + LoRA
FinBERT + LoRA
FLANG-BERT +
Price or Not
0.93
0.93
0.94
Price Up
0.90
0.92
0.92
Price Constant
0.94
0.94
0.93
Price Down
0.93
0.94
0.92
Past Price
0.93
0.92
0.93
Future Price
0.99
0.98
0.97
Past News
0.93
0.93
0.94
Future News
0.99
0.99
0.99
Asset Comparison
0.98
0.96
0.98
Mean F-1 Score
0.95
0.94
0.95
Table Breakdown of LoRA results for each label for
News Headline Classification.

---

Label
Bert + Adapter
FinBERT + Adapter
FLANG-BERT + Adapter
Price or Not
0.96
0.95
0.95
Price Up
0.94
0.91
0.95
Price Constant
0.98
0.98
0.98
Price Down
0.95
0.95
0.95
Past Price
0.95
0.94
0.95
Future Price
0.99
0.99
0.99
Past News
0.96
0.95
0.95
Future News
0.99
0.99
0.99
Asset Comparison
0.99
0.99
0.99
Mean F-1 Score
0.97
0.97
0.97
Table Breakdown of Adapter results for each label for News Headline Classification.
A.4 Model Sizes and Time
A.4.1 Financial Phrase Bank
Model
Size
Trainable Parameters
Fine-tuning Time
BERT-base
418 MB
109 523 718 (100%)
62 sec
BERT-base + LoRA
171 KB
41 478 (0.04%)
1668 sec
BERT-base + Adapterи
3.4 MB
1 487 427 (1.4%)
318 sec
FinBERT
419 MB
109 793 286 (100%)
60 sec
FinBERT LoRA
1.2 MB
11 178 (0.01%)
1736 sec
FinBERT Adapterи
MB
1 4887 427 (1.4%)
326 sec
FLANG-BERT
418 MB
109 523 718 (100%)
60 sec
FLANG-BERT + LoRA
171 KB
41 478 (0.04%)
1703 sec
FLANG-BEL BLANG-BERT+ + Adapteræ
3.4 MB
1 487 427 (1.4%)
326 sec
Table 7: Comparison of model size, number of trainable parameters and time for FPB.
A.4.2 FIQA
Model
Size
Trainable Parameters
Fine-tuning Time
BERT-base
418 MB
109 520 642 (100%)
sec
BERT-base LoRA
165 KB
38 402 (0.04%)
700 sec
BERT-base + Adapterи
3.4 MB
1 1485 889 (1.4%)
46 sec
FinBERT
419 MB
109 790 210 (100%)
84 sec
FinBERT LoRA
171 KB
38 402 (0.04%)
677 sec
FinBERT Adapterи
3.4 MB
1 4485 889 (1.4%)
47 sec
FLANG-BERT
418 MB
109 520 642 (100%)
35 sec
FLANG-BERY + LoRA
165 KB
38 402 (0.04) (0.04%)
649 sec
FLANG-BERG + Adapteræ
3.4 MB
1 485 889 (1.4%)
46 sec
Table Comparison of model size, number of trainable parameters and time for FiQA.

---

A.4.3 NER
Model
Size
Trainable Parameters
Fine-tuning Time
BERT-base
418 MB
109 489 162 (100%)
464 sec
BERT-base + LoRA
2.7 MB
699 658 (0.6%)
330 sec
BERT-base Adapters
3.4 MB
898 373 (0.8%
1117
FinBERT
419 MB
109 758 730 (100%)
410 sec
FinBERT LoRA
2.7 MB
699 658 (0.6%)
331 sec
FinBERT Adapters
3.4 MB
898 373 (0.8%)
1118 sec
FLANG-BERT
418 MB
109 489 162 (100%)
429 sec
FLANG-BERT + LoRA
2.7 MB
699 658 (0.6%)
330 sec
FLANG-BERI + Adapters
3.4 MB
898 373 (0.8%)
1118
Table 9: Comparison of model size, number of trainable parameters and time for NER.


Reviews:
[{"review_id": "Gxh4iqXioB", "reviewer": "Reviewer_mQ9a", "summary": "", "questions": "I would have liked to see the paper address more interesting research questions such as a) how does the effectiveness of LoRA/Adapter change across datasets? b) Beyond vanilla Adapter/LoRA what else can one do to do better on new domains? c) Evaluating additional parameter-efficient methods and udnerstanding trends across datasets, domains, and tasks.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Comment 1:**\nWe thank the reviewer for the remark about the paper's novelty. We agree that the contribution to existing literature might seem marginal, though believe still valuable to evaluate such established parameter-efficient finetuning approaches to unexplored domains to further consolidate their efficacy. We believed a short paper to be a good choice to address our research questions.\n\n**Response to Question a:**\nWe greatly appreciate the reviewer’s feedback and insightful suggestions. Analysing the effectiveness of LoRA/Adapter across datasets would indeed enhance the comprehensiveness of our work and add valuable insights to the paper. We will go into a more detailed analysis of PEFT methods across datasets in the revised version of the paper.\n\n**Response to Question b:**\nWe thank the reviewer for highlighting the importance of exploring strategies beyond LoRA and Adapters. Other domain adaptation strategies exist, such as unsupervised domain adaptation, that could be leveraged or even combined with parameter-efficient finetuning methods to do better on new domains. In future works we will aim to present a more comprehensive framework for improving model performance in new domains by exploring techniques such as cross-domain pre-training, transfer learning, as well as considering external sources such as knowledge graphs.\n\n**Response to Question c:**\nWe appreciate the reviewer’s keen interest in exploring additional parameter-efficient finetuning methods and identifying trends across datasets, domains and tasks. In future work we intend to include a more detailed section dedicated to evaluating various parameter-efficient finetuning approaches beyond LoRA and Adapters, as well as the combination thereof. We intend to provide a comparative analysis of these methods across diverse scenarios and extending our work with more challenging tasks. Our goal is to build up on this work and continue building an in-depth examination of trends emerging from the conducted experiments on a wider range of datasets and tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper shows that parameter-efficient finetuning methods like LoRA and Adapters are able to retain nearly all of the finetuning performance while gaining in time and resource efficiency.", "reasons_to_reject": "1. Novelty: The paper's contribution to existing literature seems marginal. Evaluating well-established parameter efficient finetuning methods on new datasets (finance domain) is helpful but in my opinion, does not constitute a significant contribution to this conference.", "questions_to_authors": "I would have liked to see the paper address more interesting research questions such as a) how does the effectiveness of LoRA/Adapter change across datasets? b) Beyond vanilla Adapter/LoRA what else can one do to do better on new domains? c) Evaluating additional parameter-efficient methods and udnerstanding trends across datasets, domains, and tasks.", "paper_topic_and_main_contributions": "This paper performs a thorough comparison of financial BERT-like models to their fully fine-tuned counterparts on financial tasks by leveraging efficient parameter-efficient tuning methods Adapters and LoRA. ", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "No concerns here, paper was well-written."}, "extra_text": {}}, {"review_id": "8dFB745CAE", "reviewer": "Reviewer_7Xqg", "summary": "", "questions": "1) The paper's appeal could be enhanced by conducting evaluations across a wider range of tasks and diverse dataset types.\n2) An examination of the scenarios where one method outperforms the other during model fine-tuning would add value. Insights into the determining factors behind such decisions should be explored.\n3) It would provide valuable insights to understand the rationale behind the selection of specific hyperparameters and epoch counts. This transparency would contribute to a deeper understanding of the experimental setup.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 1, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Comment 1:**\nWe sincerely appreciate the reviewer's feedback and acknowledge the importance of making substantial contributions to the field. While we agree that introducing new datasets, algorithms, and novel discoveries is critical, we would like to emphasize that our primary objective was to evaluate the effectiveness of the LoRA and Adapter models in the context of fine-tuning for financial data. However, we take the reviewer's suggestion seriously and will explore opportunities to expand our work by incorporating additional datasets and different models  in future work.\n\n**Response to Comment 2:**\nWe appreciate the reviewer pointing out the potential impact of varying training epoch counts on performance discrepancies. We strongly agree that this is a crucial factor to consider and ensure the validity of our findings. While conducting experiments with consistent hyperparameter settings, such as training epochs, is important, we chose the number of epochs based on a hyperparameter search that we conducted for every task. We found that the results reflect the general guidelines (https://docs.adapterhub.ml/training.html) of using a higher number of epochs for PEFT methods as seen in the LoRA literature (https://arxiv.org/pdf/2106.09685.pdf).\n\n**Response to Comment 3:**\nWe appreciate the reviewer's suggestion to broaden the evaluation across a wider spectrum of datasets. We plan to extend our experiments to include a diverse range of datasets from various domains in order to better assess the effectiveness of PEFT techniques in future work. \n\n**Response to Comment 4:**\nWe thank the reviewer for their insight regarding the potential alignment of our paper with workshops specifically focused on Finance NLP. We can consider a specialized workshop in the future.\n\n**Response to Question 1:**\nThe reviewer’s suggestion to conduct evaluations across a wider range of tasks and dataset types is highly valued. We completely agree and are committed to expanding the scope of our experiments to encompass a diverse set of tasks and datasets in future work. \n\n**Response to Question 2:**\nWe appreciate the reviewer's suggestion to investigate scenarios where one method outperforms the other during model fine-tuning. We will consider this more detailed analysis in future work in order to uncover the factors that contribute to the success of each method, determine the strengths and weaknesses of the approaches, and understand their suitability for different scenarios.\n\n**Response to Question 3:**\nWe recognize the importance of transparency in hyperparameter selection to facilitate a deeper understanding of our experimental setup. We would like to point out that we mentioned in the paper that we did a hyperparameter search for each dataset and task, and will make this clearer in the revised version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The evaluation they conducted showcases relatively persuasive findings, indicating that the PEFT approach yields outcomes akin to full fine-tuning, all while avoiding overfitting.", "reasons_to_reject": "1) The paper lacks significant contributions in terms of introducing new datasets, algorithms, or novel discoveries.\n2) Various models underwent training for differing epoch counts, and any performance discrepancies could be linked to this factor.\n3) The evaluation, as highlighted in the paper, is confined to a narrow selection of small datasets. It would be advisable to assess the approach across a broader spectrum of datasets.\n4) The paper primarily revolves around fine-tuning models for financial data, yet it doesn't introduce any substantial new insights. Hence, it might find better alignment within workshops specifically focused on Finance NLP.", "questions_to_authors": "1) The paper's appeal could be enhanced by conducting evaluations across a wider range of tasks and diverse dataset types.\n2) An examination of the scenarios where one method outperforms the other during model fine-tuning would add value. Insights into the determining factors behind such decisions should be explored.\n3) It would provide valuable insights to understand the rationale behind the selection of specific hyperparameters and epoch counts. This transparency would contribute to a deeper understanding of the experimental setup.", "paper_topic_and_main_contributions": "\nThe study assesses the effectiveness of Parameter Efficient Fine-Tuning techniques on financial datasets. Specifically, the investigation delves into the performance of LoRA and Adapter-H methods when applied to Bert-base, FinBERT, and FlangBERT models. The datasets encompass a variety of tasks: 1) Soft Contextual Data Augmentation (Classification), 2) Financial Question Answering (FiQA) (Regression), 3) Gold News Headlines (Classification), and 4) Named Entity Recognition (NER).\n\nThe primary assertion of the paper is to demonstrate that these fine-tuning approaches yield comparable performance to that achieved through complete model fine-tuning. Notably, these techniques necessitate only a fraction of the parameters to be fine-tuned, effectively mitigating the risk of overfitting.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sfAVr1XhHG", "reviewer": "Reviewer_yiyh", "summary": "", "questions": "- Question A: In Table 1, the reproduced results and the originally reported values (i.e., the second row with (repr.) and first row) are different. What is the reason for the performance gap? Please refer to the results reported in Table 2 of this work (Raj et al. 2022, https://aclanthology.org/2022.emnlp-main.148/). Are there reproducibility issues or a need for implementation details in the public code?\n- Question B: Table 2 presents the fine-tuning time to decrease the Fine-tuning time. However, the same phenomenon only appears in Headline Classification datasets. Therefore, a follow-up question RQ2 is, for \"smaller\" datasets (Appendix A.4), what is the reason (+LoRA, or +$\\text{Adapter}_H$) takes much more time for fine-tuning? For example, in Table 7, FinBERT + LoRA takes 1,736 sec, which is >10 times longer than the Baseline dataset shown in Table 2. Does setting $r$ to 1 make it hard to converge? What is the possible weakness of the parameter-efficient method in this scenario? They might be essential information for readers.\n- Question C: Will the source code be public if accepted?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Comment 1:**\nWe appreciate the reviewer's attention to the correctness of the reproduced results in Table 1. We apologise for any confusion this may have caused. We would like to point out that we have stated to be using our reproduced results of the baselines due to some discrepancies with the originally reported values. We would further direct the reader to the Appendix for more results. We agree that the results are mixed and not conclusive enough to say that one method performs better than the other. We can only see a trend that Adapters seem to perform better than LoRA on the chosen tasks. \n\n**Response to Comment 2:**\nWe thank the reviewer for highlighting the importance of having more conclusive results. In our revised manuscript, we will report on additional results between Adapter and LoRA. We would assume that because adapters keep a higher proportion of parameters compared to LoRA, that they seem more stable and will report further on this in the camera-ready version. \n\n**Response to Question A:**\nWe thank the reviewer for this question and acknowledge the importance of addressing this discrepancy. We will provide an explanation of the potential reasons for the observed differences in the revised manuscript as well as reach out to the original authors to ask why there could be some potential differences in reproducing the results.\n\n**Response to Question B:**\nWe thank the reviewer for raising questions about the differences in fine-tuning times observed across datasets and methods. As mentioned and seen in the literature, LoRA usually requires more epochs. We will make this clearer in the revised version of the paper. \n\n**Response to Question C:**\nWe fully support the principles of transparency and reproducibility in research. As such, we are committed to making our source code publicly available if our paper is accepted. We will ensure to provide well-documented code that is accessible to other researchers who would like to replicate and further explore this topic.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors provide a brief report on applying the parameter-efficient method in the finance domain. The experiments show the task-specific manner of applying LORA to different size datasets. The increasing fine-tuning time of *BERT + LORA for smaller datasets shown in A.4 is an exciting topic for the community to explore.", "reasons_to_reject": "- The correctness of the reproduced results in Table 1 needs further confirmation. The reproduced results and the originally reported values (i.e., the second row with (repr.) and the first row) are different with performance drop by up to 3% (see FinBERT/NER, 0.88 versus $\\textbf{0.77}$), which is confusing in determining whether the parameter-efficient methods are feasible. For example, for FLANG-BERT, the original (i.e., FLANG-BERT) and reproduced MSE (i.e., FLANG-BERT (repr.)) in task \"FiQA SR\" are 0.05 and 0.08, respectively. If the MSE of FLANG-BERT+ $\\text{Adapter}_H$ is 0.07, concluding \"FLANG-BERT + $\\text{Adapter}_H$ outperforms the fully fine-tuned FLANGBERT\" in L208-L211 is not convincing for readers. More details for reproducing baseline models are required in L191-L194 to give more confidence in the correctness of Table 1.\n- More conclusive results are required. For example, in section 3.1 L199-L215,  besides reporting the trend of the experiment results, the observation of why $\\text{Adapter}_H$ looks more stable than LoRA is worth reporting.", "questions_to_authors": "- Question A: In Table 1, the reproduced results and the originally reported values (i.e., the second row with (repr.) and first row) are different. What is the reason for the performance gap? Please refer to the results reported in Table 2 of this work (Raj et al. 2022, https://aclanthology.org/2022.emnlp-main.148/). Are there reproducibility issues or a need for implementation details in the public code?\n- Question B: Table 2 presents the fine-tuning time to decrease the Fine-tuning time. However, the same phenomenon only appears in Headline Classification datasets. Therefore, a follow-up question RQ2 is, for \"smaller\" datasets (Appendix A.4), what is the reason (+LoRA, or +$\\text{Adapter}_H$) takes much more time for fine-tuning? For example, in Table 7, FinBERT + LoRA takes 1,736 sec, which is >10 times longer than the Baseline dataset shown in Table 2. Does setting $r$ to 1 make it hard to converge? What is the possible weakness of the parameter-efficient method in this scenario? They might be essential information for readers.\n- Question C: Will the source code be public if accepted?", "paper_topic_and_main_contributions": "The authors report the performance of applying parameter-efficient methods, Adapters (Houlsby et al., 2019) and LoRA (Hu et al., 2021), to three backbone models: BERT-base, FinBERT, and FLANG-BERT on four finance data sets: FPB SC, FiAQ SR, Headline, and NER. The contribution of this work includes providing a preliminary study on parameter-efficient methods (i.e., $\\text{Adapter}_H$ and LoRA) applied in the finance domain. The report concludes that the parameter-efficient method decreases the domain-specific BERT's (i.e., FinBERT's and FLANG-BERT) fine-tuning time by \\~50% compared to the full fine-tuning (Table 2) for a relatively large \"Headline\" dataset (\\~10K train+val). In addition, the performance is task-specific and might drop after incorporating parameter-efficient methods (Table 1, the first row of each model and lines +LORA and +$\\text{Adapter}_H$).", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- L211 typo: FLANGBERT -> FLANG-BERT"}, "extra_text": {}}, {"review_id": "UenKAJre5E", "reviewer": "Reviewer_JNPW", "summary": "", "questions": "1. Can you extend this analysis to more PEFT approaches and types of base models?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Comment 1:**\nWe appreciate the reviewer’s feedback regarding the motivation and novelty of our study. For the camera-ready version, we will emphasize on the significance of exploring PEFT techniques beyond the general context and highlight the potential benefits of these approaches in domain-specific contexts. \n\n**Response to Comment 2:**\nWe thank the reviewer for the valuable suggestion to broaden the scope of our comparisons among PEFT approaches. We agree that including a wider range of methods would enrich the analysis and contribute to a more comprehensive understanding of PEFT approaches in the finance domain. To provide a more detailed and diverse comparison, we will incorporate these additional methods into our future work. \n\n**Response to Comment 3:**\nWe acknowledge the reviewer's insight regarding the potential benefits of exploring generative tasks and considering a more diverse range of model types in our PEFT evaluation. To address this, we will extend our study in the future to also incorporate generative tasks. Furthermore, to make the analysis of PEFT more comprehensive, we will include an evaluation of PEFT on more model types.\n\n**Response to Comment 4:**\nWe appreciate the reviewer's feedback regarding the novelty of our second research question. We believed the second research question to be relevant and to logically follow the conclusions of the first research question.\n\n**Response to Question 1:**\nWe thank the reviewer for suggesting an extension of our analysis to encompass a broader range of PEFT approaches and base models. Building upon this suggestion, we plan to conduct an in-depth evaluation of additional PEFT techniques and explore their performance across different types of base models in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper extends the existing PEFT analysis to the financial domain, and indicates competitive performance to fully supervised fine-tuning.\n", "reasons_to_reject": "1. The study of PEFT in a new domain is not motivated and has limited novelty.\n\n2. The study only contains two PEFT approaches, Adapter and LoRA. More interesting comparisons among parallel adapter [1], IA^3, pre-fix tuning, prompt tuning, and many other new PEFT approaches can make the analysis in the finance domain more existing. \n\n3. The study only focuses on classification-based finance tasks. Generative tasks can lead to more interesting results. Additionally, this study only considers encoder-only models. A more throughout evaluation study of PEFT should consider T5, and GPT-type models.\n\n4. The second research question is not novel.\n", "questions_to_authors": "1. Can you extend this analysis to more PEFT approaches and types of base models?", "paper_topic_and_main_contributions": "This paper presents a study of PEFT in financial domain tasks. It addresses two research questions: 1. how do PEFTs perform on financial tasks? 2. what are the pros and cons of using PEFTs? The author conducts adapter-tuning and LoRA-tuning in comparison to fully supervised fine-tuning across BERT-base, FinBERT, and FLANG-BERT on 4 financial domain tasks from the FLUE benchmarks. However, the first research question is not well motivated as PEFTs have shown tremendous success across domains, languages, and even modalities. Therefore, the technical contribution of this paper is minimal, and there is no additional finding on PEFT on top of existing PEFT literature. Moreover, the second research question has been addressed by individual PEFT papers. \n\nPfeiffer, J., Goyal, N., Lin, X. V., Li, X., Cross, J., Riedel, S., & Artetxe, M. (2022). Lifting the Curse of Multilinguality by Pre-training Modular Transformers. \n\nChronopoulou, A., Peters, M. E., & Dodge, J. (2022). Efficient Hierarchical Domain Adaptation for Pretrained Language Models.\nSung, Y.-L., Cho, J., & Bansal, M. (2022). VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks.\n\n\n", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., & Neubig, G. (2021). Towards a unified view of parameter-efficient transfer learning. \n\n[2] Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., & Raffel, C. A. (2022). Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.\n", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 03: they risk -> the risk / their risk\nThe bold fonts in Table 1 are misleading. The author should not exclude the original results in ranking and should also highlight the FT baseline that achieves the same highest score. "}}]

Author rebuttal:

Meta-review: The main concern raised by the reviewers is the lack of novelty. While Adapter and LoRA results for the financial domain are new and provide a data point for PEFT vs full-finetuning, the paper does not contain new ideas.

The soundness of the paper is mixed: some reviewers mention some specific problems in the methodology, and the authors highlight that they do a full grid search to find the right hyperparameter, thus providing a strong methodology. Some other issues remain partially unaddressed such as the reproduction of a core result. All in all, this is a borderline paper that could benefit from further revision and review. In this state, I would only recommend acceptance to findings if there is there are enough slots in the proceeding.
Decision: Accept-Findings