On Surgical Fine-tuning for Language Encoders
Authors: Abhilasha Lodha
Gayatri Vyankatesh Belapurkar
Saloni Chalkapurkar
Yuanming Tao
Reshmi Ghosh
Samyadeep Basu
Dmitrii M Petrov
Soundararajan Srinivasan
Venue: EMNLP 2023 Findings (2023)
Keywords: Efficient Fine Tuning
Language Models
Language Encoders
Linguistics
Optimization
Distributional Shifts
Temporal Shifts
TL;DR: Using Fisher Information Matrix diagonal score we perform efficient surgical fine-tuning for LM tasks

Abstract: Fine-tuning all the layers of a pre-trained neural language encoder (either using all the parameters or using parameter-efficient methods) is often the de-facto way of adapting it to a new task. We show evidence that for different downstream language tasks, fine-tuning only a subset of layers is sufficient to obtain performance that is close to and often better than fine-tuning all the layers in the language encoder. We propose an efficient metric based on the diagonal of the Fisher information matrix (FIM score), to select the candidate layers for selective fine-tuning. We show, empirically on GLUE and SuperGLUE tasks and across distinct language encoders, that this metric can effectively select layers leading to a strong downstream performance. Our work highlights that task-specific information corresponding to a given downstream task is often localized within a few layers, and tuning only those is sufficient for strong performance. Additionally, we demonstrate the robustness of the FIM score to rank layers in a manner that remains constant during the optimization process.

On Surgical Fine-tuning for Language Encoders
Abhilasha Lodha'; Gayatri Belapurkar'; Saloni Chalkapurkar's Yuanming Tao';
Reshmi Ghosh?, Samyadeep Basu", Dmitrii Petrov', Soundararajan Srinivasan"
University of Massachusetts, Amherst
Microsoft Corp.
3 Universitty of Maryland, College Park
Correspondence to: reshmi www.:iighhhhmimsoottttccm
Abstract
fine-tuning all layers arbitrarily might risk overfit- ting or adversely affecting the generalization abil- ity of the model during the transfer lcarning pro- cess. While, recently, the focus has shifted to devel- opment of parameter efficient approaches of fine- tuning large-la and language models (Liu et al. (2022), Lialin et al. (2023), Han et al. (2021)), these techniques still require development of 'adapter' architecture relative to target dataset. We therefore focus on developing a data-driven cri- teria to automatically identify and tune a smaller sub-set of layers using only 100 target data sam- ples.
Fine-tuning all the layers of a pre-trained neu- ral language encoder (either using all the pa- rameters or using parameter-efficient methods) often the de-facto way adapting to new task. We show evidence that for different downstream language tasks, fine-tuning only a subset of layers is sufficient obtain per- formance that is close to and often better than fine- tuee-tuning all the layers in the language en- coder. We propose efficient metrie based on the diagonal of the Fisher information ma- trix (FIM score), to select the candidate lay- ers for selective fine-tuning. We show, em- pirically on GLUE and SuperGLUE tasks and across distinct language encoders, that this met- ric can effectively select layers leading strong downstream performance. Our work highlights that task-specific information corre- sponding to given downstream task is often localized within few layers, and tuning only those is sufficient for strong performance Ad- ditionally, we demonstrate the robustness of the FIM score rank layers a manner that remains constant during the optimization pro- cess.
In this paper, we propose a simple strategy to select layers for fine-tuning for real-world NLP tasks, leveraging the Fisher Information Matrix (FIM) score, which quantifies the impact of param- eter changes on model's prediction. We further demonstrate the effectiveness of FIM score on lan- guage encoder model(s) on practical NLP tasks from GLUE and SuperGLUE benchmark, by iden- tifying a subset of layers that are most informative to adapt to the target data distribution. We find that fine-tuning parameters in identified subset of layers by FIM score, outperforms full model fine-tuning in some, and results in comparable performance to full fine-tuning in almost all the GLUE and Super- GLUE tasks. In niche scenarios, where FIM scores leads to selection of layers that contribute to sub- optimal performance comparison with full model nhe-tuning, we investigate the nuanced character- istics of the GLUE and SuperGLUP tasks through the lens of linguistic features learned during trans- fer learning process, and potential categories of target data distribution shifts that could influence the performance while surgically fine-tune.
1 Introduction
Fine-tuning of language encoders is crucial step towards applying natural language processing solu- tions real-world challenges. It allows adaptation of knowledge on target distribution after generally training on source data, but requires curation of ad- equately sized labelled dataset to gain 'new' knowl- edge while retaining information obtained during the pre-training phase.
Although preserving knowledge of target distri- bution by tuning the entire model can yield im- pressive results, it can be expensive and may in- crease data volume requirements. Additionally,
Interestingly, we find that GLUE and Super- GLUE tasks that are dependent on a simpler un- derstanding of linguistic features such as syntax and semantics as well as discourse understanding, can be surgically fine-tuned using our proposed
*Equal Contribution: order determined by alphabeti- arrangement names. 'Our code is publicly available at: Github

---

FIM score criteria. However, we find that for tasks that rely on learning more complex knowledge of both high and low level linguistic features such as textual entailment, common sense and world knowledge FIM score criteria unperformed to se- lect the relevant layers. On investigating categories of target distribution shifts that could surface in various GLUE/SuperGLUE tasks, we also find that FIM score signals at efficient tuning of parameters for the group of tasks that align closely with con- cepts of domain, environmental, and demographic shifts, but fails to perform optimally on tasks that require learning of temporal drifts in language.
the advantages of few-shot parameter-efficeen fine- tuning over in-context learning in terms of effec- tiveness and cost-efficiency. Additionally, tech- niques like prompt tuning are also considered as parameter-efficient fine-tuning methods.
A handful of studies have investigated the knowl- edge gain in fine-tuning process for Language En- coders, particularly BERT. Merchant et al. (2020), Hessel and Schofield (2021) investigated the im- pact of shuffling the order of imput tokens on the performance of the BERT model for several lan- guage understanding tasks. Sinha et al. (2021) fur- ther investigates the effectiveness of masked lan- guage modeling (MLM) pre-training and suggests that MLMs achieve high accuracy on downstream tasks primarily due to their ability to model distri- butional information.
2 Related Work
Surgical fine-tuning has been widely explored in various computer vision applications to identify definitive distribution shifts in target datasets. Lee al. (2023) explains why surgical fine-tuning could match or outperform full fine-tuning on dis- tribution shifts and proposes methods to efficiently select layers for fine-tuning. However, in natural language applications, it is challenging to define such delineations due to the rapid changes in lan- guage based on context, domain, and individuals. Several attempts have been made to conduct Pa- rameter Efficient Fine Tuning (PEFT) in natural lan- guage. Lialin et al. (2023) and Han et al. (2021) ex- plores the landscape of utilizing adapter layers and soft prompts, including additive methods, selective techniques,
However, our approach of efficient fine-tuning using the proposed FIM score criteria (that is able to capture signals from 100 target data samples), differs from all existing methods, as it focuses on helping NL. practitioners with small size target datasets to efficiently rank and select important layers for optimizing the fine-tuning process.
3 Proposed Method
3.1 Fisher Information Matrix score
The significance of a parameter can be assessed by examining how modifications to the parameter affect the model's 's output. We denote the output distribution over y generated by a model with a parameter vector 0 E Rl given input as po y)). To quantify the impact of parameter changes model's 's prediction, one approach is to compute the Fisher Information Matrix (FIM), which repre- sented by equation 1:
methods, and reparameterrizttoo-baaee whereas He et al. (2022) applies network pruning develop pruned adapter (sparse dapter). In an- other body work, Sung et al. (2021) constructed a FISH (Fisher-Induced Sparse uncHanging) mask to choose parameters with the largest Fisher infor- mation. Additionally, Hu et al. (2021) attempted to efficiently fine-tune by proposing Low Rank Adapter that reduces trainable parameters by freez- ing the weights of the pre- me-trained model and in- jecting trainable rank decomposition matrices into each layer of the architecture.
F6 - Re ~ p(e) [Ry - poly NoVo logioiy! 2yVa logaviy 2]"] (1)
where, F9 is the FIM for the model with param-
eter vector f, quantifying the impact of parameter changes on the model's prediction, Ex ~ p(x) is expectation operator over x drawn from the dis- tribution p(x), Ey ~ po(y x)) is the expectation operator over y drawn from the output distribu- tion Po .r), Vo is gradient operator with re- spect to the parameter vector 0, log po(y) r)is the logarithm of the conditional probability of y given x under the model with parameter vector 0. and Vo log (y Vo log Po x)"" is the outer product of the gradients, which is used to compute the FIM.
Some fine tuning techniques involve fine-tuning a small subset of the model parameters. Sung et al. (2022) propose way of reducing memory require- ment by introducing Ladder Side Tuning (LST). A small 'ladder side' network connected to each of the layers of the pre-trained model is trained to make predictions by taking intermediate activa- tions as input from the layers via shortcut connec- tions called ladders. Liu et al. (2022) demonstrated

---

To analyze the impact of individual layers, we aggregate the diagonal elements of the FIM using the Frobenius norm. In our experimental setup, we randomly select a small sample (100 samples) from the validation set for each task. For fine-tuning, we specifically choose the top 52 layers with the highest FIM scores. The FIM score measures the amount of information provided by observable random variable about unknown parameter in its distribution. It reflects the sensitivity of the likelihood function to changes in parameter values. A higher Fisher information score indicates that more information can be gleaned from the data re- garding the parameter, leading to a more precise estimation of the parameter. In essence, higher score suggests that the likelihood function is more responsive to changes in parameter values, improv- ing the precision of parameter estimation.
iteratively tuned parameters of top 1 to 5 most im- portant layers, in the identified ranked layer order determined by FIM score, to compare and constrast the performance against full model fine-tuning. Furthermore, to comprehensively understand scenarios where FIM scores leads sub-optimal identification of layer ranks, we investigate the sen- sitivity of GLUE/SuperGLUE tasks (representing sentiment analysis. paraphrase detection datasets, NLI, question answering, linguistic acceptabilityy, commonsense reasoning, etc.) with respect to four possible data distribution shift categories, namely: Domain shift: Comparable shift from source data in target data distribution due to differences in fields or areas of knowledge.
Environmental shift: Changes from source data in target data distribution due to differences in con- texts.
Temporal drift: Changes in use of certain lan- guage entities over time.
4 Layer-wise Fisher Information Score Does Not Change During Fine-tuning
Demographic shift: Changes in data distribu- tion across different demographic groups.
In Fig. (2), we compute the rank of distinct layers leveraging the Fisher Information Score across the fine-tuning process of BERT at different epochs. Across tasks including WSC and WIC, we find that the rank of the different layers remain more or less consistent across the entire optimization trajec- tory during fine-tuning. This shows that the layers which are important for a given task, can indeed be selected even before the start of fine-tuning and after pre-training is done. Using this observation, in the next section, we show the effectiveness of fine-tuning only the layers selected using Fisher Score the start of the fine-tuning step.
Additionally, we also qualitatively investigate influence of six primary linguistic features that are possibly influenced during the fine-tuning process depending the task, namely Semantic Under- standing, Discourse Understanding, Syntactic Un- derstanding, Co-reference Resolution and Pronoun Disambiguation, Commonsense and World Knowl- edge, and Textual Entailment and Contradiction (for more details, refer to Appendix A.2).
5.2
Does surgical fine-tuning work across NLP tasks?
5 Experiments and Results
Our objective was to empirically analyze the perfor- mance of surgical fine-tuning approach leveraging FIM on real-worid NLP tasks against full model fine-tuning. Results in Figure (synthesized from Table 4 and Table 5) show that for GLUE and Su- PerGL.UE tasks, surgical fine-tuning of identified most important layers results in comparable perfor- mance and sometimes outperforms tuning all pa- rameters in all layers of (EERTbbase-asee model on target data distribution. Furthermore, we find that by selectively fine-tuning most relevant layer(s), as identified by FIM, the resulting performance on (almost all) GLUE and SuperGLUE tasks are in the ball park range of +5% of the full fine-tuning performance.
5.1 Experimental Setup
We applied F'MM score criteria to identify the 'layer importance rankings' for BERT3 across real-world NLP tasks from GLUE and SuperGLUE (more de- tails on experimental setup and hyperparameters in Appendix A.1). Based on these identified layer rankings, we performed surgical fine-tuning and
"e select 5 layers from ranked importance in a language encoder, such as BERT-base as we find the surgical fine-tuning on at-most 5 layers across all GLUE Lup tasks results, result in comparable performance when compared against full model fine-tuning
layers
We also validated the effectiveness of FIM with RoBERTa for some tasks to understand effectiveness of FIM scores across language encoder, results in Table ?? and Table ??
We also discover that the identified layer impor-

---

2.5
soning, and the ability integrate contextual infor- mation. Fine-tuning only a subset of layers based on FIM rankings may not adequately capture the necessary information and intricate relationships between linguistic elements, leading to decreased performance on these complex tasks.
Reletive Performance (%6
-2.5
-5.0
-7.5
-10.0
-12.3
We hypothesize that complex tasks such as RTE, COPA and CB, require holistic understanding language and reasoning abilities that span across multiple layers in the model. Consequently, selec- tively fine-tuning based solely on localized knowl- edge gain identified by FIM scores may not be sufficient to achieve optimal performance.
-15.0
-17.5
COLA
MBPC
QNU
ATE
WNLI
55T2
QQP
CB
COPA
MULTIRC
WCc
WSC
Boolq
GLUE and LUperGLUE tasks
Figure I: Plot of relative performance, i.c., the percentage point difference between the performance of surgical fine- tuning and full model fine-tuning, Fine-tuning parameters in the ranked
GLUE and Super-
GLUE tasks in important layer(s) can outperform full fine-tuning which of significant importance, and in almost all tasks of GLUE and SuperGLUE, result in relative performance in the range of +5%. Only for the case RTE, CB, and COPA(showed no change) selected layers using FIM scores lead sub-optimal results.
5.3.2 Efffec of target data distributional shifts
We also investigate the effectiveness of FIM in suggesting the appropriate layer importance ranks that maximize the localization of knowledge while adapting proposed categories of distributional shifts in target GLUE/SuperGLUE tasks.
tance rank through FIM is different across settings, depending on the nature of task from GLUE and SuperGLUE benchmark.
When categorizing tasks based on their sensi- tivity to distribution shifts, it becomes evident that MNLI and MRPC tasks primarily revolve around the comprehension of semantic relation- ships within sentence pairs. These tasks exhibit a high sensitivity to shifts in the domain of dis- course, as opposed to temporal or environmental variations. Conversely, tasks such as SST-2, CoLAA, and QNLI heavily rely on contextual information to ascertain sentiment analysis accuracy, linguis- tic acceptabilitty and question answering, respec- tively. Consequently, these tasks inclined to- wards being influenced by environmental shifts rel- ative to the training data (for BERT-base) originat- ing from Wikipedia and BookCorpus. Furthermore, STSB and RTE tasks demonstrate a notable level of change in the target data distribution with time as the language reference can change.
5.3 Sensitivity of localized knowledge gain For some tasks (RTE, STSB, CB, and COPA) FIM score based selected layers under-performed in gical fine-tuning approach. Thus, attempt to in- vestigate through the lens of differences Iearned linguistic features and possible distributional shifts target data, the overall effectiveness of FIM for real-world NL. tasks.
5.3.1 Effect of linguistic features
Across the GLUE and SuperGLUE benchmarks, we observed that tasks requiring localized linguis- tic knowledge, such discourse understanding (MNLI, MRPC, WNLI, WSC, and MultiRC), syn- tactic understanding (CoLA), semantic understand- ing, and commonsense/wont world knowledge (SST-2, QNLI), can be effectively fine-tuned with fewer lo- calized parameters identified through ranked layer importance from the FIM scores.
When comparing surgical fine-tuning with full fine-tuning in Figure 1, we observe that BoolQ and MRPC outperform the full model fine-tuning, while tasks such as QNLI, CoLA, MNLI, WSC, WiC, and MultiRC yield comparable performance. In contrast, RTE and STSB underperform in the surgical fine-tuning process. This indicates that our proposed approach of utilizing FIM identify layer importance ranks works well in cases of do- main and environmental shifts but fails to adapt to temporal drifts.
However, for tasks that involve textual entail- ment (RTE) and require strong grasp of common sense/world knowledge (COPA), as well tasks focusing on understanding propositions (CB), sur- gically fine-tuning the model using FIM rankings resulted in sub-optimal performance. These tasks rely heavily on semantic understanding, logical rea-

---

Layer Rank
Epoch_o
Epoch 2
Epoch
Epoch_8
Fpoch_10
Layer BERT-base
Layer
BERT-base
Layer
BERT-base
Layer # BERT-base
Layer BERT-base
Layer BERT-base
Figure 2: Plot of layer ranking for SuperGLUE tasks determined through FIM score during various checkpoints of optimization. ie.. epoch 0. 2. 5, 8, and 10. The rank of layers with respect to Fisher Information remains constant across the optimization trajectory. Note that for task CB, the layer rankings identical across all the epochs.
COLA
WNLI
RTE
MRFC
STS-B
10.0
Layer Rank
Epoch
7.5
Eaach
Epoch_2
5.0
Epoch_5
Epoch_10
0.0
10 Layer e BERT-base
10 Layer é BERT-base
5
Layer # BERT-base
Layer BERT-base
Layer BERT-base
Figure 3: Similar SuperGLUE tasks, in Figure 2, show that in GLUE tasks well, the rank of layers with respect Fisher Information remains constant across the optimization trajectory. Especially for WNLI and RTE, layer rankings almost identical the various checkpoints.
5.4 Ranking layers using FIM score vs. optimization trajectory
tuning for many real-world natural language tasks. In future work, we plan to investigate the linguis- tic correlates of different layers in large-language models (LLMs) and the value of FIM in surfacing them.
Upon investigating the efficacy of our proposed approach even further, we observed that the rank- ing of layers for surgical fine-tuning determined through FIM scores for SuperGLUE (Figure and GLUE (Figure 3) tasks remains constant across various checkpoints the optimization trajectory.
Limitations and Future Work
The FIM score criteria proposed in this paper shows promising results on several GLUE and Su- perGLUE tasks with language encoders. However, additional experiments are needed on some of the recent very large parameter models that perform well in zero-shot settings.
In particular, we investigate the rankings at epochs 0, 2, 5, 8, and 10 and observe that for Super- GLUE and GLUE tasks, the deviation in rankings is almost negligible (deviation plots in Appendix section A.3), and in some tasks like CB, WNLI, and RTE, the trajectory identical. Thus, the arrange- ment of layers selected by the FIM score remains unchanged for the task at hand as the fine-tuning process progresses.
In addition, we plan to extend our evaluations and compare our method with existing solutions, such as Low-Rank Adaption (LoRA), to quantify the benefits of our approach.
6 Conclusion
Acknowledgements
This paper contributes to the growing body of work that demonstrate that selective fine-tuning of lan- guage models is not only efficient but also effective in many downstream tasks. Summarizing our con- tributions, strongly prove that selecting layers for finetuning based on ranking according to the FIM scores gives optimal results majority of GLUE and SuperGL.UP tasks and could thus help NLP practitioners with small datasets to efficiently select a sub-set relevant layers for optimized fine-
This work supported by GPU resources from the University of Massachusetts, Amherst, and Mi- crosoft Corporation, New England Research and Development Center.
References
Jacob Devlin, Ming-Wai Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of

---

deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. NAACL-HLTT 2019. Minneapolis, MN. USA, June 2-7, 2019, Volume (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. 2022. Lst: Ladder side-tuning for parameter and memory efficient transfer learning.
Yi-Lin Sung, Varun Nair, and Colin Raffel. 2021. Train- ing neural networks with fixed sparse masks.
Thomas Wolf, Lysandre Dehut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Fun- towicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Can- wen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface's 's transformers: State-of-the-art natural language processing. arXiv preprint arXiv: Arxiv-1910.05771.
Wenjuan Han, Bo Pang, and Ying Nian Wu. 2021. Ro- bust transfer learning with pretrained language mod- els through adapters. CoRR, abs/2108.02340.
Shwai He, Liang Ding. Daize Dong, Jeremy Zhang. and Dacheng Tao. 2022. SparseAdapter: An easy approach for improving the parameter-efficiency of adapters. In Findings of the Association for Computational Linguisties: EMNLP 2022, pages 2184-2190, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
A Appendix
A.1
Experiment Setup and Hyperparameters
Jack Hessel and Alexandra Schofield. 2021. How effective is BERT without word ordering? impli- cations for language understanding and data pri- vacy. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 204-211, Online. Association for Computa- tional Linguistics.
We used our set of hyperparameters (mentioned in Table 1) achieve close to State of the Art perfor- mance on almost all the GLUE and SuperGLUE tasks on fine finettuing the entire model. We then used this fully fine-tuned model as our baseline for com- paring the performance after fine-tuning selected layers using FIM.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. CoRR, abs/2106.09685.
Train Batch Size
16
Validation Batch Size
16
Learning Rate
5e-5
Epochs
10
Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Ku- mar. Huaxiu Yao, Percy Liang, and Chelsea Finn. 2023. Surgical fine-tuning improves adaptation to distribution shifts. In The Eleventh International Conference on Learning Representations.
Weight Decay
0.01
Table 1: Hyperparameters used
We are using standard GLUE and SuperGLUE datasets and BERT (Devlin et al., 2019) as the trans- former based language model for our experiments with FIM. For the datasets, leveraged the Hug- gingFace Transformer library (Wolf et al., 2019) and used the full validation sets corresponding the task to train the model and evaluate its perfor- mance (mentioned Table and 3). Code im- plementation is done using PyTorch and Python on Unity clusters (M40 and TitanX instances) and Google Colab.
Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky, 2023, Scaling down to scale up: A guide parameter-efficieet fine-tuning.
Haokun Liu, Derek Tam, Mugeeth Mohammed, Jay Mohta, Tenghao Huang. Mohit Bansal, and Colin Raffel. 2022. Few-shot parameter-efficient fine- tuning is better and cheaper than in-context learn- ing. In Advances in Neural Information Processing Systems.
Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, and Ian Tenney. 2020. What happens to BERT embed- dings during fine-tuning? CoRR, abs/2004. 14448.
A.2 Task Specific Observations in BERT with Fisher metrics
Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2888-2913 Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
We conducted various fine-tuning experiments on various GLUE and SuperGLUE tasks using BERT base cased model. The accuracy values obtained in these experiments are mentioned in Table and 5.
Selective Layer Fine-tuning:

---

QNLI
SST-2
CoLA
MRPC
RTE
WNLI
STSB
QQP 40,430
MNLI-M
5,463
872
1,043
408
277
71
1,500
9,815
Table 2: Validation Dataset size across GLUE tasks
CB 56
COPA 100
MultiRC 4,848
WiC
WSC 104
BoolQ 3,270
638
Table 3: Validation Dataset size across SuperGLUE tasks
(a) CoLA (Corpus of Linguistic Acceptabil- ity): CoLA focuses on grammatical ac- ceptability, requiring model to under- stand subtle linguistic nuances. It is likely that the lower layers captured in the Fisher metrics, which correspond to lower-level linguistic features, contain vital information for such tasks. By fine- tuning these layers, the model can still grasp the necessary linguistic patterns and achieve comparable performance.
ference): WNLI a pronoun resolution task that requires understanding corefer- ence. The consistent accuracies across all layers suggest that the task primar- ily relies on higher-level reasoning and discourse-level comprehension rather than fine fine-grained distinctions among lay- ers. Thus, fine-tuning smaller subset of layers may still achieve comparable performance.
(c) BoolQ (Boolean Questions): BoolQ involves answering boolean questions based on passages. As it requires logical reasoning and comprehension, the more streamlined approach of fine- tuning fewer layers (top layers given by Fisher) may enhance the model's 's ability to reason and derive accurate answers.
(b) QQP (Quora Question Pairs) and QNLI (Question Natural Language Inference): These tasks involve semantic similarity and entailment recogmition, which can be aided by lower-level features captured in the lower layers. Fine-tuning the top-5 layers from the Fisher metries may still preserve enough relevant information to make accurate predictions.
Task-specific Variations:
(c) MultiRC (Multi-Sentence Reading Com- prehension): MultiRC assesses compre- hension across multiple sentences. is possible that the lower layers encompass critical contextual information, enabling the model to capture relevant relation- ships and dependencies within the text. Fine-tuning these layers could provide sufficient context understanding for ac- curate predictions.
(a) WNLI and COPA:
The tasks exhibit con- sistent accuracies across all layers, indi- cating that the distinctions among layers may not significantly impact the perfor- mance. WNLI's focus on pronoun resolu- tion and COPA's requirement for causal reasoning might rely more on higher- level understanding and reasoning abili- ties rather than specific layer representa- tions.
(b) WSC (Winograd Schema Challenge): WSC presents a challenging task that evaluates commonsense reasoning. Re- markably, the top accuracies, full model accuracy, and bottom-1 accuracies are all exactly the same. This result suggests that the layers identified by the Fisher Information Matrix capture the essential reasoning capabilities required for effec- tive performance, underscoring the sig- nificance of higher-level features.
2. Improved Accuracy with Fewer Layers:
(a) MRPC (Microsoft Research Paraphrase Corpus): MRPC focuses paraphrase identification, which often relies lo- cal context and surface-level similarities. The top-3 layers selected from the Fisher metrics may effectively capture these as- pects, resulting in improved performance compared to the full model.
(b) WNLI (Winograd Natural Language In-

---

22
BOOLD
50
Park ocvisiD
10 Layer BERT base
2 4
4 6
8
2
4
6
10
3 10 Layor BERT base
6 Layer BERT base
2
4
0
2
8 10
Layer # BERT base
Layor
BERT base
Layer BERT base
Figure 4: Degree of layer rank deviation (on y-axis) for SUPERGLUE tasks (complementary representation to Figure 2).
Dagres of devialion
12
COLA
10
00
6
2
10 Layer Number BERT-base
5
10
Layer Number BERT-base
Layer Number BERT-base
Layer Number BERT-base
Layer Number BERT-bas
Figure 5: Degree of layer rank deviation y-axis) for GLUE tasks (complementary representation Figure 3).
A.3 FIM scores during optimization
Figures and 5 are complementary extensions to Figure 2 and Figure 3, respectively, where we present evidence of minimal layer rank deviations (y-axis) for all SuperGLUE and GLUE tasks. For CB, WSC, COPA, WNLI, and RTE, the ranking of layers is identical for epochs 0, 2, 5, and 10 of the training process.
A.4 Results from BERT-base
copa
multirc
75
001a
WSC
85
mepp
70
boolq
80
Performence
75
Performace
70-
60
50
45
Top k fine-tuned leyers
60
Figure 7: Performance of the SuperGLUE Tasks when the top layers identificd by the fisher score fine tuned
Top fine tuned layers
Figure 6: Performance of the GLUE Tasks when the top k layers identified by the tisher score fine tuned

---

Layers finetuned
QNLI
SST-2
CoLA
MRPC
RTE
WNLI
STSB
QQP
MNLI-M
State the Art
0.905
0.935
0.521
0.889
0.664
0.563
0.858
0.712
0.834
Full-model
0.905
0.923
0.582
0.846
0.620
0.563
0.574
0.907
0.829
Top
0.851
0.908
0.422
0.806
0.552
0.563
0.464
0.888
0.778
Top 2
0.873
0.904
0.483
0.826
0.555
0.563
0.486
0.901
0.798
Top 3
0.886
0.909
0.539
0.868
0.552
0.563
0.534
0.901
0.806
Top 4
0.896
0.910
0.560
0.838
0.577
0.563
0.528
0.901
0.815
Top
0.893
0.907
0.571
0.848
0.566
0.563
0.539
0.902
0.817
Bottom
0.899
0.885
0.208
0.684
0.581
0.563
0.525
0.903
0.730
Table 4: GLUE test results based on Fisher score BERT base cased
Layers finetuned
CB
COPA
MultiRC
WiC
WSC
BoolQ
State of the Art
0.836
0.706
0.700
0.695
0.643
0.774
Full-model
0.804
0.556
0.695
0.718
0.635
0.707
Top
0.625
0.556
0.681
0.652
0.635
0.667
Top 2
0.625
0.556
0.684
0.666
0.635
0.669
Top 3
0.643
0.556
0.677
0.682
0.635
0.661
Top 4
0.661
0.556
0.686
0.683
0.635
0.718
Top 5
0.661
0.556
0.687
0.683
0.635
0.680
Bottom
0.393
0.556
0.658
0.636
0.635
0.681
Table SuperGLUE test results based on Fisher score on BERT base cased


Reviews:
[{"review_id": "neLVgFhIsh", "reviewer": "Reviewer_rfrx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Editors and Reviewers,\nThank you for taking the time to review our manuscript. We appreciate your constructive feedback and the opportunity to address the concerns raised. Below, we provide detailed responses to each of your comments.\n\n1. Missing Common Baseline:\nWe understand the importance of including a common baseline such as freezing the bottom N layers. In light of this, we experimented with freezing the bottom 5 layers of the BERT-base model and compared the performance results against the selected layers using our surgical fine-tuned (FT) method. For almost all tasks in SuperGLUE (except BoolQ), the results show that the surgical fine-tuning approach can outperform the baseline on all tasks, with an average improvement of 4.7%. This demonstrates that surgical FT can better identify and optimize the important layers for fine-tuning than a simple heuristic.\n\n2. Missing Reference to AutoFreeze:\nAutoFreeze is a method that automatically freezes model blocks based on a convergence criterion during fine-tuning. It differs from our method in two aspects: (1) AutoFreeze freezes model blocks dynamically during fine-tuning, while our method freezes model layers statically before fine-tuning; (2) AutoFreeze uses a convergence criterion based on the gradient norm of model blocks, while our method uses a Fisher information matrix (FIM) score based on the sensitivity of model parameters to fine-tuning. Our FIM score metric provides a low-resource alternative to AutoFreeze with promising results and thus is applicable to scenarios where data is limited to fine-tune for specific tasks.\n\n3. Limited to BERT Model:\nWe conducted experiments with RoBERTa on a subset of tasks, evaluating its performance on GLUE tasks like CoLA, MRPC, RTE, WNLI, and SuperGLUE tasks like WiC. Notably, we observed a consistent trend: when utilizing top 1 to top 5 layers determined by FIM scores in the surgical fine-tuning process and freezing the remaining layers during fine-tuning, we consistently achieved the highest accuracy at the top 5 layers across the above-mentioned tasks. \nThe observed trend in the experiment results across different tasks in both BERT and RoBERTa models can be attributed to various factors related to the inherent properties of neural network architectures, task complexities, and fine-tuning processes. We aim to generate results from all experiments with RoBERTa and XLNet to show how our approach generalizes across different architectures, and we hope to add these results to the final version of our paper.\n\nWe believe that addressing these concerns will significantly improve our manuscript. We look forward to your further comments and hope our revisions will make our paper suitable for acceptance at EMNLP 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is well written and easy to follow\n* The surgical FT method seems to be mostly effective\n* Analysis are performed where surgical FT is not working well", "reasons_to_reject": "* The authors miss a simple common baseline of freezing bottom N layers, is surgical FT able to beat that?\n* The results are limited to the BERT model, it is not clear how would they generalize to latest LLMs\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper explores surgical fine-tuning for NLP, where a subset of layers is selected automatically for fine-tuning, thus being more time and data efficient. The authors suggest Fisher Information Matrix based criteria to select the layers to \"unfreeze\". Results on GLEU/SuperGLEU show that the method is effective on most tasks (within +-5% of full fine-tuning) except tasks that require reasoning or world knowledge.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "* The authors seem to miss one related reference [1] which also does selective layer FT, they rely on a metric to measure if a layer is close to convergence to decide on freezing. How does [1[ compare to the authors work and terms of method and performance?\n\n[1] AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning. https://arxiv.org/pdf/2102.01386.pdf"}}, {"review_id": "j1yl1nMumu", "reviewer": "Reviewer_BW1p", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewers, Thank you for taking the time to review our manuscript. We appreciate your constructive feedback and the opportunity to address the concerns raised. Below, we provide detailed responses to each of your comments.\n\nYou make an excellent point about the need for rigorous comparisons to justify our claims. We will extend our evaluations to compare our method with existing solutions, such as LoRA, to quantify the benefits of our approach.\n\nPerformance numbers for WSC, CB, and COPA (SuperGLUE) using BERT-base from the suggested surgical fine-tuning process against LoRA is comparable, whereas for some GLUE tasks such as RTE and SST2, the LoRA numbers are marginally higher than the surgical fine-tuning process as reported in (2023.repl4nlp-1.19.pdf (aclanthology.org)). We aim to do a comprehensive analysis against LoRA baselines for GLUE and SuperGLUE tasks and include our hypothesis about differences and similarities between surgical fine-tuning-based performance and LoRA due to nuances in linguistic features learned in the transfer learning process in the final version of the paper.\n\nWe believe that addressing these concerns will significantly improve our manuscript. We look forward to your further comments and hope our revisions will make our paper suitable for acceptance at EMNLP 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Efficient Finetuning of large language models is desirable as the size of these models are increasing and consequently demand mode computational power. Therefore, exploring ways to fine tune these models more efficiently is of high importance.\n- Authors investigated the effect of linguistic features and different distributional shift in the performance of surgical fine tuning after observing the differences in performance change in different tasks in FLUE and SuperGLUE. -Authors are aware of the related work and have outlined all the important related work in the paper.", "reasons_to_reject": "In the last part of the related work, authours mention: \"However, our approach of efficient fine-tuning using the proposed FIM score criteria (that is able to capture signals from ≈ 100 target data samples), 1differs from all existing methods, as it focuses on helping NLP practitioners with small size target datasets to efficiently rank and select important layers for optimizing the fine-tuning process.\" . I am not sure this argument is convincing. I think in a research community, we should welcome new ideas and new ways of addressing an issue but the comparisons should be well-justified and backed up by numbers. I wish authors have provided comparisons to some of the existing solutions such as LoRA to point out the benefits their method can hold over existing methods.", "questions_to_authors": "", "paper_topic_and_main_contributions": "- Authors propose using Fisher Information Matrix (FIM) to prune layers in a transformer based model (i.e. BERT) during the fine-tuning of tasks, tested on GLUE and SuperGLUE benchmarks.\n- They carry put analysis of why performance can be improved or drop for specific tasks in GLUE or SuperGRLUE.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OPJ7Ljljp0", "reviewer": "Reviewer_2Coy", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewers, \n\nThank you for taking the time to review our manuscript. We appreciate your constructive feedback and the opportunity to address the concerns raised. Below, we provide detailed responses to each of your comments:\n\nIn our experimentation, we uncover a consistent pattern in the ordering of layers identified by the FIM (Fisher Information Matrix) score at different stages of the optimization process. Specifically, we find that the arrangement of layers selected by the FIM metric remains unchanged for the task at hand as the fine-tuning process progresses. This intriguing discovery is further exemplified by the outcomes observed for the CB SuperGLUE task.\nFor the CB task, we scrutinize the FIM scores and rankings throughout the optimization trajectory, unveiling noteworthy insights into the dynamics of parameter importance. Across epochs, the FIM scores display minimal variation, reinforcing their stability. What's particularly interesting is the constancy of layer rankings, which persistently follow the order: ['8', '9', '7', '11', '6', '10', '0', '2', '4', '3', '5', '1'].\n\nFIM for Fine-Grained Parameter Selection:\nWe understand the concern. We specifically selected FIM for layers instead of individual parameters, as layers encode different levels of linguistic information. Fine-tuning only certain layers (selected by our FIM metric) can provide further interpretability to the fine-tuning procedure corresponding to the downstream tasks. In Section 4.3, we try to investigate the overall effectiveness of FIM for real-world NLP tasks by looking at the differences in learned linguistic features and possible distributional shifts in target data, and can attempt to see how linguistic information is being used. Moreover, our results in Sec. 4, Table 4, and Table 5 in the main paper, show that fine-tuning at the layer level leads to comparable performance to full-fine-tuning, therefore validating that using FIM at the layer level, though simple, is sufficient. \nWe will also add a section discussing the limitations of using FIM at the layer level and the potential for a more fine-grained parameter selection by using a GPT-level annotation methodology.\n\nVerification on NLG Tasks:\nYou're correct that we only validated our method on NLU tasks. Given that our study is primarily for language encoders, we focused mainly on the NLU tasks which is the appropriate setup for it. In the future version of our paper, we will extend our experiments to include NLG tasks for a more comprehensive evaluation. \n\nTypos, Grammar, Style, and Presentation:\nWe will correct the errors in lines 453 and 454 and ensure that figures are referred to correctly.\n\nWe believe that addressing these concerns will significantly improve our manuscript. We look forward to your further comments and hope our revisions will make our paper suitable for acceptance at EMNLP 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The idea sounds reseaonable.\n(2) The proposed method is easy to understand and follow.", "reasons_to_reject": "(1)  In theory, since parameters change during fine-tuning,  the importance of each parameter for fine-tuning change dynamically when evaluated by FIM. It may be inappropriate to choose which layer to be tunned statically before fine-tuning. \n\n(2) FIM can choose which fine-grained parameters to be tunned. It may be inappropriate to use it to  choose which layer only.\n\n(3) This paper only verifies their methods on NLU tasks, remaing NLG tasks unsolved.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes using FIM to select  several tuning layers of encoder-based LM to accomplish parameter-efficient tuning.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 453 and 454,  figures have not been referred correctly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces the surgical fine-tuning method that only fine-tunes a selection of layers. The authors propose Fisher Information Matrix (FIM) score for the layer selection. They compare the proposed method with full model fine-tuning and show that their method achieves strong performance in certain tasks.

As mentioned by the reviewers, the paper presents a sound idea and executes it in a straightforward experimental setup. Additionally, the authors provide further analysis to understand the impact of linguistic features, and distributional shifts on the FIM and performance of their method. 

However, as reviewers noted and also acknowledged by the authors, the paper lacks important comparisons with several baselines. This includes both similar methods such as AutoFreeze to justify that the proposed method is advantageous in certain conditions, and also existing parameter-efficient fine-tuning methods such as LoRA. This comparison should include the performance metric as well as the potential benefits of using this method such as parameter or sample efficiency.
Decision: Accept-Findings