Information Guided Regularization for Fine-tuning Language Models
Authors: Mandar Sharma
Nikhil Muralidhar
Shengzhe Xu
Raquib Bin Yousuf
Naren Ramakrishnan
Venue: COLM (2024)
Keywords: regularization, finetuning, loss landscape, information, transfer learning
TL;DR: We propose a novel (and better) approach to dropout regularization that is both task-agnostic and adds no computational overhead to the fine-tuning process.

Abstract: The pretraining-fine-tuning paradigm has been the de facto strategy for transfer learning in modern language modeling. With the understanding that task adaptation in LMs is often a function of parameters shared across tasks, we argue that a more surgical approach to regularization needs to exist for smoother transfer learning. Towards this end, we investigate how the pretraining loss landscape is affected by these task-sensitive parameters through an information-theoretic lens. We then leverage the findings from our investigations to devise a novel approach to dropout for improved model regularization and better downstream generalization. This approach, named guided dropout, is both task & architecture agnostic and adds no computational overhead to the fine-tuning process. Through empirical evaluations, we showcase that our approach to regularization yields consistently better performance, even in scenarios of data paucity, compared to standardized baselines.

Published as a conference paper at COLM 2024
Information Guided Regularization for Fine-tuning Language Models
Mandar Sharma', Nikhil Muralidhar', Shengzhe Xu! Raquib Bin Yousuf & Naren Ramakrishnan" !Department of Computer Science, Virginia Tech 2Deparmment of Computer Science, Stevens Institute of Technology mandarsharma@v edu
Abstract
The pretraining-fine-tureee paradigm has been the de facto strategy for transfer learning in modern language modeling- With the understanding that task adaptation in LMs is often function of parameters shared across tasks, we argue that a more surgical approach to regularization needs to exist for smoother transfer learning. Towards this end, we investigate how the pretraining loss landscape is affected by these task-sensitive parame- ters through an information-thhorre
lens. We then leverage the findings from our investigations to devise a novel approach to dropout for improved model regularization and better downstream generalization. This approach, named guided dropout, is both lask & archilecture agnostic and adds no compu- tational overhead the fine-tuning process. Through empirical evaluations, we showcase that our approach to regularization yields consistently better performance, even in scenarios of data paucity, compared to standardized baselines. Our codebase for reproducibilty hosted here!.
1 Introduction
Although highly-parameterizeed LMs have recently demonstrated incredible few-shot and zero-shot capabilities (OpenAI, 2023; Touvron et al., 2023), transfer learning via fine-tuning still remains crucial for building task-specific models (Sharma et al., 2022; Min et al., 2023). As such, mechanisms to improve the efficiency of fine-tuning parameter-heavy LMs have garnered much attention leading to frameworks like LoRA (Hu et al., 2021). However, the aspect of transfer learning that doesn't receive as much attention is regularization. As wwer-parameterized models are made to adapt niche tasks with few data points, effective regularization can go long way into building models that generalize better to downstream tasks. Specially in light of recent findings that LMs tend to share parameters depending on the specific downstream task (Sharma et al., 2023), we argue against regularizers that treat all LM parameters the same all neurons are equal but some neurons are more equal than others. Here, we empirically study the effects that LM parameters have on their loss geometry and propose more surgical and effective approach to regularization.
As our proposed approach is independent of the target task, it simply involves appending few lines of pre-determined code to the LM training loop to obtain a slight performance improvement on virtually any downstream task. This could have significant implications as foundational models are often used for a colossal number of downstream applications the pert-base-uncased model has been downloaded more than 1.5 billion times on HuggingFace (Wolf, 2024). Concisely, we offer the following contributions:
We We the effects that task-sensitive parameters have on the LM loss landscape through their visual geometry.
Through our findings, we propose novel information guided approach to L2 reg- ularization that its both task & architecture agnostic and adds no computational overhead to the fine-tuning process.
htttps://iithu/ http://ittuuu..ccmaaaaarrr hww.shhhw..ccaaaaaaaaaaaaaaaaaaa drirdrr-poud

---

Published as a conference paper at COLM 2024
We empirically showcase that approach to guided dropout yields consistently better performance compared to standardized baselines, specially in scenarios of data paucity.
Alongside, we also showcase that reliable estimate of the model information can be obtained in cost-effective manner simply through a small sub-sample of the training corpus.
Theoretical Foundations
The theoretical properties underpinning the mini-batch based stochastic gradient descent (SGD) has received thorough exploration from the scientific community (Ge et al., 2015; Hardt et al., 2016; Bottou et al., 2018). Specially relevant converges through the approximation of the eigenvalues of the Hessian (Keskar et al., 2016) and through its visualized geometry eLi al., 2018; Yao et al., 2020). With Fisher information as a proxy to the Hessian ($2.1), we investigate how this computationally cheaper first-order metric can lend us an understanding of how the LM loss landscape geometry is affected by its task-specific parameters ($2.2). Further, we leverage these insights to devise an information guided approach to L2 regularization via dropout (Wan et al., 2013) for transfer learning ($2.3).
is the investigation of how loss
Buluca Sonj
Nadds
(a)
(b)
(c)
Figure 1: The 3D loss landscape geometry of the pretrained BERTBASE model when: (a) all model parameters are perturbed, (b) 50% of the model parameters with the highest Fisher scores are perturbed, and (c) 50% of the model parameters with the lowest Fisher scores are perturbed. Plot (b) resembles the typical sharp minimizers that generalize poorly compared to the wide minimizers of base plot (a) and control plot (c).
2.1 A Tale of Two Matrices: The Fisher & the Hessian
Observation 1: At + 6,, the Fisher information approximates the Hessian and thus can be used understand the training dynamics and loss landscapes of LMs.
For an LM parameterized by 0, the Fisher score function s()) represents the gradient of the log-likelihood function log (x)()). The Fisher information I(0) is understood as the
covariance matrix K of the score function (1). Characteristically, the Fisher score evaluated at the true value of the model parameter 0, is O, giving us (2). However, as this requires computing the expectation over the distribution 0, which is often intractable, the Empirical Fisher (3) is computed given dataset x E with N samples.
I(0) = Ks(0) = [[(s(() E(s(O.)))(s(O)) -(s())))))))))))))))
(1)
(F(s())s(0)T)
(2)
1
alog ( a log (xx10)
(3)
N
0e d0T
2

---

Published as a conference paper at COLM 2024
Similarly, the Hessian for the LM characterized by 0 can be written as:
1 N
alog alogp(x (xx00)
llog (x
1 2² 0x pxx
H
(4)
a0
d00
d000r
Jastrzebski et al. (2017) note that the second term in (4) is negligible in the case when the conditional probability distribution of the model coincides with the conditional distribution of the training data. Mathematically, this occurs when the model parameters of the LM is close to the optimum such that - 6.- Thus, for converged pretrained LM, the covariance represented by the Empirical Fisher (3) is approximately the Hessian.
Top 5% of Parameters
Top 1% of Parameters
IN Is DKL(IN, Is)
IN * DKL(IN, Is)
Sample
0
0
μ
0r
33%
0.9864
0.0409
0.0055
0.0304
0.9767
0.0825
0.0044
0.0243
10%
0,9428
0.1219
0.0164
0,0451
0,9305
0.1665
0.0159
0.0415
5%
0.9661
0.0788
0.0126
0.0544
0.9547
0.1171
0.01
0.0407
1%
0.9233
0.1338
0.0261
0.0923
0.9111
0.1732
0.0212
0.0665
0.10%
0.7564
0.2448
0.1149
0.2272
0.7349
0.3035
0.1011
0.1757
Top 0.5% of Parameters
Top 0.25% of Parameters
Is
DKL(IN, Is)
IN
DKL Is)
J
33%
0.9643
0.1427
0.004
0.0208
0.9399
0.3163
0.0034
0.0169
10%
0.9136
0.2057
0.014
0.0348
0.8784
0.4215
0.0112
0.0274
5%
0.9445
0.1597
0.0092
0.0351
0.8998
0.4003
0.0072
0.0271
1%
0.8916
0.2504
0.0196
0.0574
0.8424
0.4822
0.017
0.048
0.10%
0.7208
0.3792
0.0898
0.1599
0.2908
0.3956
0.0612
0.1451
Table 1: The layer-wise mean μ and standard deviation o of the cross-correlation (+) and the KL-Divergence (DKL) between the Fisher information computed with the entire WikiText- 103 dataset (IN) vs that computed with the sub-sampled dataset (Is, where S << N) for the base BERT model. As noted in section 52.2, since only fraction of the parameters contain the highest concentration of Fisher scores, we evaluate the consistently of the Fisher information matrix based on the top (5, 1, 0.5, 0.25) percent of the LM parameters. Based on this table, we conclude that reliable estimates of the Fisher information can be obtained through as little as 1-5 % of the training data.
Observation 2: For LMs, reliable estimates of the Fisher information can be obtained from fractional sub-sampled corpus.
While the eigenvalues of the Fisher matrix certainly be computed effectively in terms of memory and run-time (Liao et al., 2018), we showcase that for LMs, the entire Fisher information matrix I(0) can be approximated through fractional sub-sample of the orig- inal training corpus. The pretraining corpus for BERTBASE consists of combination of the Wikipedia corpus and the book corpus (~ GB) (Devlin et al., 2019). We attain our approximation of the Empirical Fisher 1(0) for BERT through WikiText-103 (Merity et al., 2016), much smaller cut (~ 4%) BERT's original pretraining corpus. The effective results we present in section $3 are all based on this approximation. We attribute this obser- vation to the nature the pretraining LMs the redundancy that exits as a consequence of being trained on large unstructured corpora of free-form text (Dalvi et al., 2020). To further illustrate this point, in Table 1, we approximate I(0) with incrementally smaller sub-samples from WikiText-1003 We find reliable estimates of I(@) even with just 1% to 5% of the samples from WikiText (0.04%- - 0.2% the original pretraining corpus), although the quality of the estimate drops if we sub-sample below 0.01% (0.004 % of the original corpus).
3

---

Published as a conference paper at COLM 2024
2.2 Fisher Information & the LM Loss Landscape
Prior studies have used the Fisher information as substitute to the Hessian for char- acterizing the properties of stochastic gradient descent (Liao et al., 2018; Martens, 2020). To attain complementary insights, we perform an empirical study of the loss geometry of LMs as a function of the Fisher information I(0). Borrowing the filternnormalization
based visualization scheme of Li et al. (2018), we plot the loss landscape of the pretrained BERTBASE (Devlin et al., 2019) as f fa, B) (5) where the cross-entropy loss L is computed as the converged model parameters 8, are perturbed with directional vectors 5 and lj scaled by a and & respectively. The loss landscape for the BASE is depicted in Figure (a).
f(a, = L(eo + ad + Bn)
(5)
Observation For LMs, an exceedingly tiny fraction of the model parameters have signifi- cantly high Fisher scores and they have a pronounced effect on the loss geometry.
LMs are known to have task-specific peculiarities in the parametric distribution of their Fisher information (Sharma et al., 2023). We observe similar behaviors where only an exceedingly tiny fraction of the model parameters have significantly high Fisher scores (see Figure 3 a). To observe the effects of these parameters on the training loss geometry, we selectively perturb half of the parameters of 6 that lie on the higher end of the Fisher score spectrum (see Figure 1 b). The loss geometry These have been known to lead to poor generalization (Hochreiter & Schmidhuber, 1997; Keskar et al., 2016; Jastrzebski et al., 2017). On the other hand, the loss geometry of the BERT model with the entirety of 6g perturbed (Figure 1 a) resembles that of a wide minimizer that are known to generalize better. To act as control experiment strengthen findings from Figures 1 a & b, we plot the loss geometry of the same model but this time perturbing those half of 9g that lie on the lower end of the Fisher score spectrum (see Figure c). This control plot does not deviate from the wide shape of the loss geometry of the BERTBASE model in Figure 1 a. Thus, the half of BERT parameters with higher Fisher scores contribute drastically more to the loss geometry than the half with lower Fisher scores.
formed resembles typical sharp minimizer.
Reference [-110 M Porams)
Reference [-110 Params) [-44
Reference 110 Params)
(-88
Cross Entropy Loos
60% (-66 Faroms)
25% 2-27 M Paroms)
Crozs Entropy Loos
60% (- M Paroms)
40% (-64 Params)
5% (-550 K Params)
40% [-44 M Params))
1-275 Params)
-20 -15
-10
Purterbation Direction
00 01 1.5
15
20
-20 -15
-1.0 ae 1.5
Purterbation Direction
13
2.
-15 -20
-10 -Is ao 1.2 13 20
Direction
(a)
(b)
(c)
Figure 2: The loss landscape geometry of the pretrained BERT model when: (a) & (b) of parameters with the highest Fisher scores are perturbed, (c) n % of randomly chosen parameters are perturbed. Plot (a) showcases worsening minimas as we narrow down the parameters deemed essential by the Fisher score. Plot (b) showcases that the worsening stops and the landscape remains consisten the we take < 40% of the highest scoring parameters. Plot (c) acts as the control plot.
Further, incrementally focusing the model perturbation on those selective LM parameters that hold the highest Fisher scores leads to an incremental degradation in the loss landscape (see Figure 2 a). Here, as we perturb fewer and fewer parameters while focusing on the ones with highest Fisher scores (from 80% of the parameters with the highest Fisher scores all the way to 0.25% of the parameters with the highest scores), we see the loss landscape incrementally converging to flatter regions in the neighborhood of the minimum, again leading to poor generalization (Hochreiter Schmidhuber, 1997; Keskar et al., 2016; Jastrzebski et al., 2017). It is interesting to note that as we go below 40,, the effect of
4

---

Published as a conference paper at COLM 2024
parameter perturbation on the loss landscape seems to remain consistent. This implies that the 220,000 (0.25%) out of 160,000,000 BERT parameters are so vital to the model convergence that perturbing them produces pronounced of an effect perturbing 44,000,000 (40%) of the model parameters. Similar to the 3D loss geometries, to further solidify our observations, introduce control plot (see Figure 2 c). In this plot, there is consistent signal of worsening loss geometry as we perturb randomly chosen 80%, 60%, 40% of the model parameters.
Fisher Information Guided Regularization
Observation 4: For transformers, minority of the layers hold significant concentration of Fisher sensitive parameters. This observation is used to devise a surgical approach to regularization.
Shifting our attention from the atomic parameter-wisst view to an aggregate layer-wise view, yet again observe similar findings: minority of the transformer layers hold a significant concentration of Fisher sensitive parameters (see Figure 3b, see Appendix SA.2 for individual non-normalized plots). Based this observation and our findings from $2.2, we devise an approach to L2 regularization for transfer learning which we term information-guided regularization. Our idea is rather straightforward. implement L2 regularization such that the sway from current model convergence is minimal. In other words, during transfer learning, focus the regularization more on the LM layers deemed less important by the Fisher information and relax regularization on those layers deemed as vital by the same. Based on conventional popularity, we choose dropout (Srivastava et al., 2014) our L2 implementation. For simplicity, our approach to regularization is algorithmically illustrated in appendix $A.4.
1.0
Fisher Score (Nermalized)
GPT2
Fiher (Normalizeg
GPT2
0.0
BERT
0.8
T5
0.6
0.6
0.
0.0
0.9
05
1/
1.5
100
150
Transformer Parameters lin 10")
Transformer Layers
(a)
(b)
Figure 3: The sorted Fisher scores (normalized) for BERT, GPT2, and T5 based their (a) model parameters and (b) model layers (aggregated) see appendix SA.1 and $A.2 for the raw (non-normalized) plots. From (a) we observe that a fraction of LM parameters are often attributed the highest Fisher scores. Similarly, (b) provides an aggregate view showcasing how minority of the LM layers hold the majority of training information. Please see appendix $A.3 for the unsorted layer-wise Fisher score distributions these models.
Assuming a LM with E (1,.../[L]) layers such that the output of a given layer I with weights (1)) and biases b(4) can be represented as:
y =f b(1)
(6)
Here, f is the activation function. layer I followed by a standard dropout layer in the LM network, the forward operation becomes:
(0)) ~ Bernoulli(p)
(7)
(()) - 70) * (1))
(8)
5

---

Published as a conference paper at COLM 2024
Conventional fine-tuning with dropout has p constant. Given that we have ordered collection of layers S = {I( < I(12) < ... < [(11))) sorted based on their Fisher scores I()), we propose an incremental setting of layer-wise p()) such that the masking of y through Bernoulli(p()) follows a layer-wise masking schedule. We define our Bernoulli sampling probability pll) for the layer I at the jt position in S as:
p())
Σ As
(9)
p(b) Bernouli(p
(10)
where As Pupper -Pieser the step size followed by the linear increment schedule. Here, ng
Pupper Pupper E (0,1) and Plower E (0, are constants that represent the upper and lower bounds for the value that p() can take and ny represents the number of dropout layers in the LM architecture. Our approach amounts to sampling sub-networks from a larger network where the guidance imposes a sampling bias towards retaining highly informative neurons in the sub-network. This guided sampling bias leads to improved generalization by learning through better behaved loss landscape.
The determination of our layer-wise masking schedule is one-time task. After p(0 is determined for a model, the same schedule yields improvements in generalization for all downstream tasks for the model (as we demonstrate in section $3). Thus, not restricted by specific model architectures nor specific downstream tasks, our guided dropout is both task and archilecture agnostic. Further, as the the layer-wise dropout probabilities p(h) are predecided ahead of fine-tuning, guided dropout adds no computational overhend to the fine- tuning process. The p(l) implementation used for this study is linear, however, users are free to utilize their own non-linear schedules. We empirically evaluate the efficacy of our approach in the section that follows.
3 Fine-tuning LMs with Information Guided Regularization
Phang et al. (2018) state that fine-tuning BERTLARGE sometimes fails when target dataset has fewer than 10,000 training instances. This was specially apparent on a subset of the GLUE dataset (Wang et al., 2018) that have relatively low data samples CoLA, MRPC, RTE, and STS-B. Intuitively, this would make sense to converge on a small dataset. Lee et al. (2019) evaluate their regularization technique by these 4 tasks and showcase high rate of failure among random restarts? for standard dropout- specially for the CoLA task where 7 out 20 of their runs fail to converge. In our attempts to replicate this, using the exact same specification tions and the original BERT model, we found that all 30/30 of our random restarts for CoLA converged to a respectable Matthew's correlation score (see appendix $A.5 for the full table). The only differentiating factor between our experiment and theirs was the batch-size, our 8 vs their 32. There have been numerous investigations into how large batch sizes lead to generalization failures (Keskar et al., 2016; Jastrzebski et al., 2017).
overparameterized model would fail
fine-tuning BERTLARGE
However, we do borrow the idea to simulate model performance under data paucity. Instead of having an waeepparameeerized model LARGE evaluated on smaller GLUE datasets, we evaluate BERT BASE on progressively smaller cuts of those same four datasets CoLA, MRPC, RTE, and STS-B. We progressively shrink the size of the training dataset from 100% to 10% and evaluate our regularizer with standardized baselines all the way through. To establish consistency, we report results as averages across random restart runs along with the corresponding inter-quartile ranges of the runs. Figure 4 showcases the perfor-
Based on Lee et al. (2019), random restarts refer to using different train-test split of the dataset for each fine-tuning run.

---

Published as a conference paper at COLM 2024
mance of guided dropout (our regularizer) against two popular contemporary choices for regularization for the GLUE tasks.
The standard dropout regularizer (Srivastava et al., 2014) follows the masking of the input vector as shown in equation (6). Gaussian dropout (Wang & Manning, 2013)
Baselines:
follows similar idea where instead of zero-ing out parameters in the input vector, it injects Gaussian noise into the vector. The noise e draws from the Gaussian distribution (1,
where & = r with as the dropout probability. We choose 10% as Lee et al. (2019) report Dropout(0.1) to have be optimal for all GLUE tasks.
Experimental Setup: For fair comparisons to the baselines, we use the same training hyperparameters those recommended by Devlin et al. (2019): adam optimizer with learning rate = =2 10-5, P1 = 0.9, B2 = 0.999, learning rate warm-up over the first 10% of steps and linear decay of the learning rate after the warm-up steps. For each GLUE task, the exact train-test split is decided by a random seed and the output layer of BERTBASE for the task is randomly initialized from a distribution N (0, 0.022). The fine-tuning is performed for 3 epochs for all tasks and models and the results are evaluated on the development set.
0.850
0.9
0.825
PCC/r Coefficient
0.8
0.7
0.800
0.6
0.775
0.5
0.750
0.725
Dropout(0.1)
0.4
Dropout(O.1)
0.700
Gaussian (0.))
Gaussian Dropout(0.1)
0.675
Guided Dropout(0.1)
Guided Dropout(0.1)
0.650
0.0
80
Training Set Percentage (%)
Training Set Percentage (%)
(a)
(b)
Matthew's Correlation
0.6
0.65
0.5
Accuracy
0.4
0.3
0.55
Dropout(0.1)
Dropout(O.1)
Gaussian (r.1
Gaussian Dropout(0.1)
0.50
Guided Dropout(0.1)
Guided Dropout(0.1)
0.0
100
80
60
40
20
100
80
40
20
Training Set Percentage (%)
Training Set Percentage (%)
(c)
(d)
Figure 4: The performance of Guided dropout (our regularizer) vs standardized baselines on fine-tuning BERTBASE decreasing cuts of the training datasets for (a) MRPC (b) STS-B (c) RTE and (d) CoLA. Each data point in the grid is an average across 5 random restarts, as shown by the boxplots. Guided dropout yields consistently better results for all the tasks, specially under data paucity.
GLUE Tasks: Although we evaluate the performance of our model vs the baselines under simulated data-paucity for the select GLUE tasks (following Phang et al. (2018) and Lee et al. (2019), Figure 4), we still perform the standard evaluation our models with respect to the baselines for all tasks in the GLUE benchmark (see Table 2):
Single Sentence Tasks: CoLA (the Corpus of Linguistic Acceptability) (Warstadt et 2019) for grammatical fidelity, SST-2 (the Stanford Sentiment Treebank) (Socher et al., 2013) for sentiment prediction.
Similarity and Paraphrase Tasks: MRPC (the Microsoft Research Paraphrase Corpus) (Dolan Brockett, 2005), QQP (the Quora Question Pairs), and STS-B (the Semantic Textual Similarity Benchmark) (Cer et al., 2017) for semantic equivalence.
7

---

Published as a conference paper at COLM 2024
Regularizer
CoLA
STS-B
MNLI
MNLI (MM)
MRPC
Dropout (10%)
0.56170.5446
0.8605
84.5284.37
84.9584.47
80.4579.52
Gaussian Dropout (10%)
0.5782
0.8630
84.7884.66
85.3184.84
81.1180.01 80.01
Guided Dropout (10%)
0.60960.6445
0.88130.8743
85.1184.42
85.14g4.81
82.4181.44
Regularizer
QNLI
QQP
RTE
SST-2
WNLI
Dropout (10%)
89.5999.50
89,5389
64.8062.92
94.9894.93
45.0743.40
Gaussian Dropout (10%)
89. 89.08gs.90
89 89.45g9.39
66.7864.25
95.1094.97
44.3643.66
Guided Dropout (10%)
89.7189.64
89.6289.58
68.7763.00
95.2395.23
50.7045.42
Table 2: The performance of guided dropout (our regularizer) standard and Gaussian dropouts at p = 10% on GLUE tasks. For consistency, the numbers are presented as Max of results across 5 random restart runs. For CoLA, the metric presented is the Matthew's correlation coefficient and that for STS-B is an average of the Pearson's 's and Spearman-r coefficient. For all other tasks, the figures presented are averages of the accuracy and fl scores. Our regularizer outperforms both baselines on 9/10 10 tasks.
Inference Tasks: MNLI (the Multi-Genre Natural Language Inference Corpus) (Williams et al., 2018) and RTE (Recognizing Textual Entailment) for textual entail- ment, QNLI (the Stanford Question Answering Dataset) (Rajpurkar et al., 2016) for Q&A, and WNLI (the Winograd Schema Challenge) (Levesque et al., 2012) for pronoun referent selection.
4 Conclusion
Fisher information is a useful tool for characterizing the properties of stochastic gradient de- scent (Liao et al., 2018; Martens, 2020). We extend its implementation to further understand the training dynamics of LMs through the visual geometry of their loss landscape. Upon observing the pronounced effect of a select few sets of parameters on the LM loss geometry, we devised surgical approach to L2 regularization that dissuades the sway from optimal model convergence, leading to better generalization in downstream tasks. We term this regularization as information-guudde can be comprehended as sampling sub-networks from a larger network where the guidance imposes sampling bias towards retaining highly informative neurons in the sub-network. This guided sampling bias leads to improved generalization by learning through better behaved loss landscape. Although surgical, each pretrained model requires this estimate to be computed only once - the same estimate is then applied when fine-tuning to any down- stream task. Thus, making our approach both fask & model agnostic with no computational overhead added to the fine-tuning process. We experimentally validate the effectiveness of our regularizer and showcase its prowess, specially in scenarios of data paucity. We also showcase how for LMs, the Fisher information can be frugally approximated with a tiny fraction of the training corpus.
regularization. Applied to dropout, this regularization
Limitations
In our study, we extend the use of Fisher information to further understand the training dynamics of LMs through the visual geometry of their loss landscape. From our findings we devise regularization technique which we showcase to be effective for a multitude of downstream tasks. Due to the exhaustive nature of our experiments, replicating this study for multiple variants of transformer-basee computational resources, thus we chose the most popular LM BERT (Wolf et al., 2019) for our empirical evaluations. However, in section $2, we showcase that the theoretical motivations that make this applicable for the encoder-only BERT model (Devlin et al., 2019) also holds true for the decoder-only GPT2 model (Radford al.) and the encoder-decoder
LMs was not viable given our in-house
8

---

Published as a conference paper at COLM 2024
based model T5 (Raffel et al., 2020). Thus, we believe regularizer to be universally applicable.
6 Ethics Statement
While many LLM applications are fraught with ethical concerns, we can ascertain that our study does not bring forth additional complications. The datasets we use in this study are established benchmark datasets from publicly accessible websites and do not contain any personally identifiable information. Our analyses does not constitute human subjects and thus not within the purview of the IRB. A silver lining from our work is that, by adopting regularization techniques as proposed here, we can achieve better downstream generalization without added computational costs.
7 Acknowledgements
This work is supported in part by US National Science Foundation grant IIS-2312794.
References
Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM review, 60(2):223-311, 2018.
Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEzval-200) (017)), 2017.
Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov. Analyzing redundancy in pretrained transformer models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4908 4926, 2020.
Jacob Devlin, Ming-Was Weig Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, 2019.
Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005), 2005.
Rong Ge, Furong Huang. Chi Jin, and Yang Yuan. Escaping from saddle points online stochastic gradient for tensor decomposition. In Conference on Learning Theory, pp. 797-842. PMLR, 2015.
Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International Conference on Machine Learning, pp. 1225-1234. PMLR, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1:1144, 1997.
Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
Stanislaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv preprint arXit:1711.04623, 2017.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representationn, 2016.

---

Published as a conference paper at COLM 2024
Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. Mixout: Effective regularization to finetune large-scale pretrained language models. In International Conference on Learning Representations, 2019.
Hector Levesque, Ernest Davis, and Leora Morgenstemn. The winograd schema challenge. In Thirleenth International Conference on the Principles of Knowledge Representation and Reasoning, 2012.
Hao Li, Zheng Xu, Gavin laylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
Zhibin Liao, Tom Drummond, Ian Reid, and Gustavo Carneiro. Approximate fisher infor- mation matrix to characterize the training of deep neural networks. IEEE transactions on pattern analysis and machine intelligence, 42(1):15-2)) 16, 2018.
James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21 21146)):177, 2020.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel
mixture models. In International Conference Learning Representations, 2016.
Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys, 56 (2):1-40, 2023.
OpenAL. Chatgpt, 2023. URL https: //chtt. mohaat.ppeai.com
Jason Phang, Thibault Févry, and Samuel R Bowman. Sentence encoders on stilts: Sup- plementary training on intermediate labeled-data tasks. arVio preprint ar 1rXiv=1811.01088, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods Natural Language Processing, 2016.
Mandar Sharma, Ajay Gogineni, and Naren Ramakrishnan. Innovations in neural data-to- text generation: A survey. ar Xiv preprint ar 200th 2022.
Mandar Sharma, Nikhil Muralidhar, and Naren Ramakrishnan. Learning non-linguistic skills without sacrificing linguistic proficiency. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 6178-618 77981 Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023. acl-long.340.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic com positionnalyt over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, 2013.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1)11999-1958, 2014.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. Xrv preprint arX:::2307 099888..0998....8 2023.
10

---

Published as a conference paper at COLM 2024
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International Conference on Machine Learning, PP- 1058-1066. PMLR, 2013.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353 353-355, 2018.
Sida Wang and Christopher Manning. Fast dropout training. In International Conference on Machine Learning, pp- 118-126. PMLR, 2013.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 2019.
Adina Williams, Nikita Nangia, and Samuel Bowman. broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 2018.
Thomas Wolf. Did you know senrt-ased-unncaeee has been downloaded 1.5 billion times the hugging face hub top ai models hitting youtube viral videos number ranges. 16222422,, 2024.
https://x. http:hhhwwwwstttsss/sse//sssae/e
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. 2019.
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference big data (Big data), pp. 581-590. IEEE, 2020.
A Appendix
A.1 LM Parameters based on their Fisher Scores Sorted
40
GPT2
30
BERT
Score
25
15
10
0.0
0.5 1.0
1.5 2.0
Model Parameters (in 10
Figure 5: When LM parameters are sorted based on their increasing Fisher scores, we observe a that only an exceedingly tiny fraction of the model parameters have significantly high scores. This observation is consistent for all popular transformer architectures: encoder only (BERT with 0.001% of parameters having 2 0.01 score), decoder only (GPT2 with 0.02% of parameters having 2 0.01 score), and encoder-decoder models (T5 with 0.0009% of parameters having 2 0.01 score).
11

---

Published as a conference paper at COLM 2024
A.2 LM Layers based on their Fisher Scores Sorted
0.000
0.007
0.0020
Fisher Score
Fisher Score
0.005
0.004
0,0010
0.003
0.002
0.0005
0.001
0.000
23
50
75
100
125
190
175
200
140
Layers
GPT2 Layers
(a)
(b)
0.04
Fisher Score
0.03
0.02
Layers
(c)
Figure Similar to our observations in $A.1, when the layers of LM are sorted based on the aggregate Fisher score of their respective layer parameters, we observe that certain layers tend to have concentration of parameters with high Fisher scores. Again, this observation holds true for all popular transformer architectures: (a) encoder only BERT, (b) decoder only GPT2, and (c) encoder-decoder based T5.
A.3 LM Layers based on their Fisher Scores Unsorted
1.0
BERT
Fisher Score (Norma lize)
GPT2
0.8
0.6
0.4
0.2
0.0
50
100
150
200
250
Transformer Layers
Figure 7: Complementary to figure 6, while only certain layers of the model have signifi- cantly high scores, these vital layers are spread throughout the network and not concentrated towards the beginning or end of the network traditionally believed.
12

---

Published as a conference paper at COLM 2024
A.4 Algorithmic Representation of Information Guided Fine-tuning
Information- Guided Regularizaiton The Algorithm
1. For pre-training corpus C with N samples used to train a foundational model 6 let us subsample n = 1% of N samples (Page 3, Table 1)
2. With these n samples, we estimate the empirical Fisher ((@) (Page 2, Equation 3) for e. Based on ((@), we compute the layer-wise aggregate Fisher scores
If O BERT there are 37 dropout layers in its architecture. These dropout layers are arranged in the descending order of the Fisher scores of the encoder layers that precede them (dropout layer ids in the code snippet below).
4. With Puperr = 10% and Plowe 0%, we linearly decrease the dropout rate for layers proportional to the increasing Fisher scores (Page 6, Equations 9 & 10) Thus, less important layers are highly regularized and more important layers are lightly regularized. The programmatic snippet appears as the following:
List[int], P_upper float, p_lower float):
#Design linearly decreasing dropout schedule 3 dropout_probs range(p_upper, p_lower, step_size)
4 assert len(dropout_probs) == (in(dropput_layer_dds))
5 #Assign dropout probabilities based on increasing Fisher score
6- for layer_id, layer_parame in emumraat((ndel.mmoulee()
7
#If nodule the layer is dropout layer
layer_id in dropout_layer_ids:
#Find Fisher-based ranking by srrr__layer_ddsinnelllaae
dropout_layer_idd
eAssign the respective dropout probability from the schedule dropout_probs propput_probs[idk]
layer_parans. wrwpssddrooottprrbbbbioo
4. Fine-tune the model to any downstream task after modifying the dropout probabilities. Please note that the above code block is not dependent on the target task. if the foundational model e remains the same, the same dropout scheme will improve performance and all target tasks (Figure 4 and Table 2)
A.5 Fine-tuning BERT Large on CoLA
Restart Seed
Matthews Correlation
Restart Seed
Matthews Correlation
0.6357
16
0.6074
0.6164
17
0.6657
0.6127
18
0.6186
0.6412
19
0.6437
0.6060
20
0,6280
0.6730
21
0.6266
0.6482
22
0.5982
0.6206
23
0.6567
0.6475
24
0,6205
0.6555
25
0.6390
11
0.6357
26
0.6378
0.6348
27
0.6461
13
0.6710
28
0.6275
14
0.6470
29
0.6759
15
0.6396
30
0.6394
Table 3: The seed values and the respective Matthew's Correlation score for 30 random restart runs of BERTLARGE fine-tuned on CoLA.
13


Reviews:
[{"review_id": "otQXN2247x", "reviewer": "Reviewer_NchA", "summary": "The paper performs extensive empirical studies on the relationships between Fisher information and loss landscape during LM training. \nThey propose an adaptive dropout technique based on their empirical findings, which has demonstrated to be effective across multiple downstream tasks and multiple types of LMs.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our manuscript and for your positive remarks. These insightful comments help us better our research. Please allow us to address your concerns and the questions you have raised:\n\n1. Models like BERT/GPT2/T5, while superseded by larger models, are still useful to demonstrate the validity of theoretical constructs like ours (see other recent papers like [1,2]). Current limitations in resources prevent us from applying our approach to models like Llama/Mistral. However, as we believe all modern LMs to be overparameterized, our methods should transfer over:\n\n**Architectural Agnosticism:** Overparameterization drives redundancies in the training loss geometry that can be harnessed by intelligent dropout mechanisms like ours.  In section 2.3, we show that the same overparameterization that held true for the encoder-only BERT also holds true for the decoder-only GPT2 and the encoder-decoder based T5. Although modern LMs such as GPT3/4, Llama, and Mistral are significantly larger, they operate on the same decoder-only architecture as GPT2.\n\n**Prevalence of Overparameterization:** Further, recent work has shown that these larger decoder-only models are as overparameterized (if not more) than their smaller counterparts. Gromov et. al. 2024 [3] showcase that up to 50% of Llama 70B, 40% of Llama 13B, and 30% of Mistral parameters are redundant. Thus, although we currently do not have the compute required to evaluate larger models, we strongly believe our findings should generalize.\n\n2. While SSMs like Mamba [4] have showcased transformer-esque performance with faster inference and better scaling, their application to the language modality still follows the pretrain-finetune paradigm. Thus, our regularizer *should* improve downstream performance. Although gauging the *magnitude* of improvement requires a Fisher-based investigation similar to section 2. If our manuscript is accepted, we will include an investigation about the applicability of our method to non-transformer based architectures.  \n\nReferences:\n\n1. Sharma et. al. Learning Non-linguistic Skills without Sacrificing Linguistic Proficiency. ACL Proceedings. 2023.\n\n2. Lodha et. al. On Surgical Fine-tuning for Language Encoders. EMNLP Findings. 2023.\n\n3. Gromov et. al. The Unreasonable Ineffectiveness of the Deeper Layers. 2024.\n\n4. Gu et. al. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The results are very interesting to know. So I think it will be good for the community to see this paper. \n\nThe 4 key observations are very interesting and I enjoy reading about them. \n\nThe empirical results verify their hypothesis and demonstrate their proposed methods. \n\nThe experiments are solid.", "reasons_to_reject": "The experiments are somewhat limited. \n\nBERT, T5, and GPT2 are very interesting models, but they are somewhat obsolete. \n\nThe authors claim that \"Thus, we believe our regularizer to be universally applicable.\" \nBut this seems to be over-claim because there are no experiments---please correct me if I am wrong---about new generation of LMs like Llama or Mistral (even the smallest versions).", "questions_to_authors": "What do you think of its potential generalization to larger decoder-only models? \n\nWhat do you think of its potential generalization to other architectures, like SSM?", "extra_scores": {}}, {"review_id": "l8OsePyUFB", "reviewer": "Reviewer_vAfW", "summary": "This paper proposes a novel approach to regularization in the context of fine-tuning language models (LMs). By utilizing Fisher information to analyze the effects of task-sensitive parameters on the loss landscape of pretrained models, the authors introduce a method called \"guided dropout.\" This technique adjusts dropout rates layer-wise based on the Fisher information of parameters, aiming to minimize the perturbation from optimal convergence and enhance model generalization, especially under conditions of limited data availability. Their empirical results suggest that this approach outperforms traditional regularization methods across various tasks, particularly in data-scarce scenarios.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our manuscript and for your positive remarks. These insightful comments help us better our research. Please allow us to address your concerns and the questions you have raised:\n\n1. We agree that extending our findings to a greater array of models and datasets would further help our cause. With statistics presented for 5 random restarts, showcasing the efficacy of our approach under data sparsity for downstream tasks requires fine-tuning all models with several cuts of the training dataset (from 100% to 10%). Thus, to accommodate both compute and page constraints, we chose the tried and tested GLUE benchmark. However, if our manuscript is accepted, we will utilize the extra page to extend our experimentation to the SuperGLUE benchmark [1] as well as other models.\n\n2. Yes, the effectiveness of our approach indeed relies on the accurate estimation of the Fisher information. Thus, through observation 2 & table 1, we establish that for a text-based pretraining corpus (central to all LLMs), the Fisher information can be reliably estimated using as little as 1% of the training data (rows 4 & 8 in table 1), making our method tractable even for large complex datasets and models. \n\n3. The foundational principle behind our approach is the fact that modern LMs are overparameterized. This overparameterization leads to a small subset of model parameters driving the training loss geometry (observation 3, page 4). Following this, our claim for architectural agnosticism mainly stems from two observ​​ions:\n\n- In section 2.3, we show that the same overparameterization that held true for the encoder-only BERT also holds true for the decoder-only GPT2 and the encoder-decoder based T5. Although modern LMs such as GPT3/4, Llama, and Mistral are significantly larger, they operate on the same decoder-only architecture as GPT2.\n\n- Recent work has shown that these larger decoder-only models are as overparameterized (if not more) than their smaller counterparts. Gromov et. al. 2024 [2] showcase that up to 50% of Llama 70B, 40% of Llama 13B, and 30% of Mistral parameters are redundant. Thus, although we personally do not have the compute to evaluate larger models, we strongly believe our findings should generalize.\n\nReferences:\n\n1. Wang et. al. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. 2020.\n\n2. Gromov et. al. The Unreasonable Ineffectiveness of the Deeper Layers. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The use of Fisher information to guide dropout regularization provides a novel and theoretically motivated method to improve LM generalization, particularly in fine-tuning phases.\n\n2. The proposed method is versatile, applicable to various LMs (as demonstrated with BERTBASE, GPT2, and T5), and independent of the specific downstream tasks, which enhances its utility across different settings.\n\n3. The comprehensive empirical testing, including comparisons with standard regularization methods across multiple GLUE tasks, robustly supports the effectiveness of the guided dropout approach.", "reasons_to_reject": "1. While the results are promising, the experiments are limited to a specific subset of models and tasks. Extending these findings to a wider array of models and scenarios would be beneficial to substantiate the universality of the approach.\n\n2. The effectiveness of the approach is heavily reliant on the accurate estimation of Fisher information, which might not always be feasible or reliable, especially with very large models or complex datasets.\n\n3. Despite claims of architecture agnosticism, the actual implementation and effectiveness may vary significantly across different architectures, which was not fully explored beyond a few model types.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ygxSUQHdZy", "reviewer": "Reviewer_15W3", "summary": "Based on the relationship between (empirical) Fisher information and (the expected) hessian matrix, as well as the established observation that \"sharpness\" of Hessian's correlation with generalizability, this paper proposes to measure \"how well the pretrained parameters are trained\" based on the scale of Fisher score, i.e., parameters with higher Fisher scores are less well pretrained. Built on this conjecture, this paper developed a variant of dropout to apply different strengths of regularization to different layers during the fine-tuning stage. Overall I find this paper interesting and potentially useful in various downstream applications.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our manuscript and for your positive remarks. These insightful comments help us better our research. Please allow us to address your concerns and the questions you have raised:\n\n1. If our manuscript is accepted, we will append the following details to section 2:\n\n- For a network with *|W|* parameters, the cost of estimating Fisher information using *n* samples is *O(n|W|)* gradient computations in the network. We showcase that reliable estimates of Fisher information can be obtained with *n* as tiny as 1% of the pretraining corpus (table 1).\n\n\n- While our current plots showcase sorted layer-wise Fisher scores, we will append figures (see [here](https://shorturl.at/jVyou)) to showcase the raw layer-wise distributions. Interestingly, we note that vital layers are spread throughout the network and not concentrated at the beginning or end as traditionally believed.\n\n\n- We will expand on the details of the loss-landscape computation and visualisation through more elaborate writing and by mathematically extending aspects of equation 5.\n\n2. Figures 2 a & b serve to show the profound effect that a few vital parameters have on the model's loss geometry. 2(b) shows that perturbing the top 0.25% of parameters has the same effect on the loss geometry as perturbing the top 40%. You raise an important question - where is the tipping point (40% < x < 60%) such that the degradation of the loss landscape turns from gradual to sudden. We believe this warrants an \"edge of chaos\" style analysis [3] which we aim to do in future work.\n\n3. OOD generalization across similar tasks as a function of Fisher-based sharper/flatter parameters is an interesting question. There is evidence indicating that Fisher-guided pretraining-finetuning helps generalize better to OOD instances [1]. In line with this, we will evaluate OOD generalization for our method in future work.\n\n4. We agree that if applied to modern architectures that are even more overparameterized [2], the effects of guided-dropout would be even more pronounced. \nAlthough we personally do not have the compute to evaluate larger models, we strongly believe our findings should generalize.\n\nReferences:\n\n1. Sharma et. al. Learning Non-linguistic Skills without Sacrificing Linguistic Proficiency. ACL Proceedings. 2023.\n\n2. Gromov et. al. The Unreasonable Ineffectiveness of the Deeper Layers. 2024.\n\n3. Zhang et. al. Edge of Chaos as a Guiding Principle for Modern Neural Network Training. 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Simple yet effective method\n2. Illustrative analyses\n3. Efficient estimation of Fisher scores", "reasons_to_reject": "1. I think the authors can improve their writing by offering more details. Especially, I am curious about 1) what is the actual cost of Fisher score estimation? 2) which layers are with higher and lower Fisher scores? 3) In Figure 2, we see a big difference between 40&60, what happened there?\n2. Since the loss visualization is quite central to the analyses, offering a brief preliminary section would be beneficial (longer and more detailed than the part in 2.2). \n3. The paper in my opinion would be strengthen by adding more experiments: 1) adding some experiments on OOD. I wonder whether keeping flatter parameters can benefit OOD generalization, and 2) (less important) adding experiments on larger decoder models. As I understand the compute requirement here, I believe adding more \"modern\" architectures will better show the effect of guided dropout.", "questions_to_authors": "See the first point in reasons to reject.", "extra_scores": {}}, {"review_id": "XEUFMBYTuV", "reviewer": "Reviewer_bMGQ", "summary": "This paper examines the relationship between task-sensitive parameters and the loss landscape during pre-training. It investigates dropouts and introduces a new variant called 'guided dropout.' The experiments conducted demonstrate that the proposed guided dropout consistently yields better performance.\nThe clarity of the writing is fair (but with some technical details missing), however execution quality, originality of the work can be improved (see reasons to reject).", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our manuscript. Your insightful comments help us better our research. Please let us address your concerns below:\n\n1. We agree and will include this block in the revised manuscript to clarify the implementation of FIM as well as the P_upper & P_lower based schedules - see [algorithm](https://shorturl.at/buIh5).\n\n2. While we agree that these papers related to FIM & NN training would provide a fruitful background to our readers, they are solving fundamentally different problems:\n\n**Their Problem:** Using parameter-efficient fine-tuning (PEFT) via sparser models, can we *match* standard fine-tuning using the full dataset, by *computing FIM for each target task*?\n\n**Our Problem:** With the architecture unchanged, can we *beat* standard fine-tuning, especially in scenarios of data paucity, by *computing FIM only once for the source task*?\n\nNevertheless, we did the comparison suggested by the reviewer and *our method indeed beats the latest state-of-the-art in the list (Lodha 2023)  by a significantly large margin*, see [table](https://shorturl.at/j9VVy).\n\nRegarding the other baselines mentioned:\n\n- Liu 2024 use FIM as an analytical tool for their chosen domain of cross-lingual adaptation rather than as a general purpose regularizer, as done here. Further, this paper was published on Arxiv after the COLM submission deadline.\n\n- LP-FT, gradual unfreezing (similar to Liu 2024), and auto-RGN/SNR are PEFT as well. RGN/SNR (similar to Sung 2021 & Lodha 2023) are based on target tasks.\n\nTo address your questions:\n\n1) With a grid-search, we found P_lower = 0% and P_upper = 10% with a linear increment schedule to be the most effective approach. We will add grid-search results to the Appendix if our manuscript is accepted. Thank you for bringing this to our attention.\n\n2) Consider A to be the FIM with scores for each parameter computed w.r.t the full corpus. Table 1 evaluates A with matrices B_n - FIMs with scores for each parameter computed w.r.t *n* % of the corpus. From observation 3, we know that the information from FIMs we need to pull only pertains to the top *x* % of parameters. Thus, The FIM approximates B_n are not computed w.r.t a subset of parameters but are rather evaluated w.r.t them.\n  \n3) Yes. The central requirement for guided dropout to function is the fact that modern LMs are overparameterized. In section 2.3, we show that this thesis holds true for both GPT2 (decoder based) and T5 (encoder-decoder based) models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The author proposed an interesting training method by sampling layers for masking out.\n- The proposed guided dropout seems simple to implement and is effective with the BERT model.\n- The cross-correlation analysis on the effects of training data and sparsity for FIM calculation provides valuable insights, challenging common assumptions in this area.", "reasons_to_reject": "- While containing interesting experiments, the connection between FIM, sparsity, and the proposed algorithm is unclear. It would be better if the author provided an algorithm block. Currently, it is unclear how Fisher information is used as part of the algorithm.\n- The paper ignored prior literature that studied sparsity, fisher information, and transfer learning with no related work section. This leads to potential insufficient baselines, given a few examples (all can serve as baselines in fact…):\n\n  - Sung, Yi-Lin, Varun Nair and Colin Raffel. “Training Neural Networks with Fixed Sparse Masks.”   – using FIM to determine a submask for training transformers, experimented with GLUE tasks and BERT as the backbone.\n  - Liu, Chen Cecilia, Jonas Pfeiffer, Ivan Vulic and Iryna Gurevych. “FUN with Fisher: Improving Generalization of Adapter-Based Cross-lingual Transfer with Scheduled Unfreezing.” – using FIM to determine the training order of adapters for cross-lingual transfers with transformers.\n  - Lodha, Abhilasha, Gayatri Belapurkar, Saloni Chalkapurkar, Yuanming Tao, Reshmi Ghosh, Samyadeep Basu, Dmitrii Petrov and Soundararajan Srinivasan. “On Surgical Fine-tuning for Language Encoders.”  - using FIM to determine which layer to train with GLUE tasks and BERT backbone.\n\nThere are also non-FIM based varieties in terms of (temporarily) omitting layers for training such as LP-FT, gradual unfreezing or Auto-RGN/Auto-SNR in surgical fine-tuning…", "questions_to_authors": "- How are P_upper and P_lower determined, if the masking follows a schedule, what’s the typical schedule? An analysis would be nice. \n\n- Table 1: Prior work already uses a fraction of training data for estimating FIM. This experimental finding only holds when computing FIM with respect to a subset of parameters. This should be clarified further in Observation 2 in writing.  While this analysis is interesting, would you please clarify its connection with the layer-wise masking dropout discussed later in the paper?\n\n- How does the proposed method work on other type of transformers (other than BERT)?\n\n\n-----\nUpdated my score to reflect the opinion change after rebuttal.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces  and "information guided" for regularization through dropout; this is proposed as a regularization technique for fine-tuning language models. The method uses Fisher information as a proxy to the Hessian to infer attributes of the loss landscape and use that to adjust dropout rates layer-wise accordingly. The authors claim improved model generalization, especially in data-scarce scenarios, and demonstrate consistent performance gains across various GLUE tasks.

Strengths:
+ theoretically grounded approach using Fisher information for regularization
+ method applicable to various LM architectures (BERT, GPT2, T5)
+ nice analysis of loss landscape and efficient estimation of Fisher information

Weaknesses (-):
- Limited experimentation on newer, larger language models (e.g., Llama, Mistral)
- Lack of comparison to other regularization techniques beyond standard dropou
- "information theory" link pretty weak. i would suggest to rewrite these claims
- Quality drop in Section 3 compared to earlier sections
- Missing details on hyperparameter sensitivity and implementation
- weak related work section


Reviewer Consensus:
The reviewers generally view the paper positively, acknowledging its potential contribution. Some praise the strength of the first two sections on loss landscape analysis and FIM estimation. However, concerns were raised about the quality of Section 3, limited experimentation with newer models, and the absence of a related work section.

Reviewer bMGQ, initially critical, did recognized some of paper's merits. I am personally leaning towards acceptance, recognizing the paper's contribution and potential value to readers, despite some limitations.

[comments from the PCs] As much as possible, please follow up on the reviewer and AC comments to improve the paper.
Decision: Accept