Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulator
Authors: Yu Xin Li
Felix Dangel
Derek Tam
Colin Raffel
Venue: ICML 2025 spotlightposter (2025)
Keywords: Fisher
optimizers
gradient accumulators
Primary area: deep_learning->theory

Abstract: The diagonal of a model's Fisher Information Matrix (the "Fisher") has frequently been used as a way to measure parameter sensitivity.
Typically, the Fisher is estimated by computing the squared gradient of the model's outputs with respect to its parameters, averaged over a few hundred or thousand examples — a process which incurs nontrivial computational costs.
At the same time, adaptive gradient methods like the ubiquitous Adam optimizer compute a moving average of the squared gradient over the course of training.
This paper therefore explores whether an approximation of the Fisher can be obtained "for free" by recycling the squared gradient accumulator that has already been computed over the course of training.
Through a comprehensive set of experiments covering five applications of the Fisher, we demonstrate that the "Squisher" (**Squ**ared gradient accumulator as an approximation of the F**isher**) consistently performs similarly to the Fisher while outperforming baseline methods.
Additionally, we clarify the exact differences between the Squisher and the Fisher and provide empirical quantification of their respective impact.

Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulator
YuXin Li
Felix Dangel
Derek Tam
Colin Raffel
Abstract
the statistical manifold of likelihoods and scales updates according to the parameter space's curvature, informing us about the "steepness" or "flatness" of the objective func- tion at given point in the parameter space (Karakida et al., 2019). This has made the FIM a valuable tool for training neural networks, improving stability and convergence speed (Karakida & Osawa, 2020).
The diagonal of model's Fisher Information Ma- trix (the "Fisher diagonal") has frequently been used way to measure parameter sensitivity. Typically, the Fisher diagonal is estimated via squared sampled gradients of the model's likeli- hood with respect to its parameters, averaged over a few hundred or thousand examples - a process which incurs nontrivial computational costs. At the same time, adaptive gradient methods like the ubiquitous Adam optimizer compute a moving average of the squared gradient over the course of training. This paper therefore explores whether an approximation of the Fisher diagonal can be obtained "for free" by recycling the squared gradi- ent accumulator that has already been computed over the course of training. Through compre- hensive of experiments covering five applica- tions of the Fisher diagonal, demonstrate that the "Squisher" (Squared gradient accumulator an approximation of the Fisher) consistently per- forms similarly to the Fisher diagonal while out- performing baseline methods. Additionally, we clarify the exact differences between the Squisher and the Fisher diagonal and provide empirical quantification of their respective impact.
Beyond optimization, recent work has explored the use of the FIM's diagonal (the "Fisher diagonal") of parameter sensitivity (Ly et al., 2017), including appli- cations in model sparsification (Theis et al., 2018), sparse training (Sung et al., 2021), task similarity measurement (Achille al., 2019), continual learning (Kirkpatrick et al., 2017), and model merging (Matena & Raffel, 2022; Daheim et al., 2024). These applications leverage the fact that the diagonal elements of the Fisher reflect each parameter's impact on the model output, when considering parameters to be independent, providing principled approach to un- derstanding and modifying neural nets.
measure
Despite its utility, computing the Fisher diagonal introduces nontrivial computations beyond those for training. While these costs are typically comparable to training a few hundred or a few thousand examples, the Fisher diagonal requires computing, squaring, then summing per-example gradients on sampled labels. Doing so efficiently is non- trivial most deep learning frameworks. While there exist specialized solutions (Dangel et al., 2020; Osawa al., 2023), practitioners often resort sequential gradient com- putations (i.e. with "batch size of one"), which sacrifices parallelization opportunities. Additionally. computing the Fisher diagonal requires access to training data that might not be available - for example, trained models are frequently released without their associated training data. We suspect that these factors can hinder the adoption of methods that require computing the Fisher diagonal for example, Fisher Merging (Matena & Raffel, 2022) is not implemented in the ubiquitous me mergekit library (Goddard et al., 2024), it does not support merging methods that require computing gradients or accessing external data.
1. Introduction
The Fisher Information Matrix (FIM, Fisher, 1922) is fundamental concept in statistics, capturing how much in- formation an observable random variable carries about an unknown parameter. In machine learning, the FIM has been widely used in optimization, particularly in Natural Gradient Descent (NGD, Amari, 1998). Unlike standard gradient-based methods, which update parameters using the Fuclidean gradient, NGD leverages the geometry from
University Toronto& Vector Institute. Correspondence to: YuXin Li <lyx.li@ mail. ati.uutooooo.ca.....
the same time, modern neural networks are typically trained using adaptive gradient methods such as Adam (Kingma & Ba, 2014) 4) that make of accumulator vari-
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Fishers for Free?
Square then sum
Standard Fisher
Empirical Fisher > Ing.
Diagonal Fisher R > g
Squisher N No
Joint likelibood
Empiriaal likelihood
Extract diagonal
Muving average
Undo scale
Sum then square
Joint Fisher
Joint empirical Fisher { (Ng)
Joint diagonal Fisher >> g
Squared grad. accumulators v()) ovv (vvl111 1) + (1 a)
1 Eply x,, (Ng)
9 N En
g Σn 9
=== gh (1)
Figure 1. The arrwws represent various approximations of the Fisher Information Matrix. The central idea of the paper is highlighted through the recyeling symbol. Namely, we show that squared gradient accumulators can he used to approximate the Fisher diagonal (details in Section 2). Terms in blue stem from using loss function with mean reduction (u/) and should be omitted when using sum reduction (Cum). The per per-sample gradients 9n and "would-be" gradients gn are defined in Equations (3) and (5).
able to store an exponential moving average of the squared gradient over training steps. The squared gradient accumu- lator at least superficially similar to the Fisher diagonal in the sense that both compute some kind of average of squared gradients. The fact that such accumulator variables available "for free" (i.e. without requiring any additional computation) at the end of training, combined with the afore- mentioned inconveniences of computing the Fisher, raises a natural question for applications that use the Fisher as a notion of parameter importance:
of Fisher-based methods without sacrificing effective- ness.
We investigate the Squisher's limitations and trade-offs. providing deeper understanding of when and why it succeeds (Section 4).
On the whole, our work comprehensively establishes the connections between the squared gradient accumulator and the Fisher diagonal, paving the way for broader adoption of Fisher-based techniques in deep learning.
Can we recycle an optimizer's squared gradient accumulator approximate the Fisher diagonal for free?
2. Background & Motivation
Here we provide background on, and connect, the Fisher diagonal and squared gradient accumulators (Figure 1).
The superficial similarities of the Fisher diagonal and squared gradient accumulator suggest this could be trivially possible, and indeed, past work like the paper introducing Adam (Kingma & Ba, 2014) asserts without evidence that the squared gradient accumulator "is an approximation to the diagonal of the Fisher information matrix". However, closer look reveals important subtleties. Our contribution is to rigorously assess and empirically validate these nuances:
2.1. The Fisher Information Matrix and its Variants
Standard Fisher Information Matrix For simplicity, we first consider an unscaled loss of the form
N Σ (f((n. ==
Camm ( =
(I)
We discuss the non-trivial connections between the Fisher diagonal and squared gradient accumulators from adaptive optimizers, highlighting important mod- ifications that are necessary to successfully use them in place of the Fisher (Section 2).
where f(, 0) is a neural network with parameters that processes a data point H, into a prediction which is then
scored by comparison with its label yn using criterion function l. Also define the per-sample gradient 9n. = V Vfl(f(z,, 0), yn). Later, will add back the more com- mon scaling and consider L(0) Sum (0). The most common loss functions in machine learning, like square or softmax cross cross-entropy loss, permit the loss in Equation (I) to be interpreted as a negative log likelihood for a random variable y representing a target (Martens, 2020), and we can define per-example likelihood
We empirically validate the Squisher (squared gra- dient accumulator in place of the Fisher) across six settings spanning five key applications. We find that the Squisher achieves comparable performance to the Fisher diagonal and consistently outperforms the base- line across all settings (Section 3). This motivates the Squisher as a practical and efficient alternative that eliminates the computational costs and inconvenience
p(y x, 0) x exp (-e(f(x, (-((((x)),)))
(2)
2

---

Fishers for Free?
Note the relation to the per-sample gradient when y Vn
2.2. Gradient Accumulators
and x nz
Training deep neural networks is an ill-conditioned and non- convex optimization problem (Dauphin et al., 2014; Saari- nen et al., 1993). The size of typical datasets used in neural network training also necessitates stochastic optimization where different randomly sampled batches of data are used at each training step (LeCun et al., 2002; Robbins & Monro, 1951). Consequently, most optimizers used in neural net- work training incorporate some mechanism to provide adap- tivity to noise and differences in parameter sensitivity (i.e. differences in per-parameter gradient scale). One common approach is to approximate the average squared gradient for each parameter over the training steps. This squared gradient estimate can then be used e.g. to normalize each parameter update by a smoothed estimate of the gradient magnitude.
Ve log plyn
In. 0)
gn-
(3)
The standard Fisher Information Matrix (FIM) is
N
(0)
Σ
Eg ppplllan..) [
(4)
with "would-be" gradients
In == - Vo log P(0n Ln, -== Vol(f(z 0). (m) (5)
that require drawing labels in. from the model's likelihood.
Empirical Fisher Information The empirical FIM re- places the model's likelihood with the empirical likelihood implied by the data, p(y] æp. 0) - 6(y yn). It consists of per-sample gradients which do not require sampling:
Specifically, we focus optimizers with squared gradient accumulator that takes the form of an exponential moving average (EMA, Roberts, 1959) of squared gradients:'
pemp (0)
Σ Ega-oly- I
Σ 9n (6)
(1() = av((11) + + a)
1 N
(1)
(8)
where is a hyperparameter and (!) is the gradient vector for training example n at training step t. This accumula- tor mechanism, first introduced in 2012 through concur- rent developments of RMSProp (Tieleman & Hinton, 2012) and Adadelta (Zeiler, 2012), has become a fundamental technique for adapting learning rates to stabilize training dynamics. Apart from being used in the ubiquitous Adam optimizer (Kingma & Ba, 2014) and its derivatives (AdamW (Loshchilov, 2017), NAdam (Dozat, 2016), RAdam (Liu et al., 2020), AdamP (Heo et al., 2021), etc.), it is also in- cluded in FTRL (MeMahan & Streeter, 2012), AMSGrad (Reddi et al., 2018), QHM (Ma & Yarats, 2019), LAMB (You et al., 2020), and many others (Schmidt et al., 2021). In this work, we focus primarily on Adam and Adam which represent the most commonly used optimizers in con- temporary deep learning research.
While past work has argued against using the empirical FIM in second-order methods like NGD (Kunstner et al., 2019; Thomas al., 2020), it nevertheless remains popular in applications where the FIM is being used as a measure of parameter importance (c.g. Matena & Raffel, 2022; Theis et al., 2018; Sung et al., 2021; Achille et al., 2019, inter alia) due its lower computational costs. We therefore primarily focus on the empirical FIM in this work.
Diagonals Our discussion so far was concerned with Fisher information matrices, which are quadratic in the num- ber of model parameters and therefore prohibitively large in most deep learning applications. Hence, many applications only compute and use the diagonal of the FIM. Note that for matrix that is a sum of rank one matrices, A A= E; v, which is the case for all the Fisher flavors we previously dis- cussed, the diagonal is diag(A A) = Σ; vf where the square is applied elementwise. This leads to simple expressions for
2.3. The Squisher
the standard diagonal Fisher:
Joint Fisher Note that both the standard and empirical FIMs in Equations (4) and (6) require per-datum (would- be) gradients to be squared then accumulated. This is in contrast to the squared gradient accumulator of Equation (8), which first aggregates the per-datum gradients, then squares them. Lin et al. (2024) proposed new Fisher matrix which has a sum- num-then-square structure that is more compatible with the structure of squared gradient accumulators. Instead of modeling the label as single random variable y, they
N diag Fss (0)) = >
Bin Spn.p,y [
n=1
and the empirical diagonal Fisher:
diag( Fuld (0))
Σ 2h
Here, we have already included scaling factor of I/N in the loss and gradient, as we will later use this scenario in practice.
where the square is elementwise.
3

---

Fishers for Free?
consider likelihood for random vector of labels y V1-.2:--..11 .Nn) jointly from inputs X (x1,-...; XN) via
approximations of the same underlying Fisher. When using batches, can simply replace N by B in all expressions that follow.
ply X. 0)
II
ply, X 0).
Handling averaged loss functions So far, we have as- sumed an unscaled loss (1) in our discussion of Fishers. However, most implementations use an average loss
Their joint Fisher information matrix is
Fjoiin (0) := Beggppyyxxxx.. o) [99
(9)
1 (()) = N
Σ
N
1 Sum (0). (11) (U(xxn 0), Mb)) = N
with "would-be" gradients و Va P( X 0)
onnl Ve log wn 0) = Ex= 9n and in = [m] As for the standard case, their their joint empirical Fisher infor- mation matrix follows by replacing the model's likelihood over the joint labels with the likelihood implied by the data,
Note that we cannot absorb the factor 1/N into the loss func- tion E and define fscaled = 1/NÉ to reduce Equation (11) to the form of Equation (1), because only f, but not Sscaled- corresponds to probability density via Equation (2). There- fore, we will keep the I/N factor separate and use the RHS of Equation (11), which means we can re-use the standard Fisher matrices that are defined in terms of the unscaled loss Sum and scale them by 1/N. Rescaling the joint empirical Fisher matrix from Equation (10), we get
ply X, 0) > II oy,, Vo), which gives
J (0) = E IT. 50y,- vn)) [99 ] 99
(10)
with the empirical gradient g - -- Ve log plyn nn 0) Σm (o Un) o ( The
diagonal joint empirical FIM follows as
1 T 1 NVC(( (NV)C())) N 9g
N = (V
2
diag(
Femp joint (0) = 9
Σ Jn
In the diagonal Fisher case, we have
The key properties of the joint Fisher matrices in Equa- tions (9) and (10) is that they are squares of sums, and not sums of squares. This provides a theoretical motivation that the square of aggregated gradients indeed corresponds to the diagonal of a Fisher information matrix. In fact, Lin et al. (2024) show that the joint and standard Fisher coincide, Fjoine (0) (.)((), ). and hence that both views- standard and joint joint-lead to the same underlying Fisher.
N
2
1 N
NV N
9n
(12)
n=1
In practise, re-scaling is often unnecessary as many applica- tions are invariant under scaling the Fisher (Section 3).
Recyeling the squared gradient accumulator The term in the parenthesis on the RHS of Equation (12) is exactly the quantity whose EMA is computed by the squared gradient accumulator of Equation (8). This leads us to a clear path for using the Squisher, i.e. the squared gradient accumulator, as an approximation of the Fisher. Specifically, compared to the Fisher (i.e. the diagonal of the empirical FIM, as com- monly used as measure of parameter importance), the Squisher squares the average gradient over training batch rather than computing the sum of gradients over an arbi- trary collection of datapoints. The Squisher therefore more closely relates to the diagonal of the joint empirical FIM. In addition, the squaring of the average gradient introduces a factor of N difference scale as in Equation (12).
Note that the joint Fisher's dis- tribution considers a vector of random variables whose size equals the data set. When using only a subset of data, say a mini-batch XB = (....... TD), we can consider the mini- version of the joint Fisher, Fjoin (0, B), which is defined in terms of the marginal distribution p(yB B X B, 0) with YB = (y (y......y', yn). Lin et al. (2024) show that F/BF joim (0, B) is an unbiased estimation of F (0). This property allows us to estimate the Fisher on larger data set (as would be done when using the standard Fisher in applications) from gradients evaluated on a smaller amount of data (as would be done by an optimizer based on mini- batch gradients). Importantly, this estimation is unbiased if we sample labels from the model's likelihood. This ensures that mini-batching itself does not introduce bias in Fisher estimation. However, when labels are replaced with their empirical counterparts (e.g.. ground truth labels), the result- ing estimator becomes biased. Hence we can think of the standard and joint empirical Fishers as two different biased
Handling mini-batching
Separately, the Squisher is computed using an EMA of mini- batch gradients. The EMA coefficient a is typically tuned to improve training convergence and therefore might not reflect the best value for approximating the Fisher. For ex- ample, the default value of a in Adam (where it is referred to as 32) is 0.999, which results in (+ containing nontrivial contributions from a long history of gradients - the time
4

---

Fishers for Free?
constant, f.e. the number of steps reach rescaling of 1 1/c 63%, is about 10,000 steps. Averaging over such a long history both introduces contributions from gradients computed with respect to "old" parameter values and also results in a biased estimate of the corresponding joint em- pirical Fisher computed over all of the data the model has been trained on (Lin et al., 2024).
where F; and 0, are the Fisher and parameters of model i out of M models being merged. Equation (13) corresponds to parameter averaging where parameters with higher cor- responding values in the Fisher are given a higher weight when averaging. Noting that Equation (13) is invariant to rescaling all F; by the same constant, using either the Squisher (Nu())) or the squared gradient accumulator (v(1) as Fisher proxies yields the same merge.
3. Experiments
Setup We directly follow Tam et al. (2024) and merge eight variants of TS-Large-LMM-Adap (Raffel et al., 2020; Lester et al., 2021) that were fine-tuned on text datasets that have been shown to produce performant multitask models (Zhou et al., 2022). Performance is measured as the aver- age accuracy of the merged model on held-out data from the eight datasets used to fine-tune the individual models. As Fisher-free baseline, we use simple parameter averag- ing (Utans, 1996; Wortsman et al., 2022). For further exper- imental details, please see Tam et al. (2024, Section 6.2).
The above discussion reveals clear way to relate the squared gradient accumulator to a Fisher, but this relation involves various nontrivial approximations. We therefore turn to exploring whether these approximations are prob- lematic through an empirical study covering a wide range of six settings where the Fisher is used, which we outline below. We emphasize that our goal is not to show that either the Fisher or the Squisher is "better" across all of these settings, but rather to test whether or not the approximations made in formulating the Squisher result in differences in performance compared to using the Fisher itself. To ensure reliable results, we base all of our experiments on prior im- plementations. In most cases, these implementations used either Adam Adam for optimization and therefore lent themselves straightforwardly to using the Squisher. We additionally always compare to "Fisher-free" baseline, i.e. a method that does not involve computing the Fisher and therefore no additional computational costs (like the Squisher). Our high-level results are shown in Figure 2: complete fine- -ne-grained results are provided in Appendix A.1.
Results Average performance of the Fisher, Squisher, and unweighted parameter averaging are shown in Figure 2. Overall, we found Squisher merging performed consider- ably better than Fisher merging. We don't interpret this to mean that the Squisher is "better", but rather that it is due to the inherent instability in multitask merging (Tam et al., 2024; Yadav et al., 2024; Ilharco et al., 2022). Since both the Fisher and Squisher work significantly better than the Fisher-free baseline, we conclude that both provide a reliable estimate of parameter importance. Additionally, we found that Squisher merging performance could suffer if the fine-tuned models were not fully trained, likely be- cause the squared gradient accumulator had not observed sufficient training steps. We explore this factor further in Appendix A.2.
Naming: As the empirical diagonal standard Fisher from Equation (7) is the standard choice as a notion of parameter importance in many applications (Matena & Raffel, 2022; Theis et al., 2018; Sung et al., 2021; Achille et al., 2019), we will drop all prefixes and simply refer to as the "Fisher" from now on.
3.2. Model Merging by Uncertainty-Baser Gradient Matching (UBGM)
3.1. Fisher Merging
Daheim et al. (2024) uncover that "gradient mismatch" arises when merging models that fall in disparate regions of the loss landscape, potentially leading merged models to fall in high-loss areas, thereby degrading merging per- formance. Daheim et al. (2024) therefore aim to mitigate gradient mismatch by aligning parameter updates with the optimization trajectories of the individual models. Specif- ically, given base model with weights Og and Fisher Fo. and M fine-tuned models with weights O, and Fishers F, for i E (1...., M) the parameters of the merged model e are given by
Model merging aims to cheaply combine individual models into single model that inherits their capabilities (Utans, 1996; Singh & Jaggi, 2020). Fisher merging (Matena & Raffel, 2022) formulates merging as maximizing the joint likelihood of the individual models' posterior distributions over parameters. To do so, Fisher merging uses the Laplace approximation (MacKay, 2003), where the Fisher is the precision matrix of Gaussian approximation to this poste- rior. Fisher Merging then uses a closed-form solution to the likelihood maximizing problem:
MM Σ 1m
M
M
M (Fo + F) -8b)
θ- =
F
Σ
F,O:
(13)
Ô = ou +
Fo +
Σ
Fy
Σ 1=1
iml
5

---

Fishers for Free?
Fisher Merging
UBGM Merging
Fisher Pruning
FISH Mask
Task Embedding
Continual Learning
Average accuracy
60
Average accuracy
94
66
Average Accuracy
85
50
80
55
Accuracy
64
88
MRR (x 1000)
40
Accuracy
75-
62
66
65
75
58
20
60-
45
93
56
76
10
55
Fisher
Squisher
Fisher-free baseline
Figure Performance the Fisher, our proposed Squisher (i.e. using the squared gradient accumulator place of Fisher), and an applicable Fisher-free baseline across all of the settings consider. Across all settings, the Squisher performs comparably to the Fisher and outperforms the Fisher-free basclinc.
As with Fisher merging, the Squisher simply replaces F; with the corresponding model's 's squared gradient accumula- tor at the end of training.
focus on an experiment described in Section 5.3 of Lubana et al. (2020), which involves pruning a VGG-13 (Simonyan, 2014) network trained on CIFAR-100 (Krizhevsky et al., 2009). Lubana et al. (2020) originally used vanilla stochastic gradient descent for training; we therefore modified the implementation to use Adam and tuned hyperparameters to ensure comparable results. Since Fisher pruning retains the top-k parameters, it is insensitive to global rescaling and we can therefore use the squared gradient accumulator as-is for the Squisher. As a Fisher-free baseline, we consider pruning parameters at random.
Setup We exactly replicate the setup of Daheim et al. (2024) and consider the experiments focused on merging fine-tuned variants of RoBERTa (Liu et al., 2019) on four standard text classification tasks. As for Fisher merging, measure performance in terms of average accuracy the individual tasks. Daheim et al. (2024) use AdamW W for fine- tuning, and we re-use their squared gradient accumulator without modification. Further experimental details are pro- vided Daheim et al. (2024, section 4.3). Like the previous merging setting. we use parameter averaging as baseline.
Results We present the results for pruning 75% of the model parameters (i.e. reducing the model to 25% of its original size) in Figure 2 and include additional results for 25% and 50% in Appendix A.1. Across all pruning levels, we find that Squisher pruning slightly underperforms Fisher pruning but performs much better than baseline pruning with a random mask.
Results High-level results are shown in Figure 2. In the UBGM Merging setting, found that the Squisher per- formed similarly to Fisher. Interestingly, parameter averag- ing is relatively strong Fisher-free baseline in this setup, though both Fisher- and Squisher-based UBGM merging outperform it slightly. Therefore, we consider their perfor- mance to comparable in both settings.
3.4. FISH Mask
Sung et al. (2021) consider sparse training and fine-tuning to-iinnnng i.e. updating only a small subset of a model's parameters during training. They propose FISH Mask, which uses the Fisher to choose which parameters to update during training. Specifically, the parameters update are selected based their importance measured by the Fisher diagonal:
3.3. Fisher Pruning
Pruning aims to convert a standard neural network into a "sparse" network where most of the weights are zero (LeCun al., 1989; Hassibi & Stork, 1992). Theis et al. (2018) formulate pruning eliminating the parameters that least impact the loss. Given neural net trained to convergence at 0*, the optimal perturbation 0 = 0" + 5(i., such that (0), with minimal increase (p) of the loss is given by
{10]: [F(O)] 2 sort(diag( (F(()))
With the Squisher, we instead update only those parameters with the largest values in the squared gradient accumulator. As with Fisher pruning, we use random masking as Fisher- free baseline.
(0)=F(0))-4 p(8(1)) 2 8()) = F(00)
l0p?
with e; the canonical i-th basis vector. Assuming a di- agonal Fisher produces the pruning statistics p(ō(1)) [(("))), Theis et al. (2018) then retain the param- eters corresponding to the leading pruning statistics.
Setup We focus on the BERT-Large (Devlin al., 2019) tne-tuning setting described in Section 4.1 of Sung et al. (2021), which uses the FISH mask to reset parameters of fine-tuned models back to the pre-trained values (i.c. their values before fine-tuning). Specifically, 50% of the model's weights are masked, and the remaining 50% are reset to their pre-trained values rather than retaining their
Setup We base our Fisher pruning experiments on the re-implementation of Lubana et al. (2020). Specifically, we
6

---

Fishers for Free?
fine-tuned values. Fine-tuned, masked models are sepa- rately trained and evaluated on nine datasets from the GLUE benchmark (Wang et al., 2018). Full details are available in Sung et al. (2021).
Results As seen in Figure 2, the Squisher-based task em- bedding produced better mean reciprocal rank than the Fisher-based one; both outperformed the dataset size heuris- tic. This trend held across both classiiicatonn/eeressinn and question-answering datasets. This confirms that the squared gradient accumulator's values can be used as a reliable mea- sure of task similarity.
Results As shown in Figure 2, the Fisher and Squisher attain extremely similar performance when used in the FISH Mask setting. Combined with our previous finding for prun- ing, this result suggests that the squared gradient accumula- tor provides reliable way to rank parameter importance.
3.6. Elastic Weight Consolidation
Continual learning faces the challenge of catastrophic for- getting, where artificial neural networks forget previously learned tasks when training on new tasks. Elastic Weight Consolidation (EWC, Kirkpatrick et al., 2017) aims mit- igate catastrophic forgetting by using the Fisher to avoid changes to model parameters that have high influence on the performance of previously seen tasks. Specifically. EWC introduces a regularization term that rescales the squared difference between the current parameter value 0 and the learned values from the previous task(s) i by the Fisher F from the previous task:
3.5. Task Embedding
The Ves embedding represents tasks as points in a vector space in which the distance aims capture task similarity (Achille et al., 2019). Vesk2 embeddings are computed using the Fisher of a model trained on a given task and averaging the values across parameter "groups" (e.g. weight matrices). This vector approximates how sensitive different parameters are to some particular task. Vesk has been shown to be useful for predicting task similarities, such as semantic or taxonomic relations, and for determin- ing which tasks are best suited for knowledge transfer (Vu et al., 2020), i.e. whether training on task before training on task b can improve performance on task b. Specifically, the cosine similarity between task vectors Fa and F/ is
2 LEWC - - O) O). 0).
(14)
Setup We focus on task-incremental learning for this study, since EWC has been shown to have poor perfor- mance on domain- and class-incremental learning (van de Ven & Tolias, 2019). Task-incremental learning happens when the context identity is known during training, and the model must incrementally learn set of distinct tasks (Ruvolo & Eaton, 2013). We consider three standard bench- marks for this setting: split MNIST, a split of the origi- nal MNIST dataset into five contexts with two digits each (Shin et al., 2017); permuted MNIST, additional variant of MNIST transformed by applying fixed, random pixel permutation to each task (Zenke et al., 2017); and split CI- FAR100, similarly split into ten contexts with ten classes each (Krizhevsky, 2009). We use an MI. with 478,410 parameters for split MNIST, a larger MLP with 2.126, 100 parameters for Permuted MNIST, and 5- CNN with 393,088 parameters for split CIFAR-100. For each proto- col, replaced the Fisher with the Squisher in the EWC regularizer.
dsim (on(Fa.. (Fa: Fb) = doss ((Fa + Fb) Fa: (Fa + F,)-
and is used to predict task transferability, i.e. tasks with more similar task vectors are predicted as being more amenable to knowledge transfer. Since cosine distance is invariant to rescaling, to use the Fisher we simply replace F, and F, with the squared gradient accumulators from the corre- sponding models.
Setup We consider the setting from Vu et al. (2020), who use Task2 Vec to predict task transferability for intermediate- task training (Phang al., 2018; Pruksachatkum et al., 2020). Specifically, Vu et al. (2020) experiment with predicting the best intermediate task for fine-tuning BERT (Devlin et al., 2019) on wide range of datasets. We focus on the 22 classification, regression, and question-answering datasets. Since the original fine-tuned models were not released by Vu et al. (2020), we re-fine-tuned BERT on each dataset using the AdamW optimizer. As Fisher-free baseline, we use the simple and effective heuristic of ranking datasets in terms of their size (because larger datasets tend to be more beneficial for intermediate-tast transfer (Vu et al., 2020)). We evaluate each ranking method based on its mean reciprocal rank ( (Voorhees et al., 1999) which measures how a given embedding method tends to rank the best dataset for intermediate-tate transfer.
Unlike in previous settings, rescaling the Squisher does change the learning behavior in this setting as it modifies the regularization strength. For EWC, we found that scaling the Squisher computed on batches of size by N provided best performance, where N is the data set size on which the original Fisher computed. Although we lack formal theoretical justification, this serves as a useful heuristic, i.e., one can start with setting ASquisher = N AFisherr and sweep over parameters around this value in a grid search. In prac- tice, when training EWC from scratch, Apiater is unknown and must be tuned regardless, so using the Squisher does
7

---

Fishers for Free?
not introduce any additional tuning burden. We discuss the importance of adjusting the scaling in Section 4. As Fisher-free baseline, we incrementally train the model without regularization.
Table I. Time (in seconds) it took calculate the Fisher for the settings considered. The Squisher is for free, up to the cost of loading the pre-computed optimizer statisties from disk.
SETTING
TIME (s)
FISHER MERGING
2.93 -1
UBGM
4.36 10a
FISHER PRUNING
2.52
Results When used in EWC for continual learning, we find that the Squisher performs slightly better than the Fisher across all three continual learning setups (Figure 2 shows results for CIFAR 100; results for other protocols are avail- able in Appendix A. Using either the Squisher or the Fisher worked significantly better than using no parameter- wise rescaling (i.e. the identity matrix instead of the Fisher in Equation (14)), suggesting again that the Squisher does capture reliable notion of parameter importance.
FISH MASK
6.98 102
TASK EMBEDDING
5.19-107
EWC
1.25-10
4. Ablation Experiments
When relating the squared gradient accumulator to the Fisher in Section 2.3, we highlighted various approxima- tions made. Although our results from Section confirm that these approximations generally do not harm perfor- mance, we would still like to untangle each of their impacts.
3.7. Summary
Across all of the diverse experimental settings we consider, replacing the Fisher with the Squisher had little impact on performance. While performance degraded marginally in Fisher pruning, it improved in Fisher merging, and was almost identical for the other settings.
To do so, we run a series of ablation experiments in the continual learning setting from Section 3.6. We chose con- tinual learning as it was the only setting where results were dependent on appropriately rescaling the Squisher.
In all cases, both the Fisher and Squisher significantly out- performed relevant Fisher-free baselines. This validates that the approximations made when going from the Fisher to the Squisher do not significantly impact performance at least when using diagonal approximations, is common practise. More broadly, we can confirm that the Squisher meaning- fully reflects the importance of a model's 's parameters to a similar extent to the Fisher.
Squared gradient accumulator without rescaling To measure the importance of rescaling the squared gradient accumulator (as described in Scction 2.3), we measure per- formance when using the squared gradient accumulator di- rectly without tuning the À term in Equation (14). As can be seen in Table 2, not tuning the value and using the default from Fisher provides suboptimal results.
We want to emphasize that our goal is not to claim that Squisher is inherently superior or inferior to Fisher. The performance differences are problem-dependent and influ- enced by training trajectories, but overall remain small rela- tive to baseline performance. These differences are largely attributable to noise within the setting, rather than any fun- damental distinction between the two approaches. This is further explained Appendix A.2.
Changing the exponential moving average From the re- sults of Fisher merging, we observed that Squisher's perfor- mance could degrade if the fine-tuned models were not fully trained, likely because the squared gradient accumulator had not been exposed to enough training steps. We address this effect by reducing 33 in Adam, which decreases the emphasis on past squared gradients and places more weight on recent gradients. The optimal setting aligns with the default AdamW value of 0.999, while performance degra- dation was observed at 0.95. Importantly, our intent is not to tune 33 for Squisher; rather, we examine whether any standard optimizer hyperparameters can serve as drop-in approximations for the Fisher. We recommend using the 32 that best supports optimization. Notably, even with unusu- ally low values of [32. the Squisher continued to outperform the baseline, highlighting that Squisher remains effective as long as conventional training settings are used.
3.8. Runtime Analysis
The central motivation of this paper is to obtain Fishers for free. The time required to compute the Fisher varies with the model, dataset, and experimental setting; corresponding runtimes are reported in Table 1. In contrast, computing the Squisher incurs virtually no cost, it simply involves load- ing pre-computed values and, in the case of EWC, applying a scaling factor. Across all settings, Squisher's ''s runtime re- mains below 0.1 seconds, and under 1 second for EWC. A detailed breakdown of computation costs across datasets is provided in Appendix A.3.
Note that some of the ablation conclusions do not hold for split MNIST, likely due to the simplicity of the setting.
8

---

Fishers for Free?
cient computation of per-sample gradients and second-order quantities like the empirical Fisher diagonal by extending the backward pass of neural networks. While more efficient than naïve for-loop approaches, it still introduces additional overhead, requires code changes, and lacks full architectural support. For instance, it does not support layer normaliza- tion. As result, the for-loop approach remains common in Fisher-based methods. In contrast, the Squisher intro- duces no extra cost code modifications, it reuses the squared gradient accumulator already present in optimiz- ers like Adam (Kingma & Ba, 2014), making it universally supported without these limitations.
Table Results from the ablation experiment settings, averaged
over 5 trials.
SQUISHER w/o NORM.
FISHER
SQUISHER
8/ 0.95
JOINT
SPLITM
99.59
98.59
98.85
98.66
99.46
РERMM
94.55
94.47
92.05
93.37
95.59
SPLITC
75.96
75.30
61.53
72.80
75.38
Diagonal joint empirical FIM The squared gradient accu- mulator uses moving average of gradients over the course of training, whereas FIMs compute an explicit sum of gradi- ents at the end of training. To measure the importance of this difference, we explicitly square the sum of per-example gra- dients (i.e. we turn off the moving average in the Squisher), shown in Table as JOINT. We see that its performance values are close to FISHER, suggesting the joint Fisher is useful approximation.
6. Conclusion
In this paper, we present novel technique, the Squisher, that serves a free approximation for the diagonal Fisher in various Fisher-based methods. The Squisher uses gradient accumulators that are readily available during training to provide an estimate of the Fisher, thereby alleviating the costs associated with calculating the Fisher. We motivate this by formulating the empirical Fisher as a joint Fisher, and hence show its parallels with gradient accumulators, We implement the Squisher as direct replacement for the Fisher in settings where only the relative importance of pa- rameters matters, and apply a rescaling term in contexts where the magnitude of values is consequential. We show that the Squisher had comparable performance to the Fisher across five settings, and had significant improvements from the baselines. We further discuss the impact of the approxi- mations we made by considering several variants.
5. Related Work
To the best of our knowledge, there has been no prior work investigating whether the squared gradient accumu- lator can reliably be used as a measure of parameter im- portance in place of the Fisher. However, there has been ongoing research on more directly connecting adaptive op- timizers to second-order optimization. One such example is IVON (Shen et al., 2024), an extension of Adam that approximates the Hessian via weight perturbations. Notably, similar to our goal, IVON demonstrates that its accumulated statisties can serve as a substitute for the FIM in model merging tasks. However, unlike this work, using IVON's statistics involve nontrivial deviation from what is cur- rently standard practice for training neural networks.
In future work, we can explore advanced optimizers which use gradient accumulators that approximate the non- diagonal Fisher, and whether that leads to better perfor- mance. Another direction is to maintain a moving average of the Fisher over time, analogous to the exponential moving average used adaptive optimizers, rather than computing it at single post-training point. Although impractical dur- ing real training, it may be interesting to explore whether it offers better approximations for the true Fisher. By re- ducing the computational burden of Fisher-based methods, this work advances the democratization of deep learning research. Furthermore, it is currently not common practice to share optimizer weights. However, we hope this will en- courage the community to share full training configurations, hyperparameters, and optimizer state dicts, fostering more reproducible and inclusive machine learning research.
Similarly, the AdaFisher optimizer (Gomes et al., 2024) replaces Adam's second momentt estimation with a novel block-diagonal approximation of the FIM. This approach yields improved performance, emphasizing the richness of the FIM compared to the squared gradient moving average. As with IVON and the Squisher, recycling the statistics produced by AdaFisher or FAdam could yield a similarly effective replacement for the Fisher.
FAdam is another optimizer that leverages the empirical FIM (Hwang, 2024). The authors reinterpret the second- order moment in Adam through the lens of the diagonal Fisher, and propose modified optimizer to address its limi- tations. While their work offers valuable insights into the relationship between natural gradient methods and adaptive optimization, it lacks a full theoretical proof and compre- hensive empirical analysis.
A Acknowledgements
Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Insti-
BackPACK (Dangel et al., 2020) is library that enables effi-
9

---

Fishers for Free?
tute. We would like thank Reviewer WT93 for helping us formalize the equivalence between the standard and joint Fisher. This clarification strengthens our motivation to in- terpret the standard empirical Fisher and Squisher as two distinct empirical approximations of the same underlying Fisher information.
J. Arcee's mergekit: A toolkit for merging large language models. arXiv preprint arXiv:2403. 1...003...335. 13257, 2024.
Gomes, D. M., Zhang, Y., Belilovsky, E., Wolf, G., and
Hosseini, M. S. Adafisher: Adaptive second order
optimization via fisher information. arXiv preprint arXiv:2405.1633 2024.
Impact Statement
Hassibi, B. and Stork, D. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in Neural Information Processing Systems (NIPS), 1992.
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Hco, B., Chun, S.. Oh, J., Han, D., Yun, S., Kim, G., Uh, Y., and Ha, J.-W. Adamp: Slowing down the slowdown for momentum optimizers on inverinvarriatt weights, 2021.
References
Achille, A., Lam, M., Tewari, R., Ravichandran, A., Maji, S., Fowlkes, C. C., Soatto, S., and Perona, P. Tesk2vec: Task embedding for meta-learning. In Proceedings of the IEEE/CVF international conference on computer vision,
Hwang. D. Fadam: Adam a natural gradient optimizer us- ing diagonal empirical fisher information. arXiv preprint arXiv:2405. 12807, 2024.
pp. 6430 (009-443,,2200 2019.
Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing mod- els with task arithmetic. arXiv preprint arXiv:2212.04088 2022.
Amari, S.-L. Natural gradient works efficiently in learning. Neural Computation, 10(2):251-22 -276,, 1998.
Daheim, N., Möllenhoff, T., Ponti, E., Gurevych, I., and Khan, M. E. Model merging by mncertainty-based gradi- ent matching. In The Twelfth International Conference on Learning Representations. 2024.
Karakida, R. and Osawa, K. Understanding approximate fisher information for fast convergence of natural gradient descent in wide neural networks. Advances neural information processing systems, 33: 1089119110, 2020.
Dangel, F., Kunstner, F., and Hennig, P. BackPACK: Pack- ing more into backprop. In International Conference Learning Representations (ICLR), 2020.
Karakida, R., Akaho, S., and Amari, S.-i. Universal statistics of fisher information in deep neural networks: Mean field approach. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1032-1041. PMLR, 2019.
Dauphin, Y. N., Pascanu, R., Gulcehre, C., Cho, K., Gan- guli, S., and Bengio, Y. Identifying and attacking the saddle point problem in dimenssiinii non-convex op- timization. Advances in Neural Information Processing Systems, 27, 2014.
Kingma, D. and Ba, Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 114((3):3521-3526 2017.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics, 2019.
Ctrabsa-Barwwngg
Dozat, T. Incorporating nesterov momentum into adam, 2016. ICLR Workshop.
Krizhevsky, A. Learning multiple layers of features from tiny images. 2009.
Fisher, R. A. On the mathematical foundations of theoret- ical statistics. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of Math- ematical or Physical Character, 222(594-604):309-333 1922.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Kunstner, F., Hennig, P., and Balles, L. Limitations of the empirical fisher approximation for natural gradient descent. Advances in Neural Information Processing Systems, 32, 2019.
Goddard, C., Siriwardhana, S., Ehghaghi, M., Meyers, L., Karpukhin, V., Benedict, B., McQuade, M., and Solawetz,
10

---

Fishers for Free?
LeCun, Y., Denker, J., and Solla, S. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Osawa, K., Ishikawa, S., Yokota, R., Li, S., and Hoefler, T. Asdl: A unified interface for gradient preconditioning in pytorch, 2023.
LeCun, Y., Bottou, L., Orr, G. B., and Müller, K.-R. Effi- cient backprop. In Neural networks: Tricks of the trade, pp. 9-50. Springer, 2002.
Phang, J., Févry, T., and Bowman, S. R. Sentence encoders on stilts: Supplementary training on intermediate labeled- data tasks. arXiv preprint
2018.
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021.
Pruksachatkun, Y., Phang, J., Liu, H., Htut, P. M., Zhang, X., Pang. R. Y., Vania, C., Kann, K., and Bowman, S. Intermediate-task transfer learning with pretrained mod- els for natural language understanding: When and why does it work? arXiv preprint 2tti::20000002222 2020.
Lin, W., Dangel, F., Eschenhagen, R., Bae, J., Turner, E., and Makhzani, A. Can remove the square-root in adap- tive gradient methods? second-order perspective. In International Conference on Machine Learning (ICML), 2024.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, J. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(1), 2020.
Liu, L.,, Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Han, J. On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations, 2020.
Reddi, S. J., Kale, S., and Kumar, S. On the convergence of adam and beyond. In International Conference Learning Representations, 2018.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 11Xiv:1977.11692 2019.
Robbins, H. and Monro, S. A stochastic approximation method. The annals of mathematical statistics, 1951.
Roberts, S. Control chart tests based on geometric moving averages. Technometrics, 42(1), 1959.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100 2017.
Ruvolo, P. and Eaton, E. ELLA: An efficient lifelong learn- ing algorithm. In Dasgupta, S. and McAllester, D. (eds.), Proceedings of the 30th International Conference on Ma- chine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 507-515, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.
Lubana, E. S., Trivedi, P., Hougen, C., Dick, R. P., and Hero, A. O. Orthoreg: Robust network pruning using orthonor- mality regularization. arXiv preprint arXiv:2009.050/1 2020.
Ly. A., Marsman, M., Verhagen, J., Grasman, R. P., and Wagenmakers, E.-J. A tutorial on fisher information. Journal of Mathematical Psychology, 80:40-55, 2017.
Saarinen, S., Bramley, R., and Cybenko, G. Ill-conditioning in neural network training problems. SIAM Journal on Scientific Computing, 14(3), 1993.
Ma, J. and Yarats, D. Quasi-hyperbollic momentum and adam for deep learning. In International Conference Learning Representations, 2019.
Schmidt, R. M., Schneider, F., and Hennig, P. Descend- ing through a crowded balley-benchmarking deep learn- ing optimizers. In International Conference on Machine Learning. 2021.
MacKay, D. J. Information theory, inference and learning algorithms. Cambridge university press, 2003.
Martens, J. New insights and perspectives on the natural gradient method, 2020.
Shen, Y., Daheim, N., Cong, B., Nickl, P., Marconi, G. M., Bazan, C., Yokota, R., Gurevych, I. Cremers, D., Khan, M. E., et al. Variational learning effective for large deep networks. arXiv preprint at arXiv:440..16611 2024.
Matena, M. S. and Raffel, C. A. Merging models with fisher- weighted averaging. In Advances in Neural Information Processing Systems, 2022.
Shin, H., Lee, J. K., Kim, J., and Kim, J. Continual learning with deep generative replay, 2017.
McMahan, H. B. and Streeter, M. Adaptive bound optimiza- tion for online learning and stochastic optimization. In International Conference on Machine Learning (ICML), 2012.
Simonyan, K. Very deep convolutional networks for large- scale image recognition. arXiv preprint 1rXiv:1409.1533 2014.

---

Fishers for Free?
Singh, S. P. and Jaggi, M. Model fusion via optimal trans- port. Advances in Neural Information Processing Systems, 33, 2020.
models. Advances in Neural Information Processing Systems, 36, 2024.
You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., and Hsieh, C.-J. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference Learning Representations, 2020.
Sung. Y.-L., Nair, V., and Raffel, C. Training neural net- works with fixed sparse masks. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Tam, D., Bansal, M., and Raffel, C. Merging by matching models task parameter subspaces. Transactions on Machine Learning Research, 2024.
Zeiler, M. Adadelta: an adaptive learning rate method.
arXiv preprint arXiv: 1212.5701, 2012.
Theis, L., Korshunova, I., Tejani, A., and Huszár, F. Faster gaze prediction with dense networks and fisher pruning. arXiv preprint arXi:: 2018.
Zenke, F., Poole, B., and Ganguli, S. Continual learning through synaptic intelligence, 2017.
Zhou, J., Lin, Z., Zheng, Y., Li, J., and Yang, Z. Not all tasks are born equal: Understanding zero-shot generaliza- tion. The Eleventh International Conference on Learning Represenaatioss 2022.
Thomas, V., Pedregosa, F., van Merrienboer, B., Manzagol, P.-A., Bengio, Y., and Roux, N. L. On the interplay between noise and curvature and its effect optimiza- tion and generalization. In International Conference Artificial Intelligence and Statistics (AISTATS), 2020.
Tieleman, T. and Hinton, G. Lecture 6.5 rmrssppop: Divide the gradient by a running average of its recent magnitude, 2012. Coursera: Neural Networks for Machine Learning.
Utans, J. Weight averaging for neural networks and local re- sampling schemes. In AAAI-96 Workshop on Integrating Multiple Learned Models, 1996.
van de Ven, G. M. and Tolias, A. S. Three scenarios for continual learning, 2019.
Voorhees, E. M., Tice, D. M., et al. The trec-8 question answering track evaluation. In TREC, volume 1999, pp. 82, 1999.
Vu, T., Wang, T., Munkhdalai, T., Sordoni, A., Trischler, A., Mattarella-Micke A., Maji, S., and lyyer, M. Exploring and predicting transferability across nlp tasks. arXiv preprint ar Stv:20055000700 2020.
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Pro- ceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2018.
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Lopesjo-Lopes, R. Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference Machine Learning, 2022.
Yadav, P., Tam, D., Choshen, L., Raffel, C., and Bansal,
M. mies-ne Resolving interference when merging
12

---

Fishers for Free?
A. Appendix
A.1. Full Results
We show the full results all six settings as described Section
Fisher Merging
Table FISHER and SQUISHER represent the corresponding performance of the original and proposed method. LINEAR the unweighted averaging baseline.
DATASET
FISHER
SQUISHER
LINEAR
COSMOS
61.0
50.7
39.4
PAWS
50.7
50.4
48.0
QASC
73.7
85.9
77.1
57.2
60.7
43.2
QUARTZ
54.6
59.3
ROPES
12.6
36.1
45.1
SOCIALIQQ
68.7
61.0
48.0
WIKI_A
50.1
58.1
27.6
AVERAGE
53.6
58.0
48.5
Model Merging by Uncertainty-Bae -Based Gradient Matching
Table 4. FISHER refers to the accuracies using the original method, while SQUISHER represents the accuracies achieved with proposed estimation technique, LINEAR is unweighted averaging baseline. AVERAGE is direct average across datasets, while TRUE AVERAGE takes into account the number of data samples per dataset.
DATASET
FISHER
SQUISHER
LINEAR
IMDB
93.67
94.48
94.32
YELP
97.05
97.23
96.73
RT
89.68
89.02
89,02
SST2
93.58
92.78
93.35
AMAZON
95.97
96.47
95.97
AVERAGE
93.99
94.00
93.88
TRUE AVERAGE
95.91
96.40
95.92
Fisher Pruning
Table Pruning is applied to achieve three different target percentages, with each prune percentage methho pair averaged over five seeds. FISHER, SQUISHER, and RANDOM represent the accuracy of their respective pruning methods.
METHOd
FISHeR
SQUISHER
RANDOM
25%
67.0
66.7
65.9
50%
66.9
65.9
64.0
75%
64.8
63.9
58.8
13

---

Fishers for Free?
FISH Mask
Table GLUE test results with BERT-Large using FISH mask with a sparsity 0.50%. FISHER, SQUISHER, and RANDOM represent the accuracy of their respective masking methods.
TASK
FISHER
SQUISHER
RANDOM
COLA
56.5
58.8
33.5
MNLI
85.2
85.5
80.0
MNLImm
85.7
85.8
81.3
MRPC
85.5
85.9
76.8
QNLI
93.4
93.1
84.4
QQP
87.3
87.1
83.1
RTE
67.1
65.0
60.6
SST-2
92.5
92.3
88.9
STS-B
87.9
70.1
AVG
82.4
82.3
73.2
Task Embedding Since we fine-tuned our own models, our rankings did not exactly match those of Vu et al. (2020), but the scores attained were qualitatively similar.
We note that because many intermediate datasets produce similar results, the rankings be noisy.
Table We report the average rank and NDCG as defined in the original paper, where average rank is the rank assigned to the best source, and NDCG measures the overall ranking quality. We also report Mean Reciprocal Rank (average of the reciprocals of ranks associated with the source) to align with the convention that higher better, as used in other settings. In-class refers to ranking within the specific task group, and all-class refers ranking across all datascts, with number of datasets each group included in brackets.
METRIC
FISHER
SQUISHER
DATASIZE
classfiiatton/regrrssson
4.6
3.8
4.8
IN-CLASS (10)
NDCG
82.0
84.7
81.7
MRR
0.421
0.468
0.136
9
6.7
11.6
ALL-CLASS (21)
NDCG
82.5
85.9
75.0
MRR
0.312
0.261
0.105
question answering
5.3
4.5
7.4
IN-CLASS (10)
NDCG
86.3
84.0
84.4
MRR
0,460
0,563
0,319
8
5.8
10.8
ALL-CLASS (21)
NDCG
88.4
86.1
85.1
MRR
0.350
0.301
0.154
14

---

Fishers for Free?
Elastic Weight Consolidation
Table Results from all protocols and scenarios. Note that EWC performs poorly in the class setting (van de Ven & Tolias, 2019).
FISHER, SQUISHER, and RANDOM represent the accuracy of their respective continual learning methods.
SCENARIO
FISHER
SQUISHER
BASELINE
TASK
99.59
98.59
95.93
SPLIT MNIST
DOMAIN
65.75
69.87
$6.63
CLASS
19.21
19.29
19.87
TASK
94.55
94.47
78.17
PERMUTED MNIST
DOMAIN
94.61
92.32
80.18
CLASS
48.25
29.64
46.73
75.96
75.30
61.82
SPLIT CIFARI00
DOMAIN
18.79
20. 16
16.01
CLASS
5.82
5.98
5.88
A.2. Evaluating the difference in performance between the Fisher and the Squisher
The results in Figure 2 show that it is not consistently the case that either the Fisher or the Squisher performs better. This inconsistency arises due to several layers of approximation. Furthermore, the Squisher relies on a history of gradients, while the Fisher does not. Given that Adam accumulates gradient statistics over time (Kingma & Ba, 2014), is reasonable to expect that the performance of models using the Squisher depends on how long the model has been trained before extracting optimizer weights. We illustrate this effect two settings with the largest performance gap between the two methods.
Model Merging For Fisher merging, Squisher outperformed Fisher. However, the results reported in this paper based on the "final model" setting, where all datasets trained for the same number of epochs without early stopping. An alternative approach, suggested by Tam et al. (Tam et al., 2024), is to merge using the "best model", i.e. merging checkpoints corresponding to the highest individual accuracy.
Table Effect of training time for Fisher merging.
FISHER FINAL
SQUISHER
BEST
FINAL
BEST
COSMOS
61.0
62.4
50.7
31.5
PAWS
50.7
52.0
50.4
55.8
QASC
73.7
81.1
85.9
71.7
QUAIL
57.2
55.7
60.7
41.7
QUARTZ
54.6
59.6
61.1
56.7
ROPES
12.6
39.8
36.1
41.9
SOCIAL IQA
68.7
60.2
61.0
48.6
WIKI QA
50.1
61.0
58.1
92.5
AVERAGE
53.6
59.0
58.0
55.1
The results show that Squisher merging performance can suffer the fine-tuned models are not fully trained, likely because the squared gradient accumulator had not been computed over sufficiently many training steps. As shown, under the best model setting, the Fisher tends to perform better, but the Squisher performs better when final model is used. This indicates that the merging outcome is sensitive the specific checkpoints selected, making the comparison between the Squisher and the Fisher unstable. This variability is also reflected in the performance across individual datasets, which differs significantly without consistent pattern favoring one method over the other.
One possible explanation for the Squisher performing better in the final model setting is that model parameters fluctuate more during early training, making the optimizer- derived Squisher less reliable at that stage. As training progresses and the optimizer statistics stabilize, the Squisher becomes more effective, yielding better merged accuracy.
15

---

Fishers for Free?
Model Pruning For this setting, the number of training epochs before pruning varied, in other words, the number of epochs which the Adam weights are accumulated. Previously, the model was trained for 15 epochs prior to pruning. Here, present results for training with only 10 epochs. The results are averaged over 5 runs to illustrate the scale of variance relative to the model's s accuracy.
Table 10. Effect of training time for Fisher pruning.
FISHER
SQUISHER
EPOCHS TRAINED
15
10
15
10
ACCURACY
64.8
64.0
63.9
64.3
STD. DEV.
0.1
0.7
0.3
0.6
Similar to Fisher merging, changing the time trained changes which method performs better. Therefore, this supports the claim that we cannot definitively conclude which method is superior, the results exhibit considerable variability. But overall, the variation remains small relative to baseline performance and the experimental results thoroughly validate the Squisher as a zero-cost drop-in replacement for the Fisher.
A.3. Runtime Results
All results are reported in seconds. The Fisher merging and FISH Mask was trained using NVIDIA TITAN Xp, while the other settings were trained on A6000.
Fisher Merging
Table Time took to calculate the Fisher for each fine-tuned model to merged. The individual times were added together to find total runtime cost for Fisher merging.
DATASET
RUNTIME
COSMOS QA
4597.096425
SOCIAL IQA
6034.254711
PAWS
8914.329607
QUAIL
2105.06268
WIKLQA
1635.845571
QUARTZ
522.851436
QASC
1498.499418
1957.923106
TOTAL
16

---

Fishers for Free?
Model Merging by Uncertainty-Bas Gradient Matching
Table 12. Time it took to calculate the Fisher for each fine-tuned model to be merged. The individual times added together to find total runtime cost for UBGM merging.
DATASET
RUNTIME
IMDB
88.251
AMAZON
298.730116
YELP
30.893116 16
ROTTEN TOMATOES
9.417264
SST2
8.584630
TOTAL
435.876928
Fisher Pruning
Table 13. Time it took to calculate the Fisher, averaged across seeds. A pruning percentage of 75 was used for this runtime comparison, but the timing remains the across other pruning levels, since the Fisher is computed the unpruned model, which is identical regardless of the pruning percentage.
SEED
RUNTIME
2.53115
2.524477
7
2,439246
42
2.578159
1234
2.529666
AVERAGE
2.5205396
FISH Mask
Table 14. Runtime for computing the Fisher for each task. Since the model is used across tasks, the runtime remains similar due consistent model size and computational cost.
TASK
RUNTIME
COLA
87.106167
MNLI
86.589986
MRPC
88.24297
QNLI
87.799851
QQP
86.861222
RTE
87.250311
SST2
87.4 87.427422
STSB
86.98881
TOTAL
698.266745
17

---

Fishers for Free?
Task Embedding
Table Time it took calculate the Fisher for each task. The individual times were added together to find total runtime this setting.
DATASET
RUNTIME
BooLQ COMQA
928.932541
2635.093401
CQ
1150.859747
DROP
3724.8071 16
DUORC-P
2306.916999
DUORC-S
2546.056588
HOTPOTQA
8348.362174
NEWSQA
3798.1676
SQUAD-1
920.5654031
SQUAD-2
1370.587094
WIKIHOP
5012.5946521
COLA
91.747407
STS-B
61.246726
WNLI
7.841009
MNLI
4030.210978
MRPC
39.255162
QNLI
1073.20734 207344
QQP
3818.9 912368
RTE
26.929389
SST-2
697.788316
SCITAIL
245.605
SNLI
5666.327469
TOTAL 51902.01455
Continual Learning
The time it took to calculate the Fisher after each context is tracked for all three protocols. TOTAL indicates the cumulative time required to compute the Fishers across all contexts.
Table 16. Runtime for Split MNIST (in seconds)
CONTEXT
RUNTIME
83.543847
2
78.777524
83.420858
85.942129
TOTAL
331.684358
18

---

Fishers for Free?
Table 17. Runtime for Permuted MNIST (in seconds)
CONTEXT
RUNTIME
1
1695. 10917
2
49955
3
1997.16719
4
1701.106478
5
1689.53. 109
6
1687.297443
7
1662.508294
1686.655048
9
1696.493132
TOTAL
15215.518690
Table 18. Runtime for Split CIFAR-100 (in scconds)
CONTEXT
RUNTIME
1
(38.255250
2
137.803834
3
137.13763.
136.980094
5
138.475648
6
139.223644
7
138.512695
8
140.716222
9
139.287938
TOTAL
1246.392960
19


Reviews:
[{"review_id": "QsGvaRbTBE", "reviewer": "Reviewer_deFR", "summary": "This paper explores the idea of approximating the Fisher Information Matrix by using the squared gradient accumulator that is already computed in optimizers like Adam. The authors have done an excellent job of testing this approximation in six different applications where empirical Fisher is used and show that in most of the settings, this cheap approximation is good enough.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your effort and valuable feedback.\nWe fixed the Adam citation in the manuscript, thanks for pointing that out.\nPlease find our responses below and let us know if you have any follow-up questions.\n\n---\n\n> [...] while you mention that squisher is better than or on par with Fisher, you should also highlight that it is actually much cheaper to compute squisher.\n\nThanks for this actionable suggestion!\nWe will make sure to highlight this more clearly, and also include a small table in the main text that contrasts the computation times of the empirical Fisher with that of the Squisher (essentially zero).\n\n---\n\n> Q1: page 4: In the last paragraph of section 2, you talk about the effect of having $\\beta_2$ = 0.999 and mention that it results in a biased estimate. But then the discussion is not complete. Is it good or bad that the estimate is biased?\n\nThanks for bringing this to our attention, let us clarify (we will do so in the text, too):\n\nThere are two different types of bias in the Squisher.\nThe first stems from approximating the Fisher with the empirical distribution.\nThe paragraph tries to highlight the second one, which stems from the optimization algorithm's exponential moving average heuristic (using $\\beta_2$).\nWhile the empirical Fisher is computed at one point in parameter space, the Squisher contains contributions from an entire trajectory.\n\nWe can only hypothesize if these averages are good or bad (e.g. for optimization they definitely seem to help as the default value for $\\beta_2$ in Adam is $0.999$).\nInstead, we accept the presence of exponential moving averages as given, since our goal is to investigate whether the optimizer's statistics can be used as drop-ins for the Fisher.\n\nIn our ablation experiments, we found that the Squisher still works reasonably (i.e. better than the Fisher-free base line) even if we vary $\\beta_2$, but the default value seems to work best in most cases.\n\n---\n\n> Q2: For all the applications, the results section just mentions whether squisher underperforms or overperforms Fisher. But you do not reason why in every application. It might be useful to add this discussion.\n\nPlease see our response to the same question in Reviewer D2a9 (first quoted note).", "claims_and_evidence": "Yes. The authors were very careful in not claiming that the proposed approximation is consistently better than using the computationally more expensive Fisher. There is no overclaiming.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "There are no proofs in the paper.", "experimental_designs_or_analyses": "Yes. No issues with the experimental design.", "relation_to_broader_scientific_literature": "The authors have done a good job of positioning their work. They took a really simple idea of coming up with a cheap approximation for Fisher but then did an extensive empirical analysis to understand if the approximation was reasonable.", "essential_references_not_discussed": "No. I appreciate the author's effort in giving credit to Adadelta, while most papers cite RMSProp for the accumulator mechanism.", "other_strengths_and_weaknesses": "Strength:\n\n* The paper considers a computationally cheap approximation for diagonal Fisher and demonstrates that it works on par or better than the computationally costlier methods.", "other_comments_or_suggestions": "* page-2: Adam citation is not correct.\n\n* In all the results sections, while you mention that squisher is better than or on par with Fisher, you should also highlight that it is actually much cheaper to compute squisher. This is implicit but you can highlight this better in your experimental sections.", "questions_for_authors": "* Q1: page 4: In the last paragraph of section 2, you talk about the effect of having \\beta_2 = 0.999 and mention that it results in a biased estimate. But then the discussion is not complete. Is it good or bad that the estimate is biased?\n\n* Q2: For all the applications, the results section just mentions whether squisher underperforms or overperforms Fisher. But you do not reason why in every application. It might be useful to add this discussion.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XGRCvnKOlq", "reviewer": "Reviewer_WT93", "summary": "This paper proposes reusing the squared gradient estimator of adaptive gradient methods as an approximation to the Fisher information matrix, called the 'Squisher'. Through extensive evaluation in model merging, model pruning, sparse fine-tuning, task embedding, and continual learning, the authors demonstrate that the Squisher performs on par with the empirical Fisher without needing additional gradient computations.\n\n## Update after the rebuttal\n\nThe authors addressed my remaining concerns. I retain my recommendation of acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your thorough review and the various suggestions that we will make sure to incorporate into the manuscript.\n\nWe are specifically grateful to the reviewer for pointing out the equivalence between the standard and joint Fisher, which strengthens our motivation to view the standard empirical Fisher and Squisher as two different empirical approximations of the same underlying Fisher.\n\nPlease find our response to your remaining concerns below.\nLet us know if you have follow-up questions or suggestions.\n\n---\n\n> The exact model used in the EWC experiment is unspecified.\n\nWe used a 5-layer CNN with 393,088 parameters for the incremental learning task on CIFAR-100.\nWe will add details of its architecture to our draft.\nPlease let us know if you would like further clarification.\n\n---\n\n> It would also be insightful to see the performance of the true (MC-approximated) Fisher on the downstream task for comparison, as the empirical approximation loses theoretically desirable properties of the Fisher.\n\nFor the EWC setting, we actually computed the true diagonal Fisher for the results presented in the paper. As shown, Squisher provided similar performance to the true diagonal Fisher. We used the true Fisher in this case because it was part of the original implementation.\n\nWe agree that these additional experiments would be interesting for other settings. We will explore their feasibility, though some may require a large number of MC samples, making them impractically expensive.\n\nNote that this does not affect the main contributions of our paper, which is proposing a cheap replacement for the empirical Fisher diagonal, which is widely used in the tasks we study, specifically at extreme scales.\n\n---\n\n> - I prefer retaining the term \"Fisher\" for the Fisher information matrix and keeping the \"empirical\" adjective for cases when the ground-truth labels are used. These terms are often used interchangeably in the literature, and simplifying the naming in this paper could potentially add to the confusion.\n\nWe agree that, given the frequent confusion of Fisher and empirical Fisher in the community, we should make the terminology less ambiguous.\nWe will do so by introducing acronyms for the attributes **diagonal** and **empirical** to specify the Fishers used in the text more clearly.\n\n---\n\n> Q1. In the paragraph of L292, the authors discuss the fine-tuning setting described in Section 4.1 of Sung et al. (2021). According to this description, it uses the FISH mask to reset parameters. I have not found this in Section 4.1 of Sung et al. (2021); could the authors clarify what they mean by resetting parameters?\n\nFor the FISH mask setting, the diagonal Fisher is used to determine which parameters to mask by identifying those with the lowest estimated importance. The remaining parameters are retained and fine-tuned on the new task. By \"resetting parameters,\" we refer to resetting the masked weights to their original pre-trained values rather than using from their fine-tuned values.\n\nWe will clarify this in the final paper. Please let us know if you need further details.\n\n---\n\n> Q2. \"Another experiment is to maintain a moving average of Fishers\": Do the authors refer to the full Fisher information matrix as the Fisher here instead of the diagonal? Otherwise, Adam already calculates this quantity.\n\nApologies for the confusion.\n\nTypically, the diagonal Fisher is computed at a single point in time, after training for a specific number of epochs.\nThe ablation we propose is maintain a moving average over these values, which is similar to the exponential moving average in adaptive optimizers.\nThe key distinction between this approach and what Adam computes lies in the order of operations: Adam maintains a moving average of squared gradients (i.e., sum then square), whereas the Fisher is derived from the squared expectation of gradients (i.e., square then sum), as discussed in the background section.", "claims_and_evidence": "The paper's claims, mostly of a performance-centric nature, are well-supported. In particular, the paper claims that the Squisher approximator is on par with the empirical Fisher on a wide range of tasks, _not_ that it is a good approximation of either the empirical Fisher or even the true Fisher (which would require further theoretical analysis on the quality of the approximations, needing even distributional assumptions).", "methods_and_evaluation_criteria": "The authors evaluate the Squisher against both the empirical Fisher and a Fisher-free baseline. The downstream tasks include model merging and pruning, sparse fine-tuning, and continual learning. These applications all require a notion of parameter importance, making them appropriate downstream tasks. The datasets and architectures are all large-scale.", "theoretical_claims": "No such theoretical claims were made in the submission.", "experimental_designs_or_analyses": "I reviewed the experimental design in detail, in particular, how the diagonal Fisher approximations are used in the downstream tasks and on what datasets and architectures the evaluation happens. This work uses T5 and BERT Transformer models for text-based tasks, and a VGG-13 for Fisher Pruning. I am satisfied with these choices. The exact model used in the EWC experiment is unspecified.", "relation_to_broader_scientific_literature": "The Fisher Information Matrix, proposed by Fisher (1922), is a natural (degenerate) Riemannian metric on the parameter space of neural networks (Amari, 1998). While Kunstner et al. (2019) argue against the use of the _empirical_ Fisher as a _curvature estimate_, it still gained widespread use as an indicator of _parameter importance_. This work makes use of existing buffers in adaptive gradient methods to provide a cheap approximation to the Fisher. Lin et al. (2024) use this approximation to motivate the removal of the square root in adaptive gradient methods and show that it does not suffer from the problems of the empirical Fisher outlined in (Kunstner et al., 2019). This work uses this \"free\" quantity in downstream tasks ranging from continual learning to model merging.", "essential_references_not_discussed": "I am unaware of _essential_ references that the paper does not consider.", "other_strengths_and_weaknesses": "A clear strength of this paper is its empirical evaluation. Numerous diverse downstream tasks considered where the Squisher performs on par with the empirical Fisher. Further, the proposed method recycles gradient estimators inherent to adaptive gradient methods, imposing virtually no overhead for calculating the Squisher. This insight, combined with the poor baseline performance on downstream tasks, motivates the proposed method well. Lastly, the paper is enjoyable to read and has a clear structure.\n\nA weakness of this paper is the lack of motivation on why the empirical version of the \"new Fisher matrix\" of Lin et al. (2024) is a sensible alternative object to describe parameter importance. This is a conceptually simple fix: one can show that the underlying \"true\" Fisher remains the same and only the nature of the approximation differs. In detail, for $n$ i.i.d. input random variables (RVs) $X$ and output RVs $Y$, distributed according to some parametric process $p(x, y \\mid \\theta) = p(y \\mid x, \\theta)p(x)$, the \"true\" Fisher is written as $$\\begin{align}\\mathbb{E}\\_{p(X, Y \\mid \\theta)}\\left[{\\nabla}\\_{\\theta} \\log p(Y \\mid X, \\theta){\\nabla}\\_{\\theta} \\log p(Y \\mid X, \\theta)^\\top\\right].\\end{align}$$ Using the additivity of the Fisher information for i.i.d. random variables and a discrete set of inputs {$x_1, \\dots, x_N$} to approximate $p(x)$, we obtain $$\\begin{align}\\sum_n \\mathbb{E}\\_{p(y \\mid x_n, \\theta)}\\left[\\nabla\\_\\theta \\log p(y \\mid x_n, \\theta) \\nabla\\_\\theta \\log p(y \\mid x_n, \\theta)^\\top\\right].\\end{align}$$ The standard empirical Fisher approximation substitutes labels {$y_1, \\dots, y_N$} from the true generative model:\n$$\\begin{align}\\sum_n \\nabla\\_\\theta \\log p(y_n \\mid x_n, \\theta) \\nabla\\_\\theta \\log p(y_n \\mid x_n, \\theta)^\\top.\\end{align}$$\nInstead, the formula the authors use can be derived from the \"true\" Fisher by first Monte Carlo approximating $X$ with a single set of inputs $\\mathcal{X} =$ {$x_1, \\dots, x_N$}, resulting in:\n$$\\begin{align}\\mathbb{E}\\_{p(Y \\mid \\mathcal{X}, \\theta)}\\left[{\\nabla}\\_{\\theta} \\log p(Y \\mid \\mathcal{X}, \\theta){\\nabla}\\_{\\theta} \\log p(Y \\mid \\mathcal{X}, \\theta)^\\top\\right],\\end{align}$$\nthen substituting in the ground-truth labels $\\mathcal{Y} = $ {$y_1, \\dots, y_N$}:\n$${\\nabla}\\_{\\theta} \\log p(\\mathcal{Y} \\mid \\mathcal{X}, \\theta){\\nabla}\\_{\\theta} \\log p(\\mathcal{Y} \\mid \\mathcal{X}, \\theta)^\\top.$$\n\nIt would also be insightful to see the performance of the true (MC-approximated) Fisher on the downstream task for comparison, as the empirical approximation loses theoretically desirable properties of the Fisher.", "other_comments_or_suggestions": "- I prefer retaining the term \"Fisher\" for the Fisher information matrix and keeping the \"empirical\" adjective for cases when the ground-truth labels are used. These terms are often used interchangeably in the literature, and simplifying the naming in this paper could potentially add to the confusion.\n- Eq. (10): $X_n$ should be $X$. The same holds for the Joint Fisher in Fig. 1, except there, the $\\frac{1}{N}$ scaling is also incorrect.\n- Eq. (8): I suggest using $B$ instead of $N$ to prevent confusion about mini-batching vs. full-batch training.\n- Eq. (15): I suggest adding \", where squaring is performed element-wise\" after the equation. Likewise in Eqs. (15, 16) for division and multiplication.\n- L303: \"section 5.3\" -> \"Section 5.3\"\n- L293: \"section 4.1\" -> \"Section 4.1\"\n- L295: \"uses <the> FISH mask\"\n- The first paragraph of Section 3.5 might be hard to understand for readers unfamiliar with Task2Vec. I propose the following clarification: \"Task2Vec embeddings are computed using the Fisher of a model trained on a given task. ~~and averaging~~ The values are averaged across parameter ``groups'' (e.g. weight matrices), leading to a smaller-dimensional representation than the number of weights.\" In the Setup paragraph, consider elaborating on the notion of \"intermediate task\". It might be instructive to state the full pipeline (pretraining -> intermediate task -> downstream task).\n- L330: \"dataset into five contexts\"", "questions_for_authors": "1. In the paragraph of L292, the authors discuss the fine-tuning setting described in Section 4.1 of Sung et al. (2021). According to this description, it uses the FISH mask to reset parameters. I have not found this in Section 4.1 of Sung et al. (2021); could the authors clarify what they mean by resetting parameters?\n2. \"Another experiment is to maintain a moving average of Fishers\": Do the authors refer to the full Fisher information matrix as the Fisher here instead of the diagonal? Otherwise, Adam already calculates this quantity.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9TNRebnHTT", "reviewer": "Reviewer_D2a9", "summary": "This paper introduces \"Squisher,\" a method that repurposes the squared gradient accumulator from adaptive optimizers (such as Adam) to approximate the Fisher Information Matrix (FIM) without additional computational cost. The authors provide theoretical analysis connecting the squared gradient accumulator to the FIM and evaluate the approach across six settings:\n\n- Fisher Merging\n- Uncertainty-Based Gradient Matching (UBGM)\n- Fisher Pruning\n- FISH Mask (sparse training)\n- Task Embedding with Task2Vec\n- Elastic Weight Consolidation (continual learning)\n\nThe main finding is that Squisher consistently performs similarly to the traditional Fisher while outperforming Fisher-free baselines, making Fisher-based methods more practical by eliminating their computational overhead.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful feedback and suggestions.\nWe will make sure to mention AdaFisher as adaptive method that uses a diagonal Kronecker-factored approximation of the Fisher.\nPlease find our responses below and let us know if you have follow-up questions.\n\n---\n\n> Limited insight into performance differences: More analysis could explain why Squisher outperforms Fisher in some settings but underperforms in others. \n\n> Q1- In Fisher Merging experiments, Squisher significantly outperforms the original Fisher. Do you have insights into why this is the case, rather than just performing similarly? Does the Squisher have properties particularly beneficial for model merging?\n\nSince both the diagonal Fisher and Squisher rely on multiple approximation steps, it is difficult to isolate a specific factor responsible for their divergence. We want to emphasize that our goal is not to claim that Squisher is inherently superior or inferior to Fisher. The performance differences are problem-dependent and influenced by training trajectories, but overall, the variation remains small relative to baseline performance and our experimental results thoroughly validate the Squisher as a zero-cost drop-in replacement for the Fisher.\n\nLet us illustrate this point for two settings with the largest difference:\n\n- For merging, both Fisher and Squisher exhibit high variance in performance depending on the checkpoint used.\n  We will describe these variances in more detail in the text and quantify them with error bars.\n- For pruning, we will add results under varying the training duration.\n  The table below shows that decreasing the number of epochs slightly increases the Squisher's performance.\n\n  | Method | Fisher| Fisher |Squisher | Squisher|\n  | -------- | -------- | -------- | -------- | -------- |\n  | Epochs Trained     | 10     | 15     |10     | 15     |\n  | Accuracy    | 64.0   | 64.8     |64.3     | 63.9     |\n  | Std dev    | 0.7     | 0.1     |0.3     | 0.6     |\n\n---\n\n> Hyperparameter sensitivity: The ablation study shows dependence on the EMA coefficient, but provides limited guidance on optimal settings\n\n> Q2- How sensitive is Squisher to optimizer hyperparameters, particularly the EMA coefficient? Given these are typically tuned for optimization rather than Fisher approximation, do you have recommendations for non-optimal settings?\n\nThe ablation results show the effect of varying the β₂ hyperparameter. We found that lowering β₂ hurt performance, highlighting the importance of accumulating gradient information over time. The optimal setting aligns with the default AdamW value of 0.999, while performance degradation was observed at 0.95. Notably, even with an unconventionally low β₂, the Squisher estimate still outperformed the baseline, indicating that while β₂ impacts performance, it remains usable if conventional values are used during training. We will clarify this in the final report, and we can provide a more comprehensive plot of β₂ values vs accuracy if necessary.\n\n---\n\n> Some implementation details could be clearer, particularly for mini-batch scaling\n\nThanks for bringing this to our attention.\nWe will make sure to update Figure 1 to incorporate an additional stage where mini-batching is introduced. \nLet us know if this addresses your concern or if there are specific details you’d like us to clarify further.\n\n---\n\n> Q3- How does mini-batch size affect the quality of the Squisher approximation? Since the scaling factor is related to dataset size N, are there special considerations for very large datasets with small mini-batches?\n\nThis is a theoretically interesting question and we will try to add some more experiments on the EWC setting with varying batch sizes.\nPractically though, we believe that varying the batch size should not be considered a hyperparameter of Squisher, since changing it will also impact the optimization algorithm.\nIts choice should primarily be influenced by the optimization algorithm and the hardware.\nGenerally speaking, our results did not exhibit a major dependence on batch size, and we anticipate that the Squisher should provide reasonable performance for typical batch sizes.", "claims_and_evidence": "The claims are well-supported through:\n\n- Theoretical analysis connecting squared gradient accumulators to the Fisher, particularly through the \"joint Fisher\" formulation\n- Comprehensive empirical evaluation across six different applications\n- Ablation studies examining the impact of specific approximations (normalization and EMA coefficient)\n- Honest reporting of both positive and negative results\n\nThe experimental results convincingly demonstrate that Squisher can effectively replace the Fisher in various applications without significant performance degradation.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are appropriate:\n\n- The authors replicate established experimental setups from prior work for each application\n- For each setting, they compare against both the original Fisher implementation and a Fisher-free baseline\n- The evaluation metrics are standard for each application (accuracy, mean reciprocal rank, etc.)\n- Multiple random seeds are used for stochastic elements", "theoretical_claims": "I checked the theoretical derivations connecting the squared gradient accumulator to the Fisher Information Matrix. The connections drawn between different formulations (particularly the joint empirical Fisher and its relationship to the accumulator in Equations 11-15) appear sound. The scaling factor of N needed to align the Squisher with the Fisher (Equation 15) is correctly identified and explained.", "experimental_designs_or_analyses": "The experimental design is sound:\n\n- The authors base their implementations on prior work for direct comparability\n- They use multiple random seeds where appropriate (e.g., five seeds for pruning experiments)\n- Both positive and negative results are reported transparently\n- The ablation studies in Section 4 isolate specific factors affecting performance", "relation_to_broader_scientific_literature": "The paper positions its contributions well within multiple research areas:\n\n- Natural Gradient Descent and second-order optimization methods\n- Adaptive gradient methods (Adam, RMSProp, etc.)\n- Applications of Fisher diagonal as a parameter importance measure\n- The practical challenges of computing the Fisher in existing frameworks\n\nThe authors correctly identify that computational costs have hindered the adoption of Fisher-based methods, motivating their approach.", "essential_references_not_discussed": "The paper covers most relevant prior work. However, it could benefit from more discussion of recent second-order optimizers:\n\n- Related work: The paper does not mention some other recent second-order optimizers such as AdaFisher [1], which could be relevant in the context of adaptive second-order methods. Including these in the discussion would provide a more comprehensive overview of related work.\n\n[1] Gomes, D.M., Zhang, Y., Belilovsky, E., Wolf, G., & Hosseini, M.S. (2025), AdaFisher: Adaptive second order optimization via fisher information. In the thirtheenth international conference on learning representations.", "other_strengths_and_weaknesses": "Strengths:\n\n- Practical impact: Eliminates the computational overhead of Fisher-based methods\n- Broad evaluation: Demonstrates robustness across diverse applications\n- Clear theoretical grounding: Well-explained connection to the Fisher\n- Honest reporting: Presents both positive and negative results\n\nWeaknesses:\n\n- Limited insight into performance differences: More analysis could explain why Squisher outperforms Fisher in some settings but underperforms in others\n- Hyperparameter sensitivity: The ablation study shows dependence on the EMA coefficient, but provides limited guidance on optimal settings\n- Some implementation details could be clearer, particularly for mini-batch scaling", "other_comments_or_suggestions": "The paper is well-written and clearly structured. No significant typos or errors were noted.", "questions_for_authors": "1- In Fisher Merging experiments, Squisher significantly outperforms the original Fisher. Do you have insights into why this is the case, rather than just performing similarly? Does the Squisher have properties particularly beneficial for model merging?\n\n2- How sensitive is Squisher to optimizer hyperparameters, particularly the EMA coefficient? Given these are typically tuned for optimization rather than Fisher approximation, do you have recommendations for non-optimal settings?\n\n3- How does mini-batch size affect the quality of the Squisher approximation? Since the scaling factor is related to dataset size N, are there special considerations for very large datasets with small mini-batches?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hHibSnbAh9", "reviewer": "Reviewer_3M1M", "summary": "- In various contexts where a method is motivated using the Fisher Information Matrix (e.g. EWC, Fisher Pruning, etc), the paper proposes to replace squared sum of gradients, which can sometimes be cumbersome to compute, by the exponential moving average of squared sums of gradients as computed in e.g. Adam, since this quantity is already computed during training. The method is called the \"Squisher\".\n- The proposition is motivated by means of the joint empirical Fisher proposed in Lin et al. 2024\n- Empirical validation is performed on a variety of settings that employ approximate diagonal Fisher information, which show that there is no substiantial difference (sometimes slightly better, sometimes slightly worse) in downstream performance when replacing the diagonal second moment of gradient by the Squisher.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful feedback, questions, and suggested references.\nWe have updated our manuscript in accordance with your suggestions.\nPlease find our responses below.\nLet us know if you have any follow-up questions, we'd be happy to discuss further.\n\n---\n\n> Why would a data scientist or researcher use the Squisher\n\n**Cost:** The main advantage of Squisher is that it’s completely free when using an adaptive optimizer that accumulates the squared gradient (e.g., Adam(W)), which is ubiquitous in deep learning. Since this information is already stored in the optimizer’s state dict, Squisher can be obtained without any additional computation or access to training data. In contrast, computing the Fisher requires both extra computation and access to training data.\n\n**Universal support:** You are right that there exist libraries like BackPACK that compute the empirical Fisher diagonal more efficiently than a for loop.\nHowever, these alternative approaches still introduce additional cost, require modifications in the code, and do not support all architectures -- for example, BackPACK does not support layer normalization.\nAs a result, the for-loop approach remains common in existing Fisher-based methods, including those considered in our paper. Since Squisher simply recycles the already available squared gradient accumulator, it is universally supported without these limitations.\n\n--- \n\n> it seems rather limited to only study diagonal approximation of the FIM\n\nWhile recent work has explored Fisher approximations beyond the Fisher diagonal, such work has primarily targeted optimization applications (e.g. the natural gradient) as opposed to settings where the Fisher is used as a notion of parameter importance.\nAdditionally, such approximations are actually *less* relevant in modern deep learning due to scale. \nComputing, processing, or storing anything larger than the Fisher diagonal (which has as many entries as model parameters) is generally infeasible with today's large-scale models, which might have billions of parameters.\nConsequently, the Fisher diagonal remains especially relevant in today's research.\nGiven the increasing popularity of non-diagonal adaptive optimization algorithms like Shampoo/SOAP, we anticipate that their non-diagonal gradient second moments could be used to replace Kronecker-factorized curvature approximations of the Fisher, such as (E)KFAC.\nWhile we leave this exploration as future work, we believe it actually strengthens our argument for using gradient second moments from adaptive optimizers as Fisher proxies for free.\n\n---\n\n> the claimed empirical results could be a side-effect of some desirable property of your new method\n\nWhile we agree that the empirical and true Fisher can meaningfully differ (as pointed out by other works, e.g. https://arxiv.org/pdf/1905.12558), most work using the Fisher as a notion of parameter importance uses the empirical Fisher because it produces good downstream performance and is more straightforward and efficient to compute than the true Fisher.\n\nPlease also see our response to the second quoted note in Reviewer WT93.\n\n---\n\n> Why refer to the FIM's diagonal as the \"Fisher\" ?\n\nPlease see our response to the same question in Reviewer WT93 (third quoted note).\n\n--- \n\n> [...] please clarify what is new contribution compared to Lin et al.\n\nThe work of Lin et al. serves as one motivation for our work and provides perspective on Fisher approximations that first sum then square gradients. Our study aims primarily to empirically validate the connetion between the Squisher and Fisher; we do not make or claim any substantial theoretical contributions.\nNote that Lin et al. primarily focus on optimization and the role of the square root in the update of adaptive optimizers for neural network training. This is different from our setups, and their empirical findings do not overlap with ours.\n\n---\n\n> - why motivate a method using an expectation over sampled vectors y, since this expection is never actually computed ?\n> - why would it be useful to consider the Riemannian metric defined by the joint empirical FIM ?\n\nThe motivation for considering the joint Fisher which stems from considering a vector of labels is that it gives rise to a Fisher which first sums, then squares the gradient.\nThis pattern is similar to the statistics of adaptive optimizers like Adam that accumulate the square of summed gradients.\n\nAs pointed out by Reviewer WT93, the joint and standard Fishers coincide, i.e. they approximate the same underlying \"true\" Fisher.\nTherefore, the standard empirical Fisher and the joint empirical Fisher can be regarded as two different empirical approximations of the same underlying Fisher.\nSince the standard empirical Fisher is used in various problems in practise, our paper studies if the Squisher, which approximates the joint empirical Fisher, can serve as an a \"free\" drop-in replacement.", "claims_and_evidence": "The claim is rather conservative (i.e. downstream performance is sometimes better, sometimes not significantly worse) so the empirical results correctly support the claim.", "methods_and_evaluation_criteria": "My main concern regarding this paper is that it seems rather limited to only study diagonal approximation of the FIM, and I don't agree with the motivating argument that it is still difficult nowadays to obtain per example gradients e.g. using BackPACK, or other libraries easily found on Github.\n\nSpontaneously, I would have benchmarked full-fledged (i.e. not diagonal) methods using the true FIM, as can be achieved using (E)KFAC approximations e.g. [Ritter et al. 2018] for continual learning.\n\nMy feeling after reading the paper (but I might be wrong) is that it is trying to solve a problem which is not anymore so relevant in 2025's deep learning community.\n\nAnother concern is the use of the concept of \"Fisher\" where the Squisher is never actually directly compared to a ground truth diagonal Fisher. From the content of the paper alone, the claimed empirical results could be a side-effect of some desirable property of your new method, that has not much to do with true Fisher information. Even the title of the paper might be misleading.", "theoretical_claims": "No new theoretical claims are introduced in the paper as far as I can tell, everything is inherited from Lin et al. 2024.", "experimental_designs_or_analyses": "Did you check the soundness/validity of any experimental designs or analyses? Please specify which ones, and discuss any issues.\n\nA fundamental flaw in my opinion is that many benchmarked methods claim to be using Fisher Information, but actually end up making at least two stages of approximation:\n - diagonal (whereas there are efficient approximate method nowadays)\n - gradients w.r.t. training set targets, whereas the actual FIM would require gradient w.r.t. sampled targets (there exist closed form).", "relation_to_broader_scientific_literature": "The literature regarding diagonal approximate method, applied to various settings (i.e. optimization, pruning, model merging, etc) is correctly discussed. A possible improvement would be to discuss methods that use the actual FIM, and approximate methods that scale its use to large nets.", "essential_references_not_discussed": "For computing per-example gradients (methods):\nGoodfellow, I. (2015). Efficient per-example gradient computations. arXiv preprint arXiv:1510.01799.\nRochette, G., Manoel, A., & Tramel, E. W. (2020, November). Efficient Per-Example Gradient Computations in Convolutional Neural Networks. In Workshop on Theory and Practice of Differential Privacy (TPDP).\n\nClosed-form FIM that does not require sampling pseudo target classes:\nPascanu, R., & Bengio, Y. (2013). Revisiting natural gradient for deep networks. arXiv preprint arXiv:1301.3584.", "other_strengths_and_weaknesses": "A main strength of the paper is that many different setups are covered in the experiments.\n\nAnother strength is that previous material, as well as the new method, are clearly presented.", "other_comments_or_suggestions": "Why refer to the FIM's diagonal as the \"Fisher\" ? If accepted, it might be misleading for future readers, whereas Fisher Information corresponds to the full matrix.\n\nFrom the body only, please clarify what is new contribution compared to Lin et al.\nMoreover, the current paper inherits some ambiguity from Lin et al.'s paper:\n - why motivate a method using an expectation over sampled vectors y, since this expection is never actually computed ? \n - why would it be useful to consider the Riemannian metric defined by the joint empirical FIM ?", "questions_for_authors": "Why would a data scientist or researcher use the Squisher instead of more accurate approximate methods using github libraries?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes to recycle the squared-gradient accumulator used in common optimizers such as AdamW as an approximation to the Fisher matrix. Across several benchmarks, consistent improvements of the squared-gradient accumulator over the diagonal Fisher are demonstrated. 

Overall, the paper makes an important contribution and may put more awareness in the community that variance estimates computed during optimization can be reused. The contribution was highly appreciated by most reviewers. 

The main criticisms by the reviewers were: 
1) the lack of systematic comparisons whether better Fisher approximations give better downstream performance,
2) extension to non-diagonal Fishers, 

I found these to be addressed by the rebuttal, and given today's billion-parameter scale models these are hard to run and properly benchmark. 

Overall, I recommend acceptance of this work and believe that it will make a nice contribution to the conference and potentially an impact (e.g., people releasing AdamW's scale vector on HuggingFace more often along with the trained parameters). For the final version, perhaps it could be acknowledged more clearly that the idea of reusing the squared-gradient accumulator for model merging was already mentioned at the end of Section 3.3 in Daheim et al. 2024.
Decision: Accept (spotlight poster)