Merging Models with Fisher-Weighted Averaging
Authors: Michael S Matena
Colin Raffel
Venue: NeurIPS 2022 Accept (2022)
Keywords: transfer learning
parameter averaging
ensembling

Abstract: Averaging the parameters of models that have the same architecture and initialization can provide a means of combining their respective capabilities. In this paper, we take the perspective that this "merging" operation can be seen as choosing parameters that approximately maximize the joint likelihood of the posteriors of the models' parameters. Computing a simple average of the models' parameters therefore corresponds to making an isotropic Gaussian approximation to their posteriors. We develop an alternative merging procedure based on the Laplace approximation where we approximate each model's posterior as a Gaussian distribution whose precision matrix corresponds to its Fisher information. We first show that our "Fisher merging" technique provides a performance boost in settings where simple parameter averaging is currently used -- specifically, robust fine-tuning and model ensembling. Then, we compare merging to standard gradient-based transfer learning and demonstrate that merging enables a fundamentally different method for transferring capabilities across models. Specifically, we show that Fisher merging is competitive with gradient-based transfer learning approaches (while being significantly cheaper) in intermediate-task training and domain-adaptive pre-training. We also show that our merging procedure makes it possible to combine models in previously unexplored ways. We release our code to facilitate future research into methods for merging models.

Merging Models with Fisher-Weighte Averaging
Michael Matena
Colin Raffel
Department of Computer Science University of North Carolina at Chapel Hill (mmatena, craffel) unc. edu
Abstract
Averaging the parameters of models that have the same architecture and initializa- tion can provide means of combining their respective capabilities. In this paper, we take the perspective that this "merging" operation can be seen as choosing pa- rameters that approximately maximize the joint likelihood of the posteriors of the models' parameters. Computing a simple average of the models' parameters there- fore corresponds to making isotropic Gaussian approximation their posteriors. We develop an alternative merging procedure based on the Laplace approximation where we approximate each model's 's' posterior as a Gaussian distribution whose precision matrix corresponds to its Fisher information. We first show that our "Fisher merging" technique provides a performance boost in settings where simple parameter averaging is currently used specifically, robust fine-tuning and model ensembling. Then, we compare merging to standard gradient- transfer learn- ing and demonstrate that merging enables fundamentally different method for transferring capabilities across models. Specifically, we show that Fisher merging is competitive with gradient-hased transfer learning approaches (while being sig- nificantly cheaper) in intermediate-tasi training and fomain-adaptno We also show that our merging procedure makes it possible to combine models in previously unexplored ways. We release our code to facilitate future research into methods for merging models."
pre-training.
Introduction
How should we transfer knowledge and capabilities across trained models? One popular approach transfer learning [44], which fine-tunes a pre-trained model a target task through additional gradient-based training. The preparatory step of pre- pre-training the model on data-rich task ideally instills useful "knowledge" into the network's parameters, which allows the model to learn more rapidly and effectively when fine-tuned on a downstream task of interest. Transfer learning has therefore become a particularly important and omnipresent tool across many fields, including natural language processing [57, 13, 9, 52, 53, 46] and computer vision [43, 24, 68]. Recently, it has been shown that training on an "intermediate" task between pre-training and fine-tuning can further boost performance through additional transfer of capabilities from the intermediate task [47, 60, 51, 48]. Alternatively, continued self-supervised training on unlabeled domain-specialiize data can serve as form of domain adaptation [19].
All of the aforementioned transfer learning methods transfer knowledge by using trained network to initialize another network followed by iterative gradient descent. While demonstrably powerful, several drawbacks arise from this: First, improvements ancestor models cannot be passed down to descendants; instead, we must restart the whole process from the improved ancestor model, throwing away our previous work. For example, we fine-tune pre-trained model on a downstream task,
mergiinee
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Fine-tuned
Fine-tuned
Pre-trained
Intermediate
Fine-tuned
Pre-traiiedd
Pre-trained
Fine-tuned
Donor
Pre-trained
Donor
Model
Fine-tuning
Merglin
Figure 1: Merging patterns considered in this work. Left: Merging many fine-tuned models form of ensembling. Center, top: "Robust fine-tuning" [66]. where a fine-tuned model is merged with the pre-trained model to improve performance the original pre-training task. Center, bottom: Merging a fine-tuned model with a "donor" task, analogous to intermediate-tane transfer learning [47, 51]. Right: Merging intermediate-task trained model with a donor model.
but then the pre-trained model is improved through additional training, we must re-fine-tune the new model on our downstream task if Furthermore, if we gain access to checkpoint that has been fine-tuned on a useful intermediate task, we must again throw away our previous work and fine-tune from the intermediate task checkpoint. Existing methods for transfer learning also have the disadvantage of only being able to transfer information from a single model. While it may possible to train on multiple intermediate tasks sequentially, one quickly either runs into combinatorial explosion of saved checkpoints or faces the issue of "catastrophic forgetting" in continual learning [28]. In addition to slowing down experimentation by preventing reuse of work, these drawbacks impose limitations the types of transfer that can occur.
confer benefits from this additional pre-training.
A less common way of transferring capabilities across models is to simply average their parameters. This procedure, which we call "merging", generally only feasible when the models being averaged share a common architecture and initialization. Merging is the core component of the FedAvg algorithm used in Federated Learning [39], where updates a shared model computed by individual workers that are training on different datasets are combined by simpling averaging the updates. Recently, Wortsman al. [66] demonstrated that merging can be used to improve robustness to domain shift in fine-tuned models by averaging the parameters of the original pre-trained model with the fine-tuned parameters. Merging also a common way performing ensembling [49, 67], where the parameters of individual models trained on the same dataset are averaged to create single performant model.
In this work, we view model merging approximately maximizing the joint likelihood of the models' posterior distribution over parameters. Since gradient-based maximum likelihood training only provides a point estimate of the posterior, some approximation of the posterior distribution is required. When an isotropic Gaussian distribution used to approximate the posterior (with identity precision matrix and mean set to the model's parameter values). we show that maximizing the joint likelihood across models is equivalent to simply averaging their parameters. We therefore refer to merging models by averaging parameters as isotropic merging. The view of merging maximizing the joint posterior distribution may
likelihood of model posteriors suggests that using a better estimate yield improved merging results. This leads to introduce Fisher merging, which leverages the Laplace approximation by using the diagonal of each model's ' Fisher information as the precision matrix for that model's posterior.
Empirically, demonstrate that merging models with Fisher merging outperforms isotropic merging in variety of settings. We first focus on the existing applications of model ensembling [49, 67] and improving fine-tuned model robustness [66]. Then, we demonstrate for the first time that merging is viable alternative to traditional gradient-based transfer learning. Specifically, we compare merging to intermediate-task transfer learning [47, 51] and -adaiivee pre-training [19], finding that merging can achieve comparable performance at significantly lower cost. Additionally, we show that merging can provide an additional boost to models created via traditional intermediate-tass training. This provides concrete example of transfer that is fast and easy with merging but onerous

---

or impossible to do with existing methods. Diagrams of the merging patterns we consider in this work are shown in fig. I.
The rest of our paper is structured as follows: In section 2, we provide necessary background and detail our Fisher merging procedure. Section 3 provides experimental results on model ensembling, robust fine-tuning, intermediate-task training, and domain adaptation. We explore related works in section 4 and provide conclusions and thoughts on future work in section 5.
2 Weighted Parameter Averaging for Model Merging
Our focus is on procedures for model merging, i.e. averaging the parameters of models that share an architecture and initialization. In this section, we first frame the common practice averaging together model parameters as approximately maximizing the joint likelihood of model posteriors. Specifically, we show that parameter averaging corresponds to using an isotropic Gaussian as the approximate posterior for each model. We then introduce Fisher merging, which uses the model's diagonal Fisher information matrix as the precision matrix the Gaussian approximate posterior. Fisher merging be implemented by setting each merged parameter value to weighted average of the corresponding parameter values from the original models, with the weighting for each parameter determined by its Fisher information. In addition, add model-level weightings additional hyperparameters to set the relative importance of each model.
2.1 Isotropic merging
Consider the problem setting where have M trained neural networks with parameters 01,.... Aas and our goal is to create a single neural network with parameters @ that, loosely speaking, inherits the capabilities of the M trained neural networks. Assume that all of these neural networks share common architecture and had the same set of initial parameter values before being trained. Merging attacks this problem by finding the parameters 0 that maximize the joint likelihood of the posterior distributions of the M models. Unfortunately, typical neural network training procedures do not provide access posterior distribution, which necessitates approximation. If the posterior of each model is approximated via an isotropic Gaussian with mean set to the model's parameters, the optimization problem can be written 0 argmax, E, log p(0),,, I) where p(0)0,, 7) the probability distribution of the aforementioned approximate isotropic Gaussian posterior distribution used for model and / the identity matrix. This optimization problem has a closed-fore solution given by 0 J7 Σ, 0z, i.c. an average of the model parameters. Such an averaging procedure has been used in past work aiming combine model capabilities, e.g. in federated learning [39], model ensembling [49, 67], and robust fine-tuning [66].
2.2 Per-model weights
In this work, we additionally introduce model-specific scalar hyperparameters A E {l..... M) into the model merging framework described above. Specifically, we change the optimization problem 8" argmax, Σ, A₂ log p(010,, 1) where 2 0, Σ, A 1. In the case of isotropic merging, this changes the solution to 0* = E, 7,O; These hyperparameters provide control over the importance assigned each of the models that are being merged. For example, when using merging to perform ensembling we might expect each model to be equally important and therefore set X; = 1/M M for all i. On the other hand, when mimicking the setup of intermediate- training where the capabilities of a "donor" model are used to improve performance of recipient model, we might weigh the recipient model more highly. Wortsman et al. [66] introduce a similar hyperparameter a when averaging the parameters of two models and report results for varying values of a.
2.3 Laplace Approximation
Framing merging as approximate maximization of the joint posterior likelihood reveals that simple parameter averaging is implicitly using an isotropic Gaussian posterior approximation. Such an approximation may be overly simplistic and lead to degraded performance. To explore improved merging procedures, we consider improved methods for creating an approximate posterior from point estimate. Specifically, we use the Laplace approximation to the posterior, which corresponds to second-order Taylor expansion of the log density around mode [36, 10]. This leads Gaussian
3

---

approximation [e0 ") of the posterior, where H is the Hessian matrix and are the model's s trained parameter values. More precisely, we assume that the parameter values 0 of a trained neural network are local maximum of the posterior. It can then be shown that the precision matrix of the Laplace approximation is given by the Fisher information matrix of the network
The Fisher information matrix F' [16, 3] of a neural network pe (y|z) trained to predict an output from input data æ is [0] |0] positive semidefinite matrix given by the formula
Fe = E
E (y)la))
Ve Po (ogooool) logro(vlx))
(1)
can be shown that the Fisher information matrix coincides with the Hessian H at modes of the distribution [45], explaining its use in the Laplace approximation. The Fisher information matrix Fø can also be used to relate changes in the model parameters to changes in the model output by noting 20" Fod Fo6 as ō O, where DRI. denotes the KL--divergence
that E,z [DxL. (po ([po++(ylp))))))) [45].
As the full Fisher matrix takes ?( memory to store, it quickly becomes impractical for all but the smallest models. We are thus forced to use an approximation to the full Fisher in practice. this paper, we follow the common practice of using the diagonal of the Fisher matrix [28]. While other methods (e.g. [I]) exist for estimating the Fisher, we leave their exploration for future work. In our experiments, estimated the diagonal of the Fisher matrix via
N
P6
E (y(a)
(Vo log Po
N
where ,i,..., N are drawn i...d. from the dataset that was used to train the model. The expectation y be estimated via sampling from p=) computed exactly when the number of classes is small. We note that computing the Fisher requires N per-example gradients, which can be straightforwardly computed for neural networks using backpropagation. This makes computing the diagonal Fisher have roughly the same computational cost as training on N examples.
2.4 Fisher Merging
Having noted that the Laplace approximation provides a tractable way to obtain a better approximation to the posterior, we now use it to create an improved merging procedure that we call Fisher merging. Letting Fi...., Fas correspond to the diagonal approximate Fisher matrices, we construct p(0) F,) as a Gaaussan-distribuuie precision F. To obtain the merged model, we find single set of parameters that is given a high probability under all posteriors. Formally, we have
posterior over the parameters of the merged model with mean 0, and
M 8 argmax, Σ
A log ((10,, F.),
(3)
which has the closed-form solution
()) M XF 0:0) EM X,
(4)
where j = ,,,.,,,..1.. 10). Intuitively, we can think of Fisher merging as computing a weighted average of the parameter values in each model where the weighting is done according to each parameter's Fisher information. Since the Fisher information is a local property of a single parameter value, Fisher merging might be less performant when applied to models whose parameters far apart in parameter space. We therefore limit our focus to models that were trained from the same initialization.
Numerical Issues. Note that (4) can run into numerical issues when the Fisher is close to zero across all models for given parameter. In practice, we choose a privileged "target model" in all of our experiments and "default" to the parameter's 's value in the target model in these cases. An alternative would to take an average weighted only by the merging coefficients (i.e., pretend the Fisher the same across all models). In practice, the choice of a "default" value for these parameters had little impact on performance (likely because a small Fisher value implies that changing the parameter has minute effect on the model's 's outputs and is therefore relatively unimportant to the model's behavior).

---

Unmergeable Parameters, In many cases, we have some parameters from each model that do not appear in all of the models we are merging. For example. this includes having task-specific classification heads on top of a common body architecture. We handle this by only applying the merging procedure (3) to the shared body parameters and keeping the task-specific heads unchanged. Although this may lead to a distribution shift in the classification head inputs, we found it to work well in practice for the datasets and tasks we consider.
3
Experiments
Our first experimental goal is to validate that use of an improved estimate of the posterior yields improved merging performance. To test this hypothesis, we apply Fisher merging to two settings where isotropic merging has already proven successful: Model ensembling [49, 67] and robust fine-tuning [66]. Then, we demonstrate that Fisher merging provides a cheap and effective alternative to traditional transfer learning pipelines by validating its performance in intermediate- task transfer learning [47, 51] and domain- aamain-adaptive pre-training [19]. Finally, we demonstrate that merging opens up new paths of transferring capabilities across models by demonstrating boost in performance when merging an intermediate task-trained model with different donor models.
3.1 Ensembling
An existing application of isotropic merging is for ensembling, i.e. combining models trained on the same dataset to obtain better predictions. Ensembling is most commonly performed by averaging the predictions of the individual models. This form of ensembling requires computing the output of all M models in ensemble, thereby increasing the computational cost by a factor of M compared to computing the output for single model. A cheaper alternative is average the parameters of the models themselves. This approach diagrammed in fig. 1, left. Such an approach used in the classical method of Polyak averaging [49], where parameter values from the final . iterations of training are averaged. More recently, Wortsman et al. [67] introduced the "Model Soup" approach where fine-tuned models with different hyperparameter settings are averaged to improve performance. To the best of our knowledge, all aarameter-averaginn ensemble methods have used isotropic merging, i.e. an unweighted average.
To test whether Fisher merging provides a boost over isotropic merging when averaging parameters for ensembling, we consider ensembling fine-tuned checkpoints derived from the same pre-trained model. Specifically, we consider the BERT- BERT-Base model [13] fine-tuned on the RTE [8], MRPC [14], and SST-2 [59] datasets. For each dataset, Hugging Face model hub.² These checkpoints were fine-tuned with a variety of hyperparameter settings that were chosen by us, our experimental setting most closely matches the "Model Soup" approach [67]. A list of the checkpoints used is available in appendíx A. Since we do not anticipate that any member of the ensemble should be given larger weight, we set X 1/5 for all models.
five fine-tuned checkpoints downloaded from the
Our results are shown in fig. 2. We report validation set scores for Fisher merging, isotropic merging, and prediction ensembling (specifically, averaging the output probabilties of all models). Fisher merging significantly outperforms isotropic merging in all cases and attains comparable performance to prediction ensembling. Notably, performing inference after merging is M X cheaper than prediction ensembling, suggesting that merging can provide cheaper alternative to standard ensembling procedures.
3.2 Robust Fine- Fine-Tuning
Recently, Wortsman et al. [66] found that while fine-tuning a pre-trained vision model tends to improve performance on the downstream task, it also tends decreases accuracy on the original pre- training task. They therefore propose a "robust fine-tuning" procedure called WiSE-FT that computes a weighted average of the original pre-trained parameters and the fine-tuned parameters. Different weighting values produce difffrent trade-off's between pre-training and fine-tuning task performance. In some cases, robust fine-tuning can even improve performance on the original pre-training task without sacrificing performance on the downstream fine-tuning task relative to traditional fine-tuning.
hntepe:/huuginngface http://gggaacc.cccoleee...
5

---

100
Isotropic merging
Fisher merging
Average OOD accuracy
90
Output ensembling
62
Accuracy
60
58
70
56
Isotropic merging
Fisher merging
50
RTE
MRPC
SST-2
70
75 ImageNet accuracy
80
Figure 2: Validation set accuracy for ensembles of five fine-tuned BERT models using different ensembling methods the RTE, MRPC, and SST-2 datasets. Fisher merging produces a sin- gle model that performs comparably output ensembling while being 5x cheaper.
Figure 3: IID (ImageNet) and average out- of-domain (OOD) accuracy across five OOD datasets when using the WiSE-FT procedure [66] with either Fisher or isotropic merging. Dark light color indicates increasing λ from to 1.
This procedure implicitly uses isotropic merging and therefore provides another natural testbed for determining whether Fisher merging provides a boost in performance. schematic of robust fine-tuning shown in fig. 1, center top.
We use the codebase and experimental setup of Wortsman
[66] exactly, simply replacing
isotropic merging with Fisher merging. For full details of this setup, refer to Wortsman et al. [66]. As short summary, we apply WiSE-FT to the ImageNet [11, 58] pre-trained ViT-B/16 model [15]
on five out-of-domain (OOD) datasets: ImageNet-A [21]. ImageNet-R [20]. ImageNet Sketch [62]. ImageNet V2 [56], and ObjectNet [4]. Following Wortsman et al. [66]. we measure IID (ImageNet) and OOD performance when averaging together the original pre-traue rre-trained model parameters and parameters from models fine-tuned on each of the OOD datasets, varying è (the averaging weight for the pre-trained model, called by Wortsman al. [66]) from 0 to 1 in 0.1-step increments (with X2 = 1 A correspondingly decreasing from to 0). To determine whether Fisher merging confers boost in performance, compare parameter averaging using either isotropic Fisher merging.
plot the IID (ImageNet) accuracy against the average accuracy on the five OOD datasets for varying values of λι in fig. 3, with plots for individual OOD datasets in fig. 7 (appendix). Fisher merging produces significantly better trade-off between IID and OOD accuracy. In particular, Fisher merging seems general improve IID accuracy compared isotropic merging. For example, for the value of λ producing the best average OOD accuracy, Fisher merging produces about I 1% higher IID accuracy than isotropic merging.
3.3 Intermediate-tase training
Having established that Fisher merging produces better results than isotropic merging in settings where merging has been attempted before, we now explore the use of merging as an alternative to gradient-based transfer learning procedure. Specifically, we explore intermediate-task training [47, 51J, where a model is fine-tuned on an intermediate "donor" task before being trained on target task of interest. To the best of our knowledge, as a way performing intermediate-tass transfer lcarning. For the most part, intermediate- training has mainly been considered in the NLP domain; such, we limit experiments to the BERT [13] and RoBERTa [33] pre-trained language models. To enable comparison to past work, we mostly explored merging pairs of models but we interested in exploring merging more than two models future work. As in section 3.1, 1, we made use fine-tuned BERT and RoBERTa checkpoints from the Hugging Face repository [65].
prior work has considered parameter averaging
Following previous work [47, 51], we first ran experiments using BERT-base on the GLUE benchmark [61]. The GLUE benchmark consists of the sentence acceptability task Col.A [64], the sentiment
detection task SST-2 [59], the paraphrase detection tasks MRPC and QQP [14, 23], the sentence
similarity task STS-B B [7]. and the natural language inference (NLI) tasks MNLI QNLI RTE, and

---

Isotropic merging
75
Fisher merging
Standard training
Accuracy
Accuracy
70
79
65
60
CoLA
SST-2
MAPC
STS-B
QQp
MNU
QNL
77
SST-2
MRPC
STS-B
QQP
MNL
QNLI
Donor task
Donor task
Figure 4: Validation set accuracy on RTE when performing intermediate-tare training with datasets from GLUE the donor task. Dashed line denotes RTE accuracy without intermediate- task training.
Figure 5: Validation accuracy on RTE after first fine-tuning on MNLI, then fine-tuning on RTE, and finally Fisher merging with various donor task models. Dashed line denotes RTE accuracy after MNLI intermediate-task training.
WNLI [6, 54, 8, 31]. All the GLUE tasks are classification tasks except for STS-B, which is a regression task with score ranging from O to 5. To simplify computation of the Fisher, we turn STS-B into a classification task by partitioning the continuous label into 25 equally-sized buckets [53]. Following common practice, we do not experiments WNLI due to the tricks required to get a good score [12. 29]. See Wang et al. [61] for more details on these tasks and their associated metrics. We detail how obtained fine-tuned checkpoints these tasks in appendix C. We computed a diagonal Fisher approximation for each checkpoint using up to 4096 examples from the corresponding train set. Since it is not clear priori what weighting coefficients \; use in this setting, we chose X; by a grid search with 50 points, using the score on the first 2048 validation examples as the selection metric. We compare Fisher merging to isotropic merging as well standard gradient-based intermediate-task fine-tuning baseline [47]. A diagram of intermediate-task merging is shown in fig. center bottom.
In initial experiments (reported in tables A1 to A3), we performed intermediate- training for possible pair of datasets from the GLUE benchmark. Congruent with past work [47, 51, 60], found that intermediate-task training provided the most notable performance boost when the RTE dataset the target. We therefore focus RTE results in the main text. Figure 4 shows the results of intermediate- training of BERT-base with RTE as the target task and the other GLUE datasets donor tasks, using Fisher merging, isotropic merging, standard gradient-basee training. Notably, performing gradient-based intermediate-tass training hurts on some datasets, whereas merging always helps. Fisher merging gets comparable or better performance than isotropic merging with the largest gap observed when using MNLI as the intermediate task. On the other hand, merging performs worse than standard gradient-basee based training when using MNLI as the donor task.
Exploring new paths for transfer Given this performance gap. we were interested to see whether merging could provide an additional boost on top of gradient- intermediate-task training. We therefore performed Fisher merging on a BERT-base model that was first fine-tuned on MNLI and then fine-tuned on RTE. A diagram of this setup shown in fig. 1, right. This procedure does not have a direct analog in traditional gradient-based, and we will show later, performing multi-stage gradient-based intermediate-task training generally harms results.
We consider Fisher merging the intermediate-tain trained RTE model with all GLUE tasks and show the results in fig. 5. Fisher merging provides boost over gradient-based intermediate-tas! training for all tasks. Interestingly, a boost is still conferred when merging with an MNLI-trained model, suggesting that merging provides a complementary path for transferring capabilities across models.
Scaling to RoBERTa-large Seeing that merging can provide boost on top of intermediate-task training, we explored whether this boost could still obtained for a stronger model than BERT- base. We therefore applied the same procedure to a RoBERTa-large RTE model that had been

---

fine-tuned from an MNLI intermediate checkpoint. Our donor models were the original RoBERTa- large checkpoint (i.e., not fine-tuned on MNLI) fine-tuned on MRPC, RTE, STS-B, and SST-2. We additionally ran sequential gradient-based fine-tuning baseline where we started with the MNLI checkpoint, fine-tuned on the donor task, and then fine-tuned on the target task.
The results are shown in fig. 6. We find merging provides a boost in performance even on the more performant RoBERTa model. The largest boost of 2.2 points came from Fisher merging with another RTE checkpoint, which is reminiscent of using merging for ensembling. Notably, including an additional intermediate task gradient- based training significantly harmed performance compared to performing intermediate-tur -tasa training on MNLI alone. We hypothesize this is related to the phenomena of catastrophic forgetting [17], where the model's capabilities on MNLI are forgotten as it is trained on the next intermediate task. Nevertheless, this illustrates model merging's s ability to sidestep the issue of catastrophic forgetting and enable exploration of novel transfer strategies.
Costs We had previously noted that our merging procedure could potentially be substantially more efficient than standard gradient-based fine-tuning. To measure this claim concretely, we computed the FLOPs required for fine-tuning and merging an RTE checkpoint based on the heuristics described in Kaplan al. [26]. Fine-tuning BERT-base on RTE for 10 epochs would require about 5.5e 14 FL.OPs. Our merging procedures require computing the merged checkpoint (eq. (4)) and then evaluating it on the validation set with Fisher merging also requiring the estimation of the Fisher matrix (eq. (2)) beforehand. These steps require about 4.0e8, 2.0c and 9.1e13 FLOPs respectively, resulting in a roughly 6x lower total cost compared to fine-tuning for Fisher merging and 275x lower cost for isotropic merging. We note that the Fisher matrix only needs to be computed once and can be reused for subsequent merges, which amortizes the most expensive step in Fisher merging.
To explore methods for further reducing costs, experimented with using fewer examples to estimate the Fisher. Specifically, we experimented with intermediate-tan Fisher merging of BERT-base with MNLI as the donor task and RTE as the target task. The results are shown in table A4. While using the full training set to estimate the Fisher produced the best performance (73.4%). using only 256 examples to estimate the Fisher only produced mild degradation in accuracy (72.7%) and still outperformed the isotropic merging baseline. This suggests that computing the Fisher over fewer examples could further reduce computational costs without sacrificing great deal of accuracy.
3.4 Domain Adaptation
We now turn our attention to the "domain-adaptiie pre-training (DAPT) approach for domain adaptation advocated by Gururangan et al. [19], which is methodologically similar to intermediate- task training. DAPT consists of additional pre-training of an original general-puree -urpose pre-trained checkpoint domain-specific unlabeled data. We explore the benefits of merging in experimental setup similar to Gururangan et al. [19]. We focus on the biomedical (BIOMED) and computer science (CS) domains because they correspond to the classification tasks that saw the largest gains from domain- adaptive pre-training in [19]. Namely, we experimented with the CHEMPROT [30] relation classification task on the BIOMED domain. On the CS domain, we used the citation intent task of ACL-ARC [25] and the relation classification task of SCIERC [35]. Following Gururangan et al. [19], we report macro- F2 for ACL-ARC and SCIERC, and we report micro-F, for CHEMPROT. We used RoBERTa-base [33] as our bascline model. Appendix D includes full details of the pre-training, fine-tuning, and merging procedures used.
We present our results in table 1. Merging provided the largest boost on ACL-ARC, and outperformed traditional fine-tuning in this setting. We only observed minor improvement in performance on CHEMPROT and SCIERC. We note that our boosts from gradient- based fine-tuning were smaller than reported in [19]. which was likely because we were only able to train on public data and applied domain-adaptive pre-training for fewer steps. However, our results are consistent in the sense that ACL-ARC received the largest boost and CHEMPROT received the smallest boost.
4 Related Work
Like our work, elastic weight consolidation (EWC) [28] uses the Laplace approximation to the posterior over model parameters to create a regularizer to prevent catastrophic forgetting in the context of continual learning. While their framework supports the use of posteriors from multiple
8

---

Riemannian manifold Fisher merging
90
Standard training
Method
ChemProt
ACL-ARC
SciERC
Accuracy
Unmerged
82.To.3
70.53.2
81.00.4
85
Fisher
83.1o.4
73.21.7
81.30.5
Isotropic
82.8o.4
72.52.3
81.70.5
Fine-tuned
82.5p.1
71.53.0
81.61.0
80
Table 1: Domain adaptation results. "Unmerged" refers to checkpoints fine-tuned from RoBERTa- base. "Fisher" and "Isotropic" refer the result of merging those checkpoints with the domain- adaptive pre-trained (DAPT) checkpoint. "Fine- tuned" refers to models fine-tuned from the DAPT checkpoint. Subscripts provide the stan- dard deviation across five trials.
75
SST-2
MRPC
STS-B Donor task
RTE
Figure 6: Validation accuracy on RTE using the setup of fig. 5, but with RoBERTa-large instead of BERT-base. "Standard training" fine-tunes MNLI, then the donor task, then RTE. Dashed line denotes MNLI intermediate-task training.
models as well, they restrict such models to be previous checkpoints of continually trained model. EWC keeps the model from losing previously acquired knowledge while merging provides a means of directly adding new knowledge to a model.
Some other existing procedures such distillation [22] and ensembling [42] also be thought of as combining or transferring knowledge between neural networks. However, those methods represent knowledge solely through the output of models. The knowledge contained within the parameters of network will necessarily be greater than the knowledge contained in its output [2]. Hence, methods that directly combine model parameters such merging have the potential be more powerful than those methods. Furthermore, our merging procedure has an efficient and closed-form solution (eq. (4)) while distillation requires iterative gradient descent-baad based training.
Isotropic checkpoint averaging is used by federated learning [39] and Polyak averaging [49]. However, the checkpoints merged by those methods can be thought of coming from the same training run of single model. We believe we are the first to demonstrate cross-task transfer coming from checkpoint averaging and to explore it in the context of transfer learning. However, adapting ideas from federated learning such as [32, 63] could provide a fruitful avenue for future model merging research.
Natural gradient descent refers to an optimization procedure that uses KL-divergence of model predictions as distance measure rather than the Euclidean distance in parameter space employed by regular gradient descent [3]. It does this by performing gradient descent with the Fisher information matrix as its metric [45]. In practice, this amounts to using the Fisher as a preconditioner during gradient descent. Some work on natural gradient descent may prove relevant for model merging such as using Kronecker-factorrized factorized Fisher matrices as an alternative to the diagonal approximation employed in this paper [37, 18, 38). More broadly, in the field of information geometry the Fisher information matrix plays the role of a metric on a Riemannian manifold [40]. This has led explorations of model averaging using tools from information geometry, e.g. [5, 41, 50].
5 Conclusion
In this paper, we introduced Fisher merging, a way to combine the capabilities of different models by computing weighted average of their parameters. Fisher merging motivated by a novel perspective of parameter averaging as maximizing the joint likelihood of model posteriors. Through extensive experiments, we demonstrated that using the Fisher information as weight on the contribution of each parameter outperforms using an unweighted average. Furthermore, we showed that Fisher merging attains comparable and sometimes better performance than traditional gradient- based transfer learning methods at significantly lower costs. Our experiments also demonstrated various merging strategies that would onerous with traditional gradient-based training, which opens up new avenues for transferring capabilities across models. In future work, we plan to investigate different methods for approximating the Fisher information and model posteriors as well as more esoteric combinations of models.

---

Acknowledgments and Disclosure of Funding
This work was supported by the NSF CAREER award under Grant No. 2145822.
References
[1] Achille, A., Lam, M.. Tewari, R., Ravichandran, A., Maji, S., Fowlkes, C. C. Soatto, S., and Perona, Task2vec: Task embedding for meta-learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6430-6439, 2019.
[2] Achille, A.. Paolini, G., and Soatto, S. Where is the information in deep neural network? arXiv preprini urXiv: 1e0.//90512.222 /2113, 2019.
[3] Amari, S. Neural learning in structured parameter spaces-natural riemannian gradient. Advances in neural information processing systems, pp. 127-133, 1997.
[4] Barbu, A., Mayo, D., Alverio, J., Luo, W., Wang. C., Gutfreund, D., Tenenbaum, J., and Katz, B. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances Neural Information Processing Systems (NeurIPS), 2019.
[5] Bartels, L. M. Specification uncertainty and model averaging. American Journal of Political Science, 1997.
[6] Bowman, R., Angeli, G., Potts, C., and Mamning, C. D. large annotated corpus for learning natural language inference. arXiv preprint arXiv: 05326, 2015.
[7] Cer, D., Diab, M., Agirre, E., Lopez- Gazpioo I., and Specia, L. Semeval-2017 task 1: Se- mantic textual similarity-multilian multilingual and cross-lingual focused evaluation. arXiv preprint arXiv: 00055, 2017.
[8] Dagan, I., Glickman, O.,, and Magnini, B. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop. pp. 177-190. Springer, 2005.
[9] Dai. A. M. and Le, Q. V. Semi-supervised sequence learning. In Advances in neural information processing systems, 2015.
[10] Daxberger, E., Kristiadi, A., Immer, A., Eschenhagen, R., Bauer, M., and Hennig, P. Laplace redux-effortless. bayesian deep learning. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[11] Deng, J., Dong, W., Socher, R., Li, L.JJ, K., and Fei-Fei, L. ImageNet: A large-scale hierar- chical image database. In 2009 IEEE conference computer vision and pattern recognition. 2009.
[12] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 2018.
[13] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1e:::180004000 00.0000, 2018.
[14] Dolan, W. B. and Brockett, C. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (/WP2005), 2005.
[15] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D,, Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint 2rXiv:2010.11922 2020.
[16] Fisher, R. A. On the mathematical foundations of theoretical statistics. Philosophical Trans- actions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 222(594-604):399-36 1922.
10

---

[17] Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., and Bengio. Y. An empirical investigation of catastrophic forgetting in gradient-basee based neural networks. arXiv preprint arXiv:1312.6211, 2013.
[18] Grosse, R. and Martens, A kronecker-factored approximate fisher matrix for convolution layers. In Internatinal Conference on Machine Learning, pp. 573-582. PMLR, 2016.
[19] Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy. I., Downey, D., and Smith, N. A. Don't 't stop pretraining: Adapt language models to domains and tasks. arXiv preprint 10994,, 2020.
[20] Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., and Gilmer, J. The many faces of robustness: A critical analysis of out-of-distribution generalization. International Conference on Computer Vision (ICCV), 2021.
[21] Hendrycks, D., Zhao, K.. Basart, S., Steinhardt, J., and Song. D. Natural adversarial examples. Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[22] Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in neural network. arXiv preprint 1503:00500 2015.
[23] Iyer, S., Dandekar, N., and Csernai, K. First quora dataset release: Question pairs, 2017. URL quora. cttaaiistt/ss//eeeeeeeeeee Conn/Firsttuuora-Daeie Daaira-Dataeet--ee Release -wwettueeeeaaaiieeeeeeeee
https://daae quora
[24] Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and
Darrell, T. Caffe: Convolutional architecture for fast feature embedding. Proceedings of the 22nd ACM international conference on Multimedia, 2014.
[25] Jurgens, D., Kumar, S., Hoover, R., McFarland, D., and Jurafsky, D. Measuring the evolution of a scientific field through citation frames. Transactions of the Association for Computational Linguistics, 6:391-406, 2018.
[26] Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Rad- ford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08360 2020.
[27] Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
[28] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T.,
A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13)33521-33 1-356,, 2017.
[29] Koeijan, V., Cretu, A.-M., Camburu, O.-M., Yordanov, Y., and Lukasiewicz, T. A surprisingly robust trick for winograd schema challenge. arXiv preprint arXiv: 119905066900 2019.
[30] Kringelum, J., Kjaerulff, S. K., Brunak. S., Lund, O., Oprea, T. L., and Tabourcau, O. Chemprot- 3.0: a global chemical biology diseases mapping. Database, 2016, 2016.
[31] Levesque, H., Davis, E.,, and Morgenstern, I.. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning, 2012.
[32] Liu, X., Masana, M., Herranz, L., Van de Weijer, J., Lopez, A. M., and Bagdanov, A. D. Rotate your networks: Better weight consolidation and less catastrophic forgetting. In 2018 24th International Conference on Puttern Recognition (ICPR), pp. 2262-2268. IEEE, 2018.
[33] Liu. Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O.,, Lewis. M., Zettlemoyer, L., and Stoyanov, V. Roberta: robustly optimized hert pretraining approach. arXiv preprins 1692, 2019.
[34] Lo, K., Wang, I.. I.., Neumann, M., Kinney, R., and Weld, D. S. S2orc: The semantic scholar open research corpus. arXiv preprint arXiv:1911.02782, 2019.
11

---

[35] Luan, Y., He, L., Ostendorf, M., and Hajishirzi, H. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. arXiv preprint www..098889.. 2018.
[36] MacKay, D. J. A practical bayesian framework for backpropagation networks. Neural compula- lion, 4(3):448-47) 1992.
[37] Martens, and Grosse, R. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, pp. 2408-2417. PMLR, 2015.
[38] Martens, J., Ba, J., and Johnson, M. frattoedrrfacttored curvature approximations for recurrent neural networks. In International Conference on Learning Representations, 2018.
[39] McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. Communication efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pp. 1273-1282. PMLR, 2017.
[40] Nielsen, F. An elementary introduction to information geometry. Entropy, 22(10):1100, 2020.
[41] Nielsen, F. and Nock, R. Sided and symmetrized bregman centroids. IEEE transactions on Information Theory, 55(6), 2009.
[42] Opitz, D. W. and Shavlik, J. W. Actively searching for an effective neural network ensemble. Connection Science, 8(3-4):337-354 1996.
[43] Oquab, M., Bottou, L., Laptev, I., and Sivic, J. Learning and transferring mid-level image representations using convolutional neural networks. Proceedings of the IEEE conference on computer vision and pattern recognition, 2014.
[44] Pan, S. J. and Yang. Q. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345 1359, 2009.
[45] Pascanu, and Bengio, Y. Revisiting natural gradient for deep networks. arXiv preprint 1301v:1301.330 2013.
[46] Peters, M. E., Neumann, M., lyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. Deep contextualized word representations. arXiv preprint arXiv: 1802.05365, 2018.
[47] Phang, J., Févry, T., and Bowman, R. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint 1811::811.001000 2018.
[48] Phang, J., Htut, P. M., Pruksachatkun, Y., Liu, H., Vania, C., Kann, K., Calixto, I., and Bowman,
S. R. English intermediate-aane training improves zero-shot cross-lingual transfer too. arXiv preprint 2rXiv:2005.13013. 2020.
[49] Polyak, B. T. and Juditsky, A. B. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838-855, 1992.
[50] Posada, D. and Buckley, T. R. Model selection and model averaging in phylogenetics: ad- vantages of akaike information criterion and bayesian approaches over likelihood ratio tests. Systematic biology, 53(5), 2004.
[51] Pruksachatkun, Y., Phang. J., Liu, H., Htut, P. M., Zhang, X., Pang, R. Y., Vania, C., Kann, K., and Bowman, S. R. Intermediate-tur task transfer learning with pretrained models for natural language understanding: When and why does it work? arXiv preprint arXiv:2005.0m 00628, 2020.
[52] Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre-training, 2018
[53] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, J. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv preprint arXiv: 1910. 10683, 2019.
[54] Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, Squad: 100,000+ questions for machine comprehension of text. arXiv preprint a6Xiv: 05250, 2016.
12

---

[55] Rajpurkar, P., Jia, R., and Liang, P. Know what you don't 't know: Unanswerable questions for squad. arXiv preprint urXiv:1806.03822. 2018.
[56] Recht, B., Roelofs, R., Schmidt,, L., and Shankar, Do ImageNet classifiers generalize to ImageNet? In International Conference on Machine Learning (ICML), 2019.
[57] Ruder, S., Peters, M. E., Swayamdipta, S., and Wolf, T. Transfer learning in natural language processing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Tutorials, pp. 15-18, 2019.
[58] Russakovsky, O., Deng, J., Su. H., Krause, J., Satheesh, S., Ma. S., Huang, Z., Karpathy, A., Khosla, A., Bemsstein, M., et al. ImageNet large scale visual recognition challenge. International journal of computer vision, 2015.
[59] Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., A. Y., and Pous, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631- 2013.
[60] Vu, T. Wang, T., Munkhdalai, T., Sordoni, A., Trischler, A., Mattarella-Micke, A., Maji, S., and Iyyer, M. Exploring and predicting transferability across nlp tasks. arXiv preprint 2xXi:2000.00070. 2020.
[61] Wang. A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Glue: A multi- task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.0744 2018.
[62] Wang, H., Ge, S., Lipton, Z., and Xing, E. P. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[63] Wang, H., Yurochkin, M., Sun, Y., Papailiopoulos, D., and Khazaeni, Y. Federated learning with matched averaging. arXiv preprint arXiv:2002.06444) 2020.
[64] Warstadt, A., Singh, A., and Bowman, S. R. Neural network acceptability judgments. Transac- tions of the Association for Computational Linguistics, 7:625-641, 2019.
[65] Wolf, T., Debut, L.,, Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T. Louf, R., Funtowicz, M., et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint
2019.
[66] Wortsman, M., Ilharco, G., Li, M., Kim, J. W., Hajishirzi, H., Farhadi, A., Namkoong, H., and Schmidt, L. Robust fine-tuning of zero-shot models. arXiv preprint arXiv:2 2021.
[67] Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., and Schmidt, L. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. arXiv preprini 220::2203.054:: 2022.
[68] Yosinski, J., Clune, J., Bengio, Y., and Lipson, H. How transferable are features in deep neural networks? In Advances in neural information processing systems, 2014.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Settings where merging under- performs traditional gradient-basse training are covered in section 3 and also discuss caveats of our method in section 2.4.
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
13

---

2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We state that merging assumes shared architecture and initialization various points in the paper and explain why in section 2.4.
(b) Did you include complete proofs of all theoretical results? [N/A] No theoretical advances were sufficiently complex to warrant proof.
3. If you ran experiments..
(a) Did you include the code. data, and instructions needed reproduce the main perimental results (either the supplemental material or as a URL)? [Yes] Code is provided in the supplementary. All datasets are public and are downloadable with
code.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See appendix C and appendix
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See tables Al to A3.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [No] All datasets are widely-used (hundreds
or thousands of citations) public datasets.
(c) Did you include any assets either in the supplemental material URL? [Yes] We include code in the supplemental.
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "xH74EfETOf3", "reviewer": "Reviewer_dofC", "summary": "This paper explore \"merging\" models, i.e. combining the parameters of two or more models into a single model. It advances an increasing popular research direction in averaging parameters of models that share the same architecture and initialization. Their main contribution is to go beyond the way parameters are typically averaged, by having individual weights for each of the parameters. For such, the authors advocate for finding a model with the most likely set of parameters, considering the posteriors of all models being averaged. Their method is based on approximating each model's posterior as a Gaussian distribution whose precision matrix corresponds to its Fisher information. This then leads to a closed-form solution for which weights to use for each parameter when averaging multiple models. Authors additionally have global hyper-parameters which can be tuned to control the importance of each model when averaging. The authors present a series of experiments in both natural language processing and computer vision, contrasting their Fisher-weighted averaging with previous work where simple averaging is used (i.e., without any parameter-specific weights). The authors also explore new settings for combining models, in addition to existing ways like robust fine-tuning or model soups. Code  for replicating the experiments will be publicly released.\nOverall, this paper presents a solid advance in our understanding and methods for combining models. Considering that pre-training is becoming increasingly expensive and intractable to repeat, this line of work is very timely. I believe this paper would be of interest to many in the community, and I recommend its acceptance. ", "questions": "Have the authors explored larger language models? I believe this line of work is most attractive when pre-training is prohibitively expensive (if pre-training is cheap, why not just combine the datasets and pre-train again?). It would also be interesting to show that this can work with parameter-efficient fine-tuning methods like adapters.", "limitations": "The authors discuss some limitations of their method and how to address them (for instance, numerical issues and unmergeable parameters). As previously mentioned in the Strengths and Weaknesses section, I believe better understanding of when Fisher merging provides large improvements over Isotropic merging would greatly strengthen the paper. ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1) The research direction this paper explores is timely, and I believe would be of interest to many.\n\n2) The ideas presented in this work are novel, and it departs significantly from previous work.\n\n3) The experiments presented are comprehensive and solid. Compared to standard (isotropic) merging, Fisher merging provides improvements in most cases, sometimes by large amounts. Moreover, their method does not incur in any extra compute at inference time compared to the models being merged. Finally, the fact that the the authors conduct experiments both in natural language processing and computer vision showcases the generality of their findings.\n\n4) I found the experiments presented in Section 3.1 to be particularly interesting, since authors combine models with external checkpoints, created independently by multiple people. Considering model zoos like the Hugging Face model hub are growingly rapidly, this experiment is a great proof of concept of a future were we take better advantage of such resources.\n\n5) The paper provides a great review of related work, and contextualizes itself and its contributions well with respect to recent literature.\n\n**Weaknesses**\n\n1) In some cases, the improvements gained via Fisher merging (compared to Isotropic merging, which has been previously explored) are small or even inexistent. Better understanding *when* Fisher merging provides larger gains compared to Isotropic merging would greatly strengthen the paper.\n\n2) Compared to Isotropic merging, Fisher merging can require extra compute. In section 3.3, the authors claim that merging is more computationally efficient than fine-tuning, but this only holds true when assuming that all checkpoints being merged are already available. While this can be true for many datasets, there are scenarios where it does not hold, for instance, for recently released, less popular or private tasks. Moreover, the proposed procedure might be infeasible in some cases: for instance, consider fine-tuning a CLIP model on ImageNet, as in Sec. 3.2. CLIP models are public, but their training data is not. Given that the authors advocate for using data from the dataset used to train the model for computing the Fisher matrix (L. 141), how can this be done if it is not public? If other datasets can be used for computing the Fisher matrix, which ones to chose? Does the choice matter? Further exploring and discussing this would strengthen the paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fNQbWIuhnE", "reviewer": "Reviewer_c1b6", "summary": "This paper proposes a method for casting the problem of averaging parameters from multiple models with shared architecture as seeking a MAP estimate of the joint posterior with Laplace approximation for individual trained models.\nEmpirical experiments on natural language processing datasets are presented, showing its effectiveness on model ensembling, fine-tuning, and domain adaptation tasks.", "questions": "- For neural networks with batch normalization, and spectral normalization, how would the parameters be averaged in the proposed method?\n- How would the parameter uncertainty be addressed in the proposed method? ", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n#### Strengths\n- The method proposed in this work is conceptually simple and intuitive. The paper is generally well-written and easy to follow.\n- The empirical evaluation covers a wide range of tasks and metrics, including both task-specific performance and computational efficiency.\n\n#### Weaknesses\n- While I understand the point that the authors aim to propose an alternative to the plain parameter averaging approach, the introduced method is still seeking a point estimation for the joint posterior which is based on the Laplace approximation. Thus the issues of the Laplace approximation would also exist in the proposed approach, and it is hard to tell how the uncertainty in the final estimation is addressed given the results included.\n- Quite a few experimental results are presented, however, only on natural language datasets and tasks. As a general parameter averaging approach, it would be beneficial to consider other data modalities as well.\n- As discussed in the first point about the limitations of the Laplace approximation, for efficient estimation of the Fisher matrix, diagonal approximation is made which leaves the structure of the task ignored.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EarQILFwDze", "reviewer": "Reviewer_PTzq", "summary": "The paper studies the problem of how to effectively average existing deep learning models, which has broad application background in transfer learning. The authors proposed to derive a **geometric centroid** of the given models based on information geometry. Based on certain simplifications (such as diagonal Fisher information matrix, FIM), the resulting model is in closed form based on the FIM in eq.(4). On real-world transfer learning experiments, the authors showed that the proposed method is better than empirical averaging.", "questions": "Sec. 2. related expressions regarding the posterior can be better formulated and clearly written. For example, why $\\theta^{\\star}=\\sum_{i}\\log{}p(\\theta|\\theta_i,I)$? and why $p(\\theta|\\theta_i,I)$ is the posterior? What is the associated prior? These formulations can be improved.\n\nThe presented work is a simple application of Fisher information into modern machines. Most of the cited papers are from recent deep learning communities. The idea of using information geometry for model averaging is not new. Related literature where similar ideas were known and used should be better acknowledged.  For example, see  [1][2][3].\n\n[1] \"Specification Uncertainty and Model Averaging\", Larry M. Bartels, 1997.\n\n[2] \"Sided and Symmetrized Bregman Centroids\", Nielsen & Nock, 2009.\n\n[3] \"Model Selection and Model Averaging in Phylogenetics: Advantages of Akaike Information Criterion and Bayesian Approaches Over Likelihood Ratio Tests\", David Posada, Thomas R. Buckley, 2004.\n\nThere should be more discussion on why it works better than the empirical averaging approach. Instead, the authors introduced the method \"as is\" without sufficient theoretical arguments or motivation.", "limitations": "The paper is theoretical and does not have a negative societal impact as far as I can tell.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n- The proposed method is efficient based on a diagonal approximation of the Fisher matrix (as in Adam).\n- Strong experimental results with extensive study in modern deep learning models.\n\nCons:\n\n- The paper can be regarded as an application of information geometry into deep learning. The theoretical depth can be improved to explain why it works well.\n- Some historical literature on model selection and information geometry can be more carefully investigated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DDrJJbBGLLr", "reviewer": "Reviewer_XQHE", "summary": "Transfer learning is ubiquitous in deep learning methods, and gradient-based transfer learning is usually dominant in these areas where a pre-trained network in fine-tuned for a task. The issue with this approach is that after fine-tuning the network becomes better at the current task at hand but might 'forget' the previous task. To alleviate this issue the authors (along with others) suggest taking the 'average' of the two networks (or more networks if other fine-tuned networks are available). This approach combines the strengths of multiple models such that they retain their performance on their respective individual tasks. The authors extend this approach by considering the weighted mean of the different models (at an individual parameter level) and guide the weighing of the parameters using their respective information matrices.", "questions": "Can the following two sentences be clarified a bit more?\n\n- \"Since the Fisher information is a local property of a single parameter value, we limit our focus to models that were trained from the same initialization.\" Do you refer to \"same pre-trained model\" being fine-tuned multiple times? Why is this relevant if the parameter spaces are exactly the same?\n\n- \"a small Fisher value ultimately means that the parameter is relatively unimportant to the model’s behavior.\" I will guess that a small information will imply that the parameter cannot be estimated reliably from the data, and thus, can be removed from the \"averaging\". Is this a suitable interpretation?\n\nCan Figure 3 be elaborated a bit more:\n\n- Can you show the transition of regularization values (perhaps using different transparency).\n- What are the values that simultaneously\n\nFigure 4 shows that isotropic merging and Fisher merging perform similar to each other on an average, and standard fine-tuning can outperform these approaches on some occasions. Can\nthe authors provide some guidelines on when one of these three approaches should be considered?\n~                                                                                                ", "limitations": "Some limitation of the method, e.g., numerical issues have been addressed in the paper. It will be great to address situations where Fisher merging performs better than isotropic merging and vice versa.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper tackles an interesting and relevant problem.\n- The paper presents significant empirical evidence.\n\nWeaknesses\n- The idea presented is straightforward and the main contribution of the paper is empirical results corroborating the proposed framework, and thus, lacks theoretical arguments.\n- Some intuitions and sentences in the paper are not obvious.\n- The arguments made in the paper are a bit difficult to follow.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6u6xrhW0hJ", "reviewer": "Reviewer_Dr1z", "summary": "The paper proposes a novel method for merging (pre-trained) models with the same architecture and initialisation set up. The merging strategy for obtaining the new model based on the previous ones is based on the Laplace approximation, and particularly, on the Fisher information matrix. In the manuscript, the advantages of merging via Fisher information matrix is illustrated within simpler strategies based on isotropic Gaussian approximations or just output ensembles. Experimental results using (very) large models -- i.e. BERT, show a significant performance while at the same time being simple and x5-x6 cheaper than other approaches.", "questions": "Some additional questions that I would appreciate to be answered.\n\n- *Q1.* If I understand correctly, once each $m$-th model is trained, we are able to compute the Fished matrix in a *posthoc* manner. Once F\n Is computed/approximated, we can use it for training the new model merging. Does this last merging re-visit old data? Or F is assumed to be fixed and known without the need of re-accessing $x$ or $y$?\n\n- *Q2.* This is just a curiosity, but as far as I know, BERT models use self-attention layers in their architecture. Do these type of layers/operations introduce unmergeable parameters? In other recent Laplace papers, some type of layers could not be considered due to the difficulties for computing Hessians.\n\n- *Q3.* How many models are being merged in the experiments? And how large is the space of parameters? Could the authors provide some numbers on the validation set accuracy of the individual pre-trained models? \n\n- *Q4.* Are convolutional layers considered also as parameters to be merged?\n", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths.**\n\nThe idea of merging models to transfer capabilities and knowledge to new models without the need of exhaustive re-training from the same data is sound and novel. The paper positively illustrates the problem considered and related work is well reviewed in my opinion. While the methodology section could be slightly improved (see weaknesses), the experimental section is strong, mainly due to the consideration of big models or other neural network (NN) based architectures in the results. While being simple, performance seems promising. In general, I had not seen results on merging such large models like BERT or relative ones. Also details on FLOPs or computational cost saved from merging is a good point for the community.\n\n**Weaknesses.**\nIn my opinion, there are two small points of weakness:\n\n- *Section 2:* The methodology described in Sec. 2 for the isotropic + Fisher merging could be improved, or at least connect better with probabilistic modelling. For me, was not difficult to follow and key details are included, but perhaps some intermediate details or equations are missing. For instance, equation in L102 introduces densities $p(\\theta|\\theta_i, I)$, not being 100% clear enough where they come from. Similarly, in L129-L132, Fisher matrix is introduced within data $x,y$ which are not previously defined. \n\n- *Laplace Approximation:* There is a recent interest in the community on Laplace approximation and Hessian-based methods, mainly due to their utility for providing alternatives to Bayesian NNs (approximating posterior on the parameters). Even being a different goal, I find the paper somehow connected, and some references could be perhaps included (Laplace Redux, + previous and follow-up approaches). Additionally, I am a bit surprised about the lack of difficulties for computing the Fisher information matrix (even being diagonalised) in such enormous architectures.. I would appreciate if authors could elaborate a bit more on this and the computational cost.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept