Rethinking Layer-wise Model Merging through Chain of Merges

## Abstract

Abstract Fine-tuning pretrained models has become a standard pathway to achieve state-of-the-art performance across a wide range of domains, leading to a proliferation of task-specific model variants. As the number of such specialized models increases, merging them into a unified model without retraining has become a critical challenge. Existing merging techniques operate at the level of individual layers, thereby overlooking the inter-layer dependencies inherent in deep networks. We show that this simplification leads to distributional mismatches, particularly in methods that rely on intermediate activations, as changes in early layers are not properly propagated to downstream layers during merging. We identify these mismatches as a form of internal covariate shift , comparable to the phenomenon encountered in the initial phases of neural networks training. To address this, we propose Chain of Merges (CoM), a layer-wise merging procedure that sequentially merges weights across layers while sequentially updating activation statistics. By explicitly accounting for inter-layer interactions, CoM mitigates covariate shift and produces a coherent merged model through a series of conditionally optimal updates. Experiments on standard benchmarks demonstrate that CoM achieves state-of-the-art performance.

## 1 Introduction

The availability of large-scale pretrained models has reshaped machine learning (Radford et al., 2021; Touvron et al., 2023), with fine-tuning emerging as the most accessible path to obtaining state-of-the-art performance across diverse domains (Raffel et al., 2020; Wang et al., 2019). As these foundation models are increasingly adapted to specialized tasks and datasets, a natural question arises: how can we combine task-specific checkpoints without retraining? This challenge, broadly referred to as model merging (Ilharco et al., 2023; Matena & Raffel, 2022; Wortsman et al., 2022a), has recently proven effective for achieving modularity, knowledge reuse, and efficient deployment.

Since specialized modules are typically trained independently, there is no guarantee that their weights can be seamlessly combined (Yadav et al., 2023; Stoica et al., 2023). In practice, naive strategies such as weight averaging (McMahan et al., 2017; Wortsman et al., 2022a) often lead to strong performance degradation when combining heterogeneous models (Tang et al., 2024; Daheim et al., 2024; Tam et al., 2024). To tackle this challenge, the literature has proposed a wide range of heuristics, spanning techniques that mitigate interference (Yadav et al., 2023; Yu et al., 2024), align parameters via permutation-based matching (Ainsworth et al., 2023; Singh & Jaggi, 2020), preserve important weights (Matena & Raffel, 2022; Lee et al., 2025), and perform interpolation within orthonormal or task-adaptive subspaces (Marczak et al., 2025; Gargiulo et al., 2025; Tam et al., 2024). While these approaches achieve decent performance, they typically rely on problem-specific assumptions and extended hyperparameter search, lacking a unifying theoretical foundation.

A different line of work focuses on aligning model activations at the layer level (Stoica et al., 2023; Jin et al., 2023; Tatro et al., 2020; Jordan et al., 2022), typically by permuting or modifying parameters to facilitate compatibility. While these approaches lay a foundation for more principled model composition, they overlook a key challenge: layers in deep networks are not independent, but conditioned on the outputs of preceding computation. Merging them independently can introduce inconsistencies across the network. In fact, modifying early-layer parameters through merging can shift the distribution of their output activations, resulting in unexpected inputs for downstream layers. This triggers a butterfly effect, where even small discrepancies accumulate as they propagate through the network, leading to escalating mismatches and consequent performance degradation.

We identify this issue as a form of internal covariate shift (ICS) (Ioffe & Szegedy, 2015), a well-known problem in training dynamics where rapidly shifting early-layer activations produce unstable output distributions that hinder downstream learning (Arpit et al., 2016). Analogously, we refer to its manifestation in model merging as merging covariate shift (MCS), which occurs when an early layer is altered through merging, causing abrupt shifts in the inputs to subsequent layers.
While merging covariate shift may be tolerable for methods operating entirely in parameter space – since they adjust weights directly without depending on intermediate activations – it becomes a critical issue for methods that rely on activation statistics. These approaches rely on layer inputs, which inevitably change when preceding layers are merged. Yet, they merge all layers simultaneously using activations computed before merging, failing to account for the resulting distributional shifts and consequently undermining performance.

To address this challenge, we propose Chain of Merges (CoM): a recursive merging approach that begins at the input layer and iteratively updates parameters until reaching the last one. Specifically, we propose updating activation statistics after each merging step, replacing the original task-specific activations with those produced by the partially merged model. This process explicitly captures inter-layer dependencies and ensures global consistency, providing a framework applicable to any activation-based merging methodology. Building on this, we follow Jin et al. (2023) and cast parameter merging as a layer-wise distillation problem, where the merged weights are optimized to replicate the activation distributions of the original task-specific modules. This problem admits a closed-form solution for linear layers, which constitute a substantial portion of transformer-based architectures and are typically the only layers optimized during fine-tuning (e.g., LoRA-style adaptation keeps all other weights fixed (Hu et al., 2022)).

Our main contributions can be summarized as follows:

- •
We identify and analyze the presence of internal covariate shift, empirically showing that activation mismatches accumulate across layers.
- •
We introduce Chain of Merges (CoM), a novel merging technique that progressively distills parameters by updating activation statistics, ensuring consistency across the network.
- •
We evaluate CoM on standard benchmarks and show it consistently outperforms existing methods, achieving state-of-the-art performance across diverse language and vision settings.

## 2 Background and Motivation

Model merging aims to combine a collection of $N$ models, all sharing an identical architecture, independently trained on distinct input datasets. Each model comprises $L$ linear layers, which are the target of the merging procedure. For a given layer $l\in\{1,2,\dots,L\}$, the set of corresponding weight matrices is denoted as $\{{\bm{W}}_{i}^{l}\}_{i=1}^{N}$, all having the same dimensions.

RegMean (Jin et al., 2023) proposes to find a single linear transformation, ${\bm{W}}_{M}^{l}$, that best approximates the behavior of the original layers when applied to their respective inputs $\{{\bm{X}}_{i}^{l}\}_{i=1}^{N}$. This is accomplished by minimizing the following objective function:

$$ $\operatorname*{minimize}~\Omega^{l}=\sum_{i=1}^{N}{\left\|{\bm{W}}^{l}_{M}{\bm{X}}_{i}^{l}-{\bm{W}}_{i}^{l}{\bm{X}}_{i}^{l}\right\|^{2}_{2}}.$ (1) $$

By differentiating $\Omega^{l}$ with respect to ${\bm{W}}_{M}^{l}$ and setting the result to zero, we can obtain a closed-form solution for the optimal merged layer:

$$ ${\bm{W}}_{M}^{l}=\left(\sum_{i=1}^{N}{\bm{W}}_{i}^{l}{\bm{X}}_{i}^{l}{{\bm{X}}_{i}^{l}}^{\top}\right)\left(\sum_{i=1}^{N}{\bm{X}}_{i}^{l}{{\bm{X}}_{i}^{l}}^{\top}\right)^{-1}.$ (2) $$

Here, each $\mathbf{X}_{i}^{l}\in\mathbb{R}^{d\times\text{samples}}$ is the input data to the $l^{\text{th}}$ layer of the $i^{\text{th}}$ model, and the corresponding Gram matrix ${\bm{X}}_{i}^{l}{{\bm{X}}_{i}^{l}}^{\top}\in\mathbb{R}^{d\times d}$ captures the pairwise correlations between individual examples.

## 3 Methodology

#### Merging Covariate Shift.

When using [Equation 2](https://arxiv.org/html/2508.21421v2#S2.E2), the resulting ${\bm{W}}_{M}^{l}$ closely matches the outputs of the original task-specific layers in isolation. However, merging all layers simultaneously based on the initial inputs overlooks the dependencies between successive layers. Specifically, the inputs ${\bm{X}}_{i}^{l}$ to the $l^{\text{th}}$ layer in [Equation 2](https://arxiv.org/html/2508.21421v2#S2.E2) correspond to the activations of the $(l-1)^{\text{th}}$ layer. Once the original parameters ${\bm{W}}_{i}^{l-1}$ are replaced with their merged counterparts ${\bm{W}}_{M}^{l-1}$, these activations shift accordingly. As a result, the statistics used to merge layer $l$ no longer align with the distribution actually produced after layer $(l-1)$ has been merged. This mismatch induces a shift in activation statistics, analogous to internal covariate shift, which we refer to as merging covariate shift (MCS).

### 3.1 Chain of Merges

#### Recursive Dependence.

To address MCS, we revise the closed-form solution presented in [Equation 2](https://arxiv.org/html/2508.21421v2#S2.E2). Instead of relying on the inputs ${\bm{X}}_{i}^{l}$ – that is, the activations produced by the preceding unmerged layer – we employ the activations produced by the preceding merged layer, $\hat{{\bm{X}}}_{i}^{l}$. These represent the actual inputs received by layer $l$ during inference, once all preceding layers have been merged. Formally, we define the pre- and post-merging inputs to layer $l$ of model $i$ as:

$$ ${\bm{X}}_{i}^{l}\;=\;\bm{\sigma}^{l-1}\left({\bm{W}}_{i}^{l-1}{\bm{X}}_{i}^{l-1}\right),\qquad\text{and}\qquad\hat{{\bm{X}}}_{i}^{l}\;=\;\bm{\sigma}^{l-1}\left({\bm{W}}_{M}^{l-1}\hat{\bm{X}}_{i}^{l-1}\right),$ (3) $$

where $\bm{\sigma}^{l}$ denotes the (possibly composite) activation function connecting layers $l$ and $(l+1)$. Substituting ${\bm{X}}_{i}^{l}$ for $\hat{{\bm{X}}}_{i}^{l}$ yields the revised expression for the merged weights:

$$ ${\bm{W}}_{M}^{l}=\left(\sum_{i=1}^{N}{\bm{W}}_{i}^{l}\hat{{\bm{X}}}_{i}^{l}\hat{{\bm{X}}}_{i}^{l\vskip-28.45274pt\mkern-0.5mu\raisebox{1.65764pt}{$\scriptstyle\top$}}\right)\left(\sum_{i=1}^{N}\hat{{\bm{X}}}_{i}^{l}\hat{{\bm{X}}}_{i}^{l\vskip-28.45274pt\mkern-0.5mu\raisebox{1.65764pt}{$\scriptstyle\top$}}\right)^{-1}.$ (4) $$

This substitution induces a recursive dependence: the inputs to layer $l$ now depend on the outputs of layer $(l-1)$, which are themselves computed using the merged weights ${\bm{W}}_{M}^{l-1}$ and the inputs to layer $(l-1)$. In turn, these depend on the merged outputs of layer $(l-2)$, and so forth. As a result, at every preceding layer all unmerged activations and parameters must be replaced by their merged counterparts, propagating the correction backward through the entire network.

#### Initial step.

The recursive chain starts at the point where either the weights or the inputs are fixed. Such a base case occurs at the first layer, where the inputs correspond to raw data and are unaffected by prior merging. Consequently, the merged weights can be directly computed using [Equation 2](https://arxiv.org/html/2508.21421v2#S2.E2) as:

$$ ${\bm{W}}_{M}^{1}=\Bigl(\sum_{i=1}^{N}{\bm{W}}_{i}^{1}{\bm{X}}_{i}^{1}{{\bm{X}}_{i}^{1}}^{\!\top}\Bigr)\Bigl(\sum_{i=1}^{N}{\bm{X}}_{i}^{1}{{\bm{X}}_{i}^{1}}^{\!\top}\Bigr)^{-1}.\vskip-5.0pt$ (5) $$

#### Recursive step.

The merged weights from the initial step are used to propagate consistent activations forward through the network.
For subsequent layers $l=2,\dots,L$, the algorithm proceeds recursively, alternating between computing the activations $\hat{{\bm{X}}}_{i}^{\,l}$ and updating the merged weights ${\bm{W}}_{M}^{\,l}$ according to [Equation 4](https://arxiv.org/html/2508.21421v2#S3.E4). By ensuring that the computed statistics at each stage reflect the accumulated effect of all preceding merges, this auto-regressive scheme fully mitigates merging covariate shift throughout the network. Importantly, this recursive formulation incurs no extra cost beyond the computation required for [Equation 4](https://arxiv.org/html/2508.21421v2#S3.E4), as discussed in [appendix C](https://arxiv.org/html/2508.21421v2#A3).

### 3.2 Gram‑based importance

Although the merged weight matrix produced by our strategy is optimal with respect to the regression objective, it must be regarded as an approximation rather than an exact reconstruction of the original task-specific weights. This limitation comes from the structural compression of multiple linear transformations into a single one of fixed dimensionality, which necessarily discards some representational capacity. As a result, a residual regression error is inevitable, introducing perturbations into the activations of the merged model across all tasks.

#### Task Importance.

The objective of model merging is to retain the performance across all tasks; however, the relative importance of each task is not uniform. Tasks that are semantically similar to the pretraining naturally benefit from the representations of the base model, as their data

Figure: Algorithm 1 – Chain of Merges

distribution aligns with that seen during pretraining. In contrast, those that are semantically distant cannot rely on pretrained features and face a higher risk of performance degradation after merging. Preserving these tasks is more critical, as they stand to lose the most if their task-specific weights are not adequately incorporated into the merged model. To account for this asymmetry, we assign each task–layer pair an importance weight $\omega^{\,l}_{i}$, which should reflect how strongly the merged weights have to be biased toward preserving task $i$ when merging layer $l$. The key challenge is to derive a layer-wise surrogate measure for this importance without relying on downstream evaluation data.

#### Cosine similarity as a proxy.

To quantify a task’s semantic distance from pretraining, we use the overall cosine similarity of the layer’s pretrained input features ${\bm{X}}_{i}^{l}$, which reflects redundancy in task-specific representations. Highly correlated features align in similar directions, deviating from the isotropic structure of pretraining, and indicating greater task importance as the task is harder to preserve. In contrast, weakly correlated features suggest a distribution that is closer to pretraining, making the task easier to preserve. This is consistent with prior work (Cogswell et al., 2015; Morcos et al., 2018), demonstrating that decorrelated features enhance generalization, and empirically validated in [Section 5](https://arxiv.org/html/2508.21421v2#S5).

In practice, we use the merged model’s features rather than the pretrained ones, as our merging procedure already computes the Gram matrix of each layer’s inputs ${\bm{G}}^{\,l}_{i}$ ([Equation 4](https://arxiv.org/html/2508.21421v2#S3.E4)). Specifically, when constructing ${\bm{G}}^{\,l}_{i}$, we L2-normalize the input features $\hat{{\bm{X}}}_{i}^{l}$ to effectively produce a cosine similarity matrix, and define task importance as the absolute sum of the off-diagonal terms of ${\bm{G}}^{\,l}_{i}$:

$$ $\omega^{\,l}_{i}=\sum_{p<q}\big|({\bm{G}}^{\,l}_{i})_{pq}\big|,\qquad\text{with }{\bm{G}}^{\,l}_{i}=\tilde{{\bm{X}}}_{i}^{l}\tilde{{\bm{X}}}_{i}^{l\mkern-0.5mu\raisebox{1.65764pt}{$\scriptstyle\top$}},\qquad\tilde{{\bm{X}}}_{i}^{l}=\frac{\hat{{\bm{X}}}_{i}^{l}}{\|\hat{{\bm{X}}}_{i}^{l}\|_{2}}.$ (6) $$

This metric measures the overall cosine similarity between intermediate activations. Larger $\omega^{\,l}_{i}$ indicates stronger correlations, meaning the task has diverged more during fine-tuning and is more critical to retain, while smaller $\omega^{\,l}_{i}$ reflects near-orthogonal features, suggesting the task is semantically close to the pretrained model.

#### Activation Normalization.

Our approach requires the inversion of $\sum_{i=1}^{N}\hat{{\bm{X}}}_{i}^{l}\hat{{\bm{X}}}_{i}^{l\mkern-0.5mu\raisebox{1.65764pt}{$\scriptstyle\top$}}$, whose conditioning critically affects the numerical stability of the solution. In Transformer architectures, activation Gram matrices are frequently ill-conditioned due to the inherent low-dimensionality of the underlying token representations (Barbero et al., 2024; Arefin et al., 2024), especially as fine-tuning typically occurs within a low-dimensional subspace (Aghajanyan et al., 2021; Kumar et al., 2022). Drawing on prior work showing that layer normalization (Ba et al., 2016) mitigates representational collapse (Wu et al., 2024), we replace the gram matrix $\hat{{\bm{X}}}_{i}^{l}\hat{{\bm{X}}}_{i}^{l\mkern-0.5mu\raisebox{1.65764pt}{$\scriptstyle\top$}}$ with the cosine similarity matrix ${\bm{G}}^{\,l}_{i}$, which is computed from normalized features and therefore better conditioned. The complete procedure of the proposed methodology is summarized in [Algorithm 1](https://arxiv.org/html/2508.21421v2#alg1).

#### Importance-weighted merging.

Finally, to incorporate task- and layer-specific importance, we extend our objective $\Omega$ ([Equation 1](https://arxiv.org/html/2508.21421v2#S2.E1)) with the weighting factor $\omega^{\,l}_{i}$, biasing the merged layer toward more sensitive tasks. The resulting merging rule for ${\bm{W}}_{M}^{\,l}$ is given by:

$$ $\operatorname*{arg\,min}_{{\bm{W}}_{M}^{\,l}}~\sum_{i=1}^{N}{\omega_{i}^{\,l}\left\|{\bm{W}}^{l}_{M}\tilde{{\bm{X}}}_{i}^{l}-{\bm{W}}_{i}^{l}\tilde{{\bm{X}}}_{i}^{l}\right\|^{2}_{2}}\;=\;\Bigl(\sum_{i=1}^{N}\omega_{i}^{\,l}\,{\bm{W}}_{i}^{\,l}\,{\bm{G}}_{i}^{\,l}\Bigr)\Bigl(\sum_{i=1}^{N}\omega_{i}^{\,l}\,{\bm{G}}_{i}^{\,l}\Bigr)^{-1}.$ (7) $$

The complete procedure of the proposed method is summarized in [Algorithm 1](https://arxiv.org/html/2508.21421v2#alg1), with the full derivation of [Equation 7](https://arxiv.org/html/2508.21421v2#S3.E7) provided in [appendix A](https://arxiv.org/html/2508.21421v2#A1).

## 4 Experimental Study

### 4.1 Evaluation settings

#### Vision experiments.

We evaluate our approach in the vision domain using the benchmark of Ilharco et al. (2023), which involves merging checkpoints from eight classification datasets: Stanford Cars (Krause et al., 2013), DTD (Cimpoi et al., 2014), EuroSAT (Helber et al., 2019), GTSRB (Stallkamp et al., 2011), MNIST (LeCun et al., 2002), RESISC45 (Cheng et al., 2017), SUN397 (Xiao et al., 2016), and SVHN (Netzer et al., 2011).

#### Language experiments.

Following Stoica et al. (2025), we assess model generalization to the language domain on six datasets: SNLI (Bowman et al., 2015), MultiNLI (Williams et al., 2017), SICK (Marelli et al., 2014), SciTail (Khot et al., 2018), RTE (Wang et al., 2019), and QNLI (Wang et al., 2019). In SNLI, MultiNLI, and SICK, the task is to classify the relationship between a premise and a hypothesis as entailment, contradiction, or neutral. SciTail, RTE, and QNLI only involve two labels, so the outputs space during fine-tuning and evaluation is restricted accordingly.

#### Evaluated approaches.

We compare our method with ten leading techniques in the model-merging domain. Task-Arithmetic (TA) (Ilharco et al., 2023), TIES (Yadav et al., 2023), and DARE (Yu et al., 2024) operate by directly merging task vectors in weight space. Improving on them, Consensus TA (Wang et al., 2024) prunes task-specific checkpoints to retain only shared parameters before merging, while LiNeS (Wang et al., 2025) scales parameter updates by layer depth to preserve both general features and task-specific representations. Among SVD-based techniques, KnOTS (Stoica et al., 2025) aligns weights to improve merging, enhancing existing methods like TIES and DARE. Instead, TSV (Gargiulo et al., 2025) compresses the weights and estimates task interference to guide the merging process, while Iso-C (Marczak et al., 2025) perform isotropization, decomposing the weights via SVD and reconstructing them with equal singular values. RegMean (Jin et al., 2023) aligns model parameters by solving a closed-form regression problem across all linear layers, while FisherAVG (Matena & Raffel, 2022) combines models using Fisher Information as importance weights. Finally, MaTS (Tam et al., 2024) extends these two by leveraging conjugate gradient optimization to align models within their respective task-parameter subspaces.

#### Implementation Details.

Consistent with Stoica et al. (2025), the vision-task backbones for all methods examined are variants of the CLIP encoder: specifically, ViT-B/32 and ViT-L/14 (Dosovitskiy et al., 2021). For natural language tasks, we utilize Llama 3-8B (Grattafiori et al., 2024). Each model is fine-tuned using LoRA (Hu et al., 2022), applied solely to attention modules (i.e., query, key, value, and output projection layers) with rank $r=16$. To ensure both reproducibility and fair comparison, we employ the fine-tuned checkpoints provided by Stoica et al. (2025). For NLP tasks we use bfloat16 – shown to generally outperform float16 (Kalamkar et al., 2019) – except during Gram-matrix inversion, where float32 is used to ensure numerical stability. Following the original benchmark, we report the average normalized accuracy of the merged model across all datasets. Since the normalization is performed relative to the fine-tuned performance, values can exceed 100% on some tasks.

For constructing the Gram matrices, we sample examples in a balanced manner across tasks and classes; if the number of classes exceeds the available examples, classes are randomly subsampled. To mitigate conditioning issues in matrix inversion (see [Section 3](https://arxiv.org/html/2508.21421v2#S3)), we use the Moore–Penrose pseudoinverse together with Tikhonov regularization (Hoerl & Kennard, 1970).

**Table 1: Normalized accuracies of merged models on vision tasks with different architectures. ViT-B/32 (above) and ViT-L/14 (below). Best results in bold, second-best underlined.**
| ViT-B/32 | Cars | DTD | ESAT | GTSRB | MNIST | RESISC | SUN | SVHN | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TA | 81.97 | 73.72 | 48.97 | 42.24 | 53.12 | 71.50 | 97.46 | 41.25 | 63.78 |
| TIES | 82.37 | 72.72 | 49.91 | 36.62 | 57.16 | 69.38 | 96.92 | 44.56 | 63.70 |
| DARETIES | 82.14 | 73.72 | 49.35 | 37.78 | 56.63 | 70.14 | 97.35 | 42.12 | 63.65 |
| Consensus TA | 81.21 | 75.45 | 52.53 | 40.33 | 56.65 | 71.72 | 98.41 | 41.46 | 64.72 |
| LiNeS | 81.82 | 73.99 | 49.53 | 41.08 | 53.01 | 71.39 | 97.63 | 48.55 | 63.63 |
| FisherAVG | 80.27 | 73.36 | 66.82 | 38.89 | 71.92 | 69.67 | 95.63 | 63.73 | 70.04 |
| RegMean | 79.89 | 71.07 | 37.56 | 41.82 | 62.71 | 71.23 | 95.73 | 68.17 | 66.02 |
| MaTS | 80.08 | 74.09 | 79.24 | 39.02 | 73.10 | 69.38 | 95.04 | 50.16 | 70.01 |
| TSV | 83.44 | 75.55 | 50.99 | 45.03 | 59.31 | 73.33 | 96.40 | 49.23 | 66.66 |
| Iso-C | 80.16 | 83.03 | 51.44 | 74.76 | 70.72 | 79.98 | 98.96 | 48.12 | 70.66 |
| KnOTSTIES | 83.75 | 74.45 | 50.36 | 47.31 | 67.01 | 71.79 | 96.51 | 50.62 | 67.73 |
| KnOTS${}_{\text{{DARE-TIES}}}$ | 82.01 | 72.90 | 44.15 | 45.54 | 60.59 | 70.89 | 95.56 | 47.64 | 64.91 |
| CoM (ours) | 90.37 | 87.31 | 90.83 | 90.70 | 99.42 | 94.56 | 89.44 | 98.29 | 92.62 |
| ViT-L/14 | Cars | DTD | ESAT | GTSRB | MNIST | RESISC | SUN | SVHN | Average |
| TA | 80.01 | 79.50 | 65.59 | 59.98 | 82.20 | 79.55 | 86.71 | 64.74 | 74.79 |
| TIES | 79.65 | 78.28 | 64.43 | 61.10 | 83.82 | 79.42 | 87.45 | 69.94 | 75.51 |
| DARETIES | 79.70 | 78.82 | 64.99 | 60.63 | 83.92 | 79.32 | 87.07 | 69.84 | 75.53 |
| Consensus TA | 81.88 | 81.32 | 68.60 | 64.55 | 85.04 | 81.56 | 86.90 | 63.72 | 76.70 |
| LiNeS | 80.89 | 79.88 | 65.25 | 59.74 | 81.86 | 79.37 | 86.69 | 64.37 | 74.65 |
| FisherAVG | 76.61 | 77.07 | 48.72 | 48.45 | 89.93 | 78.39 | 86.95 | 96.45 | 75.32 |
| RegMean | 77.81 | 77.98 | 53.08 | 53.91 | 59.75 | 78.87 | 86.91 | 70.53 | 69.85 |
| MaTS | 77.80 | 78.44 | 57.89 | 55.19 | 85.15 | 79.52 | 86.35 | 87.44 | 75.97 |
| TSV | 82.38 | 80.11 | 66.12 | 68.18 | 85.46 | 83.02 | 87.89 | 70.76 | 77.99 |
| Iso-C | 86.83 | 86.94 | 80.65 | 78.36 | 92.09 | 87.88 | 88.50 | 68.69 | 83.70 |
| KnOTSTIES | 82.47 | 80.26 | 64.65 | 68.85 | 88.48 | 82.37 | 88.18 | 76.63 | 78.99 |
| KnOTS${}_{\text{{DARE-TIES}}}$ | 80.41 | 79.65 | 64.84 | 62.95 | 85.33 | 79.73 | 87.55 | 71.00 | 76.43 |
| CoM (ours) | 88.72 | 87.85 | 87.27 | 92.77 | 99.37 | 90.96 | 83.71 | 97.81 | 91.06 |

### 4.2 Results

#### Vision tasks.

We evaluate CoM on the eight-dataset vision benchmark using two CLIP backbones of different capacity: ViT-B/32, and ViT-L/14 ([Table 1](https://arxiv.org/html/2508.21421v2#S4.T1)). Across both settings, simple parameter-space methods (TA, TIES, DARE, and their variants) exhibit limited effectiveness, with average normalized accuracies in the mid-60s for ViT-B/32 and mid-70s for ViT-L/14. These approaches perform poorly on challenging datasets, such as GTSRB and SVHN, where their accuracy often drops below $50\%$. More sophisticated baselines achieve stronger results: RegMean and TSV provide moderate improvements, while Iso-C consistently emerges as the best-performing baseline, attaining $70.66\%$ on ViT-B/32 and $83.70\%$ on ViT-L/14. However, Iso-C’s gains are inconsistent across datasets: i.e., it performs competitively on Cars and DTD, yet collapses on SVHN, where accuracy drops under $50\%$ with ViT-B/32.
In contrast, CoM delivers and consistent improvements across both architectures. On ViT-B/32, CoM achieves an average accuracy of $91.71\%$, surpassing the best baseline by over $20$ points. The gains are especially remarkable on tasks where other methods fail: CoM nearly preserves the original fine-tuned accuracy on MNIST ($99.6\%$) and SVHN ($98.6\%$), compared to Iso-C’s $~70\%$ and $~48\%$, respectively.

Scaling to the larger ViT-L/14 backbone improves the performance of all methods, narrowing the gap between simple parameter-space baselines and more sophisticated approaches. In this setting, CoM does not increase its normalized accuracy relative to ViT-B/32, yet it preserves a clear advantage, achieving $91.02\%$ compared to the second-best Iso-C at $83.70\%$. Notably, CoM boosts GTSRB accuracy above $90\%$, whereas the second-best baseline remains below $80\%$, highlighting that CoM not only scales effectively with model size but also mitigates dataset-specific weaknesses, yielding a merged model that consistently approaches the performance of the original task-specific modules.

**Table 2: Normalized accuracies of merged models on language with Llama3-8B. Best results are highlighted in bold, second-best underlined.**
| Method | SNLI | MNLI | SICK | QNLI | RTE | SCITAIL | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| TA | 93.57 | 95.28 | 87.96 | 68.71 | 100.00 | 96.73 | 90.38 |
| TIES | 95.17 | 96.19 | 84.18 | 74.18 | 100.00 | 96.78 | 91.08 |
| DARETIES | 94.76 | 96.80 | 78.39 | 72.08 | 98.39 | 96.20 | 89.44 |
| Consensus TA | 93.62 | 93.58 | 91.43 | 66.82 | 101.61 | 97.66 | 90.79 |
| LiNeS | 93.31 | 95.25 | 88.74 | 71.13 | 100.00 | 96.59 | 90.84 |
| RegMean | 97.67 | 96.32 | 79.79 | 65.17 | 96.78 | 89.77 | 87.58 |
| TSV | 95.38 | 95.12 | 88.83 | 76.80 | 101.61 | 97.56 | 92.55 |
| Iso-C | 55.00 | 39.04 | 76.54 | 55.90 | 46.77 | 69.25 | 57.08 |
| KnOTSTIES | 91.82 | 94.19 | 92.97 | 78.57 | 100.00 | 97.61 | 92.53 |
| KnOTS${}_{\text{{DARE-TIES}}}$ | 91.62 | 96.72 | 74.90 | 84.75 | 99.48 | 99.13 | 91.10 |
| CoM (ours) | 99.30 | 97.74 | 99.93 | 100.12 | 101.61 | 99.46 | 99.69 |

#### Language tasks.

Results on the six natural language benchmarks with LLaMA 3-8B ( [Table 2](https://arxiv.org/html/2508.21421v2#S4.T2)) show that merging is generally less destructive in this domain, as weight-space baselines such as Task Arithmetic already retain high performance ($90.38\%$ on average). TIES and DARE provide small gains (about $91\%$), while representation-aware approaches yield slightly stronger results: KnOTS+TIES and TSV reach $92.5\%$, the best among prior techniques. A clear outlier is Iso-C, which collapses to $57.08\%$, likely due to incompatibility between its singular value re-scaling and LoRA-style fine-tuning.

CoM achieves near-perfect merged performance, with an average normalized accuracy of $99.65\%$, outperforming the strongest baseline by over $7$ points. It preserves or even slightly exceeds the original task-specific accuracies: normalized scores surpass $100\%$ on some datasets (QNLI and RTE), indicating that the merged model can improve upon the individually fine-tuned checkpoints. This contrasts sharply with all prior baselines, which exhibit noticeable performance drops across different tasks. Together with the vision results, these findings demonstrate that CoM serves as a universally effective merging strategy, yielding merged models that best preserve or even surpass the performance of their specialized counterparts.

## 5 Model Analysis

**Table 3: Impact of individual components in both vision and language tasks. MCS denotes solving merging covariate shift, Norm refers to activation normalization, and Sim indicates Similarity-based importance weighting.**
| Components | Architecture |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| MCS | Norm | Sim | ViT-B/32 | ViT-L/14 | Llama3-8B |
| ✗ | ✗ | ✗ | 66.02 | 69.85 | 87.58 |
| ✓ | ✗ | ✗ | 82.53 | 83.55 | 99.45 |
| ✓ | ✓ | ✗ | 83.51 | 86.80 | 99.60 |
| ✓ | ✓ | ✓ | 92.62 | 91.06 | 99.69 |

#### Impact of Individual Components.

While CoM is primarily designed to mitigate merging covariate shift during composition, it also incorporates additional components that contribute to overall performance. We quantify the contribution of each component via an ablation study, presented in [Table 3](https://arxiv.org/html/2508.21421v2#S5.T3), systematically removing them to assess their individual impact on performance (we report the Average column). Our results indicate that addressing merging covariate shift alone is sufficient to achieve state-of-the-art performance. However, other components also play a significant role. In particular, weighting by the off-diagonal norm leads to substantial improvements in vision domains, while proving less critical for language tasks. We attribute this discrepancy to the inherent differences between the two settings: language tasks involve highly similar distributions, sharing the same label space across all datasets, whereas vision tasks correspond to classification problems defined on entirely different classes and domains. A more detailed discussion is provided in [appendix B](https://arxiv.org/html/2508.21421v2#A2). Finally, although activation normalization has a comparatively smaller impact, it consistently enhances performance across all benchmarks by ensuring numerical stability of the solution.

#### CoM – Number of examples.

Since CoM leverages input data to estimate task-specific Gram matrices, we perform an ablation study to determine the amount of data required for reliable performance. [Table 4](https://arxiv.org/html/2508.21421v2#S5.T4) reports the influence of the number of samples used to estimate the Gram matrices on each task. A sufficiently large sample size is essential to ensure numerical stability of the matrices, which allows the inversion in [Equation 2](https://arxiv.org/html/2508.21421v2#S2.E2) to produce meaningful results. Empirically, we observe that stability can be maintained with as few as $2$ samples per task. As shown in [Table 4](https://arxiv.org/html/2508.21421v2#S5.T4), CoM already surpasses the current state of the art with only $2$ samples and approaches near-maximal performance with $100$ samples, highlighting its data efficiency and robustness across tasks.

**Table 4: Performance (Average column) of CoM varying the number of examples used for each task.**
| # of samples | 2 | 5 | 10 | 50 | 100 | 200 | 300 | 400 | 500 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ViT-B/32 | 75.35 | 83.86 | 87.26 | 90.61 | 91.86 | 92.20 | 92.21 | 92.38 | 92.40 |
| ViT-L/14 | 81.77 | 87.21 | 88.95 | 90.57 | 90.99 | 90.97 | 90.99 | 91.00 | 91.06 |
| Llama3-8B | 96.15 | 97.03 | 98.50 | 98.62 | 98.95 | 99.04 | 99.50 | 99.48 | 99.33 |

#### Measuring MCS.

Covariate shift refers to changes in data distributions. Therefore, it is necessary to define a suitable distribution over the network activations in order to quantify this phenomenon. Following Huang & Yu (2020), we model the distribution of activation outputs as a multivariate Gaussian and evaluate MCS using the Earth Mover’s (EM) Distance (Villani, 2008), also referred to as the squared 2-Wasserstein distance or the Fréchet distance (Dowson & Landau, 1982). This choice is convenient, as the EM distance admits a closed-form solution for Gaussian distributions.
The total merging covariate shift for the $l^{\text{th}}$ layer can be measured as:

$$ $\operatorname{MCS}^{l}=\sum_{i=1}^{N}~\left\|\mu_{i,l}-\hat{\mu}_{i,l}\right\|_{2}^{2}+\operatorname{Tr}\left[\Sigma_{i,l}+\hat{\Sigma}_{i,l}-2\left(\hat{\Sigma}_{i,l}^{\nicefrac{{1}}{{2}}}\,\Sigma_{i,l}\,\hat{\Sigma}_{i,l}^{\nicefrac{{1}}{{2}}}\right)^{\nicefrac{{1}}{{2}}}\right],$ (8) $$

where $\mu_{i,l}$ and $\hat{\mu}_{i,l}$ denote the empirical means, while $\Sigma_{i,l}$ and $\hat{\Sigma}_{i,l}$ represent the empirical covariance matrices of the inputs ${\bm{X}}_{i}^{l}$ and $\hat{{\bm{X}}}_{i}^{l}$ ([Equation 3](https://arxiv.org/html/2508.21421v2#S3.E3)), respectively. To investigate the presence of MCS during model merging, we measure it using [Equation 8](https://arxiv.org/html/2508.21421v2#S5.E8) and report the results in [Figure 2](https://arxiv.org/html/2508.21421v2#S5.F2). The results indicate that MCS is present across all layers and tends to increase with depth, as earlier layers influence subsequent ones and the mismatch accumulates. We analyze the two projections before and after attention (the only fine-tuned layers) separately, as they show slightly different behaviors.

Figure: Figure 1: Merging covariate shift across layers.
Refer to caption: x1.png

#### Feature correlation.

To motivate similarity-weighted merging beyond its empirical effectiveness, we investigate whether the semantic distance between tasks and pretraining varies and whether it correlates with our weighting factor $\omega_{i}^{\,l}$. We estimate this distance using the performance gap between each task-specific model (evaluated in isolation) and the zero-shot performance of the base model, which reflects how much fine-tuning can improve the considered task. In [Figure 2](https://arxiv.org/html/2508.21421v2#S5.F2), we compare this accuracy gap with our similarity-based weighting factor, averaged across layers to produce a single value per task. The two measures, normalized for visual comparison, exhibit a consistent correlation across vision datasets and architectures, supporting the rationale behind our weighting scheme. Results for textual datasets are provided in [appendix B](https://arxiv.org/html/2508.21421v2#A2).

## 6 Comparison with Related Work

Pioneering model merging techniques rely on linear interpolation of model parameters. FedAvg McMahan et al. (2017) introduces this approach in the context of Federated Learning, assuming a shared initialization across models. Building on this idea, Model Soups Wortsman et al. (2022a) propose a greedy strategy that incrementally incorporates models into the mixture only when they improve validation performance; WiSE-FT Wortsman et al. (2022b) enhances fine-tuning by weighting model updates to boost generalization and robustness across tasks, and Task Arithmetic Ilharco et al. (2023) enables personalized model editing, allowing finer control over individual contributions. Despite their simplicity and effectiveness, these methods can suffer performance degradation due to conflicting parameters Tang et al. (2024); Du et al. (2024); Tam et al. (2024).

To address this, a family of approaches seeks to reduce interference by applying heuristics prior to merging: TIES Yadav et al. (2023) addresses parameter redundancy by pruning and aligning parameter signs; DARE Yu et al. (2024) applies stochastic parameter dropping and rescaling; Consensus Merging Wang et al. (2024) learns task-specific masks to preserve performance with fewer parameters; LiNeS Wang et al. (2025) adjusts parameter magnitudes based on their layer depth within the network; and Git Rebasin Ainsworth et al. (2023) leverages Linear Mode Connectivity, arguing that fine-tuned models lie within the same loss basin up to a permutation of their parameters.

In parallel, another line of research argues that model ensembling should be performed within subspaces that maximize alignment between parameters. Within this framework, TSV Gargiulo et al. (2025) compresses parameters into a low-rank structure and approximates whitening by solving the Procrustes orthogonality problem. KnOTS Stoica et al. (2025) performs merging in an aligned parameter space using singular value decomposition, while ISO-C Marczak et al. (2025) enforces layer-wise isotropic matrices by rescaling singular values to produce singular vectors of equal magnitude. While these methodologies rely on heuristics applied directly to parameters, CoM aligns activations layer-wise, enabling more precise model merging that optimally preserves task-specific features while maintaining the original architecture.

A complementary research direction focus on aligning model features to facilitate merging. Neuron Alignment Tatro et al. (2020) uses layer-wise regression to compute bipartite matches between neurons, aligning intermediate activations. ZIPIt! Stoica et al. (2023) applies intermediate feature-based permutations to align layers, while Optimal Transport Fusion Singh & Jaggi (2020) formulates alignment as an optimal transport problem, computing soft matching between activation distributions. DF-Merge Lee et al. (2025) combines parameter importance with scaling, using Bayesian optimization to tune the coefficients, while Fisher-weighted averaging Matena & Raffel (2022) directly averages parameters weighted by the Fisher Information Matrix Fisher (1922) to maximize alignment of final features. Regmean Jin et al. (2023) derives a closed-form solution to match activations layer-wise, while MaTS Tam et al. (2024) unifies the two preceding approaches under a common linear system and proposes solving it via conjugate gradient rather than in closed form. Although these methods are grounded in solid theoretical foundations, they overlook inter-layer dependencies, leading to merging covariate shift. Our Chain of Merges addresses this by sequentially updating activation statistics layer-by-layer, thus preserving network consistency.

## 7 Conclusions and Future Work

In this work, we identify and address Merging Covariate Shift (MCS), a form of internal covariate shift that emerges when merging layers independently in methodologies that rely on activation statistics. To mitigate the adversarial effect of this phenomenon, we propose Chain of Merges (CoM), a novel approach that updates activation statistics autoregressively, capturing inter-layer dependencies and fully eliminating MCS throughout the network. Empirical results on standard vision and language benchmarks demonstrate that CoM consistently outperforms existing methods across diverse architectures and domains.

Looking ahead, we plan to extend the proposed framework to a wider range of activation-based merging techniques. We also see promising opportunities for applying CoM in Federated Learning, where efficiently merging local clients’ updates is essential for centralized performance.

## References

- Aghajanyan et al. (2021)
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer.
Intrinsic dimensionality explains the effectiveness of language model fine-tuning.
In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), *Association for Computational Linguistics*, pp. 7319–7328, Online, August 2021. Association for Computational Linguistics.
doi: 10.18653/v1/2021.acl-long.568.
URL [https://aclanthology.org/2021.acl-long.568/](https://aclanthology.org/2021.acl-long.568/).
- Ainsworth et al. (2023)
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa.
Git re-basin: Merging models modulo permutation symmetries.
In *ICLR*, 2023.
- Arefin et al. (2024)
Md Rifat Arefin, Gopeshh Subbaraj, Nicolas Gontier, Yann LeCun, Irina Rish, Ravid Shwartz-Ziv, and Christopher Pal.
Seq-vcr: Preventing collapse in intermediate transformer representations for enhanced reasoning.
*International Conference on Learning Representations*, 2024.
- Arpit et al. (2016)
Devansh Arpit, Yingbo Zhou, Bhargava Kota, and Venu Govindaraju.
Normalization propagation: A parametric technique for removing internal covariate shift in deep networks.
In *International Conference on Machine Learning*, pp. 1168–1176. PMLR, 2016.
- Ba et al. (2016)
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.
Layer normalization.
*arXiv preprint arXiv:1607.06450*, 2016.
- Barbero et al. (2024)
Federico Barbero, Andrea Banino, Steven Kapturowski, Dharshan Kumaran, Joao GM Araújo, Alex Vitvitskyi, Razvan Pascanu, and Petar Velickovic.
Transformers need glasses! information over-squashing in language tasks.
*URL https://arxiv. org/abs/2406.04267*, 2024.
- Bowman et al. (2015)
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning.
The snli corpus.
In *Conference on Empirical Methods in Natural Language Processing (EMNLP)*, 2015.
- Cheng et al. (2017)
Gong Cheng, Junwei Han, and Xiaoqiang Lu.
Remote sensing image scene classification: Benchmark and state of the art.
*Proceedings of the IEEE*, 105(10):1865–1883, 2017.
- Cimpoi et al. (2014)
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.
Describing textures in the wild.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pp. 3606–3613, 2014.
- Cogswell et al. (2015)
Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, and Dhruv Batra.
Reducing overfitting in deep networks by decorrelating representations.
*arXiv preprint arXiv:1511.06068*, 2015.
- Daheim et al. (2024)
Nico Daheim, Thomas Möllenhoff, Edoardo Maria Ponti, Iryna Gurevych, and Mohammad Emtiyaz Khan.
Model merging by uncertainty-based gradient matching.
*International Conference on Learning Representations*, 2024.
- Dosovitskiy et al. (2021)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.
An image is worth 16x16 words: Transformers for image recognition at scale.
In *International Conference on Learning Representations*, 2021.
- Dowson & Landau (1982)
DC Dowson and BV666017 Landau.
The fréchet distance between multivariate normal distributions.
*Journal of multivariate analysis*, 12(3):450–455, 1982.
- Du et al. (2024)
Guodong Du, Junlin Lee, Jing Li, Runhua Jiang, Yifei Guo, Shuyang Yu, Hanting Liu, Sim K Goh, Ho-Kin Tang, Daojing He, et al.
Parameter competition balancing for model merging.
*Advances in Neural Information Processing Systems*, 37:84746–84776, 2024.
- Fisher (1922)
Ronald A Fisher.
On the mathematical foundations of theoretical statistics.
*Philosophical transactions of the Royal Society of London. Series A, containing papers of a mathematical or physical character*, 222(594-604):309–368, 1922.
- Gargiulo et al. (2025)
Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodola.
Task singular vectors: Reducing task interference in model merging.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp. 18695–18705, 2025.
- Grattafiori et al. (2024)
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al.
The llama 3 herd of models.
*arXiv preprint arXiv:2407.21783*, 2024.
- Helber et al. (2019)
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.
Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.
*IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing*, 12(7):2217–2226, 2019.
- Hoerl & Kennard (1970)
Arthur E Hoerl and Robert W Kennard.
Ridge regression: Biased estimation for nonorthogonal problems.
*Technometrics*, 12(1):55–67, 1970.
- Hu et al. (2022)
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.
LoRA: Low-rank adaptation of large language models.
In *ICLR*, 2022.
- Huang & Yu (2020)
You Huang and Yuanlong Yu.
An internal covariate shift bounding algorithm for deep neural networks by unitizing layers’ outputs.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 8465–8473, 2020.
- Ilharco et al. (2023)
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi.
Editing models with task arithmetic.
In *ICLR*, 2023.
- Ioffe & Szegedy (2015)
Sergey Ioffe and Christian Szegedy.
Batch normalization: Accelerating deep network training by reducing internal covariate shift.
In *International conference on machine learning*, pp. 448–456. pmlr, 2015.
- Jin et al. (2023)
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng.
Dataless knowledge fusion by merging weights of language models.
In *ICLR*, 2023.
- Jordan et al. (2022)
Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur.
Repair: Renormalizing permuted activations for interpolation repair.
*arXiv preprint arXiv:2211.08403*, 2022.
- Kalamkar et al. (2019)
Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, et al.
A study of bfloat16 for deep learning training.
*arXiv preprint arXiv:1905.12322*, 2019.
- Khot et al. (2018)
Tushar Khot, Ashish Sabharwal, and Peter Clark.
Scitail: A textual entailment dataset from science question answering.
In *Proceedings of the AAAI conference on artificial intelligence*, volume 32, 2018.
- Krause et al. (2013)
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.
3d object representations for fine-grained categorization.
In *Proceedings of the IEEE international conference on computer vision workshops*, pp. 554–561, 2013.
- Kumar et al. (2022)
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang.
Fine-tuning can distort pretrained features and underperform out-of-distribution.
*International Conference on Learning Representations*, 2022.
- LeCun et al. (2002)
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.
Gradient-based learning applied to document recognition.
*Proceedings of the IEEE*, 86(11):2278–2324, 2002.
- Lee et al. (2025)
Sanwoo Lee, Jiahao Liu, Qifan Wang, Jingang Wang, Xunliang Cai, and Yunfang Wu.
Dynamic fisher-weighted model merging via Bayesian optimization.
In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), *Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics*, pp. 4923–4935, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics.
ISBN 979-8-89176-189-6.
doi: 10.18653/v1/2025.naacl-long.254.
URL [https://aclanthology.org/2025.naacl-long.254/](https://aclanthology.org/2025.naacl-long.254/).
- Marczak et al. (2025)
Daniel Marczak, Simone Magistri, Sebastian Cygert, Bartłomiej Twardowski, Andrew D Bagdanov, and Joost van de Weijer.
No task left behind: Isotropic model merging with common and task-specific subspaces.
*arXiv preprint arXiv:2502.04959*, 2025.
- Marelli et al. (2014)
Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli.
Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment.
In *Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014)*, pp. 1–8, 2014.
- Matena & Raffel (2022)
Michael S Matena and Colin A Raffel.
Merging models with fisher-weighted averaging.
*Advances in Neural Information Processing Systems*, 2022.
- McMahan et al. (2017)
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.
Communication-efficient learning of deep networks from decentralized data.
In *Artificial intelligence and statistics*. PMLR, 2017.
- Morcos et al. (2018)
Ari S Morcos, David GT Barrett, Neil C Rabinowitz, and Matthew Botvinick.
On the importance of single directions for generalization.
*arXiv preprint arXiv:1803.06959*, 2018.
- Netzer et al. (2011)
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al.
Reading digits in natural images with unsupervised feature learning.
In *NIPS workshop on deep learning and unsupervised feature learning*, volume 2011, pp.  7. Granada, 2011.
- Radford et al. (2021)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.
Learning transferable visual models from natural language supervision.
In *ICML*, 2021.
- Raffel et al. (2020)
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.
Exploring the limits of transfer learning with a unified text-to-text transformer.
*Journal of machine learning research*, 21(140):1–67, 2020.
- Singh & Jaggi (2020)
Sidak Pal Singh and Martin Jaggi.
Model fusion via optimal transport.
*Advances in Neural Information Processing Systems*, 33:22045–22055, 2020.
- Stallkamp et al. (2011)
Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel.
The german traffic sign recognition benchmark: a multi-class classification competition.
In *The 2011 international joint conference on neural networks*, pp. 1453–1460. IEEE, 2011.
- Stoica et al. (2023)
George Stoica, Daniel Bolya, Jakob Bjorner, Pratik Ramesh, Taylor Hearn, and Judy Hoffman.
Zipit! merging models from different tasks without training.
*arXiv preprint arXiv:2305.03053*, 2023.
- Stoica et al. (2025)
George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, and Judy Hoffman.
Model merging with svd to tie the knots.
*International Conference on Learning Representations*, 2025.
- Tam et al. (2024)
Derek Tam, Mohit Bansal, and Colin Raffel.
Merging by matching models in task parameter subspaces.
*Transaction on Machine Learning research*, 2024.
- Tang et al. (2024)
Anke Tang, Li Shen, Yong Luo, Nan Yin, Lefei Zhang, and Dacheng Tao.
Merging multi-task models via weight-ensembling mixture of experts.
*arXiv preprint arXiv:2402.00433*, 2024.
- Tatro et al. (2020)
Norman Tatro, Pin-Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai.
Optimizing mode connectivity via neuron alignment.
*Advances in Neural Information Processing Systems*, 33:15300–15311, 2020.
- Touvron et al. (2023)
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.
Llama: Open and efficient foundation language models.
*arXiv:2302.13971*, 2023.
- Villani (2008)
Cédric Villani.
*Optimal transport: old and new*, volume 338.
Springer, 2008.
- Wang et al. (2019)
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.
Glue: A multi-task benchmark and analysis platform for natural language understanding.
In *ICLR*, 2019.
- Wang et al. (2024)
Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jimenez, François Fleuret, and Pascal Frossard.
Localizing task information for improved model merging and compression.
*International Conference on Machine Learning*, 2024.
- Wang et al. (2025)
Ke Wang, Nikolaos Dimitriadis, Alessandro Favero, Guillermo Ortiz-Jimenez, Francois Fleuret, and Pascal Frossard.
Lines: Post-training layer scaling prevents forgetting and enhances model merging.
*International Conference on Learning Representations*, 2025.
- Williams et al. (2017)
Adina Williams, Nikita Nangia, and Samuel R Bowman.
A broad-coverage challenge corpus for sentence understanding through inference.
*arXiv preprint arXiv:1704.05426*, 2017.
- Wortsman et al. (2022a)
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al.
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.
In *International Conference on Machine Learning*, 2022a.
- Wortsman et al. (2022b)
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al.
Robust fine-tuning of zero-shot models.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 7959–7971, 2022b.
- Wu et al. (2024)
Xinyi Wu, Amir Ajorlou, Yifei Wang, Stefanie Jegelka, and Ali Jadbabaie.
On the role of attention masks and layernorm in transformers.
*Advances in Neural Information Processing Systems*, 37:14774–14809, 2024.
- Xiao et al. (2016)
Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva.
Sun database: Exploring a large collection of scene categories.
*International Journal of Computer Vision*, 119(1):3–22, 2016.
- Yadav et al. (2023)
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal.
Ties-merging: Resolving interference when merging models.
*Advances in Neural Information Processing Systems*, 36:7093–7115, 2023.
- Yu et al. (2024)
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li.
Language models are super mario: Absorbing abilities from homologous models as a free lunch.
In *Forty-first International Conference on Machine Learning*, 2024.

## Appendix

## Appendix A Derivation of the similarity-weighted merging solution

We want to minimize the following objective with respect to the matrix
${\bm{W}}_{M}^{\,l}$:

$$ $\tilde{\Omega}_{l}=\sum_{i=1}^{N}\omega_{i}^{\,l}\,\|{\bm{W}}_{M}^{\,l}\tilde{{\bm{X}}}_{i}^{\,l}-{\bm{W}}_{i}^{\,l}\tilde{{\bm{X}}}_{i}^{\,l}\|_{2}^{2}.$ $$

To simplify the notation, define ${\bm{G}}_{i}^{\,l}=\tilde{{\bm{X}}}_{i}^{\,l}(\tilde{{\bm{X}}}_{i}^{\,l})^{\top}$.
Expanding the norm and applying standard rules of matrix differentiation, the gradient of the objective with respect to ${\bm{W}}_{M}^{\,l}$ is:

$$ $\nabla_{{\bm{W}}_{M}^{\,l}}\tilde{\Omega}_{l}=2\Biggl[{\bm{W}}_{M}^{\,l}\Bigl(\sum_{i=1}^{N}\omega_{i}^{\,l}{\bm{G}}_{i}^{\,l}\Bigr)-\Bigl(\sum_{i=1}^{N}\omega_{i}^{\,l}{\bm{W}}_{i}^{\,l}{\bm{G}}_{i}^{\,l}\Bigr)\Biggr].$ $$

The minimizer is obtained by setting the derivative equal to zero as:

$$ ${\bm{W}}_{M}^{\,l}\Bigl(\sum_{i=1}^{N}\omega_{i}^{\,l}{\bm{G}}_{i}^{\,l}\Bigr)=\sum_{i=1}^{N}\omega_{i}^{\,l}{\bm{W}}_{i}^{\,l}{\bm{G}}_{i}^{\,l}.$ $$

Finally, we can solve for ${\bm{W}}_{M}^{\,l}$ by multiplying on the right hand side with the inverse of $\sum_{i=1}^{N}\omega_{i}^{\,l}{\bm{G}}_{i}^{\,l}$:

$$ ${\bm{W}}_{M}^{\,l}=\Biggl(\sum_{i=1}^{N}\omega_{i}^{\,l}{\bm{W}}_{i}^{\,l}{\bm{G}}_{i}^{\,l}\Biggr)\Biggl(\sum_{i=1}^{N}\omega_{i}^{\,l}{\bm{G}}_{i}^{\,l}\Biggr)^{-1}.$ $$

If the latter matrix is singular, the Moore–Penrose pseudoinverse should be used instead.

## Appendix B Supplementary Ablations for Textual Datasets

#### Similarity-based importance.

In [Figure 4](https://arxiv.org/html/2508.21421v2#A2.F4), we report the similarity-based importance analysis for textual tasks (SNLI, MNLI, SICK, QNLI, RTE, and SciTail). Following the approach used for vision datasets, we compute the performance gap between task-specific fine-tuned models and the zero-shot performance of the base Llama3-8B model, and compare it to the similarity-based weighting factor $\omega_{i}^{\,l}$ averaged across layers.

In contrast to vision tasks, textual datasets exhibit a relatively uniform importance distribution: both performance gaps and similarity weights vary little across tasks. As a result, the correlation between the two measures is weaker, and similarity-weighted merging provides limited benefit. These findings highlight that the effectiveness of similarity-based weighting depends on task heterogeneity: greater diversity relative to the pretraining distribution amplifies its impact on the merged model.

#### Measuring MCS.

We extend the analysis from [Section 5](https://arxiv.org/html/2508.21421v2#S5) to Llama3-8B, computing MCS for each layer using the same methodology. As illustrated in [Figure 4](https://arxiv.org/html/2508.21421v2#A2.F4), textual datasets display merging covariate shifts that are broadly comparable to those observed in vision models.

Figure: Figure 3: Merging covariate shift across layers.
Refer to caption: x3.png

Figure: (a) Peak memory usage and merging computational cost for Vit-B/32.
Refer to caption: x5.png

## Appendix C Computational and Memory Cost

We evaluate the efficiency of the proposed methodology in terms of computational complexity and memory usage, comparing it against established techniques. For the three architectures used, [Figures 5(a)](https://arxiv.org/html/2508.21421v2#A2.F5.sf1), [5(b)](https://arxiv.org/html/2508.21421v2#A2.F5.sf2) and [5(c)](https://arxiv.org/html/2508.21421v2#A2.F5.sf3) (log-scale) report the number of FLOPs as a measure of computational cost, and the theoretical memory overhead in GB. We denote our methodology with the minimum number of samples achieving state-of-the-art performance as CoM_fast, and the variant using the optimal number of samples for maximum performance as CoM_best. The results show that CoM achieves performance comparable to alternative merging strategies. For clarity, methods such as Task Arithmetic, TIES, DARETIES, Consensus TA, and LiNeS are omitted from the computational plot, since their complexity is negligible.

A notable comparison is with RegMean. Although our approach shares a similar formulation, it differs in the update rule by using the merged activations instead of the task-specific ones. This key distinction improves accuracy while keeping the number of forward passes unchanged: RegMean requires one forward pass per task-specific model to compute the Gram matrices, whereas CoM performs the same number of passes on the merged model. Finally, CoM requires fewer examples in practice, resulting in consistently lower computational cost, as shown in [Table 4](https://arxiv.org/html/2508.21421v2#S5.T4).

These findings highlight that the proposed recursive scheme provides an effective balance between efficiency and performance, ensuring state-of-the-art accuracy while keeping both computation and memory overhead limited.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - **Task similarity coefficient ablation**: in the text of the paper authors specify that the similarity coefficient is computed on actuation of the merged model, for efficiency reason. Would the similarity coefficients computed on the finetuned models  activations be highly correlated to the one obtained on the merge models? 


- **Robustness to sampling**: I think it would be useful to get variance across different sampling for each tasks, by e.g. averaging and reporting std on the results in Table (REF TABLE) across different samplings.

- **Balanced sampling across tasks**: when the tasks have different complexity, could a different number of samples be needed? 

- **Scalability** What's the scalability of the method as opposed to averaging based methods? could it be employed e.g. on larger LLMs?

- **Merging LoRA weights ablation** Did you try to merge the LoRA parameters of the fine-tuned models instead of the original weights? (Assuming they have the same bottleneck dimensions across tasks)

- **Suggestion**: I would personally move the experiment on covariate shift earlier in the paper as a motivation to introduce the approach. It would be also useful to add the same measure on the merged model accounting for the order of layers to show that the covariance shift is lower in that case.
soundness: 2
strengths: - **Simple and effective contribution** Taking into account the propagation of error throughout layers in the merging, is a simple, well justified by the and very effective modification according to the result presented. 

- **Strong measured performance**: performance on the models and datasets tested is consistently stronger with respect to the baselines tested.

- **Comprehensive experiments**: the paper compares with 10+ baselines on the vision and text domain in the merging experiments, showing strong performance and good coverage , with the only exception of few unmentioned works (see weakness section)

- **Ablations**: ablations experiments are useful to characterize the contribution of each component of the method proposed, showing the large impact of correcting the covariate shift between layers.
summary: The paper proposes an algorithm to perform model merging on transformer based models fine-tuned on different tasks. The main contribution proposed is to take into account the order of the merging to avoid error propagation throughout the layers, identified as covariance shift in feature space by the authors. The proposed solution is to merge one layer at a time from the initial layer to the last, instead of considering merging each layer independently and in parallel. The merging strategy is based on (Jin et al 2022) where preactivation features on each task (i.e. inner products between weights and features)  are approximated by a shared weight matrix on feature space, which has a closed form solution. The author propose further to weight the contribution of each task using gram matrix statistics of the features. Experiments are performed on vision and text benchmarks, showing good improvements on tested dataset.
weaknesses: - **Task similarity coefficient**: according to what written in the paper the task similarity coefficient is computed on the normalized Gram   $XX^T$ matrices, by counting the magnitude of the off-diagonal elements. The authors support this choice claiming that larger correlations between features indicate higher distance to the pretrained features Gram matrix and hence, higher task complexity. It is not clear to me why this is the case: why near orthogonal samples in the fine-tuned would imply good task generalization? Distance is not directly measure w.r.t. Gram Matrices son the pretrained model as far as I understand. Moreover, isn't this meant to be computed on the covariance matrix $X^TX$, to test orthogonality of the features rather than samples? Please clarify.

- **Missing related works**: I think there are some missing related works [a,b] from previous methods that take into account layerwise dependency and would matter to discuss and compare to: especially [a] which proposes to merge task adaptively by weighting the contribution of each layer, being related to the layer ordering approach considered in the paper and improving over (Jing et al 2022). 


- **Testing on out of distribution settings** I think the lack of OOD experiments is a missed opportunity to confirm the performance of the method. this could be done by simply leaving out some of the tasks and testing the accuracy them, similarly as done in (Jin et al 2022) or [a].

- **Missing absolute performance of the models**. I believe the paper would benefit from reporting the absolute performances of the fine-tuned model and base models on the different task in the Appendix as a reference to understand how well these perform. 


_[a] Yang, Enneng, et al. "Adamerging: Adaptive model merging for multi-task learning.", ICLR 2024_

_[b] Xu, Jing, Jiazheng Li, and Jingzhao Zhang. "Scalable Model Merging with Progressive Layer-wise Distillation.”, ICML 2025_

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. For Eq. 5, even for the first linear layer, there are attention layers before the linear layers in the same residual block. As a result, shouldn’t the input still be affected by prior merging (on the attention layers in the same block, whose merging process is not explained in the paper) as well, making the notations in Eq. 5 inaccurate?

[1] Editing Models with Task Arithmetic
[2] TIES-Merging: Resolving Interference When Merging Models
[3] Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
[4] Localizing Task Information for Improved Model Merging and Compression
[5] LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging
soundness: 3
strengths: 1. The paper is clearly-written and easy to understand.
2. The method demonstrates good empirical results in both vision and NLP, including the results obtained with Llama-3.
summary: This paper proposes CoM, which recursively merges the linear layer weights from first layer to last layer during model merging. It improves Regmean by addressing internal covariate shift and demonstrates good empirical performance.
weaknesses: 1. The proposed method CoM offers modest novelty. It extends Regmean by fixing the internal covariate shift problem. The novelty primarily lies in providing the inputs of the merged model rather than the individual models for Regmean merging algorithm.

2. The performance of the baseline methods seem concerningly low (Table 1). The performance of most merging-based methods, including TA [1], Ties [2], DARE [3] seem to be notably lower than their reported results in the original papers. Furthermore, the methods which are designed to improve over TA, such as TIES, DARE, Consensus [4], LiNeS [5], underperform or have the same performance as TA. Furthermore, the manuscript does not provide implementation details to the compared baselines, which are needed in this case as the the baselines do not perform as expected.

Note: If the reason for the baseline methods’ low performance is due to applying LoRA (which case also needs to be clarified), I suggest the authors add results for standard fine-tuning as well, following the literatures in model merging [1-5]. It is important to evaluate the performance of the proposed method in the standard fine-tuning scenario as well.

3. The computational cost of TA, TIES, DARE, Consensus and LiNeS are missing in the computational cost comparison. It is unfair to omit them simply because their computational cost is negligible (Section C). The computational cost of the proposed methods need to be compared to the light-weight methods as well to clearly demonstrate the tradeoff to the viewers.

4. Many important experimental details are not given. 

4.1 The implementation details of the baselines, as stated before.

4.2 Where did the authors get the input samples for calculating the gram matrix, i.e., are they the same or different samples in the test set?

4.3 How is the method merging the layers that are not linear layers (e.g., the attention layers)? Does it follow the Regmean setup to average their weights?

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: - Why is the proposed method only applied on LoRA-fine-tuned models? How is the performance when applied to full-fine-tuned models?
- What is the actual number of examples used? 
- Do examples come from training set or test set?
- The memory overhead seems to be very similar to TA. This is a bit confusing and surprising. If the Gram matrix is computed on at least 500 samples, wouldn't there be more memory usage? What's the actual memory usage?
- How is the computational cost and latency compared to previous works mentioned in Table 1?
soundness: 3
strengths: - The proposed idea is well-motivated, based on the internal covariate shift problem.
- The displayed experimental results show strong performance of the proposed method.
- The paper is well-written and easy to follow.
summary: The paper introduces a new methodology for model merging. In particular, the paper notes that previous layer-wise merging works have neglected the well-known internal covariate shift problem, resulting in a distributional mistmatches. As such, the paper proposes a Chain of Merges (CoM) method, which sequentially merges weights layer by layer. Experimental results demonstrate that the proposed method provides very high normalized accuracy.
weaknesses: - The paper does not explicitly explain the number of examples used in the method. One may guess that the number of examples is 500 based on the ablation study results in Table 4, but the performance of ViT-B/32 and Llama3-8B do not match with the main results in Table 1 and Table 2.
- Also, the paper does not specify which set the examples come from.
- The paper only reports the normalized accuracy, without reporting the actual performance. Considering how the proposed method is applied on LoRA-fine-tuned models, the results as is do not give a full picture as to how the proposed method actually performs in comparison on other works. 
- The paper does not show large-scale experiments (14 tasks or 20 tasks, as done so in Consensus TA (Wang et al., 2024)).
- The paper does not compare against more recent state-of-the-art methods, such as EMR-Merging [A].
- Some works compared in the table originally reported the results with full-fine-tuned models, to the best of the reviewer's knowledge. The comparison does not seem completely fair.


[A] Huang et al., Emr-merging: Tuning-free high-performance model merging. NeurIPS 2024.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: - What is the performance of CoM on models finetuned without adapters?
- Can the authors provide the absolute accuracies (not just normalized) for all experiments?
soundness: 1
strengths: - Model merging is a critical and practical research area for making AI systems more modular.
- The identification of MCS is an interesting conceptual contribution.
- The paper is well-written and clear.
- The empirical results, within the context of LoRA-finetuned models, appear strong.
summary: The paper introduces Chain of Merges (CoM), a new layer-wise model merging technique. The authors identify "merging covariance shift" (MCS) as a key problem in which merging layers independently induces activation shifts that negatively affect subsequent layers. CoM addresses this problem by merging layers sequentially, while updating activation statistics to ensure consistency.
weaknesses: - **Misleading scope**. The paper's entire experimental validation is performed exclusively on LoRA-finetuned models. This fundamental design choice is only mentioned deep in the "Implementation Details" (Section 4.1). The abstract, introduction, and methodology sections all speak in general terms of merging "fine-tuned models" and "task-specific checkpoints," which strongly implies full fine-tuning. The paper is therefore not evaluated as a general model-merging technique but as a LoRA-merging technique, and its contributions must be re-evaluated in this much narrower context.
- **Invalid SOTA comparisons**. This LoRA-only setup invalidates the claims of "state-of-the-art" performance in model merging. The majority of the cited literature and the baselines themselves were developed and evaluated in the context of full-parameter fine-tuning. A clear example is TSV: the authors report ~78% normalized accuracy (on ViT-L/14), whereas the original TSV paper reports ~97% on the same benchmark (likely starting from an even higher absolute accuracy – which is not reported).
- **No absolute accuracy**. The paper indeed reports only normalized accuracy. The omission of absolute accuracy is a critical flaw. Normalized accuracy alone can be misleading, as it may mask poor overall performance. Given the baseline discrepancies, reporting also absolute accuracy is essential for a fair comparison.


Decision: Reject