RegMean++: Enhancing Effectiveness and Generalization of Regression Mean for Model Merging

## Abstract

Abstract Regression Mean (RegMean), an approach that formulates model merging as a linear regression problem, aims to find the optimal weights for each linear layer in the merge model by minimizing the discrepancy in predictions between the merge and candidate models. RegMean provides a precise closed-form solution for the merging problem; therefore, it offers explainability and computational efficiency. However, RegMean merges each linear layer independently, overlooking how the features and information in the earlier layers propagate through the layers and influence the final prediction in the merge model. In this paper, we introduce RegMean++ , a simple yet effective alternative to RegMean, that explicitly incorporates both intra- and cross-layer dependencies between merge models’ layers into RegMean’s objective. By accounting for these dependencies, RegMean++ better captures the behaviors of the merge model. Extensive experiments demonstrate that RegMean++ consistently outperforms RegMean across diverse settings, including in-domain (ID) and out-of-domain (OOD) generalization, sequential merging, large-scale tasks, and robustness under several types of distribution shifts. Furthermore, RegMean++ achieves competitive or state-of-the-art performance compared to various recent advanced model merging methods. Our code is available at https://github.com/nthehai01/RegMean-plusplus.

## 1 Introduction

As pretrain-finetune paradigm becomes the foundation of modern machine learning, the number of pre-trained and fine-tuned task-specific models (candidate models) is growing at an unprecedented pace. Model merging (Matena and Raffel 2022; Wortsman et al. 2022; Ilharco et al. 2022; Jin et al. 2022; Yadav et al. 2023; Yang et al. 2024b, a; Yadav et al. 2024), an emerged approach that aims to combine multiple candidate models into a single unified model (merge model) with multi-tasking capabilities, without incurring the computational overhead of traditional multi-task learning (MTL) or full access to original training data.

Regression Mean (RegMean; Jin et al. (2022)), an explainable and computationally efficient model merging method, formulates weight fusion as a closed-form regression problem that minimizes the differences between the outputs of merge model and those of each candidate model.
RegMean leverages the inner-product matrices of the input features at each linear layer, including those within the MLP components and attention heads of transformer layers in the candidates. Additionally,
RegMean decreases non-diagonal entries in these matrices to enhance stability during merging.
The other types of transformer layers’ weights are merged by simply averaging across candidate models.
RegMean offers several benefits, including privacy-preserving, computational efficiency, and model-agnostic.
These advantages make RegMean one of the most practical merging methods.

However, we discuss an important caveat of RegMean is that it operates by independently applying the closed-form solution to linear layers of the transformer layer, fundamentally ignoring how features and information are processed and propagated through layers in the merge model. These intra- and cross-layer dependencies are crucial for maintaining good representations. The second drawback of RegMean is its memory and storage overhead. RegMean requires memory to store the inner-product matrices, which—for the large-scale models—can rival the number of parameters. In this paper, we aim to address these limitations:
(1) We introduce RegMean++, an alternative to RegMean, which incorporates both intra- and cross-layer dependencies of the merge model’s layers into the RegMean merging objective. RegMean++ consistently improves in ID performance, OOD generalization, and demonstrates sustainability to sequential merging or large-scale tasks. Moreover, RegMean++ shows stronger robustness under various types of distribution shifts with reduced representation bias.
(2) We conduct layer-wise analysis and find that (i) merging using linear layers from middle and deep transformer layers preserves over $98\%$ accuracy compared to using all linear layers. In contrast, earlier transformer layers appear less important for merging, as using linear layers in them leads to significant accuracy degradation.
(ii) Mid-depth layers consistently surpass the last layer in merging performance. This result highlights that the mid-depth layers may serve as more reliable sources of meaningful features for model merging. (iii) Merging using linear layers in the MLP modules consistently outperforms using linear layers in the attention heads.
(3) We benchmark RegMean++ against eleven advanced model merging methods. Experiment results show that RegMean++ achieves competitive or state-of-the-art performance across diverse settings, highlighting its generalization and effectiveness relative to existing approaches.

## 2 Background and Related Work

#### Model merging.

Model Soups (or parameter averaging) (Wortsman et al. 2022; Choshen et al. 2022), a well-known approach that performs merging by simply taking the average of all candidate models’ parameters, for enhancing distribution robustness (Wortsman et al. 2022), creating merge models with multi-task or multi-modality capabilities (Sung et al. 2023; Ilharco et al. 2022; Yadav et al. 2023).

Task Arithmetic (Ilharco et al. 2022) introduces “task vector”, a new concept that quantifies the task-specific knowledge of a candidate model, by measuring the difference between the candidate model parameters and the base model parameters. However, Yadav et al. (2023) point out that Task Arithmetic suffers from conflicts where different task vectors update the same parameters in opposite directions. They proposed TIES-Mering, a three-step method that first removes low-magnitude (noisy) parameters, then resolves sign conflicts, and finally aggregates only the non-conflicting parameters. DARE (Yu et al. 2023) proposes to pre-process each task vector independently by applying a Bernoulli mask to randomly zero out its parameters, effectively performing a dropout-like pruning before merging. TSV-M (Gargiulo et al. 2025) extracts task singular vectors via singular value decomposition (SVD) on layer-wise weight updates, selects top-$k$ components for a low-rank representation, and applies whitening to decorrelate subspaces before merging. Marczak et al. (2025) argue that effective merging depends on how well merge updates span each task’s principal subspaces. They propose Iso-C, which sums task-specific updates, performs SVD, and replaces singular values with their average to create an isotropic spectrum. Iso-CTS, an Iso-C’s variant, adds top singular directions from each task’s residual, then orthogonalizes and isotropically scales the result.

Inspired by test-time adaptation schemes (Wang et al. 2020; Liang, He, and Tan 2025), AdaMerging (Yang et al. 2024b) adaptively learns the merging coefficient for each layer of each task vector by minimizing the entropy on unlabeled test data, using it as a surrogate objective to refine the merge model’s performance across multiple tasks. DOGE AM (Wei et al. 2025) applies adaptive projective gradient descent at test time by jointly tuning a small modification vector and layer-wise merging coefficients to minimize prediction entropy on unlabeled inputs. DOGE AM projects updates orthogonally to a shared subspace to resolve task conflicts without hurting the shared knowledge.

Other approaches rely on data statistics, such as Fisher Merging (Matena and Raffel 2022) that requires computing Fisher information matrices. RegMean (Jin et al. 2022) gets rid of expensive gradient computation, formulates merging as a regression problem, then comes up with a computationally efficient and explainable closed-form solution.

#### Notations and problem formulation.

We denote matrices by boldface uppercase letters (e.g., $\bm{X}$, $\bm{W}$, $\bm{\Lambda}$), scalars by lowercase letters (e.g., $\alpha$, $l$). For operators, we denote $||\cdot||$ the Euclidean norm and $\text{tr}(\cdot)$ the trace of a matrix. Following  Jin et al. (2022), we consider the training-free merging framework. In such scenarios, we have access to a pool of multiple candidate models. Each candidate model is denoted as $f_{i}:\mathbb{R}^{N_{i}\times d}\to\mathbb{R}^{N_{i}\times|C|}$, and is fine-tuned on a task-specific dataset $\mathcal{D}_{i}=\{(\bm{X}_{i},\bm{Y}_{i})\}$. Here, $\bm{X}_{i}\in\mathbb{R}^{N_{i}\times d}$ is the batch-input of $N_{i}$ samples each of dimensionality $d$; and $\bm{Y}_{i}\in\mathbb{R}^{N_{i}\times|C|}$ is the corresponding target outputs, where $C$ is the set of classes. Our target is to find a merging function that takes a set of $K$ candidate models $f_{i}$, $i\in[1..K]$ and some data points, e.g., the task-specific training samples or held-out out-of-domain samples, as the inputs and returns a merge model $f_{{M}}$. We assume that the model architecture for all models is the same.

#### Regression Mean (RegMean; (Jin et al. 2022 ) )

Figure: Figure 1: Comparison between RegMean and RegMean++ for model merging. RegMean++ leverages representations from the merge model for merging, enabling accurate alignment with its behavior.
Refer to caption: x1.png

formulates merging as an optimization problem that minimizes the prediction differences between the merge model and candidate models. More concretely, for each linear layer in a transformer layer $l$ of candidate model $f_{i}$, denoted as $\bm{W}^{(l)}_{i}$, given the input feature $\bm{X}^{(l)}_{i}$, RegMean minimizes the following regularized loss:

$$ $\displaystyle\mathcal{L}^{\text{RegMean}}=\sum_{i=1}^{K}\underbrace{||(\bm{W}^{(l)}_{M})^{\top}\bm{X}^{(l)}_{i}-(\bm{W}^{(l)}_{i})^{\top}\bm{X}^{(l)}_{i}||^{2}}_{\text{\char 192}}$ $\displaystyle+\sum_{i=1}^{K}\underbrace{\text{tr}\left[(\bm{W}^{(l)}_{M}-\bm{W}^{(l)}_{i})^{\top}\bm{\Lambda}^{(l)}_{i}(\bm{W}^{(l)}_{M}-\bm{W}^{(l)}_{i})\right]}_{\text{\char 193}},$ (1) $$

where $\bm{W}^{(l)}_{M}$ is the merge linear layer’s weights at the same position as $\bm{W}^{(l)}_{i}$ in the model $f_{i}$, $\bm{\Lambda}^{(l)}_{i}=\frac{1-\alpha}{\alpha}\text{diag}\left((\bm{X}^{(l)}_{i})^{\top}\bm{X}^{(l)}_{i}\right)$ is the diagonal matrix, where $0\leq\alpha\leq 1$ is a predetermined scaling factor. Minimizing the loss $\mathcal{L}^{\text{RegMean}}$ in Eqn. [1](https://arxiv.org/html/2508.03121v1#S2.E1) means finding $\bm{W}^{(l)}_{M}$ that ➀ approximates the behaviors of all candidate models while ➁ enforcing a regularization that keeps linear layer’s weights of the merge model $\bm{W}^{(l)}_{M}$ close to those of the candidate model $\bm{W}^{(l)}_{i}$. Minimizing $\mathcal{L}^{\text{RegMean}}$ describes a linear regression problem, where the inputs are $[\bm{X}^{(l)}_{1},...,\bm{X}^{(l)}_{K}]$ and the target outputs are $[(\bm{W}^{(l)}_{1})^{\top}\bm{X}^{(l)}_{1},...,(\bm{W}^{(l)}_{K})^{\top}\bm{X}^{(l)}_{K}]$. This objective has a closed-form solution given by:

$$ $\displaystyle\bm{W}^{(l)}_{M}=\left[\sum_{i=1}^{K}\left(\widehat{\bm{G}}^{(l)}_{i}\right)\right]^{-1}\sum_{i=1}^{K}\bm{(}\widehat{\bm{G}}^{(l)}_{i})\bm{W}^{(l)}_{i},$ (2) $$

where $\widehat{\bm{G}}^{(l)}_{i}=\alpha\bm{G}^{(l)}_{i}+(1-\alpha)\bm{\Lambda}^{(l)}_{i}=\alpha(\bm{X}^{(l)}_{i})^{\top}\bm{X}^{(l)}_{i}+(1-\alpha)\bm{\Lambda}^{(l)}_{i}$. Proof can be found in Jin et al. (2022) Appendix A. Other types of parameters in the transformer layer are merged using simple averaging.

## 3 The RegMean++

### 3.1 Motivation

Our first start is to revisit the underlying RegMean’s merging mechanism. RegMean operates by independently applying its closed-form solution to linear layers, including those within MLPs and attention heads, across all candidate models. More concretely, let us fine-grain denote $\bm{X}^{(l,j)}_{i}$ be the input features of the $j$-th linear layer $\bm{W}^{(l,j)}_{i}$ in model $f_{i}$ at transformer layer $l$. RegMean computes the $j$-th merge linear layer weights $\bm{W}^{(l,j)}_{M}$ by Eqn. [2](https://arxiv.org/html/2508.03121v1#S2.E2). In this closed-form solution, the merge weight $\bm{W}^{(l,j)}_{M}$ is determined by the individual weights and data statistics $\bm{G}^{(l,j)}_{i}=(\bm{X}_{i}^{(l,j)})^{\top}\bm{X}_{i}^{(l,j)}$, which captures the dependencies among input features across all candidate models. However, this procedure overlooks the information flow and feature transformations that occur both within each layer and across layers of the merge model—dynamics that are crucial for maintaining the merge model’s utilities and generalization.

Figure: Algorithm 1 RegMean++ for Model Merging

### 3.2 Algorithm

We introduce RegMean++, a simple yet effective alternative to RegMean, that incorporates the notion of intra- and cross-layer dependencies between layers of the merge model into RegMean’s objective. Comparison between RegMean and RegMean++ is described in Figure [1](https://arxiv.org/html/2508.03121v1#S2.F1). Similar to RegMean, given a $j$-th linear layer at the transformer layer $l$, RegMean++ computes the inner product matrix as $\bm{G}_{i}^{(l,j)}=(\bm{X}^{(l,j)}_{i})^{\top}\bm{X}^{(l,j)}_{i}$. The key difference between RegMean++ and RegMean lies in how input feature $\bm{X}^{(l,j)}_{i}$ is obtained:
For input features that are activations (cushion representations between transformer layers), RegMean++ computes $\bm{X}^{(l,j)}_{i}$ based on the activations produced by the previous merge layer $f_{M}^{(l-1)}$ in the merge model, that is, $\bm{X}^{(l)}_{i}=f_{M}^{(l-1)}(\bm{X}^{(l-1)}_{i})$ while RegMean relies on the activations produced by the previous candidate layer $f_{i}^{(l-1)}$ in the candidate model, that is, $\bm{X}^{(l)}_{i}=f_{i}^{(l-1)}(\bm{X}^{(l-1)}_{i})$.
All other parameters in the transformer layer, such as embeddings and biases, are merged via averaging. RegMean++ inherits RegMean’s advantages, but introduces a trade-off between model performance and computational cost. During the statistic‐collection phase, RegMean++ incurs additional forward passes, which are computed in the merge model to collect the inner-product matrices, yet the merging time equals that of RegMean. Our RegMean++ pseudocode is described in Algorithm [1](https://arxiv.org/html/2508.03121v1#alg1).

## 4 Experiments

**Table 1: Performance of all merging methods for ViT-B/32 measured on the 8-task benchmark. The global best, local best, and global runner-up are marked. See Appendix Table 8 and Table 9 for the results of ViT-B/16 and ViT-L/14.**
| Method | SUN397 | Cars | RESISC45 | EuroSAT | SVHN | GTSRB | MNIST | DTD | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Fine-tuned | $75.0$ | $78.3$ | $95.2$ | $99.0$ | $97.3$ | $98.9$ | $99.6$ | $79.7$ | $90.3$ |
| MTL | $72.3$ | $76.6$ | $92.2$ | $97.9$ | $95.5$ | $97.7$ | $99.3$ | $77.7$ | $88.6$ |
|  | Data-Free Methods |  |  |  |  |  |  |  |  |
| Model Soups | $65.4$ | $62.4$ | $70.6$ | $75.7$ | $64.5$ | $55.0$ | $86.3$ | $50.6$ | $66.3$ |
| Task Arithmetic | $57.0$ | $55.7$ | $64.7$ | $73.3$ | $77.9$ | $68.5$ | $96.1$ | $47.1$ | $67.5$ |
| TIES-Merging | $67.0$ | $64.2$ | $74.3$ | $74.5$ | $77.7$ | $69.4$ | $94.1$ | $54.0$ | $71.9$ |
| TSV-M | $67.6$ | $71.6$ | $84.7$ | ${\color[rgb]{.75,0,.25}93.4}$ | ${\color[rgb]{.75,0,.25}91.9}$ | ${\color[rgb]{.75,0,.25}92.5}$ | ${\color[rgb]{.75,0,.25}\underline{98.9}}$ | $63.8$ | ${\color[rgb]{.75,0,.25}83.1}$ |
| DOGE TA | $67.7$ | $69.9$ | $81.9$ | $89.8$ | $86.2$ | $86.8$ | $98.3$ | $63.8$ | $80.6$ |
| Iso-C | $\underline{71.0}$ | $73.9$ | $86.0$ | $89.6$ | $84.8$ | $90.8$ | $98.2$ | $65.8$ | $82.5$ |
| Iso-CTS | 71.1 | 74.6 | ${\color[rgb]{.75,0,.25}86.6}$ | $89.1$ | $83.4$ | $90.4$ | $98.1$ | ${\color[rgb]{.75,0,.25}68.5}$ | $82.7$ |
|  | Training-Free Methods |  |  |  |  |  |  |  |  |
| Fisher Merging | $67.4$ | $67.6$ | $75.4$ | $70.5$ | $76.5$ | $62.2$ | $87.9$ | $55.3$ | $70.3$ |
| RegMean | $68.6$ | $70.0$ | $84.6$ | $\underline{95.4}$ | $\underline{92.6}$ | $83.4$ | $98.4$ | $66.1$ | $82.4$ |
| RegMean++ (Ours) | ${\color[rgb]{.75,0,.25}69.3}$ | ${\color[rgb]{.75,0,.25}70.5}$ | ${\color[rgb]{.75,0,.25}\underline{86.7}}$ | 96.1 | 94.1 | ${\color[rgb]{.75,0,.25}90.4}$ | 99.0 | ${\color[rgb]{.75,0,.25}\underline{68.7}}$ | ${\color[rgb]{.75,0,.25}\underline{84.4}}$ |
|  | Test-Time Adaption |  |  |  |  |  |  |  |  |
| Layer-wise AdaMerging | $67.8$ | $71.1$ | $83.9$ | $92.3$ | $87.8$ | $\underline{93.3}$ | $98.2$ | $66.8$ | $82.6$ |
| DOGE AM | ${\color[rgb]{.75,0,.25}70.6}$ | ${\color[rgb]{.75,0,.25}\underline{74.5}}$ | 88.7 | ${\color[rgb]{.75,0,.25}93.7}$ | ${\color[rgb]{.75,0,.25}91.4}$ | 95.5 | ${\color[rgb]{.75,0,.25}98.8}$ | 73.0 | 85.8 |

#### Models and datasets.

Following prior works (Ilharco et al. 2022; Yang et al. 2024b; Wei et al. 2025), we evaluate the effectiveness of multi-task model merging methods on eight standard image classification datasets including: SUN397 (Xiao et al. 2016), Stanford Cars (Cars) (Krause et al. 2013), RESISC45 (Cheng, Han, and Lu 2017), EuroSAT (Helber et al. 2019), SVHN (Netzer et al. 2011), GTSRB (Stallkamp et al. 2011), MNIST (LeCun et al. 1998), and DTD (Cimpoi et al. 2014).

Following Wang et al. (2024), we employ $12$ additional tasks for measuring the sustainability of merging methods. Those include: Flowers102 (Nilsback and Zisserman 2008), PCAM (Veeling et al. 2018), FER2013 (Goodfellow et al. 2013), OxfordIIITPet (Parkhi et al. 2012), STL10 (Coates, Ng, and Lee 2011), CIFAR100 (Krizhevsky et al. 2009), CIFAR10 (Krizhevsky et al. 2009), Food101 (Bossard, Guillaumin, and Van Gool 2014), FashionMNIST (Xiao, Rasul, and Vollgraf 2017), EMNIST (Cohen et al. 2017), KMNIST (Clanuwat et al. 2018), and RenderedSST2 (Socher et al. 2013b).

We assess the performance of merging methods on three CLIP model variants (Radford et al. 2021) with ViT-B/32, ViT-B/16, and ViT-L/14. For the candidate models, we employ off-the-shelf checkpoints from Tang et al. (2024a). Details of these datasets’ description and models can be found in Appendix A.1 and Appendix A.2, respectively.

#### Comparison methods.

We compare RegMean++ against $11$ recent advanced model merging methods spanning three categories: (1) Data-Free methods: Model Soups (Wortsman et al. 2022), Task Arithmetic (Ilharco et al. 2022), TIES-Merging (Yadav et al. 2023), TSV-M (Gargiulo et al. 2025), DOGE TA (Wei et al. 2025), Iso-C and Iso-CTS (Marczak et al. 2025). (2) Training-Free methods: Fisher Merging (Matena and Raffel 2022) and RegMean (Jin et al. 2022). (3) Test-Time Adaptation: AdaMerging (Yang et al. 2024b) and DOGE AM (Wei et al. 2025). In addition, we also consider the MTL as an upper-bound performance. Details about these methods can be found in Appendix A.3.

#### Experimental setup.

We employ FusionBench (Tang et al. 2024a) for merging evaluation.
We report the accuracy and normalized accuracy (See Appendix B.1 for details) over the test split of all tasks. Hyperparameters for settings are specified in their respective subsections, and further detailed in Appendix B.2. Due to space constraints, we present key results in the main text and defer the full experimental setups and additional results to Appendix B.3 and Appendix C, respectively. Our implementation and guidelines for reproducing those results are attached to the supplemental materials.

Figure: Figure 2: Average normalized accuracy of all merging methods for ViT-B/32, ViT-B/16, and ViT-L/14; evaluated on different numbers of tasks (up to $20$ tasks).
Refer to caption: x2.png

Figure: Figure 3: Sequential merging performance of RegMean++ (dark orange) and RegMean (cornflower blue) for ViT-B/32, ViT-B/16, and ViT-L/14. Results show the mean of average accuracy on all $20$ tasks across five different task sequences.
Refer to caption: x3.png

## 5 Results and Analysis

### 5.1 Main Results

Performance of RegMean++ and other methods on eight tasks is shown in Table [1](https://arxiv.org/html/2508.03121v1#S4.T1). We observe that RegMean++ consistently surpasses RegMean on all tasks, achieving an average improvement of $2.0\%$, and demonstrates gains of $1.2\%$ and $0.6\%$ when evaluated on ViT-B/16 and ViT-L/14, respectively. Compared to other data-free and training-free methods, RegMean++ achieves competitive or the best performance. Furthermore,
RegMean++, despite requiring no access to test-time data or optimization, can rival or surpass test-time adaptation methods: outperforming Layer-wise AdaMerging ($84.4\%$ vs. $82.6\%$), and close to DOGE AM ($85.8\%$). RegMean++ achieves the best results on three specific tasks—EuroSAT ($96.1\%$), SVHN ($94.1\%$), and MNIST ($99.0\%$). These results validate that leveraging the intra- and cross-layer dependencies provides a significant advantage in the merging process.

### 5.2 Sustainability to Large-Scale Tasks

In this section, following Wang et al. (2024), we evaluate the sustainability of merging methods when scaling the number of tasks up to $20$. A higher number of merging tasks, a higher level of complexity and conflict between candidate models. Merging methods are thus expected to maintain high accuracy under this large-scale task setting.
Since evaluating all possible task combinations is computationally expensive, we fix the order of tasks and add four tasks one by one. See Appendix B.3 for the experiment setting and details of tasks. Note that these experiments are conducted independently; that is, for each algorithm, the merging process is re-executed from scratch whenever new tasks are added. Performance of merging methods for ViT-B/32, ViT-B/16, and ViT-L/14 is visualized in Figure [2](https://arxiv.org/html/2508.03121v1#S4.F2). We observe that RegMean++, along with RegMean, TSV-M, Iso-C, Iso-CTS, AdaMering, and DOGE AM demonstrate strong sustainability as the number of tasks increases. In contrast, Model Soups, Task Arithmetic, TIES-Merging, Fisher Merging, and DOGE TA exhibit a noticeable decline in accuracy as the number of tasks increases.

**Table 2: ID and OOD performance of ViT-B/32, ViT-B/16, and ViT-L/14. We report the mean of the average accuracy over five runs. The global best, local best, and global runner-up are marked.**
| Method | ViT-B/32 | ViT-B/16 | ViT-L/14 |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| ID | OOD | ID | OOD | ID | OOD |  |
| Model Soups | $70.2$ | $50.6$ | $75.4$ | $\underline{58.9}$ | $82.7$ | $\underline{67.9}$ |
| Task Arithmetic | $73.7$ | $42.6$ | $80.2$ | $54.1$ | $84.7$ | $61.8$ |
| TIES-Merging | $73.0$ | 51.7 | $77.7$ | 59.9 | $84.7$ | 68.3 |
| TSV-M | ${\color[rgb]{.75,0,.25}84.8}$ | $45.7$ | ${\color[rgb]{.75,0,.25}88.2}$ | $54.8$ | $91.3$ | $62.1$ |
| DOGE TA | $83.1$ | $49.4$ | $86.5$ | $56.0$ | $89.8$ | $65.8$ |
| Iso-C | $83.1$ | $49.3$ | $87.8$ | $58.2$ | $92.4$ | $65.9$ |
| Iso-CTS | $82.8$ | $48.6$ | $88.1$ | $58.2$ | ${\color[rgb]{.75,0,.25}\underline{92.8}}$ | $65.2$ |
| Fisher Merging | $74.3$ | ${\color[rgb]{.75,0,.25}\underline{51.0}}$ | $78.4$ | ${\color[rgb]{.75,0,.25}58.0}$ | $83.7$ | $64.2$ |
| RegMean | $84.6$ | $48.3$ | $88.0$ | $56.5$ | $91.5$ | $64.5$ |
| RegMean++ (Ours) | ${\color[rgb]{.75,0,.25}\underline{86.2}}$ | $48.9$ | ${\color[rgb]{.75,0,.25}\underline{88.9}}$ | $56.7$ | ${\color[rgb]{.75,0,.25}91.9}$ | ${\color[rgb]{.75,0,.25}64.7}$ |
| Layer-wise AdaMerging | $84.5$ | ${\color[rgb]{.75,0,.25}48.9}$ | $87.2$ | ${\color[rgb]{.75,0,.25}55.5}$ | $91.8$ | ${\color[rgb]{.75,0,.25}65.3}$ |
| DOGE AM | 87.4 | $47.7$ | 89.6 | $54.9$ | 93.0 | $63.4$ |

### 5.3 Sequential Merging

Merging from scratch when new candidate models come is computationally expensive and infeasible. Sequential merging is a practical scenario where tasks arrive over time. In this setting, we evaluate the performance of RegMean and RegMean++ by merging the first four candidate models in a predefined task sequence, then progressively merging the result model with the next four, repeating this process until all $20$ tasks are merged. Figure [3](https://arxiv.org/html/2508.03121v1#S4.F3) presents the performance of RegMean and RegMean++ for all three models, averaged over five different task sequences (see Appendix B.3 for details). RegMean++ demonstrates greater improvements when more candidate models are merged. Especially for ViT-B/32 and ViT-B/16, where the performance gaps become more apparent as the number of merged tasks increases. Further analysis indicates that RegMean++ better accommodates new tasks while exhibiting reduced forgetting on the early ones, indicated in Appendix Figure 8.

**Table 3: Performance of merging methods of ViT-B/32 on corrupted test data. The global best, local best, and global runner-up are marked.**
| Method | Clean Test Set | Corrupted Test Set |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Motion | Impulse | Gaussian | Pixelate | Spatter | Contrast | JPEG | Avg. |  |  |
| Model Soups | $76.0$ | $64.6$ | $56.9$ | $58.1$ | $28.5$ | $61.3$ | $64.7$ | $66.3$ | $57.2$ |
| Task Arithmetic | $77.5$ | $65.9$ | $58.9$ | $59.6$ | $29.7$ | $63.5$ | $66.0$ | $67.8$ | $58.8$ |
| TIES-Merging | $73.3$ | $63.2$ | $54.5$ | $56.2$ | $28.1$ | $57.7$ | $63.8$ | $64.4$ | $55.4$ |
| TSV-M | ${\color[rgb]{.75,0,.25}88.3}$ | ${\color[rgb]{.75,0,.25}78.9}$ | ${\color[rgb]{.75,0,.25}\underline{69.9}}$ | ${\color[rgb]{.75,0,.25}69.1}$ | ${\color[rgb]{.75,0,.25}37.8}$ | ${\color[rgb]{.75,0,.25}\underline{75.4}}$ | ${\color[rgb]{.75,0,.25}77.2}$ | ${\color[rgb]{.75,0,.25}80.2}$ | ${\color[rgb]{.75,0,.25}69.8}$ |
| DOGE TA | $86.1$ | $77.3$ | $66.0$ | $66.2$ | $37.5$ | $71.5$ | $76.1$ | $77.7$ | $67.5$ |
| Iso-C | $84.8$ | $75.9$ | $62.2$ | $63.9$ | $35.0$ | $69.2$ | $76.5$ | $75.3$ | $65.4$ |
| Iso-CTS | $84.9$ | $75.7$ | $61.9$ | $63.5$ | $34.2$ | $69.2$ | $76.5$ | $75.4$ | $65.2$ |
| Fisher Merging | $79.1$ | $67.0$ | $59.8$ | $60.7$ | $29.3$ | $64.9$ | $67.9$ | $69.0$ | $59.8$ |
| RegMean | $89.1$ | $79.7$ | $69.1$ | $67.3$ | $37.1$ | $75.2$ | $78.0$ | $80.9$ | $69.6$ |
| RegMean++ (Ours) | ${\color[rgb]{.75,0,.25}\underline{89.7}}$ | ${\color[rgb]{.75,0,.25}\underline{81.8}}$ | 70.0 | ${\color[rgb]{.75,0,.25}68.4}$ | ${\color[rgb]{.75,0,.25}37.9}$ | 75.9 | ${\color[rgb]{.75,0,.25}79.6}$ | ${\color[rgb]{.75,0,.25}\underline{82.8}}$ | ${\color[rgb]{.75,0,.25}70.9}$ |
| Layer-wise AdaMerging | $88.4$ | $81.3$ | ${\color[rgb]{.75,0,.25}69.0}$ | $\underline{71.2}$ | $\underline{41.3}$ | $74.5$ | $\underline{80.2}$ | $80.1$ | $\underline{71.1}$ |
| DOGE AM | 90.9 | 85.0 | $65.7$ | 73.4 | 44.2 | ${\color[rgb]{.75,0,.25}\underline{75.4}}$ | 83.8 | 84.2 | 73.1 |

### 5.4 Out-of-Domain Generalization

**Table 4: Region-specific merging and component-specific merging performance of RegMean++ and RegMean for ViT-B/32, ViT-B/16, and ViT-L/14 across eight tasks.**
| Components | ViT-B/32 | ViT-B/16 | ViT-L/14 |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| RegMean | RegMean++ | RegMean | RegMean++ | RegMean | RegMean++ |  |
| All | $82.4$ | $84.4$ | $86.0$ | $87.2$ | $90.4$ | $91.0$ |
| Early | $67.1$ | $67.8$ | $73.9$ | $74.8$ | $81.2$ | $81.5$ |
| Middle | $74.0$ | 77.2 | 78.9 | 81.0 | $85.6$ | 86.6 |
| Deep | 74.4 | $75.7$ | $78.7$ | $79.5$ | 85.8 | $86.4$ |
| Middle & Deep | $80.7$ | $83.5$ | $84.4$ | $86.5$ | $89.4$ | $90.5$ |
| Attention heads | $72.9$ | $76.0$ | $78.0$ | $80.1$ | $85.0$ | $86.5$ |
| MLPs | 78.8 | 81.4 | 82.7 | 84.5 | 88.1 | 88.9 |

In this section, we evaluate the OOD generalization of merging methods. We randomly select two from eight tasks to serve as OOD tasks, while the remaining six are used for merging.
Table [2](https://arxiv.org/html/2508.03121v1#S5.T2) reports the ID and OOD accuracy of all merging methods evaluated on three ViT models. Each result is the mean of the average accuracy over five runs. Across all models, RegMean++ consistently achieves strong ID performance, outperforming RegMean by a margin of $+1.6$, $+0.9$, and $+0.4$ points for ViT-B/32, ViT-B/16, and ViT-L/14, respectively. RegMean++ slightly outperforms RegMean for all three models on OOD tasks. Notably, TIES-Merging achieves the best OOD accuracy across ViT-B/32 ($51.7\%$), ViT-B/16 ($59.9\%$), and ViT-L/14 ($68.3\%$), but low performance in ID tasks, highlighting a trade-off between OOD and ID generalization in merging. Overall, RegMean++ offers a good trade-off between ID and OOD generalization across all models, slightly outperforming RegMean, and demonstrating competitive performance without requiring computation or access to test data, as for test-time adaptation methods.

### 5.5 Robustness Against Distribution Shifts

Figure: Figure 4: Layer-wise merging performance of RegMean++ (dark orange) and RegMean (cornflower blue) for ViT-B/32, ViT-B/16, and ViT-L/14. Results show average accuracy across eight tasks.
Refer to caption: x4.png

We first note that the notion of distribution shift is very broad and can exhibit in many forms, such as covariance shift, label shift, concept shift, class-conditional shift, etc. Here, following Tang et al. (2024b); Yang et al. (2024b); Hendrycks and Dietterich (2019), we evaluate merging methods’ robustness on test data by employing seven types of noises (covariance shift), including Motion Blur, Impulse Noise, Gaussian Noise, Pixelate, Spatter, Contrast, and JPEG Compression. Table [3](https://arxiv.org/html/2508.03121v1#S5.T3) shows the accuracy of merging methods for ViT-B/32 under corrupted test data. We observe that RegMean++ achieves superior performance as it surpasses all training-free and data-free methods on both the clean test set and the corrupted test sets. A similar trend can also be observed for ViT-B/16 and ViT-L/14 in Appendix Table 10 and Table 11, respectively. These results highlight that RegMean++ not only performs effectively on ID and OOD tasks but is also robust to distribution shift.

### 5.6 Effects of Merging in Different Spaces

One might ask:
(1) Which component—attention heads or MLPs—contributes more effectively to the merging performance?
(2) How does the choice of transformer layers influence the effectiveness of merging?

In this section, we measure the effects of merging in different spaces (i.e., different layers and components) for RegMean++ and RegMean. We perform the following empirical experiments: (1) region-specific merging: all the linear layers from a specific set of transformer layers grouped by position in the model are used for merging and compare performance across three configs: (i) early layers ($1,2,3,4$), (ii) middle layers ($5,6,7,8$), and deep layers ($9,10,11,12$). (2) Layer-wise merging: all the linear layers from a specific transformer layer are used for merging. (3) Component-specific merging: all the linear layers in MLP components or attention heads are used for merging. Note that across these empirical experiments, only the selected linear layers are merged using the respective merging methods, while simple averaging is applied for the other linear layers.

We report the results in Table [4](https://arxiv.org/html/2508.03121v1#S5.T4) and Figure [4](https://arxiv.org/html/2508.03121v1#S5.F4). We defer additional results and analysis of other merging methods to Appendix C. Overall, in all merging scenarios—region-specific, layer-wise, and component-specific—RegMean++ consistently improves over RegMean. These results validate the enhanced capability of RegMean++ in exploiting the layer’s dependencies in the merge model for better merging. Furthermore, we observe the following important insights.

#### Merging using middle and deep layers preserves overall performance.

For all models and merging methods, using the middle and deep layers (layers $5–12$ for ViT-B/32 and ViT-B/16, and layers $8–24$ for ViT-L/14) achieves high performance, closely matching that of using all layers. For example, RegMean++ achieves $98\%$, $99\%$, and $99\%$ of the full-layer accuracy for ViT-B/32 ($83.5/84.4$) and ViT-B/16 ($86.5/87.2$), and ViT-L/14 ($90.5/91.0$). When merging is performed using only the middle or deep layers, performance remains above $90\%$ of the full-layer. In contrast, early layers contribute less, with a notable drop in accuracy.

#### MLP module linear layer merging outperforms attention head merging.

Component-specific merging analysis shows that merging using linear layers from the MLP modules yields higher accuracy than that from attention heads across all models. This implies that MLPs may contain richer semantic representations for merging. This observation aligns with previous findings (Geva et al. 2021; Meng et al. 2022; Chen et al. 2024), which have shown that MLPs serve as dictionaries of factual and task-relevant knowledge in transformer models.

#### Intermediate (middle and deep) layers surpass the last layer in merging performance.

Figure [4](https://arxiv.org/html/2508.03121v1#S5.F4) shows that middle layers consistently surpass the deep layers and the last layer for merging. This suggests that the deep layers and final layer may be specialized for task-specific, while the middle layers serve as sources of more meaningful features beneficial for merging.

### 5.7 Effects of Data Characteristics

**Table 5: Accuracy of Fisher Merging (Fisher), RegMean (RM), and RegMean++ (RM++) for ViT-B/32, ViT-B/16, and ViT-L/14 on three scenarios: random ID samples, ID class imbalance, and OOD samples. We report average accuracy across $8$ tasks.**
| Characteristics | ViT-B/32 | ViT-B/16 | ViT-L/14 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Fisher | RM | RM++ | Fisher | RM | RM++ | Fisher | RM | RM++ |  |
| ID (random) | $70.3$ | $82.4$ | $84.4$ | $75.6$ | $86.0$ | $87.2$ | $82.4$ | $90.4$ | $91.0$ |
| Class Imbalance | $69.2$ | $75.5$ | $76.5$ | $74.5$ | $80.6$ | $81.6$ | $77.3$ | $86.0$ | $86.4$ |
| OOD Samples | $68.5$ | $66.5$ | $65.5$ | $73.9$ | $70.2$ | $71.9$ | $80.6$ | $78.9$ | $79.6$ |

Data plays the central role in the training-free merging framework. However, full access to training datasets is often restricted in practice due to privacy concerns. In this section, we investigate the effects of data characteristics on merging performance with three different experiment settings: (1) number of samples: we randomly select samples from the training set of each task, (2) class imbalance: we randomly select samples from a random class in the training set of each task, and (3) OOD samples: we randomly select samples from the ImageNet database (Deng et al. 2009) as an OOD dataset for all tasks.

Figure: Figure 5: Left: Effects of the number of ID samples on merging performance of training-free merging methods for ViT-B/32. Right: Representation bias of RegMean and RegMean++ for ViT-B/32. See Appendix Figure 10 for results for ViT-B/16 and ViT-L/14.
Refer to caption: x5.png

#### Effects of OOD samples.

Table [5](https://arxiv.org/html/2508.03121v1#S5.T5) indicates that if OOD samples are used for merging, Fisher Merging demonstrates a stable performance. In contrast, RegMean and RegMean++ exhibit reduced accuracy, particularly for ViT-B/32 (e.g., RegMean++ drops from 84.4% to 65.5%). This reflects a limitation of regression-based methods when the merging data distribution is misaligned with the task domains.

#### Effects of the number of samples.

Figure [5](https://arxiv.org/html/2508.03121v1#S5.F5) (left) and Appendix Figure 6 show that increasing the number of ID samples used for merging yields only a small improvement. The accuracy quickly saturates. This result implies that the effectiveness of merging is influenced more by the quality of the selected samples than by their quantity. Even using a small number of ID samples can achieve near-optimal performance in training-free settings.

#### Effects of class imbalance.

Table 5 shows that class imbalance can cause a significant accuracy drop for RegMean and RegMean++, whereas Fisher Merging remains relatively stable.

### 5.8 Analysis on Representation Bias

We compute the Euclidean distance between the feature representations (the activations from the last layer) in the merge model and candidate models measured on the task-specific test datasets. A merge model with less representation bias is likely better (Yang et al. 2024a).
As shown in Figure [5](https://arxiv.org/html/2508.03121v1#S5.F5) (right), RegMean++ improves feature alignment across tasks, further supporting the evidence of performance gains as discussed in previous sections.

## 6 Conclusion

This paper introduces RegMean++, an improved version of RegMean that incorporates intra- and cross-layer dependencies into the merging objective. RegMean++ addresses RegMean’s limitations in feature propagation and memory overhead. Extensive experiments demonstrate that RegMean++ achieves robust ID performance, improved OOD generalization, and strong scalability, outperforming or matching state-of-the-art merging methods across a wide range of settings.

## References

- Bossard, Guillaumin, and Van Gool (2014)
Bossard, L.; Guillaumin, M.; and Van Gool, L. 2014.
Food-101–mining discriminative components with random forests.
In *Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13*, 446–461. Springer.
- Chen et al. (2024)
Chen, Y.; Cao, P.; Chen, Y.; Wang, Y.; Liu, S.; Liu, K.; and Zhao, J. 2024.
Cracking factual knowledge: A comprehensive analysis of degenerate knowledge neurons in large language models.
*arXiv preprint arXiv:2402.13731*.
- Cheng, Han, and Lu (2017)
Cheng, G.; Han, J.; and Lu, X. 2017.
Remote sensing image scene classification: Benchmark and state of the art.
*Proceedings of the IEEE*, 105(10): 1865–1883.
- Choshen et al. (2022)
Choshen, L.; Venezian, E.; Slonim, N.; and Katz, Y. 2022.
Fusing finetuned models for better pretraining.
*arXiv preprint arXiv:2204.03044*.
- Cimpoi et al. (2014)
Cimpoi, M.; Maji, S.; Kokkinos, I.; Mohamed, S.; and Vedaldi, A. 2014.
Describing textures in the wild.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, 3606–3613.
- Clanuwat et al. (2018)
Clanuwat, T.; Bober-Irizar, M.; Kitamoto, A.; Lamb, A.; Yamamoto, K.; and Ha, D. 2018.
Deep learning for classical japanese literature.
*arXiv preprint arXiv:1812.01718*.
- Coates, Ng, and Lee (2011)
Coates, A.; Ng, A.; and Lee, H. 2011.
An analysis of single-layer networks in unsupervised feature learning.
In *Proceedings of the fourteenth international conference on artificial intelligence and statistics*, 215–223. JMLR Workshop and Conference Proceedings.
- Cohen et al. (2017)
Cohen, G.; Afshar, S.; Tapson, J.; and Van Schaik, A. 2017.
EMNIST: Extending MNIST to handwritten letters.
In *2017 international joint conference on neural networks (IJCNN)*, 2921–2926. IEEE.
- Deng et al. (2009)
Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009.
Imagenet: A large-scale hierarchical image database.
In *2009 IEEE conference on computer vision and pattern recognition*, 248–255. Ieee.
- Gargiulo et al. (2025)
Gargiulo, A. A.; Crisostomi, D.; Bucarelli, M. S.; Scardapane, S.; Silvestri, F.; and Rodola, E. 2025.
Task singular vectors: Reducing task interference in model merging.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, 18695–18705.
- Geva et al. (2021)
Geva, M.; Schuster, R.; Berant, J.; and Levy, O. 2021.
Transformer Feed-Forward Layers Are Key-Value Memories.
In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*. Association for Computational Linguistics.
- Goodfellow et al. (2013)
Goodfellow, I. J.; Erhan, D.; Carrier, P. L.; Courville, A.; Mirza, M.; Hamner, B.; Cukierski, W.; Tang, Y.; Thaler, D.; Lee, D.-H.; et al. 2013.
Challenges in representation learning: A report on three machine learning contests.
In *Neural information processing: 20th international conference, ICONIP 2013, daegu, korea, november 3-7, 2013. Proceedings, Part III 20*, 117–124. Springer.
- Helber et al. (2019)
Helber, P.; Bischke, B.; Dengel, A.; and Borth, D. 2019.
Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.
*IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing*, 12(7): 2217–2226.
- Hendrycks and Dietterich (2019)
Hendrycks, D.; and Dietterich, T. 2019.
Benchmarking Neural Network Robustness to Common Corruptions and Perturbations.
In *International Conference on Learning Representations*.
- Ilharco et al. (2022)
Ilharco, G.; Ribeiro, M. T.; Wortsman, M.; Schmidt, L.; Hajishirzi, H.; and Farhadi, A. 2022.
Editing models with task arithmetic.
In *The Eleventh International Conference on Learning Representations*.
- Jin et al. (2022)
Jin, X.; Ren, X.; Preotiuc-Pietro, D.; and Cheng, P. 2022.
Dataless Knowledge Fusion by Merging Weights of Language Models.
In *The Eleventh International Conference on Learning Representations*.
- Kingma and Ba (2014)
Kingma, D. P.; and Ba, J. 2014.
Adam: A method for stochastic optimization.
*arXiv preprint arXiv:1412.6980*.
- Krause et al. (2013)
Krause, J.; Stark, M.; Deng, J.; and Fei-Fei, L. 2013.
3d object representations for fine-grained categorization.
In *Proceedings of the IEEE international conference on computer vision workshops*, 554–561.
- Krizhevsky et al. (2009)
Krizhevsky, A.; et al. 2009.
Learning multiple layers of features from tiny images.
- LeCun et al. (1998)
LeCun, Y.; eon Bottou, L.; Bengio, Y.; et al. 1998.
Gradient-Based Learning Applied to Document Recognition.
*PROC. OF THE IEEE*, 1.
- Liang, He, and Tan (2025)
Liang, J.; He, R.; and Tan, T. 2025.
A comprehensive survey on test-time adaptation under distribution shifts.
*International Journal of Computer Vision*, 133(1): 31–64.
- Marczak et al. (2025)
Marczak, D.; Magistri, S.; Cygert, S.; Twardowski, B.; Bagdanov, A. D.; and van de Weijer, J. 2025.
No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces.
In *Forty-second International Conference on Machine Learning*.
- Matena and Raffel (2022)
Matena, M. S.; and Raffel, C. A. 2022.
Merging models with fisher-weighted averaging.
*Advances in Neural Information Processing Systems*, 35: 17703–17716.
- Meng et al. (2022)
Meng, K.; Bau, D.; Andonian, A.; and Belinkov, Y. 2022.
Locating and editing factual associations in gpt.
*Advances in neural information processing systems*, 35: 17359–17372.
- Netzer et al. (2011)
Netzer, Y.; Wang, T.; Coates, A.; Bissacco, A.; Wu, B.; Ng, A. Y.; et al. 2011.
Reading digits in natural images with unsupervised feature learning.
In *NIPS workshop on deep learning and unsupervised feature learning*, volume 2011, 4. Granada.
- Nilsback and Zisserman (2008)
Nilsback, M.-E.; and Zisserman, A. 2008.
Automated flower classification over a large number of classes.
In *2008 Sixth Indian conference on computer vision, graphics & image processing*, 722–729. IEEE.
- Parkhi et al. (2012)
Parkhi, O. M.; Vedaldi, A.; Zisserman, A.; and Jawahar, C. 2012.
Cats and dogs.
In *2012 IEEE conference on computer vision and pattern recognition*, 3498–3505. IEEE.
- Radford et al. (2021)
Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021.
Learning transferable visual models from natural language supervision.
In *International conference on machine learning*, 8748–8763. PmLR.
- Radford et al. (2019)
Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I.; et al. 2019.
Language models are unsupervised multitask learners.
- Socher et al. (2013a)
Socher, R.; Perelygin, A.; Wu, J.; Chuang, J.; Manning, C. D.; Ng, A.; and Potts, C. 2013a.
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank.
In *Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing*, 1631–1642. Seattle, Washington, USA: Association for Computational Linguistics.
- Socher et al. (2013b)
Socher, R.; Perelygin, A.; Wu, J.; Chuang, J.; Manning, C. D.; Ng, A. Y.; and Potts, C. 2013b.
Recursive deep models for semantic compositionality over a sentiment treebank.
In *Proceedings of the 2013 conference on empirical methods in natural language processing*, 1631–1642.
- Stallkamp et al. (2011)
Stallkamp, J.; Schlipsing, M.; Salmen, J.; and Igel, C. 2011.
The German traffic sign recognition benchmark: a multi-class classification competition.
In *The 2011 international joint conference on neural networks*, 1453–1460. IEEE.
- Sung et al. (2023)
Sung, Y.-L.; Li, L.; Lin, K.; Gan, Z.; Bansal, M.; and Wang, L. 2023.
An Empirical Study of Multimodal Model Merging.
In *Findings of the Association for Computational Linguistics: EMNLP 2023*, 1563–1575.
- Tang et al. (2024a)
Tang, A.; Shen, L.; Luo, Y.; Hu, H.; Du, B.; and Tao, D. 2024a.
Fusionbench: A comprehensive benchmark of deep model fusion.
*arXiv preprint arXiv:2406.03280*.
- Tang et al. (2024b)
Tang, A.; Shen, L.; Luo, Y.; Yin, N.; Zhang, L.; and Tao, D. 2024b.
Merging Multi-Task Models via Weight-Ensembling Mixture of Experts.
In *International Conference on Machine Learning*, 47778–47799. PMLR.
- Veeling et al. (2018)
Veeling, B. S.; Linmans, J.; Winkens, J.; Cohen, T.; and Welling, M. 2018.
Rotation equivariant CNNs for digital pathology.
In *Medical image computing and computer assisted intervention–mICCAI 2018: 21st international conference, granada, Spain, September 16-20, 2018, proceedings, part II 11*, 210–218. Springer.
- Wang et al. (2020)
Wang, D.; Shelhamer, E.; Liu, S.; Olshausen, B.; and Darrell, T. 2020.
Tent: Fully Test-Time Adaptation by Entropy Minimization.
In *International Conference on Learning Representations*.
- Wang et al. (2024)
Wang, K.; Dimitriadis, N.; Ortiz-Jiménez, G.; Fleuret, F.; and Frossard, P. 2024.
Localizing task information for improved model merging and compression.
In *Proceedings of the 41st International Conference on Machine Learning*, 50268–50287.
- Wei et al. (2025)
Wei, Y.; Tang, A.; Shen, L.; Hu, Z.; Yuan, C.; and Cao, X. 2025.
Modeling multi-task model merging as adaptive projective gradient descent.
In *International conference on machine learning*.
- Wortsman et al. (2022)
Wortsman, M.; Ilharco, G.; Gadre, S. Y.; Roelofs, R.; Gontijo-Lopes, R.; Morcos, A. S.; Namkoong, H.; Farhadi, A.; Carmon, Y.; Kornblith, S.; et al. 2022.
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.
In *International conference on machine learning*, 23965–23998. PMLR.
- Xiao, Rasul, and Vollgraf (2017)
Xiao, H.; Rasul, K.; and Vollgraf, R. 2017.
Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.
*arXiv preprint arXiv:1708.07747*.
- Xiao et al. (2016)
Xiao, J.; Ehinger, K. A.; Hays, J.; Torralba, A.; and Oliva, A. 2016.
Sun database: Exploring a large collection of scene categories.
*International Journal of Computer Vision*, 119: 3–22.
- Yadav et al. (2023)
Yadav, P.; Tam, D.; Choshen, L.; Raffel, C. A.; and Bansal, M. 2023.
Ties-merging: Resolving interference when merging models.
*Advances in Neural Information Processing Systems*, 36: 7093–7115.
- Yadav et al. (2024)
Yadav, P.; Vu, T.; Lai, J.; Chronopoulou, A.; Faruqui, M.; Bansal, M.; and Munkhdalai, T. 2024.
What Matters for Model Merging at Scale?
*arXiv preprint arXiv:2410.03617*.
- Yang et al. (2024a)
Yang, E.; Shen, L.; Wang, Z.; Guo, G.; Chen, X.; Wang, X.; and Tao, D. 2024a.
Representation Surgery for Multi-Task Model Merging.
In *Forty-first International Conference on Machine Learning*.
- Yang et al. (2024b)
Yang, E.; Wang, Z.; Shen, L.; Liu, S.; Guo, G.; Wang, X.; and Tao, D. 2024b.
AdaMerging: Adaptive Model Merging for Multi-Task Learning.
In *The Twelfth International Conference on Learning Representations*.
- Yu et al. (2023)
Yu, L.; Yu, B.; Yu, H.; Huang, F.; and Li, Y. 2023.
Language models are super mario: Absorbing abilities from homologous models as a free lunch.
In *Forty-first International Conference on Machine Learning*.

## Appendix A Datasets, Models, and Merging Methods

### A.1 Datasets

#### Standard datasets.

Task descriptions and statistics of datasets used for the standard $8$-task image classification benchmark(^1^11https://huggingface.co/collections/tanganke/the-eight-image-classification-tasks-6644ce0376c0a469f6928507) are described as follows:

- •
SUN397 (Xiao et al. 2016) contains more than $100,000$ images of $397$ categories for benchmarking scene understanding. The number of images varies across categories, but there are at least $100$ images each.
- •
Stanford Cars (Cars) (Krause et al. 2013) has $16,185$ images in total of $196$ types of cars and evenly split for training and testing sets.
- •
RESISC45 (Cheng, Han, and Lu 2017) is developed for remote sensing image scene classification. This dataset covers $45$ scene classes with $700$ images of size $256\times 256$ for each.
- •
EuroSAT (Helber et al. 2019) is used for land use and land cover classification using Sentinel-2 satellite images of size $64\times 64$, consisting of $27,000$ images covering $10$ classes.
- •
SVHN (Netzer et al. 2011) is a street view house number classification benchmark, containing more than $600,000$ RGB images of $10$ printed digits in size $32\times 32$ cropped from house number plates.
- •
GTSRB (Stallkamp et al. 2011) is a German traffic sign recognition benchmark consisting of over $50,000$ images of $43$ classes of traffic signs in varying light and background conditions.
- •
MNIST (LeCun et al. 1998), a well-known classical dataset for hand-written digit classification with $60,000$ training and $10,000$ testing images of size $28\times 28$ in $10$ classes of numbers.
- •
DTD (Cimpoi et al. 2014) is a collection of $5,640$ images across $47$ categories of textures in the wild, annotated with human-centric attributes.

#### Additional datasets.

Besides the standard $8$-task scenario, we follow the previous work and further extend our experimental scenario to $20$ tasks(^2^22https://huggingface.co/collections/tanganke/image-classification-datasets-662abda7d75efe6b0e6b43da). The new $12$ tasks are listed as follows:

- •
Flowers102 (Nilsback and Zisserman 2008) contains $102$ flower categories that are popular in the United Kingdom, with $1,020$ training and $6,149$ testing images. The images have varying poses and light conditions.
- •
PCAM (PatchCamelyon) (Veeling et al. 2018) consists of more than $300$M color images in size of $96\times 96$ pixels extracted from histopathologic scans of lymph node sections. Each of them is annotated with a binary class indicating the presence of metastatic tissue.
- •
FER2013 (Goodfellow et al. 2013) is developed for facial expression recognition. The images are grayscale and have a size of $48\times 48$ pixels, describing seven different kinds of emotions. The training and testing split consists of $28,709$ and $7,178$ samples, respectively.
- •
OxfordIIITPet (Parkhi et al. 2012) is a $37$-category pet dataset with roughly 200 images for each category, and is equally divided for both training and testing splits. The images vary in scale, pose, and lighting conditions.
- •
STL10 (Coates, Ng, and Lee 2011) is primarily built for unsupervised image recognition tasks covering $10$ classes. Hence, the number of labeled images is quite small: $500$ training and $800$ testing images for each class. All of them are in $96\times 96$ pixel resolution.
- •
CIFAR100 (Krizhevsky et al. 2009) consists of color images categorized in $100$ general classes, each class contains $600$ images, and each image is in size $32\times 32$. There are $50,000$ training images and $10,000$ testing images.
- •
CIFAR10 (Krizhevsky et al. 2009) is similar to CIFAR100, except it has $10$ classes.
- •
Food101 (Bossard, Guillaumin, and Van Gool 2014) contains of $101$ food categories, with $101,000$ images. For each class, $750$ images are for training and $250$ are for testing. Only the testing images are manually reviewed. The training images contain noise mostly from intense colors, and sometimes are mislabelled.
- •
FashionMNIST (Xiao, Rasul, and Vollgraf 2017) is designed as a drop-in replacement benchmark for the original MNIST, thereby inheriting the same structure as MNIST.
- •
EMNIST (Cohen et al. 2017) is an extended version of MNIST. EMNIST contains images of both characters and digits. We choose to use only the EMNIST Letters split, which contains around $145,000$ images evenly distributed in $26$ classes of the alphabet letters.
- •
KMNIST (Clanuwat et al. 2018), yet another version of MNIST, represents $10$ Japanese Hiragana characters.
- •
RenderedSST2 (Socher et al. 2013b; Radford et al. 2019) is used for evaluating the models’ capability on optical character recognition. The images are rendered from sentences in the Stanford Sentiment Treebank v2 (Socher et al. 2013a), with black texts on a white background in $448\times 448$ resolution. Each image is labeled as positive or negative based on the mood expressed in the text, and the number of images for both classes is nearly balanced. There are $6,920$ training and $1,821$ testing images.

### A.2 Models

We employ off-the-shelf fine-tuned checkpoints from the previous work (Tang et al. 2024a), covering three architectures of pre-trained CLIP model (Radford et al. 2021): ViT-B/32(^3^33https://huggingface.co/collections/tanganke/clip-vit-b-32-on-the-eight-image-classication-tasks-6644d0c476c0a469f693cf91), ViT-B/16(^4^44https://huggingface.co/collections/tanganke/clip-vit-b-16-on-the-eight-image-classification-tasks-66cd54d8332ce5c7468ab5f8), and ViT-L/14(^5^55https://huggingface.co/collections/tanganke/clip-vit-l-14-on-the-eight-image-classification-tasks-6644d2b014331c746683de63). The indicators $32$, $16$, and $14$ mean the size of patches in pixels that an input image is divided into. We only merge the vision encoding part of these architectures, while the text encoding part is kept unchanged. The number of parameters for the vision encoding part of these architectures is $87.5$M, $85.8$M, and $303$M, respectively.

### A.3 Merging Methods

The merging methods we employ for comparison are listed in three groups as follows:

- 1.
Data-Free Methods:
•
Model Soups (Wortsman et al. 2022) is the most straightforward approach that simply takes the average of candidate models’ parameters to produce a merge model.
•
Task Arithmetic (Ilharco et al. 2022) introduces a concept called “task vector”, which is the difference between the fine-tuned model’s parameters and the pre-trained parameters. A multi-tasking task vector is defined as the sum of those task vectors and is scaled by a coefficient before being added back to the pre-trained model’s parameters to produce a merge model.
•
TIES-Merging (Yadav et al. 2023) proposes to trim the small values of task vectors, then resolve sign conflicts before adding back to the pre-trained parameters.
•
TSV-M (Gargiulo et al. 2025) compresses the task vectors using singular value decomposition (SVD) to reduce the interference between task vectors at the layer level before merging.
•
DOGE TA (Wei et al. 2025) is a variant of Task Arithmetic, where DOGE, an iterative algorithm minimizing the gap between the merge model and the candidates while retaining the shared knowledge, is integrated.
•
Iso-C and Iso-CTS (Marczak et al. 2025). The former applied SVD on the merge task vector to identify the directions amplified by multiple tasks, i.e., common subspace. The latter further incorporates task-specific subspaces for retaining unique task features.
- 2.
Training-Free Methods:
•
Fisher Merging (Matena and Raffel 2022) produces the merge models by taking the weighted average of candidate models, with the weighting factors determined by the Fisher information matrices.
•
RegMean (Jin et al. 2022) proposes a closed-form solution for merging multiple linear layers, then applies this idea to the transformer models.
- 3.
Test-Time Adaptation:
•
Layer-wise AdaMerging (Yang et al. 2024b) adaptively learns the merging coefficients introduced by Task Arithmetic in the layer-wise or task-wise manner by using unsupervised entropy minimization on unlabeled test datasets.
•
DOGE AM (Wei et al. 2025) is another variant of AdaMerging, where DOGE is integrated.

Figure: Figure 6: Effect of the amount of data needed for one candidate model when merging ViT-B/16 and ViT-L/14.
Refer to caption: x7.png

Figure: (a) Clean
Refer to caption: assets/image_corruption/clean.jpg

## Appendix B Experimental Details

### B.1 Normalized Accuracy Metric

To avoid distortions caused by differences in value ranges, we report normalized accuracy for experiments on large-scale tasks. The normalized accuracy for each task is computed relative to the accuracy of its corresponding candidate model, then averaged over tasks as:

$$ $\textnormal{Avg. Norm. Accuracy}=\frac{1}{K}\sum_{i=1}^{K}\frac{\textnormal{acc}[f_{M}(\bm{X}_{i})]}{\textnormal{acc}[f_{i}(\bm{X}_{i})]}.$ (3) $$

### B.2 Hyperparameters

#### RegMean and RegMean++.

Both require two hyperparameters: the number of samples per task and the scaling factor $\alpha$. As illustrated in main text Figure 5 (left) and Figure [6](https://arxiv.org/html/2508.03121v1#A1.F6), increasing the number of samples for each task gives a relatively small improvement on performance across architectures. We choose to use $256$ samples for calculating the task’s inner-product matrices with a batch size of $32$ as default. For the scaling factor $\alpha$, Table [6](https://arxiv.org/html/2508.03121v1#A0.T6) shows that a higher $\alpha$ delivers better performance. However, when $\alpha=1.0$, i.e., no scaling applied, the degradation happens and the overall accuracy is almost zeroed out. This phenomenon is consistent with the insight from Jin et al. (2022). Therefore, $\alpha=0.95$ is the optimal value.

#### Other methods.

We follow the suggestions on hyperparameters in the original works and set all of the hyperparameters below as default across experiments.

- •
For Task Arithmetic (Ilharco et al. 2022), the merging coefficient $\lambda=0.3$ is used.
- •
For TIES-Merging (Yadav et al. 2023), top-$20\%$ highest-magnitude parameters are retained for each task vector, then these trimmed task vectors are merged with the merging coefficient $\lambda=0.3$.
- •
For TSV-M (Gargiulo et al. 2025), the task scaling factor $\alpha=1.0$ is used.
- •
For DOGE TA (Wei et al. 2025), the modification vector $\Delta$ is optimized on $400$ iterations with a learning rate $1e-4$ via Adam optimizer  (Kingma and Ba 2014), the global magnitude of merging coefficient $\eta=0.07$, the shared subspace basis size is set as the rank of the shared subspace divided by $6$, and top-$30\%$ highest-magnitude parameters are retained for each task vector.
- •
For Iso-C (Marczak et al. 2025), the task scaling factor $\alpha$ is set to $1.30$, $1.40$, and $1.50$ for ViT-B/32, ViT-B/16, and ViT-L/14, respectively.
- •
For Iso-CTS (Marczak et al. 2025), the task scaling factor $\alpha$ is set to $1.50$, $1.60$, and $1.90$ for ViT-B/32, ViT-B/16, and ViT-L/14, respectively; the size of the common subspace is set as its rank multiplied by $0.8$.
- •
For Fisher Merging (Matena and Raffel 2022), the number of samples per task is $256$ and the batch size is $32$ for calculating the Fisher information matrix.
- •
For Layer-wise AdaMerging (Yang et al. 2024b), the Adam optimizer is used with a learning rate of $1e-3$ for updating the merging coefficients on $1,000$ iterations with the batch size of $16$.
- •
For DOGE AM (Wei et al. 2025), its hyperparameters are the same as DOGE TA and Layer-wise AdaMerging.

Figure: Figure 8: Details on sequential-merging performance comparison of RegMean and RegMean++ for three models. Each entry is the average accuracy on a group of four tasks, averaged over five runs.
Refer to caption: x8.png

### B.3 Details for Experimental Settings

#### Orders of tasks for sustainability evaluation.

We assess the merging performance by varying the number of tasks $n\in\{2,4,8,12,16,20\}$. For every iteration $i$, we get first $n_{i}$ tasks from a fixed sequence of $20$ tasks, perform merging, and then evaluate the performance on the $n_{i}$ involved tasks only. These experiments are conducted independently and re-executed from scratch for every iteration. The fixed sequence of $20$ tasks is as follows:

SUN397, Stanford Cars, RESISC45, EuroSAT, SVHN, GTSRB, MNIST, DTD, Flowers102, PCAM, FER2013, OxfordIIITPet, STL10, CIFAR100, CIFAR10, Food101, FashionMNIST, EMNIST, KMNIST, RenderedSST2.

#### Procedure for sequential merging.

We assess the merging performance in a scenario where tasks arrive sequentially. We choose to merge four tasks at a time. Specifically, we first merge four task-specific models to obtain an initial merge model. Then, this merge model is further merged with the new four task-specific models, using the ID data for calculating its merging statistics. That is, a mixture of task-specific datasets is constructed, where the number of samples used for each task is simply defined as $256$ divided by the total number of tasks involved so far. This merging process is repeated until all of $20$ task-specific models are merged. Right after every merge model is obtained, we evaluate its performance on all $20$ tasks.

To determine the order of merging tasks, we generate a batch of different task combinations and choose five task sequences among them such that every non-overlapping group of four tasks in a task sequence does not exist in the other sequences. The five task sequences are as follows:

- •
PCAM, FER2013, OxfordIIITPet, RenderedSST2, GTSRB, FashionMNIST, SUN397, CIFAR100, EuroSAT, Stanford Cars, MNIST, STL10, DTD, Flowers102, CIFAR10, Food101, KMNIST, EMNIST, SVHN, RESISC45.
- •
CIFAR100, SUN397, EMNIST, EuroSAT, RESISC45, Food101, Flowers102, PCAM, RenderedSST2, Stanford Cars, CIFAR10, GTSRB, MNIST, DTD, KMNIST, FashionMNIST, STL10, SVHN, OxfordIIITPet, FER2013.
- •
EuroSAT, RenderedSST2, SUN397, FashionMNIST, Food101, KMNIST, OxfordIIITPet, DTD, PCAM, FER2013, Flowers102, MNIST, RESISC45, Stanford Cars, CIFAR10, STL10, GTSRB, EMNIST, SVHN, CIFAR100.
- •
EMNIST, RESISC45, MNIST, CIFAR10, FashionMNIST, SVHN, KMNIST, STL10, GTSRB, EuroSAT, SUN397, PCAM, Flowers102, FER2013, OxfordIIITPet, Food101, DTD, RenderedSST2, Stanford Cars, CIFAR100.
- •
GTSRB, Stanford Cars, SUN397, FashionMNIST, CIFAR10, EMNIST, SVHN, FER2013, OxfordIIITPet, Food101, MNIST, RenderedSST2, DTD, CIFAR100, Flowers102, PCAM, KMNIST, STL10, EuroSAT, RESISC45.

#### Unseen tasks for evaluating the OOD generalization ability.

We report the performance over five held-out sets for OOD tasks: {MNIST, DTD}, {SVHN, GTSRB}, {RESISC45, EuroSAT}, {SUN397, Cars}, and {Cars, RESISC45}. Meanwhile, the remaining six tasks in the standard benchmark serve as ID tasks. These sets are chosen such that the overlapping rate between OOD sets is the least.

#### Example of corrupted images for evaluating the robustness.

Figure [7](https://arxiv.org/html/2508.03121v1#A1.F7) demonstrates seven corrupted variants for a clean image drawn from the Stanford Cars dataset.

#### ImageNet sampling (OOD sampling).

Due to its massive volume, we randomly select $256$ from the first $10,000$ samples of this database. These selected samples are used as proxy task-specific data for all of the candidate models.

#### Class-imbalance sampling.

We simulate a data-limited scenario where only a single class from each task-specific dataset is utilized for merging. We execute five runs for each of the training-free methods, using different classes from the set $\{0,1,2,3,4\}$ for each run. For example, the first run involves class 0; meaning that for a task dataset, at most $256$ samples labeled as 0 are randomly selected to serve as data for merging.

**Table 7: Region-specific merging and component-specific merging performance of data-free methods for ViT-B/32, ViT-B/16, and ViT-L/14; where Task Arithmetic is denoted as TA and TIES-Merging is denoted as TIES. Results show average accuracy across $8$ tasks.**
| Components | ViT-B/32 | ViT-B/16 | ViT-L/14 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TA | TIES | TSV-M | DOGE TA | Iso-C | Iso-CTS | TA | TIES | TSV-M | DOGE TA | Iso-C | Iso-CTS | TA | TIES | TSV-M | DOGE TA | Iso-C | Iso-CTS |  |
| All | $67.5$ | $71.9$ | $83.1$ | $80.6$ | $82.5$ | $82.7$ | $77.1$ | $77.6$ | $87.1$ | $84.7$ | $87.9$ | $88.3$ | $80.5$ | $83.8$ | $90.6$ | $88.7$ | $92.1$ | $92.7$ |
| Early | $60.5$ | $65.8$ | $66.6$ | $66.5$ | $70.7$ | $71.7$ | $70.7$ | $72.7$ | $73.2$ | $72.8$ | $77.3$ | $78.3$ | $76.6$ | $79.8$ | $80.1$ | $80.4$ | $83.9$ | $84.8$ |
| Middle | $70.1$ | $69.0$ | $74.9$ | $72.8$ | $74.7$ | $75.0$ | 76.4 | $74.1$ | $80.2$ | $79.0$ | $81.5$ | $81.6$ | $80.9$ | $81.7$ | $85.9$ | $84.4$ | $87.3$ | $87.9$ |
| Deep | 72.1 | 70.6 | 78.5 | 76.3 | 77.0 | 76.9 | 76.4 | 75.5 | 82.6 | 81.1 | 83.4 | 83.9 | 84.1 | 82.7 | 88.1 | 86.4 | 88.8 | 88.7 |
| Middle & Deep | $74.1$ | $73.1$ | $83.4$ | $81.0$ | $81.6$ | $81.0$ | $79.1$ | $77.3$ | $87.0$ | $84.9$ | $87.0$ | $87.0$ | $84.0$ | $84.2$ | $90.9$ | $88.6$ | $91.4$ | $91.8$ |
| Attention heads | 71.1 | $69.2$ | $76.0$ | $75.0$ | $73.6$ | $73.9$ | 76.6 | $74.7$ | $80.7$ | $80.9$ | $79.5$ | $79.7$ | 83.6 | $82.2$ | $87.1$ | $85.8$ | $87.0$ | $88.1$ |
| MLPs | $67.4$ | 70.1 | 80.3 | 76.2 | 80.1 | 80.6 | $75.6$ | 75.7 | 84.5 | 81.6 | 85.6 | 86.1 | $79.3$ | 82.4 | 88.6 | 86.2 | 90.9 | 91.8 |

### B.4 Hardwares

All of the experiments were conducted on either an A100 GPU $40$GB memory or an A40 GPU $48$GB memory.

## Appendix C Additional Results and Analysis

### C.1 Performance on the 8 8 -Task Benchmark

We provide the performance comparison of RegMean++ and other methods on the $8$-task benchmark for ViT-B/16 and ViT-L/14 in Table [9](https://arxiv.org/html/2508.03121v1#A4.T9) and Table [9](https://arxiv.org/html/2508.03121v1#A4.T9), respectively.

### C.2 Sequential Merging

Along with the average performance on all $20$ tasks shown in main text Figure 3, we additionally provide a more fine-grained analysis of RegMean and RegMean++ in Figure [8](https://arxiv.org/html/2508.03121v1#A2.F8). Each entry in these heat maps visualizes the mean of average accuracy on a non-overlapping group of four tasks across five different task sequences. Along diagonals, which correspond to groups of current merging tasks, RegMean++’s performance matches or surpasses that of RegMean. Furthermore, RegMean++ also exhibits a slightly enhanced ability to retain performance on earlier tasks, as indicated by the entries below those diagonals.

### C.3 Robustness Against Distribution Shifts

We provide the performance of RegMean++ and other methods on the robustness analysis for ViT-B/16 and ViT-L/14 in Table [11](https://arxiv.org/html/2508.03121v1#A4.T11) and Table [11](https://arxiv.org/html/2508.03121v1#A4.T11), respectively.

### C.4 Merging in Different Spaces for Data-Free Methods

#### Merging using middle and deep layers preserves overall performance or even outperforms all layers.

The results are reported in Table [7](https://arxiv.org/html/2508.03121v1#A2.T7). For Task Arithmetic, TIES-Merging, TSV-M, and DOGE TA using both middle and deep layers achieves highly competitive performance, or even surpasses that of using all layers. The most noticeable performance differences can be observed on Task Arithmetic, where it consistently improves over full-layer merging by $6.6$, $2.0$, and $3.5$ points for ViT-B/32, ViT-B/16, and ViT-L/14, respectively. Meanwhile, merging both middle and deep layers preserves from $98\%$ to $99\%$ performance for both Iso-C and Iso-CTS. Additionally, applying merging to only deep layers retains more than $92\%$ overall performance, and is better than merging using middle or early layers. This trend can also be observed in layer-wise merging as visualized in Figure [10](https://arxiv.org/html/2508.03121v1#A4.F10).

#### MLP module linear layer merging is not always the best.

Component-specific merging analysis for data-free methods shows that using linear layers of the MLP modules still yields better performance than that of the attention heads, except for Task Arithmetic, as indicated in Table [7](https://arxiv.org/html/2508.03121v1#A2.T7).

### C.5 Representation Bias

Figure: Figure 9: Representation bias of ViT-B/16 and ViT-L/14.
Refer to caption: x9.png

Figure [9](https://arxiv.org/html/2508.03121v1#A3.F9) shows a comparison of the representation bias when applying RegMean and RegMean++ on ViT-B/16 and ViT-L/14.

### C.6 Full Results on Varying Data Characteristics Analysis

In main text Table 5, we have provided the average accuracy across $8$ tasks. In Figure [11](https://arxiv.org/html/2508.03121v1#A4.F11), we provide a detailed performance of data-free methods under different effects of data characteristics.

## Appendix D Limitations and Future Work

One of the main limitations of this work is the lack of consideration of larger models with billions of parameters. In addition, our approach treats all knowledge equally, without carefully distinguishing common and task-specific knowledge. Thus, exploring methods that can separate and handle these types of knowledge remains a promising direction.

**Table 8: Performance of all merging methods for ViT-B/16 measured on the $8$-task benchmark. The global best, local best, and global runner-up are marked.**
| Method | SUN397 | Cars | RESISC45 | EuroSAT | SVHN | GTSRB | MNIST | DTD | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Reference Results |  |  |  |  |  |  |  |  |
| Fine-tuned | $78.9$ | $85.9$ | $96.6$ | $99.0$ | $97.6$ | $99.0$ | $99.7$ | $82.3$ | $92.3$ |
|  | Data-Free Methods |  |  |  |  |  |  |  |  |
| Model Soups | $68.7$ | $69.0$ | $75.1$ | $83.3$ | $75.0$ | $62.6$ | $93.8$ | $51.2$ | $72.3$ |
| Task Arithmetic | $65.9$ | $68.3$ | $75.5$ | $84.5$ | $88.9$ | $82.0$ | $98.1$ | $54.0$ | $77.1$ |
| TIES-Merging | $70.7$ | $71.2$ | $79.9$ | $87.5$ | $83.3$ | $76.3$ | $96.4$ | $55.5$ | $77.6$ |
| TSV-M | $73.1$ | $80.7$ | $89.7$ | $96.2$ | ${\color[rgb]{.75,0,.25}94.1}$ | $94.1$ | 99.1 | $69.7$ | $87.1$ |
| DOGE TA | $70.8$ | $77.5$ | $85.9$ | $95.1$ | $92.7$ | $91.4$ | $\underline{98.8}$ | $65.3$ | $84.7$ |
| Iso-C | $\underline{75.9}$ | $\underline{82.9}$ | $\underline{92.3}$ | ${\color[rgb]{.75,0,.25}\underline{96.3}}$ | $91.1$ | $94.5$ | $98.7$ | $71.2$ | $87.9$ |
| Iso-CTS | 76.2 | 83.8 | 92.6 | $96.0$ | $90.9$ | ${\color[rgb]{.75,0,.25}94.7}$ | $98.6$ | ${\color[rgb]{.75,0,.25}\underline{73.7}}$ | 88.3 |
|  | Training-Free Methods |  |  |  |  |  |  |  |  |
| Fisher Merging | $68.5$ | $69.7$ | $73.6$ | $\underline{96.3}$ | $78.8$ | $73.3$ | $90.4$ | $54.0$ | $75.6$ |
| RegMean | $72.6$ | $78.8$ | $89.2$ | $\underline{96.3}$ | $\underline{94.9}$ | $90.0$ | $\underline{98.8}$ | $67.9$ | $86.0$ |
| RegMean++ (Ours) | ${\color[rgb]{.75,0,.25}72.8}$ | ${\color[rgb]{.75,0,.25}78.9}$ | ${\color[rgb]{.75,0,.25}89.3}$ | 97.3 | 96.0 | ${\color[rgb]{.75,0,.25}93.0}$ | 99.1 | ${\color[rgb]{.75,0,.25}71.0}$ | ${\color[rgb]{.75,0,.25}87.2}$ |
|  | Test-Time Adaption Methods |  |  |  |  |  |  |  |  |
| Layer-wise AdaMerging | $70.7$ | $79.8$ | $86.5$ | $93.4$ | $93.7$ | $\underline{95.6}$ | $98.1$ | $62.7$ | $85.1$ |
| DOGE AM | ${\color[rgb]{.75,0,.25}72.6}$ | ${\color[rgb]{.75,0,.25}82.4}$ | ${\color[rgb]{.75,0,.25}90.5}$ | ${\color[rgb]{.75,0,.25}94.6}$ | ${\color[rgb]{.75,0,.25}94.8}$ | 96.4 | ${\color[rgb]{.75,0,.25}98.6}$ | 75.8 | ${\color[rgb]{.75,0,.25}\underline{88.2}}$ |

**Table 10: Performance of merging methods for ViT-B/16 on corrupted test data. The global best, local best, and global runner-up are marked.**
| Method | Clean Test set | Corrupted Test set |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Motion | Impulse | Gaussian | Pixelate | Spatter | Contrast | JPEG | Avg. |  |  |
|  | Data-Free Methods |  |  |  |  |  |  |  |  |
| Model Soups | $81.9$ | $73.6$ | $62.0$ | $64.3$ | $34.4$ | $64.3$ | $73.2$ | $72.6$ | $63.5$ |
| Task Arithmetic | $84.0$ | $75.8$ | $64.1$ | $65.9$ | $35.4$ | $66.5$ | $75.2$ | $74.6$ | $65.3$ |
| TIES-Merging | $78.4$ | $69.2$ | $57.5$ | $59.7$ | $30.3$ | $60.7$ | $68.7$ | $68.6$ | $59.2$ |
| TSV-M | ${\color[rgb]{.75,0,.25}92.3}$ | ${\color[rgb]{.75,0,.25}86.0}$ | ${\color[rgb]{.75,0,.25}74.6}$ | ${\color[rgb]{.75,0,.25}\underline{73.3}}$ | ${\color[rgb]{.75,0,.25}42.0}$ | ${\color[rgb]{.75,0,.25}77.5}$ | ${\color[rgb]{.75,0,.25}84.1}$ | ${\color[rgb]{.75,0,.25}84.8}$ | ${\color[rgb]{.75,0,.25}74.6}$ |
| DOGE TA | $90.5$ | $83.7$ | $71.9$ | $70.7$ | $40.7$ | $74.5$ | $82.1$ | $82.0$ | $72.2$ |
| Iso-C | $90.3$ | $83.3$ | $67.8$ | $69.2$ | $39.4$ | $73.3$ | $82.6$ | $81.0$ | $70.9$ |
| Iso-CTS | $90.7$ | $83.7$ | $67.9$ | $69.6$ | $38.7$ | $74.1$ | $83.2$ | $81.5$ | $71.2$ |
|  | Training-Free Methods |  |  |  |  |  |  |  |  |
| Fisher Merging | $81.4$ | $73.6$ | $58.4$ | $59.9$ | $33.6$ | $63.7$ | $72.4$ | $72.0$ | $61.9$ |
| RegMean | $92.6$ | $86.2$ | 75.6 | ${\color[rgb]{.75,0,.25}\underline{73.3}}$ | $41.9$ | ${\color[rgb]{.75,0,.25}\underline{77.7}}$ | ${\color[rgb]{.75,0,.25}\underline{84.7}}$ | $84.8$ | $74.9$ |
| RegMean++ (Ours) | 93.2 | ${\color[rgb]{.75,0,.25}\underline{87.3}}$ | $75.3$ | $72.7$ | ${\color[rgb]{.75,0,.25}\underline{42.9}}$ | $77.4$ | $84.2$ | ${\color[rgb]{.75,0,.25}\underline{86.5}}$ | ${\color[rgb]{.75,0,.25}\underline{75.2}}$ |
|  | Test-Time Adaptation |  |  |  |  |  |  |  |  |
| Layer-wise AdaMerging | $90.9$ | $84.6$ | $71.7$ | $73.0$ | $\underline{42.9}$ | $75.7$ | $83.3$ | $83.6$ | $73.5$ |
| DOGE AM | ${\color[rgb]{.75,0,.25}\underline{93.1}}$ | 87.6 | ${\color[rgb]{.75,0,.25}\underline{75.4}}$ | 76.0 | 46.4 | 79.3 | 86.2 | 86.6 | 76.8 |

Figure: Figure 10: Layer-wise merging performance of data-free methods for ViT-B/32. ViT-B/16, and ViT-L/14. Results show average accuracy across $8$ tasks.
Refer to caption: x10.png

Figure: Figure 11: Accuracy of data-free methods when using class imbalance samples and OOD samples (ImageNet) for merging. For class imbalance, we report the mean of accuracy over five different classes.
Refer to caption: x11.png

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: (1) The reported performance for SOTA baselines such as Iso-C [1] and TSV-M[2] appears considerably lower than in their original papers. Is this due to a different set of model checkpoints being used? If so, could you verify whether the same trends hold when evaluating on the exact checkpoints used in those works?

References:

[1] Marczak, D., Magistri, S., Cygert, S., Twardowski, B., Bagdanov, A. D., & van de Weijer, J. (2025). No task left behind: Isotropic model merging with common and task-specific subspaces. arXiv preprint arXiv:2502.04959.

[2] Gargiulo, A. A., Crisostomi, D., Bucarelli, M. S., Scardapane, S., Silvestri, F., & Rodola, E. (2025). Task singular vectors: Reducing task interference in model merging. In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 18695-18705).
soundness: 3
strengths: [1] The paper is clearly written. The idea of using representations from the merged model—rather than the originals—is a neat and effective trick that substantially boosts merging performance.

[2] The evaluation is comprehensive, covering multiple vision benchmarks and providing thoughtful analyses.
summary: This paper presents RegMean++, an extension of RegMean that augments the merging objective with both intra-layer and cross-layer dependency modeling. The method is evaluated across diverse scenarios—ID and OOD settings, sequential merging, and merging with corrupted data—and is accompanied by a detailed layer-wise importance analysis of the merging process.
weaknesses: [1] The study is confined to vision tasks. Demonstrating results on LLMs—where model merging is widely practised—would significantly strengthen the paper’s impact.

[2] While RegMean++ advances over RegMean, the sequential-merging results should also be compared to state-of-the-art methods tailored for this setting to establish competitiveness.

---

confidence: 4
contribution: 1
initial_rating: 0
presentation: 1
questions: What is the motivation of this paper, and what are its main contributions?
soundness: 1
strengths: Extensive and well-controlled empirical campaign: Main benchmark covers 8 diverse image tasks, 3 model sizes, and 11 baselines. Sustainability under scale tested up to 20 tasks and sequential arrival. Robustness evaluated on seven corruption types and class-imbalanced or ImageNet-OOD samples.
summary: This paper proposes an improvement over RegMean. However, it lacks clear motivation, presents limited novelty, and provides an insufficiently explained methodology.
weaknesses: 1)This paper lacks clear motivation, presents limited novelty, and provides an insufficiently explained methodology.
2)No theoretical grounding for the “corrected” statistics: The manuscript claims RegMean++ “incorporates intra- and cross-layer dependencies” but offers no proof or even informal argument that using merged-model activations minimises a meaningful objective that couples layers (Sec. 3.1–3.2). Consequently, convergence, optimality, or error bounds with respect to the true multi-task risk are absent.
3)Computational cost brushed aside: RegMean++ needs an extra forward pass through the growing merged model for every layer to collect statistics (Algorithm 1, line 5). No wall-clock or FLOP comparison is given; the abstract claims “no computational overhead of re-training” but omits this non-trivial overhead relative to RegMean.
4)Scalability to larger models unaddressed: All experiments use ViT-B/32, B/16, L/14 (≤303 M params). The limitation section itself flags billion-scale models as future work, so current evidence does not support generalisability to LLM or large multimodal scenarios.
Code and checkpoints are not yet released, which limits immediate reproducibility.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: In Table 1, the original paper reports 85.86 for TSV-M and 86.3 for ISO-C, while this paper reports 83.1 and 82.5, respectively. What causes this discrepancy in the reported results?
soundness: 3
strengths: - This paper points out that RegMean considers only single-layer information during merging, while ignoring the role of cross-layer information flow.
- The paper compares the proposed method with existing approaches in terms of accuracy, out-of-distribution generalization, and performance under distribution shift scenarios.
summary: This paper proposes a method called RegMean++ for multi-task model merging, which is built upon the previous RegMean work.
weaknesses: - This paper is an improvement based on RegMean, and the main difference lies in the input features; thus, its novelty is limited.
- The paper only validates the method on ViT architectures and simple image classification tasks, lacking verification on LLMs and text generation tasks.
- The experiments do not provide a comparison of time costs among different model merging methods.
- In Table 2, it is unclear why the proposed method outperforms RegMean on OOD data; this conclusion lacks deeper analysis or theoretical explanation.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: * Compared with baseline RegMean (which is relatively cheap), how much extra computation or memory cost does RegMean++ incur (both at merging time, and at inference time if relevant)? Are there any deployment concerns?
soundness: 3
strengths: * The authors identify a meaningful shortcoming in RegMean: the layer-wise independence assumption ignores downstream effects and feature propagation. The articulation of this gap is clear.



* According to the submission summary, the method is evaluated across multiple axes (ID vs. OOD, sequential merging, large-scale tasks, robustness) and consistently outperforms the baseline (RegMean).
summary: This paper addresses the problem of model-merging: combining multiple task-specific models into a unified model without retraining. The authors engage with an existing method called RegMean (Regression Mean) which treats each linear layer in the merge model independently to minimize prediction error. They argue that RegMean neglects cross-layer and intra-layer dependencies (i.e., how features propagate through layers and how earlier layers affect later ones). This paper makes the following contributions:


* A new method called RegMean++ which extends RegMean by explicitly incorporating both intra-layer dependencies and cross-layer dependencies into the regression objective (i.e., modelling how layer outputs and later layers interact).


* Extensive empirical evaluation showing that RegMean++ outperforms RegMean across diverse settings: in-domain (ID) and out-of-domain (OOD) generalization, sequential merging (adding tasks one after another), large-scale tasks, and robustness under various distribution shifts. 

* A demonstration that this methodology competes with or even beats recent advanced model-merging methods (beyond just the baseline RegMean) in some cases.
weaknesses: * While the regression‐mean merging paradigm is attractive, it implicitly assumes linear relationships between the candidate models and the merged model’s weights. The world of deep networks is highly nonlinear and features propagate in complex ways; how valid is the linear regression assumption in deep networks? Extra discussion needed.


* The paper claims to model intra‐ and cross‐layer dependencies, but it may be unclear exactly how much additional modelling is done (e.g., are inter‐layer weights estimated, correlation structures learned?) and how scalable that is to very deep architectures.


* It would strengthen the work to include analyses of when RegMean++ fails, e.g., tasks that are very dissimilar or when layer dependencies are minimal, or whether the benefit is marginal in some cases—and what trade‐offs exist.


Decision: Reject