StatsMerging: Statistics-Guided Model Merging via Task-Specific Teacher Distillation

## Abstract

Abstract Model merging has emerged as a promising solution to accommodate multiple large models within constrained memory budgets. We present StatsMerging , a novel lightweight learning-based model merging method guided by weight distribution statistics without requiring ground truth labels or test samples. StatsMerging offers three key advantages: (1) It uniquely leverages singular values from singular value decomposition (SVD) to capture task-specific weight distributions, serving as a proxy for task importance to guide task coefficient prediction; (2) It employs a lightweight learner StatsMergeLearner to model the weight distributions of task-specific pre-trained models, improving generalization and enhancing adaptation to unseen samples; (3) It introduces Task-Specific Teacher Distillation for merging vision models with heterogeneous architectures, a merging learning paradigm that avoids costly ground-truth labels by task-specific teacher distillation. Notably, we present two types of knowledge distillation, (a) distilling knowledge from task-specific models to StatsMergeLearner ; and (b) distilling knowledge from models with heterogeneous architectures prior to merging. Extensive experiments across eight tasks demonstrate the effectiveness of StatsMerging . Our results show that StatsMerging outperforms state-of-the-art techniques in terms of overall accuracy, generalization to unseen tasks, and robustness to image quality variations.

## 1 Introduction

Computer vision has witnessed transformative progress fueled by deep learning, particularly through the development and adoption of large-scale pre-trained models. Architectures like Convolutional Neural Networks (CNNs) (Krizhevsky et al., 2012; He et al., 2016; Simonyan and Zisserman, 2014), Vision Transformers (ViTs) (Dosovitskiy et al., 2021b; Touvron et al., 2021), and hybrid approaches (Liu et al., 2022) pre-trained on massive datasets
have become the cornerstone of modern vision applications. Large-scale models leveraging multi-modal pre-training, such as CLIP (Radford et al., 2021)) or generative models like GANs (Goodfellow et al., 2014) and Diffusion Models (Ho et al., 2020; Rombach et al., 2022) have further pushed the boundaries of visual understanding and synthesis, enabling the use of pre-trained backbones to a wide range of downstream vision applications. Fine-tuning these powerful base models has become the dominant practice in a wide range of computer vision tasks. This success, however, leads to a practical challenge: the proliferation of numerous specialized pre-trained weights and model checkpoints (Cao et al., 2025), most of which share the same foundational ViT or CNN backbones. Managing this growing collection incurs significant storage overhead, complicates deployment, and represents a missed opportunity to consolidate the related, yet specialized, knowledge contained within these models (Wortsman et al., 2022), particularly on compute-constrained platforms such as edge devices (Cao et al., 2024b; Singh et al., 2024). While Multi-Task Learning (MTL) (Vandenhende et al., 2022b; Zhang and Yang, 2021) aims to create versatile single models for vision tasks, it often demands complex joint training strategies, concurrent access to diverse datasets, and careful architecture design to balance performance across disparate tasks.

Model merging offers a compelling post-hoc alternative, seeking to combine independently trained models without expensive retraining. However, while techniques for model merging have gained traction, particularly in Natural Language Processing (NLP) (Yadav et al., 2023a; Ilharco et al., 2023), adapting these techniques in computer vision domain has far less explored. A straightforward approach of simple weight averaging (Wortsman et al., 2022) often fails in vision tasks due to the complex, hierarchical visual feature representations, task-specific optimizations, and the presence of intricate noise patterns that lead to sharp, non-convex loss minima (Izmailov et al., 2018). Recent methods in this direction (Matena and Raffel, 2022; Jin et al., 2023; Yang et al., 2023; Padmanabhan et al., 2023) neglect the importance of weight distribution.

This paper introduces a novel model merging framework specifically designed to address the aforementioned challenges within computer vision. We propose StatsMerging, a weight distribution statistics-guided merging approach that moves beyond simple parameter averaging or task-vector manipulation. StatsMerging leverages the statistical features models pre-trained on prior tasks for merged. In particular, we compute salient statistics extracted by leverage Singular Value Decomposition (SVD) to capture the dominant properties of the learned feature spaces. This statistical information, intrinsically capturing aspects of the pre-trained model distributions, guides the merging process by learning a compact Multilayer Perceptron (MLP), coined StatsMergeLearner that predicts adaptive merging coefficients ($\lambda$) shown in Fig. [1](https://arxiv.org/html/2506.04567v1#S1.F1). This allows the merging to be guided by the weight landscape, rather than treating coefficients as free parameters requiring external tuning data.

Figure: Figure 1: Compared to prior works, StatsMerging uniquely learns the merging coefficients using StatsMergeLearner, taking advantage of statistical features of weigts pre-trained on prior tasks. Notably, while both AdaMerging and StatsMerging are presented in the task-wise level in c) and d) for simplicity of illustration, the same principle can be applied at the layer-wise level for fine-grained adaptation.
Refer to caption: x1.png

We make four significant contributions summarized as follows:

- •
We propose StatsMerging(^1^11Our code is available at [https://github.com/statsmerging/statsmerging.](https://github.com/statsmerging/statsmerging)), a novel model merging framework guided by model weight statistics, leveraging singular values extracted via Singular Value Decomposition (SVD) to predict merging coefficients $\lambda$.
- •
We design the lightweight StatsMergeLearner to learn model merging coefficients $\lambda$ estimation based on weight statistical features, through a newly proposed Task-Specific Teacher Distillation paradigm without manually-annotated labels.
- •
We introduce the first heterogeneous architectural merging method, which distills knowledge from models with non-identical architectures into the unified target architecture.
- •
Extensive experiments demonstrate the effective of our proposed StatsMerging++, an extended version of StatsMerging, which achieves $94.5\%$ average accuracy on merging models from eight tasks, outperforming the state-of-the-art WEMoE ($89.4\%$) by a substantial margin of $5.1\%$.

## 2 Related Work

### 2.1 Multi-Task Learning

Multi-Task Learning (MTL) (Vandenhende et al., 2022a) represents a paradigm for training a single model to perform multiple tasks concurrently. While MTL aims to create unified models capable of handling diverse objectives, it typically requires careful design of network architectures, computationally expensive training, access to large and diverse datasets, and intricate task balancing strategies (Zhang and Yang, 2021). Model merging offers a compelling alternative by enabling the combination of independently trained models, without the need for extensive retraining or simultaneous access to multi-task datasets or privacy-restricted data (Jin et al., 2023).

### 2.2 Multi-Task Merging

Early approaches to model merging often involved simple heuristics like Weight Averaging (Wortsman et al., 2022), Ties-Merging (Yadav et al., 2023a), and Arithmetic Merging (Ilharco et al., 2023). While straightforward to implement, these methods typically lack awareness of the weight distributions and learned representations within the models, leading to suboptimal performance in the merged model compared to individually fine-tuned models or unified models trained from scratch. (Wortsman et al., 2022) demonstrate that naive weight averaging could significantly degrade performance, highlighting the challenges in consolidating knowledge from independently trained networks. Methods explored in natural language processing (Yadav et al., 2023b; Ilharco et al., 2023) have also shown promise by learning interpolation weights.

More recent efforts in model merging have introduced various strategies to efficiently combine multiple models without retraining. Approaches such as ZipIt (Zhang et al., 2024a), EMR-Merging (Huang et al., 2024), and Training-Free Pretrained Model Merging methods (Sun et al., 2025; Chen et al., 2024) emphasize data-free, tuning-free methodologies, often leveraging weight-space heuristics or task-vector alignment. Techniques like Pareto Merging (Chen and Kwok, 2025), MAP (Li et al., 2024), and $C^{2}M^{3}$ (Crisostomi et al., 2024) formulate model merging as a multi-objective or constrained optimization problem to preserve task performance across domains. Other works such as Parameter Competition Balancing (Du et al., 2024) and Sharpness-Aware Fine-Tuning (Lee et al., 2025) address parameter interference during merging. Meanwhile, methods like LayerMerge (Kim et al., 2024) and MERGE3 (Mencattini et al., 2025) aim to improve scalability and computational efficiency, making merging feasible on consumer-grade hardware. WEMoE (Tang et al., 2024) ensembles shared and task-specific MLPs with input-conditioned routing in a layer-wise, data-free manner. Representation Surgery (Yang et al., 2024b) introduces a scheme to alleviate the problem of representation bias while Evolutionary Model Merge (Akiba et al., 2025) employs evolutionary algorithms to optimize model merging recipes.

These methods, however, do not explicitly leverage the weight distribution of the models being merged, a key distinction from our proposed approach. The gap often lies in effectively unifying the diverse and task-specific feature representations learned by individual models into a single, high-performing entity without extensive learning.

### 2.3 Merging Methods in Computer Vision

The application of model merging techniques in computer vision is relatively less explored compared to natural language processing (Yadav et al., 2023b; Ilharco et al., 2023). Computer vision models, particularly deep convolutional neural networks (CNNs) (Krizhevsky et al., 2012; He et al., 2016; Simonyan and Zisserman, 2014) and Vision Transformers (ViTs) (Dosovitskiy et al., 2021a; Touvron et al., 2021), learn complex, hierarchical feature representations that are highly sensitive to task-specific optimizations (Izmailov et al., 2018). Simple averaging techniques often fail due to the non-convex nature of the loss landscape and the divergence of learned feature spaces across different visual tasks. Recent advancements (Matena and Raffel, 2022; Yang et al., 2023) have shown potential, but often lack explicit mechanisms to account for the unique properties inherent in visual data and architectures, such as spatial relationships in CNNs (Cao et al., 2023, 2024a) or attention mechanisms in ViTs (Ye et al., 2023; Tang et al., 2025). Our work addresses these limitations by introducing a novel merging framework that leverages internal model weight statistics to guide the merging process, making it more adaptable and effective across diverse computer vision tasks and architectures.

**Table 1: Summary of system characteristics in recent works. *: Optional. TT: Test-Time.**
| Method | No | Layer | TT | Heterogeneous |
| --- | --- | --- | --- | --- |
|  | Manual Label | Level | Adaptability | Architecture |
| Traditional MTL | ✗ | * | ✗ | ✗ |
| Task Arithmetic | ✓ | ✗ | ✗ | ✗ |
| Ties-Merging | ✓ | ✗ | ✓ | ✗ |
| Fisher Merging | ✓ | ✗ | ✗ | ✗ |
| RegMean | ✓ | ✗ | ✗ | ✗ |
| EMR-MERGING | ✓ | ✓ | ✗ | ✗ |
| AdaMerging | ✓ | ✓ | ✓ | ✗ |
| Representation Surgery | ✓ | ✓ | ✓ | ✗ |
| WEMoE | ✓ | ✓ | ✓ | ✗ |
| StatsMerging (Ours) | ✓ | ✓ | ✓ | ✓ |

In summary, our method StatsMerging enjoys several advantages compared to prior works shown in Table [1](https://arxiv.org/html/2506.04567v1#S2.T1): (1) no human annotated labels are required for weight distribution learning; (2) It operates at a fine granularity, specifically at the layer-wise level; (4) it allows for test-time adaptability; (5) it facilitates extension to heterogeneous architectures.

## 3 Methodology

### 3.1 Preliminaries

Notations: A deep neural network is parameterized by a set of weights $\theta=\{\theta_{1},\theta_{2},\ldots,\theta_{L}\}$ that learns the mapping from an input data $x_{i}\in\mathbb{R}^{d}$ to a predicted value $\hat{y_{i}}\in\mathbb{R}^{D}$: $f_{\theta}(x_{i})\rightarrow\hat{y_{i}}$. Of these, $\theta^{l}$ represents the $l$-th $l\in\{1,2,\ldots,L\}$ layer weights where $L$ is the number of layers of the model $f_{\theta}$, $d$ denotes an input data $x_{i}$’s dimension. For classification problems, $y_{i}$ is the class label and $D$ is the number of classes, while for regression problems, $D$ is the dimension of the output vector $y_{i}$.

The weights of a pre-trained model (e.g., ViT or ResNet) are denoted as $\theta_{pre}=\{\theta_{pre}^{1},\theta_{pre}^{2},\ldots,\theta_{pre}^{L}\}$. The weights fine-tuned on a specific training data $\{x_{i},y_{i}\}^{N^{tr}_{k}}_{i=1}$ for task $k$ is recorded as $\theta_{k}=\{\theta_{k}^{1},\theta_{k}^{2},\ldots,\theta_{k}^{L}\}$ where $N^{tr}_{k}$ is the number of training samples.

Problem Formulation: The problem of model merging is formulated as given $K$ tasks’ training data, find a way to combine weights $\{\theta_{k}\}_{k=1}^{K}$ fine-tuned for $K$ tasks previously to obtain a new weight $\theta_{m}$ without undergoing the retraining process, while the new model $f_{\theta_{m}}$ is capable of performing well on $K$ tasks jointly.

It is assumed that all $K$ fine-tuned weights and the merged weight share the same neural network architecture. Therefore, the core question is how to linearly combine $\{\theta_{k}\}_{k=1}^{K}$ to obtain $\theta_{m}$. In the task level, the model merging problem is finding a set of coefficients $\lambda_{k}\in\{\lambda_{1},\lambda_{2},\ldots,\lambda_{K}\}$ such that the merged model weights $\theta_{m}=\sum_{k=1}^{K}\lambda_{k}\theta_{k}$ for model $f_{\theta_{m}}$ perform well on all $K$ tasks. In the layer level, it becomes searching for a set of coefficients $\lambda_{k}^{l}\in\{\lambda_{1}^{1},\lambda_{1}^{2},\ldots,\lambda_{1}^{L},
\lambda_{2}^{1},\lambda_{2}^{2},\ldots,\lambda_{2}^{L},\ldots,\lambda_{K}^{L}\}$ to obtain the merged model $\theta_{m}=\sum_{k=1}^{K}\sum_{l=1}^{L}\lambda_{k}^{l}\theta_{k}^{l}$ that maintain high performance on $K$ tasks.

### 3.2 Weight Statistics-Guided Model Merging

In this section, we describe the main intuition and techniques of our proposed method: StatsMerging. Our core idea is that given the distribution of pre-trained weights $\theta_{k}$, we can learn a function $g(\theta_{k})\rightarrow\lambda_{m}$ to predict the merging coefficients $\lambda_{m}$. We argue that weight distribution plays an important role in model merging. However, directly using the raw weights $\theta_{k}$ as input is impractical due to the high dimension of $\theta_{k}$. We posit that such information can be represented by weight statistics. These statistical features contain key information regarding the amount of weights $\theta_{k}$ for a task $k$ to be merged to the final model. We highlight the key differences with prior works in Fig. [1](https://arxiv.org/html/2506.04567v1#S1.F1).

Weight Statistics: For a pre-trained weight $\theta_{k}$ on task $k$, we compute the mean $\mu_{\theta_{k}}$ and variance $\sigma^{2}=Var(\theta_{k})$ to represent its center and breadth, as well as its magnitude $m=||\theta_{k}||$. The underlying intuition is based on the observation that merging performance is largely influenced by high-magnitude parameters (Yadav et al., 2023a). In addition, we extract the singular values $\sigma^{\prime}_{i}$ from Singular Value Decomposition (SVD):

$$ $W_{{k}}=U_{{k}}\Sigma_{{k}}V^{\top}_{{k}}$ (1) $$

where $W_{\theta_{k}}$ represents the matrix of the model parameter $\theta_{k}$.
By default, we use rank $3$ from $\Sigma_{{k}}$ to form weight statistics. Motivated by prior findings on the effectiveness of SVD in neural network pruning (Goetschalckx et al., 2018; Abid et al., 2002; Kim and Yoo, 2025), we hypothesize that singular values encapsulate essential information regarding the weight distribution, which can guide the allocation of weights from $\theta_{k}$ during merging.

Combining all together, the weight statistics feature vector $S_{{k}}$ is formed as

$$ $S_{{k}}=stats(\theta_{k})=[\mu,\sigma^{2},m,\sigma_{r}^{\prime}]$ (2) $$

where $stats()$ extracts the statistical features from the weight $\theta_{k}$, $\sigma_{r}$ represents the singular value vector given rank $r$: $\sigma_{r}^{\prime}=[\sigma_{1}^{\prime},\sigma_{2}^{\prime},\ldots,\sigma_{r}
^{\prime}]$.

Notably, the Equation [3](https://arxiv.org/html/2506.04567v1#S3.E3) above is task-wise while we also introduce layer-wise formulation for layer $l$:

$$ $S_{k}^{l}=stats(\theta_{k}^{l})=[\mu,\sigma^{2},m,\sigma_{r}^{\prime}]^{l}$ (3) $$

where the layer-wise statistics features of pre-trained model from task $k$ layer ${l}$ is computed.

StatsMergeLearner (SML): We adopt a multilayer perceptron (MLPs) to learn to predict the merging coefficients $\lambda$ given weight statistics feature vector $S_{{k}}$ as input. In the task-wise mode, the StatsMergeLearner is denoted as $SML(S_{{k}})$:

$$ $\lambda_{k}=SML(S_{{k}})=g(stats(\theta_{k}))$ (4) $$

where $\lambda_{k}$ is a scalar representing the merging coefficient of Task $k$ model.
In the layer-wise mode, the StatsMergeLearner is denoted as $M(S_{{k}})$:

$$ $\lambda_{k}^{l}=SML(S_{k}^{l})=g(stats(\theta^{l}_{k}))$ (5) $$

where $\lambda_{k}$ is a vector containing $L$ layers’ coefficients and $\lambda_{k}^{l}$ refers to the coefficient of layer $l$ in the $k$ pre-trained model. By default, we use a two-layer MLP to implement the StatsMergeLearner.

Optimization Objective.
To train StatsMergeLearner, in the standard supervised training paradigm, we freeze the weights for each task $\theta_{k}$ and apply the cross-entropy loss function $L_{CE}$ on the aggregated dataset:

$$ $\mathcal{L}_{\mathrm{CE}}^{SL}=-\sum^{C_{m}}_{c=1}y_{c}\log(\hat{y}_{c}))$ (6) $$

where $\hat{y}_{c}$ is the prediction from the merged model for class $c$, $C_{m}$ is the total number of classes in the aggregated dataset.

### 3.3 Task-Specific Teacher Distillation

We present a novel Task-Specific Teacher Distillation training paradigm to train the StatsMergeLearner (SML) for model merging as illustrated in Fig. [2](https://arxiv.org/html/2506.04567v1#S3.F2) and detailed in Algorithm 1.

Figure: Figure 2: Knowledge Distillation Diagram. StatsMergeLearner (SML) learns the merging coefficients $\lambda$ by minimizing the loss between the merged model’s predictions and pseudo labels generated by task-specific teacher models. During inference, only the merged model in StatsMerging is used to predict class labels.
Refer to caption: x2.png

Our key intuition is that each pre-trained model $\theta_{k}$ is already good at its own task dataset $\{x_{i},y_{i}\}_{k}\ \in D_{k}$, therefore we regard it ($\theta_{k}$) as the Task-Specific Teacher $T_{k}$. Subsequently, the predictions $\hat{y}_{i,k}$ from the model trained on task $k$ serves as sufficiently reliable pseudo labels for the validation dataset sample $\{x_{i},y_{i}\}_{k}$ from the same task. We aggregate such pairs $\{x_{i},\hat{y}_{i,k}\}_{k}$ to construct the merged dataset to train $SML()$. It is important to note that all samples for $SML()$ learning are collected from the validation set only. We highlight the key benefit of this approach that enables dataset preparation without relying on human-annotated labels. The predicted class label in one-hot encoded format. Therefore, the cross-entropy loss is applied:

$$ $\mathcal{L}_{\mathrm{CE}}=-\sum^{C_{m}}_{c=1}\hat{y}_{c,k}\log(\hat{y}_{c})).$ (7) $$

Algorithm 1. Unified Statistics-Guided Model Merging via Task-Specific Teacher Model Distillation

The use of a simple cross-entropy loss function allows for extending to other vision tasks and architectures detailed in Sec. [4.2](https://arxiv.org/html/2506.04567v1#S4.SS2).

## 4 Experiments

### 4.1 Experimental Setup

In this section, we present the experimental setup following AdaMerging (Yang et al., 2023) and evaluation results used to compare our method against recent baselines.

Datasets and Models:
Our experiments include eight image classification tasks with datasets SUN397 (Xiao et al., 2016), Stanford Cars (Krause et al., 2013), RESISC45 (Cheng et al., 2017), EuroSAT (Helber et al., 2019), SVHN (Netzer et al., 2011), GTSRB (Stallkamp et al., 2011), MNIST (LeCun et al., 1998), DTD (Cimpoi et al., 2014), and CIFAR10 (Krizhevsky, 2009) (^2^22In the remainder of the paper, the abbreviations shown in brackets are used to denote each task dataset: SUN397 (SU), Cars (CA), RESISC45 (RE), EuroSAT (EU), SVHN (SV), GTSRB (GT), MNIST (MN) and DTD (DT).) We use ViT-B/32 CLIP (Radford et al., 2021) as the pre-trained backbone. Individual task-specific models are obtained by training on each dataset separately. For merging models with different architectures, we first distill them into a single backbone before applying our merging method.

Baselines and Metrics:
We compare against standard baselines including Individual Training, Traditional Multi-Task Learning (MTL) (Zhang and Yang, 2021), Weight Averaging (Wortsman et al., 2022), Task Arithmetic (Ilharco et al., 2023), Fisher Merging (Matena and Raffel, 2022), RegMean (Jin et al., 2023), Ties-Merging (Yadav et al., 2023a), EMR-MERGING (Huang et al., 2024), AdaMerging (Yang et al., 2023), Representation Surgery (Yang et al., 2024b), SurgeryV2 (Yang et al., 2024c), and WEMoE (Tang et al., 2024). The primary evaluation metric is the average accuracy (Avg Acc) on the test sets of all tasks. The evaluation is conducted on eight different vision classification tasks.

StatsMergeLearner Training Detail:
Our MLP-based StatsMergeLearner learns to predict layer-wise or task-wise merging weights coefficients ($\lambda$) based on weight statistics from individual task models. The StatsMergeLearner is trained for 500 epochs using Adam, with a learning rate of $1e-3$ and a StepLR scheduler (factor 0.1 every 100 epochs), which translates to around only 3 hours to merge 4 ViTs, offering the practicality and advantage of applying our technique for practitioners without spending days or weeks for training (Zhang and Yang, 2021; Padmanabhan et al., 2023). We train the StatsMergeLearner primarily using knowledge distillation from the aggregated dataset without human annotated labels described in Sec. [3.3](https://arxiv.org/html/2506.04567v1#S3.SS3), optimized with either Cross-Entropy (Mao et al., 2023) or KL Divergence loss (Kullback and Leibler, 1951).

### 4.2 Merging Performance

In this section, we present a comprehensive evaluation of our approach in comparison to state-of-the-art task vector merging methods, assessing its superiority across several fundamental aspects: Multi-task merging performance, generalization to unseen tasks and heterogeneous architectures.

Substantially Higher Merging Performance. The main results of merging performance of ViT-B/32 models on eight tasks are presented in this section, detailed (^3^33Please refer to the Appendix for experimental details, including the full list of tasks, datasets, and baselines.) in Table [2](https://arxiv.org/html/2506.04567v1#S4.T2). We present two levels of granularity: Task-Wise (TW) and Layer-Wise (LW). Our method StatsMerging achieved an average accuracy (Avg Acc) of $76.4\%$ and $94.5\%$ in both TW and LW (StatsMerging++) levels, outperforming the state-of-the-art (SOTA) method AdaMerging++ and WEMoE by a large margin of $2.7\%$ and $5.1\%$. While finer granularity is generally associated with improved merging performance (Yang et al., 2023), our LW StatsMerging++, operating at the Layer-wise level, surpasses EMR-Merging (Huang et al., 2024) which is based on the finer Parameter-wise granularity. We attribute the improvements to the ability of StatsMergeLearner to
adapt weight coefficients based on their weight statistics to the merged model. In addition, the use of pseudo labels from task-specific teachers $\{T_{1},T_{2},\ldots,T_{k}\}$ provides stronger signals for StatsMergeLearner to better assign weight coefficients $\lambda$ than the entropy minimization approach in the AdaMerging++.

**Table 2: Multi-task merging performance (Avg Acc $\%$) when merging ViT-B/32 models on eight tasks. Results of our method StatsMerging are shaded in gray. Bold and underscore indicate the highest and second-highest scores in each column within each group under Task-wise and Layer-wise settings. TW: Task-wise. LW: Layer-wise. PW: Parameter-wise.**
| Method | SU | CA | RE | EU | SV | GT | MN | DT | Avg Acc |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Pre-Trained | 62.3 | 59.7 | 60.7 | 45.5 | 31.4 | 32.6 | 48.5 | 43.8 | 48.0 |
| Individual | 75.3 | 77.7 | 96.1 | 99.7 | 97.5 | 98.7 | 99.7 | 79.4 | 90.5 |
| Traditional MTL | 73.9 | 74.4 | 93.9 | 98.2 | 95.8 | 98.9 | 99.5 | 77.9 | 88.9 |
|  | Task-Wise |  |  |  |  |  |  |  |  |
| Weight Averaging | 65.3 | 63.4 | 71.4 | 71.7 | 64.2 | 52.8 | 87.5 | 50.1 | 65.8 |
| Task Arithmetic | 55.2 | 54.9 | 66.7 | 78.9 | 80.2 | 69.7 | 97.3 | 50.4 | 69.1 |
| Fisher Merging | 68.6 | 69.2 | 70.7 | 66.4 | 72.9 | 51.1 | 87.9 | 59.9 | 68.3 |
| RegMean | 65.3 | 63.5 | 75.6 | 78.6 | 78.1 | 67.4 | 93.7 | 52.0 | 71.8 |
| Ties-Merging | 59.8 | 58.6 | 70.7 | 79.7 | 86.2 | 72.1 | 98.3 | 54.2 | 72.4 |
| TW AdaMerging | 58.0 | 53.2 | 68.8 | 85.7 | 81.1 | 84.4 | 92.4 | 44.8 | 71.1 |
| TW AdaMerging++ | 60.8 | 56.9 | 73.1 | 83.4 | 87.3 | 82.4 | 95.7 | 50.1 | 73.7 |
| TW StatsMerging (Ours) | 61.3 | 70.0 | 74.2 | 85.2 | 87.5 | 82.5 | 96.2 | 54.2 | 76.4 |
|  | Layer-Wise |  |  |  |  |  |  |  |  |
| LW AdaMerging | 64.5 | 68.1 | 79.2 | 93.8 | 87.0 | 91.9 | 97.5 | 59.1 | 80.1 |
| LW AdaMerging++ | 66.6 | 68.3 | 82.2 | 94.2 | 89.6 | 89.0 | 98.3 | 60.6 | 81.1 |
| LW AdaMerging w/ Surgery | 69.8 | 71.0 | 88.9 | 98.1 | 91.7 | 96.5 | 98.8 | 73.6 | 86.1 |
| LW AdaMerging w/ SurgeryV2 | 74.7 | 71.4 | 95.1 | 99.6 | 96.8 | 98.9 | 99.6 | 78.3 | 89.3 |
| WEMoE | 74.1 | 77.4 | 93.7 | 99.1 | 96.2 | 98.9 | 99.6 | 76.4 | 89.4 |
| LW StatsMerging (Ours) | 67.4 | 74.1 | 82.9 | 91.1 | 89.8 | 94.7 | 98.3 | 77.5 | 84.5 |
| LW StatsMerging++ (Ours) | 92.4 | 95.4 | 95.1 | 92.9 | 94.6 | 98.7 | 98.5 | 88.4 | 94.5 |
|  | Parameter-wise |  |  |  |  |  |  |  |  |
| EMR-MERGING | 75.2 | 72.8 | 93.5 | 99.5 | 96.9 | 98.1 | 99.6 | 74.4 | 88.7 |

It is worth noting that StatsMerging outperforms the Individual setting. The observed improvements can be attributed to learning (1) from a broader diversity of scenes and (2) the implicit noise patterns across the aggregated data (Yang et al., 2024a), which StatsMergeLearner effectively leverages to enhance cross-domain generalization and surpass the performance of the teacher models (Nagarajan et al., 2023; Starodubcev et al., 2024).

Significantly Enhanced Generalization. A merged model is expected to generalize to unseen tasks by strategically transferring the knowledge from the combined set of old tasks. We benchmark such generalization ability of StatsMerging against four strong baselines: Task Arithmetic, Ties-Merging, AdaMerging, and AdaMerging++. We follow the same evaluation protocol in AdaMerging training on two groups of tasks, each group consisting of six seen tasks, and testing on two unseen tasks.

**Table 3: Generalization results (Avg Acc $\%$) on two unseen tasks when merging Layer-Wise ViT-B/32 models on six tasks. StatsMerging: shaded in gray. Bold: top score. Underscore: 2nd-highest score.**
|  | Seen Tasks |  | Unseen Tasks |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | SU | CA | RE | DT | SV | GT | Avg Acc | MN | EU | Avg Acc |
| Task Arithmetic | 63.3 | 62.4 | 75.1 | 57.8 | 84.6 | 80.4 | 70.6 | 77.2 | 46.2 | 61.7 |
| Ties-Merging | 67.8 | 66.2 | 77.2 | 56.7 | 77.1 | 70.9 | 69.3 | 75.9 | 43.3 | 59.6 |
| AdaMerging | 65.2 | 65.9 | 88.5 | 61.1 | 92.2 | 91.5 | 77.4 | 84.0 | 56.1 | 70.0 |
| AdaMerging++ | 68.2 | 67.6 | 86.3 | 63.6 | 92.6 | 89.8 | 78.0 | 83.9 | 53.5 | 68.7 |
| StatsMerging | 69.1 | 71.3 | 86.7 | 75.2 | 93.2 | 95.7 | 81.9 (+3.9) | 85.1 | 56.4 | 70.8 (+0.8) |
| Method | SU | CA | GT | EU | DT | MN | Avg Acc | RE | SV | Avg Acc |
| Task Arithmetic | 64.0 | 64.0 | 75.2 | 87.7 | 57.0 | 95.7 | 73.9 | 52.3 | 44.9 | 51.1 |
| Ties-Merging | 68.0 | 67.1 | 67.7 | 78.4 | 56.5 | 92.8 | 71.8 | 58.7 | 49.2 | 53.9 |
| AdaMerging | 67.1 | 67.8 | 94.8 | 94.4 | 59.6 | 98.2 | 80.3 | 50.2 | 60.9 | 55.5 |
| AdaMerging++ | 68.9 | 69.6 | 91.6 | 94.3 | 61.9 | 98.7 | 80.8 | 52.0 | 64.9 | 58.5 |
| StatsMerging | 69.6 | 73.3 | 96.1 | 95.4 | 74.1 | 97.2 | 84.3 (+3.5) | 54.2 | 67.1 | 60.7 (+2.2) |

Details are presented in Table [3](https://arxiv.org/html/2506.04567v1#S4.T3), where in both groups our proposed StatsMerging achieved $70.8\%$ and $60.7\%$, significantly outperforming AdaMerging by $0.8\%$ and $2.2\%$. Such improvements are attributed to both (1) the careful feature design of weight statistics that captures the dominant information regarding weight distributions from pre-trained models, which potentially helps reduce noise from each task dataset; and (2) the joint training from all old tasks on the task-specific teacher-distilled labels, enabling the implicit learning of task-agnostic and task-specific features that can benefit the generalization ability.

Extension to Heterogeneous Architectures. Our StatsMerging offers the first and unique advantage without the assumption of architectural identity in prior works (Wortsman et al., 2022; Ilharco et al., 2023; Yadav et al., 2023a; Matena and Raffel, 2022; Jin et al., 2023).
To verify the performance of varying architectures, we conduct experiments on ResNet50 (RN) and ViT-B/32 (VT) to represent Convolutional Neural Network (CNN) and Vision Transformer (ViT) architectures.

In particular, we distill fine-tuned VT teachers into a RN (Khanuja et al., 2021) student on three diverse tasks of CIFAR-10 (CI), EuroSAT (EU) and Stanford Cars (CA) with the distillation loss:

$$ $\mathcal{L}\;=\;\alpha\,\mathcal{L}_{\mathrm{CE}}(y,\hat{y})\;+\;(1-\alpha)\,T ^{2}\,\mathcal{L}_{\mathrm{KL}}\!\bigl{(}\sigma(\tfrac{z}{T}),\;\sigma(\tfrac{ z_{t}}{T})\bigr{)},$ (8) $$

where $\mathcal{L}_{\mathrm{KL}}$ denotes KL-Divergence, $z$ is logit, $T=4.0$ represents temperature, $\alpha=0.7$ is the weight balance of two sub-losses. CI is used due to the available pre-trained RN weights. Remarkably, the distilled RN matches its VT teacher’s accuracy, achieving $76.4\%$ (VT: $77.7\%$) for CA and $94.5\%$ for EU (VT: $99.7\%$) despite the architectural difference shown in Table [4](https://arxiv.org/html/2506.04567v1#S4.T4). We then apply our StatsMerging to combine the CI–trained RN and its distilled variants. We merge multiple task models into a single RN using the merging coefficients inferred by StatsMergeLearner, yielding a $7.6\%$ average improvement over the vanilla Task-Arithmetic of $73.7\%$ and achieving $81.3\%$ average accuracy.

**Table 4: Multi-task merging performance (Avg Acc %) of models in heterogeneous architectures: ResNet50 (RN) & ViT-B/32 (VT). StatsMerging: shaded in gray.**
| Method | CI | CA | EU | Avg Acc |
| --- | --- | --- | --- | --- |
| Backbone | RN | VI | VI | - |
| Distilled | - | RN | RN | - |
| Individual | 97.8 | 77.7 | 99.7 | 91.7 |
| Distilled | - | 76.4 | 94.5 | - |
| Weight Averaging | 77.1 | 56.4 | 64.9 | 59.4 |
| Ties-Merging | 76.5 | 52.8 | 80.1 | 69.8 |
| Task Arithmetic | 81.4 | 61.6 | 78.2 | 73.7 |
| LW StatsMerging | 87.2 | 68.4 | 88.4 | 81.3 |

### 4.3 StatsMerging Analysis

Label & Loss Function Study.

We conduct a loss function study on ViT-B/32 (4) models merged from four tasks, as shown in Table 5. Observe that StatsMerging trained on pseudo labels via Task-Specific Teacher Distillation (KD) achieves similar performance to StatsMerging trained on ground-truth labels (GT), with $88.5\%$ and $81.2\%$ average accuracy in TW and $90.4\%$ and $83.5\%$ in LW levels.

| Loss | Level | CA | EU | RE | GT | Avg Acc |
| --- | --- | --- | --- | --- | --- | --- |
| GT | TW | 73.2 | 94.2 | 91.1 | 95.6 | 88.5 |
| KD | TW | 64.2 | 88.6 | 85.2 | 86.7 | 81.2 |
| GT | LW | 75.6 | 96.3 | 92.1 | 97.6 | 90.4 |
| KD | LW | 68.7 | 91.6 | 87.2 | 93.5 | 83.5 |

Table 5: Multi-task performance (Avg Acc %) of StatsMerging when merging ViT-B/32 (4) models across four tasks. StatsMerging shaded in gray. GT: Ground Truth. KD: Knowledge Distillation. TW: Task-wise. LW: Layer-wise.

Statistical Feature Ablation Study.

We conduct an ablation study on the statistical features. Results in Table 6 show that combining all statistical features improves merging performance, validating our design choice. Notably, the singular values $\sigma^{\prime}$ improve the multi-task performance in both same and different architecture settings by $3.0\%$ and $3.2\%$ increase of average accuracy, justifying our design choice of using SVD.

| Same Architecture | Different Architecture |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $\mu_{\theta_{k}}$ | $\sigma^{2}$ | $m$ | $\sigma^{\prime}$ | Avg Acc | $\mu_{\theta_{k}}$ | $\sigma^{2}$ | $m$ | $\sigma^{\prime}$ | Avg Acc |
| ✓ |  |  |  | 83.4 | ✓ |  |  |  | 76.2 |
| ✓ | ✓ |  |  | 84.1 (+0.7) | ✓ | ✓ |  |  | 77.5 (+1.3) |
| ✓ | ✓ | ✓ |  | 87.2 (+3.1) | ✓ | ✓ | ✓ |  | 78.1 (+0.6) |
| ✓ | ✓ | ✓ | ✓ | 90.2 (+3.0) | ✓ | ✓ | ✓ | ✓ | 81.3 (+3.2) |

Table 6: Multi-task performance (Avg Acc %) of StatsMerging when ablating statistical features of ViT-B/32 (4) models on four tasks: CA, EU, RE & GT. Bold: top score. StatsMerging: shaded in gray.

Coefficient Analysis. We visualize the heatmap of ViT-B/32 (4) across eight tasks in Fig. [3](https://arxiv.org/html/2506.04567v1#S4.F3). We make several key observations: (1) the common recurring pattern of coefficients $\lambda$ across all eight tasks from earlier (left) to deeper (right) layers aligns with the repeated self-attention blocks in the ViT architecture, e.g. Multi-Head Self-Attention (MHSA), MLP (Feed-Forward Network), and LayerNorm, etc, demonstrating the need of various coefficients for various types of layers; (2) The sparse non-uniform coefficient distributions (various colors like Layer $13$, $19$ or $25$) suggests that merging layers can be more efficient at some specific layers instead of using one coefficient for an entire pre-trained model, justifying the our granularity choice of Layer-Wise over Task-Wise level; (3) some task-specific coefficient distributions verify the necessity of assigning distinct merging coefficients across tasks in various layers, such as in Layer 5 vs. 147. Such distributions reflect the various visual representations for different semantics learned across both layers and tasks. More visualizations are provided in the Appendix for in-depth analysis.

Figure: Figure 3: Heatmap of StatsMerging merging coefficients $\lambda$ of ViT-B/32 (4) across eight tasks. X-axis: layer index. Y-axis: Tasks. Coefficients are normalized to sum to 1.
Refer to caption: x3.png

## 5 Conclusion

Model merging offers a compelling post-hoc advantage to reduce memory storage from a corpus of large pre-trained models. We propose StatsMerging, a novel merging technique guided by model weight statistical features learned through Task-Specific Teacher Distillation without relying on human annotated samples. The key intuition lies in the guidance of weight statistics using a lightweight MLP learner, dubbed StatsMergeLearner, to infer merging coefficients. Comprehensive experiments demonstrate the effectiveness of our proposed StatsMerging++ (extended version of StatsMerging) in model mering from eight diverse tasks, achieving $94.5\%$ average accuracy and surpassing the state-of-the-art approach WEMoE ($89.4\%$) by a large margin of $5.1\%$.

## Appendix A Experiment Settings

This section presents a comprehensive overview of the datasets, baseline methods, and training procedures.

Task. A task is referred to the specific problem or objective that a model is designed to solve. In this paper, a task is defined as classifying images within a given dataset.

Dataset Details. This study follows the multi-task model merging protocol from Task Arithmetic (Ilharco et al., 2023), Ties-Merging (Yadav et al., 2023a) and AdaMerging (Yang et al., 2023) on eight image classification datasets. The details are provided below:

- •
SUN397 (SU) (Xiao et al., 2016): a scene classification dataset consisting of 397 classes and a total of 108,754 images, with each class containing a minimum of 100 images.
- •
Stanford Cars (CA) (Krause et al., 2013): a car classification benchmark dataset comprosing 196 categories and 16,185 images in total. For each category, the dataset is evenly divided into training and test sets in a 1:1 ratio.
- •
RESISC45 (RE) (Cheng et al., 2017): a remote sensing image scene classification benchmark with 45 scene classes and 31,500 images. Approximately 700 images are included in each class.
- •
EuroSAT (EU) (Helber et al., 2019): a 10-class satellite image classification dataset with 27,000 labeled and geo-referenced images.
- •
SVHN (SV) (Netzer et al., 2011): a real-world digit classification dataset derived from house numbers in Google Street View images. This datasets consists of 10 classes with 73,257 training samples and 26,032 test samples. Additional 531,131 samples are available for training.
- •
GTSRB (GT) (Stallkamp et al., 2011): a traffic sign classification dataset consisting of 43 classes and more than 50,000 samples in total.
- •
MNIST (MN) (LeCun et al., 1998): a benchmark dataset for image classification, containing grayscale images of handwritten digits across 10 classes. It includes 60,000 training and 10,000 test images, with a balanced number across classes.
- •
DTD (DT) (Cimpoi et al., 2014): a texture classification dataset consisting of 47 classes and a total of 5,640 images, with approximately 120 images per class.

Baseline Details.
We evaluate performance using eight comparison baselines and four alternative configurations of our method.

- •
Individual: Each task is handled by an independently fine-tuned model with no interference between tasks. However, this approach cannot perform multiple tasks simultaneously.
- •
Traditional MTL: This approach aggregates the original training data from all tasks to train a single multi-task model. It serves as a reference upper bound for evaluating model merging performance.
- •
Weight Averaging: A simple model merging technique that averages the parameters of multiple models directly. It is typically considered a lower bound for model merging performance.
- •
Fisher Merging (Matena and Raffel, 2022): This method computes the Fisher Information Matrix to assess parameter importance, guiding the model merging process based on these importance scores.
- •
RegMean (Jin et al., 2023): This approach introduces a regularization constraint during merging, enforcing the $L_{2}$ distance between the merged model and individual models to remain small.
- •
Task Arithmetic (Ilharco et al., 2023): This method is the first to propose the concept of “task vectors” and merges these vectors into a pre-trained for model merging.
- •
Ties-Merging (Yadav et al., 2023a): This approach addresses task conflicts in Task Arithmetic (Ilharco et al., 2023) by removing redundant parameters and resolving sign conflicts through a three-step procedure: Trim, Elect Sign, and Disjoint Merge.
- •
EMR-MERGING (Huang et al., 2024): A tuning-free method that merges models in three steps, by selecting a unified parameter sign (Elect), aligning task-specific parameters via masking (Mask), and adjusting their magnitudes with task-specific scaling factors (Rescale).
- •
AdaMerging (Yang et al., 2023): This method builds on Task Arithmetic (Ilharco et al., 2023) by employing an unsupervised method to automatically learn merging coefficients for each task vector.
- •
AdaMerging++ (Yang et al., 2023): An extension of Ties-Merging (Yadav et al., 2023a) that uses an unsupervised approach to learn task-specific merging coefficients.
- •
AdaMerging w/ Surgery (Yang et al., 2024b): A task-specific lightweight module that reduces representation bias through unsupervised optimization of merged-to-individual model epresentation alignment based on AdaMerging only at the final layer of the model.
- •
AdaMerging w/ SurgeryV2
(Yang et al., 2024c): This framework extends Representation Surgery (Yang et al., 2024b) to all intermediate layers through layer-wise transformations.
- •
WEMoE (Tang et al., 2024): This approach merges most parameters and upscales Transformer MLP layers to a weight-ensembling mixture of experts (MoE) module.
- •
StatsMerging (Ours): A lightweight learning-based method guided by the weight distribution statistical features (stats) of task-specific pre-trained weight models, including the mean, variance, magnitude and singular values. This method employs StatsMergeLearner to learn stats by knowledge distillation from task-specific teachers without manual labels from a small portion of validation data.
- •
StatsMerging++ (Ours): An extended version of StatsMerging trained on more validation data.

Training Details. We follow the same training procedure outlined in AdaMerging (Yang et al., 2023).

- •
Task-Specific Teacher: For each task, we utilize its corresponding Individual model as the Teacher.

Code is available at [https://github.com/statsmerging/statsmerging](https://github.com/statsmerging/statsmerging).

## Appendix B Extended Experiments

### B.1 Robustness Evaluation

We evaluate the robustness of StatsMerging against Task Arithmetic (Ilharco et al., 2023) and AdaMerging (Yang et al., 2023) under three image corruption scenarios: Motion Blur, Impulse Noise, and Gaussian Noise. The corrupted test sets are constructed following the protocols outlined in (Yang et al., 2023; Hendrycks and Dietterich, 2019). We assess performance on four datasets: Stanford Cars (CA) (Krause et al., 2013), EuroSAT (EU) (Helber et al., 2019), RESISC45 (RE) (Cheng et al., 2017), and GTSRB (GT) (Stallkamp et al., 2011). Results are reported in Table [5](https://arxiv.org/html/2506.04567v1#A2.T5). Overall, StatsMerging consistently outperforms the baselines. On the clean test set, it achieves a $2.4\%$ accuracy improvement over AdaMerging. Under corrupted conditions, StatsMerging yields performance gains of $3.1\%$, $6.3\%$, and $4.3\%$ for Motion Blur, Impulse Noise, and Gaussian Noise, respectively.

**Table 5: Robustness results when merging ViT-B/32 models on four tasks. StatsMerging: shaded in gray. Bold: top score. Values are reported in %.**
| Method | CA | EU | RE | GT | Avg Acc |
| --- | --- | --- | --- | --- | --- |
|  | Clean Test Set |  |  |  |  |
| Task Arithmetic | 66.9 | 94.7 | 82.6 | 75.1 | 79.8 |
| AdaMerging | 73.7 | 96.1 | 85.8 | 96.3 | 88.0 |
| StatsMerging | 75.6 | 96.3 | 92.1 | 97.6 | 90.4 (+2.4) |
|  | Motion Blur |  |  |  |  |
| Task Arithmetic | 65.3 | 68.1 | 80.0 | 64.2 | 69.4 |
| AdaMerging | 71.2 | 74.6 | 82.7 | 94.1 | 80.6 |
| StatsMerging | 73.5 | 76.9 | 89.2 | 95.2 | 83.7 (+3.1) |
|  | Impulse Noise |  |  |  |  |
| Task Arithmetic | 62.1 | 49.1 | 72.7 | 40.4 | 56.1 |
| AdaMerging | 67.2 | 30.8 | 75.9 | 77.5 | 62.8 |
| StatsMerging | 70.4 | 50.4 | 77.6 | 78.1 | 69.1 (+6.3) |
|  | Gaussian Noise |  |  |  |  |
| Task Arithmetic | 63.6 | 55.4 | 75.9 | 49.4 | 61.1 |
| AdaMerging | 69.9 | 41.2 | 80.6 | 76.0 | 66.9 |
| StatsMerging | 71.2 | 53.6 | 82.1 | 78.0 | 71.2 (+4.3) |

### B.2 Label Type and Loss Function Analysis

In this section, we analyze the performance of training StatsMergeLearner on two types of pseudo labels: (1) Soft Pseudo Labels, and (2) Hard Pseudo Labels, the former of which is commonly employed in knowledge distillation frameworks (Gou et al., 2021; Hinton et al., 2015) especially for classification tasks. Formally, we present two versions of our training losses:

Soft Pseudo Labels (SPL): The predicted class probability distribution. Thus we use Kullback–Leibler divergence (KL-Div) (Kullback and Leibler, 1951) loss function:

$$ $\mathcal{L}_{\mathrm{KL}}=\sum_{c=1}^{C_{m}}p_{c,k}\log\left(\frac{p_{c,k}}{q_ {c}}\right)$ (9) $$

where $p_{c,k}$ is the predicted probability of class $c$ from the pre-trained model $\theta_{k}$ on task $k$, and $q_{c}$ is the predicted probability of class $c$ from the merged model $\theta_{m}$.

Hard Pseudo Labels (HPL): The predicted class label in one-hot encoded format. Therefore, the cross-entropy loss is applied:

$$ $\mathcal{L}_{\mathrm{CE}}=-\sum^{C_{m}}_{c=1}\hat{y}_{c,k}\log(\hat{y}_{c}))$ (10) $$

Results are shown in [6](https://arxiv.org/html/2506.04567v1#A2.T6).
We highlight two key observations: (1) Training StatsMergeLearner with Hard Pseudo Labels (HPL) using cross-entropy loss (KD CE) yields performance comparable to training with ground-truth labels (GT CE), achieving $81.2\%$ vs. $88.5\%$ at the task-wise (TW) level and $83.5\%$ vs. $90.4\%$ at the layer-wise (LW) level. Importantly, StatsMerging eliminates the need for manually annotated labels, validating our intuition of leveraging task-specific teacher knowledge for supervision. (2) When trained on Soft Pseudo Labels (SPL) using KL-Divergence loss (KL-Div), StatsMergeLearner underperforms relative to HPL with cross-entropy, obtaining $73.3\%$ vs. $81.2\%$ at the TW level and $52.4\%$ vs. $83.5\%$ at the LW level, respectively.

We hypothesize that the observed performance drop is due to noisy inter-class relationships within the aggregated dataset  (Yuan et al., 2021). While a detailed investigation of these relationships is beyong the scope of this work on model merging, we believe it presents promising directions for future research.

Identifies "regularization samples" where soft labels degrade performance due to poor calibration or noisy class relationships. Proposes weighted soft labels to mitigate these issues.

**Table 6: Multi-task performance (Avg Acc $\%$) of StatsMerging when merging ViT-B/32 (4) models on four tasks. StatsMerging: shaded in gray. GT: Ground Truth. KD: Knowledge Distillation. GL: Granularity level. TW: Task-wise. LW: Layer-wise.**
| GL | Loss | CA | EU | RE | GT | Avg Acc |
| --- | --- | --- | --- | --- | --- | --- |
| TW | GT CE | 73.2 | 94.2 | 91.1 | 95.6 | 88.5 |
| TW | KD KL-Div | 56.5 | 97.6 | 56.5 | 82.4 | 73.3 |
| TW | KD CE | 64.2 | 88.6 | 85.2 | 86.7 | 81.2 |
| LW | GT CE | 75.6 | 96.3 | 92.1 | 97.6 | 90.4 |
| LW | KD KL-Div | 53.1 | 41.4 | 65.9 | 49.1 | 52.4 |
| LW | KD CE | 68.7 | 91.6 | 87.2 | 93.5 | 83.5 |

### B.3 Efficient Inference

StatsMergeLearner is designed to be lightweight, introducing minimal spatial and computational overhead to the overall merging process. As shown in Table [7](https://arxiv.org/html/2506.04567v1#A2.T7), it contains only 10.99M parameters, requires 2.95 GFLOPs, and achieves an inference time of 5.26 ms on an NVIDIA RTX A6000 GPU.

**Table 7: Model Size and Computational Overhead of StatsMergeLearner.**
| $\#$Params (M) | GFLOPs | Inference Time (ms) |
| --- | --- | --- |
| 10.99 | 2.95 | 5.26 |

### B.4 Training Curve

The training curve is shown in Figure [4](https://arxiv.org/html/2506.04567v1#A2.F4).

Figure: Figure 4: StatsMerging++ Training Accuracy Curve.
Refer to caption: x4.png

### B.5 Future Work and Limitations

In this work, we focus on vision-based classification tasks, leaving extensions to other domains, such as object detection (Tan et al., 2020), super-resolution (Sun et al., 2022), and image and video restoration (Liang et al., 2021; Merugu et al., 2025), for future work. Additionally, expanding this approach to language tasks, particularly large language models (LLMs)  (Yang et al., 2024a; Song et al., 2024; Zhang et al., 2024b; Tie et al., 2025; Kallini et al., 2025), as well as to multi-modal learning (Zhu et al., 2025; Du et al., 2025; Bousselham et al., 2024; Lin et al., 2024), represents a promising direction for further research.

## 

## References

- Abid et al. (2002)
Sabeur Abid, Farhat Fnaiech, and Mohamed Najim.
A new neural network pruning method based on the singular value decomposition and the weight initialisation.
In *2002 11th European Signal Processing Conference*, pages 1–4. IEEE, 2002.
- Akiba et al. (2025)
Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha.
Evolutionary optimization of model merging recipes.
*Nature Machine Intelligence*, pages 1–10, 2025.
- Bousselham et al. (2024)
Walid Bousselham, Felix Petersen, Vittorio Ferrari, and Hilde Kuehne.
Grounding everything: Emerging localization properties in vision-language transformers.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 3828–3837, 2024.
- Cao et al. (2023)
Bryan Bo Cao, Lawrence O’Gorman, Michael Coss, and Shubham Jain.
Data-side efficiencies for lightweight convolutional neural networks.
*arXiv preprint arXiv:2308.13057*, 2023.
- Cao et al. (2024a)
Bryan Bo Cao, Abhinav Sharma, Lawrence O’Gorman, Michael Coss, and Shubham Jain.
A lightweight measure of classification difficulty from application dataset characteristics.
In *International Conference on Pattern Recognition*, pages 439–455. Springer, 2024a.
- Cao et al. (2024b)
Bryan Bo Cao, Abhinav Sharma, Manavjeet Singh, Anshul Gandhi, Samir Das, and Shubham Jain.
Representation similarity: A better guidance of dnn layer sharing for edge computing without training.
In *Proceedings of the 30th Annual International Conference on Mobile Computing and Networking*, pages 2242–2244, 2024b.
- Cao et al. (2025)
Bryan Bo Cao, Lawrence O’Gorman, Michael Coss, and Shubham Jain.
Few-class arena: A benchmark for efficient selection of vision models and dataset difficulty measurement.
In *Proceedings of the International Conference on Learning Representations (ICLR)*, 2025.
URL [https://openreview.net/forum?id=2ET561DyPe](https://openreview.net/forum?id=2ET561DyPe).
- Chen et al. (2024)
I Chen, Hsu-Shen Liu, Wei-Fang Sun, Chen-Hao Chao, Yen-Chang Hsu, Chun-Yi Lee, et al.
Retraining-free merging of sparse mixture-of-experts via hierarchical clustering.
*arXiv preprint arXiv:2410.08589*, 2024.
- Chen and Kwok (2025)
Weiyu Chen and James T. Kwok.
Pareto merging: Multi-objective optimization for preference-aware model merging.
In *Proceedings of the 42nd International Conference on Machine Learning (ICML)*, 2025.
URL [https://arxiv.org/abs/2408.12105](https://arxiv.org/abs/2408.12105).
- Cheng et al. (2017)
Gong Cheng, Junwei Han, and Xiaoqiang Lu.
Remote sensing image scene classification: Benchmark and state of the art.
*Proceedings of the IEEE*, 105(10):1865–1883, 2017.
- Cimpoi et al. (2014)
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.
Describing textures in the wild.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 3606–3613, 2014.
- Crisostomi et al. (2024)
Donato Crisostomi, Marco Fumero, Daniele Baieri, Florian Bernard, and Emanuele Rodola.
$c^{2}m^{3}$: Cycle-consistent multi-model merging.
*Advances in Neural Information Processing Systems*, 37:28674–28705, 2024.
- Dosovitskiy et al. (2021a)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Georgiou, et al.
An image is worth 16x16 words: Transformers for image recognition at scale.
2021a.
- Dosovitskiy et al. (2021b)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.
An image is worth 16x16 words: Transformers for image recognition at scale.
In *International Conference on Learning Representations*, 2021b.
- Du et al. (2024)
Guodong Du, Junlin Lee, Jing Li, Runhua Jiang, Yifei Guo, Shuyang Yu, Hanting Liu, Sim Kuan Goh, Ho-Kin Tang, Daojing He, et al.
Parameter competition balancing for model merging.
*arXiv preprint arXiv:2410.02396*, 2024.
- Du et al. (2025)
Yiyang Du, Xiaochen Wang, Chi Chen, Jiabo Ye, Yiru Wang, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Zhifang Sui, et al.
Adamms: Model merging for heterogeneous multimodal large language models with unsupervised coefficient optimization.
*arXiv preprint arXiv:2503.23733*, 2025.
- Goetschalckx et al. (2018)
Koen Goetschalckx, Bert Moons, Patrick Wambacq, and Marian Verhelst.
Efficiently combining svd, pruning, clustering and retraining for enhanced neural network compression.
In *Proceedings of the 2nd International Workshop on Embedded and Mobile Deep Learning*, pages 1–6, 2018.
- Goodfellow et al. (2014)
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.
Generative adversarial nets.
In *Advances in neural information processing systems 27 (NIPS 2014)*, pages 2672–2680, 2014.
- Gou et al. (2021)
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao.
Knowledge distillation: A survey.
*International Journal of Computer Vision*, 129(6):1789–1819, 2021.
- He et al. (2016)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 770–778, 2016.
- Helber et al. (2019)
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.
Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.
*IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing*, 12(7):2217–2226, 2019.
- Hendrycks and Dietterich (2019)
Dan Hendrycks and Thomas Dietterich.
Benchmarking neural network robustness to common corruptions and perturbations.
In *Proceedings of the International Conference on Learning Representations (ICLR)*, 2019.
- Hinton et al. (2015)
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.
Distilling the knowledge in a neural network.
*arXiv preprint arXiv:1503.02531*, 2015.
- Ho et al. (2020)
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models.
In *Advances in Neural Information Processing Systems 33 (NeurIPS 2020)*, pages 6840–6851, 2020.
- Huang et al. (2024)
Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang.
Emr-merging: Tuning-free high-performance model merging.
*Advances in Neural Information Processing Systems*, 37:122741–122769, 2024.
- Ilharco et al. (2023)
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi.
Editing models with task arithmetic.
In *The Eleventh International Conference on Learning Representations*, 2023.
- Izmailov et al. (2018)
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson.
Averaging weights leads to wider optima and better generalization.
*arXiv preprint arXiv:1803.05407*, 2018.
UAI 2018.
- Jin et al. (2023)
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng.
Dataless knowledge fusion by merging weights of language models.
In *The Eleventh International Conference on Learning Representations*, 2023.
- Kallini et al. (2025)
Julie Kallini, Shikhar Murty, Christopher D. Manning, Christopher Potts, and Róbert Csordás.
Mrt5: Dynamic token merging for efficient byte-level language models.
In *Proceedings of the 13th International Conference on Learning Representations (ICLR 2025)*, 2025.
URL [https://openreview.net/forum?id=VYWBMq1L7H](https://openreview.net/forum?id=VYWBMq1L7H).
- Khanuja et al. (2021)
Simran Khanuja, Melvin Johnson, and Partha Talukdar.
Mergedistill: Merging pre-trained language models using distillation.
*arXiv preprint arXiv:2106.02834*, 2021.
- Kim and Yoo (2025)
Hyeonjin Kim and Jaejun Yoo.
Singular value scaling: Efficient generative model compression via pruned weights refinement.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 39, pages 17859–17867, 2025.
- Kim et al. (2024)
Jinuk Kim, Marwa El Halabi, Mingi Ji, and Hyun Oh Song.
Layermerge: neural network depth compression through layer pruning and merging.
*arXiv preprint arXiv:2406.12837*, 2024.
- Krause et al. (2013)
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.
3d object representations for fine-grained categorization.
In *Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW)*, pages 554–561. IEEE, 2013.
ISBN 978-1-4799-3022-7.
doi: 10.1109/ICCVW.2013.77.
- Krizhevsky (2009)
Alex Krizhevsky.
Learning multiple layers of features from tiny images.
Technical report, University of Toronto, 2009.
URL [https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf](https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf).
- Krizhevsky et al. (2012)
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.
Imagenet classification with deep convolutional neural networks.
In *Advances in neural information processing systems*, volume 25, pages 1097–1105, 2012.
- Kullback and Leibler (1951)
Solomon Kullback and Richard A Leibler.
On information and sufficiency.
*The Annals of Mathematical Statistics*, 22(1):79–86, 1951.
- LeCun et al. (1998)
Yann LeCun, Corinna Cortes, and Christopher JC Burges.
The mnist database of handwritten digits.
*http://yann.lecun.com/exdb/mnist/*, 1998.
- Lee et al. (2025)
Yeoreum Lee, Jinwook Jung, and Sungyong Baik.
Mitigating parameter interference in model merging via sharpness-aware fine-tuning.
*arXiv preprint arXiv:2504.14662*, 2025.
- Li et al. (2024)
Lu Li, Tianyu Zhang, Zhiqi Bu, Suyuchen Wang, Huan He, Jie Fu, Yonghui Wu, Jiang Bian, Yong Chen, and Yoshua Bengio.
Map: Low-compute model merging with amortized pareto fronts via quadratic approximation.
*arXiv preprint arXiv:2406.07529*, 2024.
URL [https://arxiv.org/abs/2406.07529](https://arxiv.org/abs/2406.07529).
- Liang et al. (2021)
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte.
Swinir: Image restoration using swin transformer.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pages 1833–1844, 2021.
- Lin et al. (2024)
Ziyi Lin, Dongyang Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Wenqi Shao, Keqin Chen, Jiaming Han, et al.
Sphinx: A mixer of weights, visual embeddings and image scales for multi-modal large language models.
In *European Conference on Computer Vision*, pages 36–55. Springer, 2024.
- Liu et al. (2022)
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.
A convnet for the 2020s.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 11976–11986, 2022.
- Mao et al. (2023)
Anqi Mao, Mehryar Mohri, and Yutao Zhong.
Cross-entropy loss functions: Theoretical analysis and applications.
In *International conference on Machine learning*, pages 23803–23828. PMLR, 2023.
- Matena and Raffel (2022)
Michael S Matena and Colin A Raffel.
Merging models with fisher-weighted averaging.
*Advances in Neural Information Processing Systems*, 35:17703–17716, 2022.
- Mencattini et al. (2025)
Tommaso Mencattini, Adrian Robert Minut, Donato Crisostomi, Andrea Santilli, and Emanuele Rodola.
Merge^3: Efficient evolutionary merging on consumer-grade gpus.
*arXiv preprint arXiv:2502.10436*, 2025.
- Merugu et al. (2025)
Ranjith Merugu, Mohammad Sameer Suhail, Akshay P Sarashetti, Venkata Bharath Reddy Reddem, Pankaj Kumar Bajpai, and Amit Satish Unde.
Joint flow and feature refinement using attention for video restoration.
*arXiv preprint arXiv:2505.16434*, 2025.
- Nagarajan et al. (2023)
Vaishnavh Nagarajan, Aditya K Menon, Srinadh Bhojanapalli, Hossein Mobahi, and Sanjiv Kumar.
On student-teacher deviations in distillation: does it pay to disobey?
*Advances in Neural Information Processing Systems*, 36:5961–6000, 2023.
- Netzer et al. (2011)
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al.
Reading digits in natural images with unsupervised feature learning.
In *NIPS workshop on deep learning and unsupervised feature learning*, volume 2011, page 4. Granada, 2011.
- Padmanabhan et al. (2023)
Arthi Padmanabhan, Neil Agarwal, Anand Iyer, Ganesh Ananthanarayanan, Yuanchao Shu, Nikolaos Karianakis, Guoqing Harry Xu, and Ravi Netravali.
Gemel: Model merging for $\{$Memory-Efficient$\}$,$\{$Real-Time$\}$ video analytics at the edge.
In *20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23)*, pages 973–994, 2023.
- Radford et al. (2021)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.
Learning transferable visual models from natural language supervision.
In *Proceedings of the 38th International Conference on Machine Learning (ICML)*, volume 139 of *Proceedings of Machine Learning Research*, pages 8748–8763. PMLR, 2021.
- Rombach et al. (2022)
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.
High-resolution image synthesis with latent diffusion models.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 10684–10695, 2022.
- Simonyan and Zisserman (2014)
Karen Simonyan and Andrew Zisserman.
Very deep convolutional networks for large-scale image recognition.
*arXiv preprint arXiv:1409.1556*, 2014.
- Singh et al. (2024)
Manavjeet Singh, Sri Pramodh Rachuri, Bryan Bo Cao, Abhinav Sharma, Venkata Bhumireddy, Francesco Bronzino, Samir R Das, Anshul Gandhi, and Shubham Jain.
Ovida: Orchestrator for video analytics on disaggregated architecture.
In *2024 IEEE/ACM Symposium on Edge Computing (SEC)*, pages 135–148. IEEE, 2024.
- Song et al. (2024)
Woomin Song, Seunghyuk Oh, Sangwoo Mo, Jaehyung Kim, Sukmin Yun, Jung-Woo Ha, and Jinwoo Shin.
Hierarchical context merging: Better long context understanding for pre-trained llms.
*arXiv preprint arXiv:2404.10308*, 2024.
- Stallkamp et al. (2011)
Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel.
The german traffic sign recognition benchmark: a multi-class classification competition.
In *The 2011 international joint conference on neural networks*, pages 1453–1460. IEEE, 2011.
- Starodubcev et al. (2024)
Nikita Starodubcev, Dmitry Baranchuk, Artem Fedorov, and Artem Babenko.
Your student is better than expected: Adaptive teacher-student collaboration for text-conditional diffusion models.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 9275–9285, 2024.
- Sun et al. (2022)
Long Sun, Jinshan Pan, and Jinhui Tang.
Shufflemixer: An efficient convnet for image super-resolution.
*Advances in Neural Information Processing Systems*, 35:17314–17326, 2022.
- Sun et al. (2025)
Wenju Sun, Qingyong Li, Yangli-ao Geng, and Boyang Li.
Cat merging: A training-free approach for resolving conflicts in model merging.
*arXiv preprint arXiv:2505.06977*, 2025.
- Tan et al. (2020)
Mingxing Tan, Ruoming Pang, and Quoc V Le.
Efficientdet: Scalable and efficient object detection.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 10781–10790, 2020.
- Tang et al. (2024)
Anke Tang, Li Shen, Yong Luo, Nan Yin, Lefei Zhang, and Dacheng Tao.
Merging multi-task models via weight-ensembling mixture of experts.
In *Proceedings of the 41st International Conference on Machine Learning*, pages 47778–47799, 2024.
- Tang et al. (2025)
Anke Tang, Enneng Yang, Li Shen, Yong Luo, Han Hu, Bo Du, and Dacheng Tao.
Merging models on the fly without retraining: A sequential approach to scalable continual model merging.
*arXiv preprint arXiv:2501.09522*, 2025.
- Tie et al. (2025)
Guiyao Tie, Zeli Zhao, Dingjie Song, Fuyang Wei, Rong Zhou, Yurou Dai, Wen Yin, Zhejian Yang, Jiangyue Yan, Yao Su, et al.
A survey on post-training of large language models.
*arXiv preprint arXiv:2503.06072*, 2025.
- Touvron et al. (2021)
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Hervé Jégou, and Alexandre Sablayrolles.
Training data-efficient image transformers & distillation through attention.
In *International Conference on Machine Learning (ICML)*, pages 10347–10357. PMLR, 2021.
- Vandenhende et al. (2022a)
Simon Vandenhende, Stamatios Georgoulis, Leander Arras, Luc Van Gool, and Radu Timofte.
Multi-task learning for computer vision: Recent advances and future directions.
*IEEE Transactions on Pattern Analysis and Machine Intelligence*, 44(10):6488–6513, 2022a.
- Vandenhende et al. (2022b)
Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool.
Multi-task learning for dense prediction tasks: A survey.
*IEEE Transactions on Pattern Analysis and Machine Intelligence*, 44(7):3614–3633, 2022b.
- Wortsman et al. (2022)
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt.
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.
In *Proceedings of the 39th International Conference on Machine Learning (ICML)*, volume 162 of *Proceedings of Machine Learning Research*, pages 23965–23998. PMLR, 2022.
- Xiao et al. (2016)
Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva.
Sun database: Exploring a large collection of scene categories.
*International Journal of Computer Vision*, 119:3–22, 2016.
- Yadav et al. (2023a)
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal.
Ties-merging: Resolving interference when merging models.
*Advances in Neural Information Processing Systems*, 36:7093–7115, 2023a.
- Yadav et al. (2023b)
Sachin Yadav, Chitta Malaviya, Graham Neubig, and Puneet Agarwal.
Merging transformers without training via a convex combination of parameter subsets.
pages 41105–41125, 2023b.
- Yang et al. (2023)
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao.
Adamerging: Adaptive model merging for multi-task learning.
*arXiv preprint arXiv:2310.02575*, 2023.
- Yang et al. (2024a)
Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao.
Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities.
*arXiv preprint arXiv:2408.07666*, 2024a.
- Yang et al. (2024b)
Enneng Yang, Li Shen, Zhenyi Wang, Guibing Guo, Xiaojun Chen, Xingwei Wang, and Dacheng Tao.
Representation surgery for multi-task model merging.
In *Proceedings of the 41st International Conference on Machine Learning*, pages 56332–56356, 2024b.
- Yang et al. (2024c)
Enneng Yang, Li Shen, Zhenyi Wang, Guibing Guo, Xingwei Wang, Xiaocun Cao, Jie Zhang, and Dacheng Tao.
Surgeryv2: Bridging the gap between model merging and multi-task learning with deep representation surgery.
*arXiv preprint arXiv:2410.14389*, 2024c.
- Ye et al. (2023)
Peng Ye, Chenyu Huang, Mingzhu Shen, Tao Chen, Yongqi Huang, Yuning Zhang, and Wanli Ouyang.
Merging vision transformers from different tasks and domains.
*arXiv preprint arXiv:2312.16240*, 2023.
- Yuan et al. (2021)
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng.
Rethinking soft labels for knowledge distillation: A bias–variance tradeoff perspective.
In *Proceedings of the International Conference on Learning Representations (ICLR)*, 2021.
URL [https://openreview.net/forum?id=6x_osD4AX9](https://openreview.net/forum?id=6x_osD4AX9).
- Zhang et al. (2024a)
Qitian Zhang, Mitchell Wortsman, Simon Kornblith, Rohan Taori, Tatsunori Hashimoto, Benjamin Recht, and Yair Carmon.
Zipit! merging models from different tasks without training.
In *International Conference on Learning Representations (ICLR)*, 2024a.
- Zhang and Yang (2021)
Yu Zhang and Qiang Yang.
A survey on multi-task learning.
*IEEE Transactions on Knowledge and Data Engineering*, 34(12):5586–5609, 2021.
- Zhang et al. (2024b)
Yuxin Zhang, Yuxuan Du, Gen Luo, Yunshan Zhong, Zhenyu Zhang, Shiwei Liu, and Rongrong Ji.
Cam: Cache merging for memory-efficient llms inference.
In *Forty-first International Conference on Machine Learning*, 2024b.
- Zhu et al. (2025)
Didi Zhu, Yibing Song, Tao Shen, Ziyu Zhao, Jinluan Yang, Min Zhang, and Chao Wu.
Remedy: Recipe merging dynamics in large vision-language models.
In *The Thirteenth International Conference on Learning Representations*, 2025.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 4
presentation: 3
questions: (1) Effect of extra validation data: The jump from LW-StatsMerging to LW-StatsMerging++ is substantial. Do other adaptation methods (e.g., Adamerging) show similar improvements as validation data increases? Additionally, since StatsMerging does not introduce any new hyperparameters of its own, what is the role of validation data? 


(2) Data sources: What exact data are used for training StatsMergeLearner and for distillation? Is it the training samples from each task? Also, is the setup strictly unlabeled for both training and validation, or is any labelled data employed?


(3) Training cost: Line 332 states StatsMergeLearner is trained for 500 epochs, which seems heavy compared to, e.g., Adamerging’s ~500 steps, which is okay since it also outperforms it, but should not be regarded as a lightweight method. Do (a) the number of models being merged and (b) backbone size (e.g., ViT-L/14) influence the number of training steps used to train the StatsMergeLearner?
soundness: 2
strengths: (1) String Performance: Consistently outperforms Adamerging and WEMoE on the reported benchmarks.

(2) Practical pipeline: The proposed Merge+Distill framework provides a workable path to architectural homogenisation, enabling broader applicability of merging.
summary: The paper introduces StatsMergeLearner, a network that predicts scaling coefficients for model merging from statistical features of model weights, trained using pseudo-labels generated by teacher models. It also proposes a Merge+Distill framework to homogenise model architectures—a prerequisite for merging—and reports strong gains over prior work across vision and language tasks.
weaknesses: (1) Limited task scale: Evaluation is restricted to the 8-task vision suite; it omits the widely used 14- and 20-task benchmarks from [1] on ViT-B/32, ViT-B/16, and ViT-L/14, which are important for understanding scaling with respect to task count and backbone size.

(2) Baselines: Missing comparisons to recent data-free merging methods such as Isotropic Merging[2], TSV-M[3], and KnOTS[4].

(3) Ablations: Task-level StatsMerging is introduced but not compared against layer-wise StatsMerging across settings, making it unclear what the difference in performance looks like.

References:

 [1] Wang, Ke, et al. "Localizing task information for improved model merging and compression." arXiv preprint arXiv:2405.07813 (2024).

[2] Marczak, D., Magistri, S., Cygert, S., Twardowski, B., Bagdanov, A. D., & van de Weijer, J. (2025). No task left behind: Isotropic model merging with common and task-specific subspaces. arXiv preprint arXiv:2502.04959.

[3] Gargiulo, A. A., Crisostomi, D., Bucarelli, M. S., Scardapane, S., Silvestri, F., & Rodola, E. (2025). Task singular vectors: Reducing task interference in model merging. In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 18695-18705).

[4] Stoica, G., Ramesh, P., Ecsedi, B., Choshen, L., & Hoffman, J. (2024). Model merging with svd to tie the knots. arXiv preprint arXiv:2410.19735.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: See weaknesses above.
soundness: 2
strengths: 1. The paper is well-organized and easy to follow.

2. The experiments demonstrate that the proposed method achieves promising results.
summary: This paper proposes a lightweight, learning-based model merging method named StatsMerging. The core idea is to adaptively predict the merging coefficients based on the weight distribution information.
weaknesses: 1. **Unclear motivation.** The usage of weight distributions plays a central role in the proposed method and serves as its main motivation. However, the paper lacks empirical or theoretical evidence to support the importance or effectiveness of this design choice.

2. **Limited technical contributions.** The use of knowledge distillation has been extensively studied in the context of model merging and other areas of machine learning. Similarly, learning adaptive merging coefficients has also been well explored in prior model merging research, which limits the novelty of the proposed approach.

3. **Reliance on training data.** The proposed method requires access to additional training data, whereas data-free model merging approaches have already been widely studied. This reliance may weaken the practical advantage of the proposed method.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: - Why is a rank of 3 chosen for SVD? Is it necessary to add ablation experiments on rank to identify the tradeoff between the overhead caused by rank and performance?
- Can a graph showing the relationship between the amount of training data and performance be provided? Additionally, can the specific amounts of data used for StatsMerging and StatsMerging++ be confirmed?
- Can the reason for the inconsistent results of LW StatsMerging++ in Table 3 and Table 7 of the Appendix be clearly explained? Alternatively, can the corrected experimental results be provided after re-evaluation?
soundness: 2
strengths: - The article demonstrates the absolute advantages of this method across various task sets and datasets through extensive experiments.
- The method proposed in the article does not require manually annotated labels; instead, it leverages pseudo-labels generated by existing models. This eliminates the need for manual annotation and provides certain support for the subsequent expansion of different scenarios.
- The structural design of SML is very simple with low training costs, and its computational cost is much lower compared to other model merging methods.
summary: This paper introduces StatsMerging, a statistics-guided model merging approach designed to consolidate multiple task-specific deep models—primarily in vision and NLP domains—without requiring ground truth labels or access to test samples. The architecture leverages Singular Value Decomposition (SVD)-derived statistics (mean, variance, norm, top singular values) from model weights and trains a lightweight StatsMergeLearner (SML) to predict adaptive merging coefficients through a novel task-specific teacher distillation process. The method extends to heterogeneous architectures and is validated across eight vision and seven NLP tasks, showing improvements over several baselines in average accuracy and robustness to data/label noise.
weaknesses: - There is a lack of certain theoretical or experimental explanations. For instance, it does not clarify the reason for choosing rank=3 when performing SVD.
- The experimental details are insufficient. For example, it fails to specify the exact amount of data used for training StatsMerging and StatsMerging++ respectively.
- There may be issues with the experimental results. For instance, the results of LW StatsMerging++ in Table 3 and Table 7 of the Appendix are inconsistent.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: 1.Why does the accuracy increase so dramatically at about 420 steps as shown in Figure 8?
soundness: 3
strengths: 1. The use of SVD singular values as a proxy for task/layer importance is a reasonable extension of existing merging techniques. This is also used in some recently published model merging methods, such as TSVM (task singular vector merging).
2. Handling heterogeneous architectures via distill-then-merge is a practical contribution.
summary: The paper proposes StatsMerging, a method for merging pre-trained models using weight distribution statistics to guide the learning of merging coefficients.
weaknesses: Below are some weaknesses of the manuscript and suggestions for improve the manuscript:

1. The averaged performance of individual fine-tuned models could be shown in Figure 4 using a vertical line.
2. Lack comparison with recent state-of-the-art training-free model merging methods, such as TSVM (task singular vector merging) and RegMean++. 
3. Mirror typos: 
    - At line 371 on page 7, should "MEMoE" be "WEMoE"?
    -  At line 226-235 on page 5, please verify that all instances of the symbols $\sigma_{r}$ and $\sigma’_{r}$ are used correctly.
4. SVD rank is fixed at 3 without ablation—why not 1, 5, or 10? How sensitive is performance to rank choice?
5. Weight statistics are used but no justification for why these capture "task importance" better than alternatives like Fisher information used in Fisher merging.


Decision: Reject