Scalable Model Merging with Progressive Layer-wise Distillation
Authors: Jing Xu
Jiazheng Li
Jingzhao Zhang
Venue: ICML 2025 poster (2025)
Keywords: Model Merging
Task Vector
Distillation
Primary area: general_machine_learning->transfer_multitask_and_metalearning
TL;DR: We propose an effective and scalable model merging algorithm based on progressive layer distillation.

Abstract: Model merging offers an effective way to integrate the capabilities of multiple fine-tuned models. However, the performance degradation of the merged model remains a challenge, particularly when none or few data are available. This paper first highlights the necessity of domain-specific data for model merging by proving that data-agnostic algorithms can have arbitrarily bad worst-case performances. Building on this theoretical insight, we explore the relationship between model merging and distillation, introducing a novel few-shot merging algorithm, ProDistill (Progressive Layer-wise Distillation). Unlike common belief that layer-wise training hurts performance, we show that layer-wise teacher-student distillation not only enhances the scalability but also improves model merging performance. We conduct extensive experiments to show that compared to existing few-shot merging methods,ProDistill achieves state-of-the-art performance, with up to 6.14\% and 6.61\% improvements in vision and NLU tasks. Furthermore, we extend the experiments to models with over 10B parameters, showcasing the exceptional scalability of ProDistill.

Scalable Model Merging with Progressive Layer-wise Distillation
Jing Xu 12 Jiazheng Li
Jingzhao Zhang 12
Abstract
Performance Vision Tasks
Performance Language Tesks
SUN397
1.0
CuLAA
.10
Model merging offers an effective way to in- tegrate the capabilities of multiple fine-tuned models. However, the performance degrada- tion of the merged model remains a challenge, particularly when none or few data are avail- able. This paper first highlights the necessity of domain-specific data for model merging by proving that data-agnostic algorithms can have arbitrarily bad worst-case performance. Build- ing on this theoretical insight, we explore the relationship between model merging and distil- lation, introducing a novel few-shot merging al- gorithm, ProDistill (Progressive Layer-wise Distillation). Unlike common belief that layer- wise training hurts performance, show that layer-wise teacher-student distillation not only en- hances the scalability but also improves model merging performance. We conduct extensive ex- periments to show that compared to existing few- shot merging methods, ProDi ProDistiio sttil 11 achieves state-of-the-ard performance, with up to 6.14% and 6.61% improvements in vision and NLU tasks. Furthermore, we extend the experiments to mod- els with over 10B parameters, showcasing the exceptional scalability of ProDistill.
OTP
0.9
Cars
RIE
0.9
SST2
0'8
0.7
Mhis
NES
QNUA
MICC
GTSRE
MILLL
S15-8
SVHN
QCF
Roher Herging
AdMMerging Protival (Curs)
RagMtan
Localle-and-SSe
Figure 1. ProDistill consistently outperforms other meth- ods across nearly all considered tasks. The performance metrics for each task are normalized and then clipped minimum value of 0.5 for better visualization.
able on platforms like Hugging Face. Depending on the specific downstream datasets, fine-tuned models excel in specialized abilities, such as mathematics or coding. How- ever, complex tasks often require the integration of multiple abilities. For example, solving an advanced math problem may necessitate the assistance of computer programs to pro- duce accurate solutions. While multi-task learning (Caruana, 1997; Misra al., 2016; Sener & Koltun, 2018; Liu et al., 2019) address this challenge, it requires access to fine- tuning data and incurs significant computational overhead during retraining. On the other hand, model ensembling (Di- etterich et al., 2002; Kurutach et al., 2018: Ganaie et al., 2022) avoids retraining but introduces substantial storage overhead due to the need to deploy multiple models.
1. Introduction
Large-scale pre-trained models have revolutionized deep learning, achieving remarkable success across various do- mains such as language (Brown et al., 2020; Team et al., 2023; Touvron et al., 2023a) and vision (Dosovitskiy, 2020; Ramesh et al., 2021). Meanwhile, an increasing number of fine-tuned checkpoints are being made publicly avail-
Model merging (Yang et al., 2024a; Goddard al., 2024; Tang al., 2024) offers an elegant solution these chal- lenges. The work of Ilharco et al. (2022) finds that the differ- ence between fine-tuned and pre-trained weights, which they name task vectors, exhibits arithmetic properties, such as ad- dition and negation, which correspond to changes in model capabilities. Therefore, model merging can be achieved by taking a weighted average of the model weights, as illus- trated in Figure 2. This is connected to the linear mode connectivity (Frankle et al., 2020; Mirzadeh et al., 2020) of neural networks.
'Institute for Interdisciplinary Information Sciences, Ts- inghua University "Shanghai Qizhi Institute 'School of Com- puter Science. Beijing Institute of Technology. Correspon- dence Jing Xu 2 wwwaiii.snnguu..cu.... azheng <Fooeverlastingg 220 outlook. com>> Jingzhao Zhang <jingzhaoz& @ mail. tsinguu..chhhcccocoo.
Ji-
Although model merging improves storage efficiency and data protection, the performance the merged model can degrade, especially when the number of models scales up.
Proceedings of the 4 International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Scalable Model Merging with Progressive Layer-wise Distillation
Expert 1
Expert
Expert
Merged Feature
Merged Layer
Merged Feature
Merged Layer i+1
Coef 1
Coef
Coef
MSE
MSE
Pretrained Model
Merged Model
Fine-tuned Feature
Fine-tuned Layer
Fine-tuned Feature
Fine-tuned Layer |+1
Task
Task 2
Task T
Merging Layer
Merging Layer
Figure 2. Left: Overview of model merging. Each expert corresponds task vector 0; Oa. which is scaled by its corresponding merging coefficient A, and summed to get the merged model. Right: Illustration of ProDistill, The merged model layer and each fine-tuned model layer take input the merged feature and the fine-tuned feature, respectively. The MSE loss between these outputs used to update the merged model layer. The output features serve as inputs for merging the subsequent layer.
Recent studies (Matena & Raffel, 2022; Jin et al., 2022; Yang et al., 2023) propose various methods to handle this issue, many of which require a few-shot validation dataset that contains domain- specific information of downstream tasks. This seems to contradict the data-free nature of task arithmetic. In light of this, we raise the question:
in ProDistill is decomposed layer by layer (Bengio et al., 2006; Kulkarni & Karande, 2017; Hettinger et al., 2017; Karkar et al., 2024; Sakamoto & Sato, 2024), en- abling the algorithm to avoid traditional end-to-end training and instead progressively train each layer of the model. See Figure 2 for an illustration.
We conduct extensive experiments to evaluate the perfor- mance of P ProDist: ittiistill 11 across various tasks, architectures, and scales. Compared to both training-based and training- free baselines, ProDist: ittilll ill achieves a notable 6. 14% increase in absolute performance for vision tasks and 6.61% % increase for natural language understanding tasks.
Is domain-specifet data necessary for model merging?
We provide an affirmative theoretical answer to this question. We prove. for the first time, that the worst-case performance of any data-agnostic model merging algorithm can be ar- bitrarily bad, even for a simple linear model. Therefore, although data-agnostic algorithm achieves great empirical success, is theoretically reasonable assume access at least few-shot dataset.
Furthermore, ProDistill demonstrates improved data and computation efficiency, and incurs significantly lower memory costs. This makes ProDistill scalable to large model sizes. We apply ProDistill to merge large lan- guage models with over 10B parameters. To the best of our knowledge, this is the first time a teaining-based merging algorithm has been scaled such large model size.
Building on these theoretical insights, we next address the empirical challenge of few-shot model merging by exploring the following question:
How to fully leverage the data and the fine-tuned models improve merging performance?
We summarize our contribution follows:
We provide the first theoretical analysis the necessity of domain-specific data for model merging, proving its critical role in ensuring effective merging performance.
To this end, frame model merging teacher-student distillation problem, where the goal is to transfer the knowl- edge from multiple fine-tuned models (teacher) into the merged model (student). However, directly applying ex- isting distillation algorithms often results in large training memory overhead, particularly for large language models with billions of parameters.
We propose ProDistill. novel model merging algorithm that leverages teacher-student distillation to progressively merge model layers.
We conduct comprehensive empirical analyses to demonstrate the state-of-the-art performance of ProDistill on wide variety of tasks. Our exper- iments highlight the data, computation, and memory efficiency of the proposed method.
To address this challenge, propose novel model merg- 1 (Progressive Layer-wise Dis- rillation). ProDistill implements distillation through an activation matching strategy, where the merging coef- ficients are trained to minimize the activation distance be- tween teacher and student models. The training objective
ing algorithm, ProDi still
Code is available JingXoTHD/Scalatt Model, Moodel_errgingLe with Progress le
https://gittubb.cm//
2

---

Scalable Model Merging with Progressive Layer-wise Distillation
2. Preliminaries
rithms have shown significant empirical success, we prove that their worst-case performance can be arbitrarily poor, even for simple linear models.
We consider model merging in a pretrain-to-finee fonnttttte--teeeeeeeee setup. Let 0n denote the weights of a pre-trained model. Consider set of T' tasks, each with a model 0, fine-tuned from Og. Model merging aims to combine the knowledge learned by task-specific models 0 into unified model 0, which preserves the generalization ability of the pre-trained model and incorporates the specialized knowledge from each task.
3.1. Hardness Results for Fixed Models
Consider the following simplified setup for model merging. Suppose we have two tasks with datasets D1, D2 and loss function ((.,)). Let f1, f2 denote two models to merge, and let M denote a data-agnostic model merging algorithm, which we assume to be deterministic for simplicity.
Task vectors. A key insight in this setup is the task vectors. The lask vector for the i-th task is defined as т, 0 On- An effective model merging method (Ilharco et al., 2022; Zhang et al., 2023) is to compute a weighted average of task vectors and add it back to the pre-trained model:
The following hardness result states that for any such al- gorithm M, one can always construct adversarial datasets such that the merging performance is arbitrarily bad.
Theorem 3.1. There exist task and loss function such that for any data-age aata-agnostic model merging algorithm M. any pair of models fi + J2, and any E, C >> 0, there exists two datasets D1, D2, such that f1, f2 have a near-zero loss on D2 and D2, respectively:
Ô= 0o + Σ A O(T.).
where A denotes the merging coefficients, and ((-) is an optional transformation function applied to the task vectors.
The merging coefficients X; can operate at different gran- ularities. Common approaches include task-wise granular- ity (Ilarrco et al., 2022), which assigns single merging coefficient to each task, and layer-wise granularity (Yang et al., 2023), which assigns a coefficient to each layer of the models. In this paper, we take one step forward and con- sider element-wise granularity. Specifically, the merging coefficient A has the same dimensionality as 0,, and element- wise multiplication X; т; is performed for each task. In this paper, we do not apply additional transforma- tions o()) to the task vectors, as our method is parallel to the transformation-based methods.
(Di.f)) ((D2,f2)
but the merged model f MMi1 f2) has constant loss D2 U UD2
((Di U D2 2
On the other hand, there exists ground truth model f that has near-zero loss on D2 and D2:
((D U f2,
Theorem 3.1 is proved by adversarially constructing lin- ear regression instances based on the merged model. The complete proof' are deferred to Appendix A.
Notations. We use D, to denote few-shot unlabeled vali- dation dataset for each task that is possibly available. Define .) as the feature mapping of the model parameterized by 0, which gives the vectorized embedding of all interme- diate layers. Let L denote layer number. For model weights i and layer index 1, the (-th layer and (((h) (0(()) to denote the feature mapping function defined by this layer.
3.2. Hardness Results for Learned Models
Theorem 3.1 assumes fixed models fi and fz, which can deviate from real-world scenarios where models are trained on datasets. To address this, we extend the analysis to cases where models are learned using an algorithm L, with fi = L(Di), f2 = L(D2).. We prove the following result.
g( to denote the parameter of
3. Theoretical Limitations on Data-Agnostic Model Merging
Theorem 3.2. There exist a task, a loss function and a learning algorithm L, such that for any data data-agnostic model merging algorithm M and any E, C >> O. there exist two adversarial datasets D1, D2, such that Si L(D1), S2 ((D2) have a near-zero loss on D2 and D, respectively:
Model merging algorithms can be broadly classified into two categories based on data availability: data-agnostic algorithms, which only use the weights of pre-trained and fine-tuned models (e.g., Ilharco et al. (2022); Yadav et al. (2024); Yu al. (2024b)), and data- detandeeenddnt algorithms, which require access to a validation set (e.g., Matena & Raffel (2022); Jin et al. (2022)). While data-agnostic algo-
e(D1, Si),, f2),,,)),,,
but the merged model f = MM[h [2) has a constant loss on Dj UD2
"The layer-wise in the title and algorithm name does not refer the granularity A, but instead refers to the training procedure.
((D) U C.
3

---

Scalable Model Merging with Progressive Layer-wise Distillation
4. A Practical Algorithm for Few-Shot Model Merging
Algorithm 1: ProPProtiiiii (Progressive Layer-wise
Distillation)
Input: Pre-trained model weights Po, Fine-tuned model
In the previous section, we prove that data availability is crucial for effective model merging. Next, we propose a practical merging algorithm designed for such data-available settings. We start with a naïve distillation algorithm that directly minimizes the embedding distance, and build upon it to develop the main algorithm of this paper.
weights 0z unlabeled validation sets {D:}
Output: Merging coefficients ( 1 i - 1 1111115 L. Initialize ( = [(x, .) : æ E D:). and t; 0. 0, for i == 1, I.....T. for p 1 11=1tm to L do
Solve the objective function using gradient descent:
(2 R=1
arg min
Σ
4.1. Model Merging as Knowledge Distillation
D.
(a [__ x=1
(#L-2))ED(-))
Model merging can be viewed through the lens of knowl- edge distillation, a perspective that remains underexplored within the community. In this context, the teacher models correspond to fine-tuned models, and the goal is to create student model that integrates their knowledge and performs well on the downstream tasks.
10! + Σ x 11) o
oT² .2)
- (0
for: i =1= to T
Update D. using:
A common strategy in knowledge distillation is to align the internal features of the teacher and student models (Young et al., 2022: Jin et al., 2024). This coincides with recent findings in the model merging community, which show that the performance of the merged model 0 is positively cor- related with the similarity between its embeddings =(00,au a:) and those of the fine-tuned models ( æ). (Zhou et al., 2023; Yang et al., 2024b). Building on this insight, we pro- pose to align the merged model and fine-tuned models in the feature space by solving the following problem:
D (100 E oTς 21). ,1).
==1 x (0 zz) E 1
end
end
return
( for sungiiT,1sIE
Σ
min A
On the other hand, the model learned on the merged dataset f" = L(D1 U D2) that has near-zero loss on DD and D:2:
Ay
DT
1=1
RED;
T
Oo + Σ
A o T1,
(0,,
(I)
j=1
C(Di DD22 S.
The objective minimizes the f2 distance between the inter- nal embeddings of the merged model 0, + ビド= X; o T, and those of the fine-tuned model 0,, over the unlabeled validation set D, for each task. Intuitively, this encour- ages the merged model to behave similarly each fine- tuned model, in its specific input domain. The objective can be solved using standard optimization algorithms such as Adam (Kingma, 2014) or AdamW (Loshchilov, 2017) on the merging coefficients X;
The hard instance constructed in the proof involves solv- ing a sinear-separable classification problem with max- margin classifiers. A broad range of algorithms fall into this category, including traditional support vector machine (SVM) (Cortes, 1995) and gradient descent algorithms that have max- mar-mn implicit bias (Nacson et al., 2019; Lyu & Li, 2019). The key insight is that the learning process often discards certain information, such as distant data points out- side the margin. This enables adversarial manipulation of the datasets degrade merging performance.
Throughout this paper, we refer to directly minimizing Ob- jective as Disti offers several insights:
Despite its simplicity,
Remark 3.3. Theorem 3.1 and 3.2 are both worst-case anal- yses. They highlight the fundamental limitations of data- agnostic model merging algorithms, but do not contradict their empirical success. Instead, these results underscore the importance of data availability in achieving robust and consistent merging performance.
1. A daptiie Merging Coefficient. Gradient descent on A; can be interpreted as selecting the appropriate merging co- efficients, whose empirical importance has been repeatedly highlighted in recent works (Jin et al., 2022; Yang et al., 2023; Gauthier- Carin et al., 2024). Notably, the element-
4

---

Scalable Model Merging with Progressive Layer-wise Distillation
wise granularity of X gives the merged model greater ex- pressive power to fit the objective.
the following rule:
D = ((x, : DD).
2. Fine-grained Model Merging via Distillation. Mini- mizing 1 can also be viewed as a way to distill the knowl- edge from the fine-tuned teacher models into a merged stu- dent model. Unlike standard distillation algorithms (Hinton, 2015), Objective leverages task vectors T; as a prior on the trainable parameters. To justify this design choice, we further show in Appendix C.3 that in the few-shot setup:
D.! { (100 (08² + + f or +f .1))
!1)
j=l
5o(0)
(2), E ED!
121. 1.
This design better approximates the global distillation ob- jective I, distinguishing it from previous merging algo- rithms based on feature alignment (Jin et al., 2022; Yang et al., 2024b; Dai et al., 2025), which align the output of merged model and fine-tuned model under the same input. As demonstrated in Appendix C.4, the incorporation of dual inputs is critical for achieving high performance in layer- wise training.
Feature-based distillation loss provides stronger su- pervision compared with logit- distillation;
Optimizing the scaling coefficients yields better results, compared with directly optimizing the model weights.
4.2. Efficient Implementation by Progressive Layer-wise Distillation
We refer this algorithm as ProDistill, short for Progressive Layer-wise Distillation. The pscudocode for ProDistill is given in Algorithm 1. Compared to DistillMerge, ProDistill offers substantial effi- ciency gains. When merging a specific layer, ProDi i st 11 only requires memory for the task vector and merging co- efficients of the current layer, rather than the entire model. Furthermore, the forward and backward passes are also re- stricted within individual layers. Interestingly, unlike the common belief that layer-wise training leads to performance degradation, we show in Appendix C.2 that ProDistill outperforms its end-to-end counterpart Dist DistillMerge.. Meger
While Equation is a reasonable objective for training the merging coefficients, directly optimizing this objective in- curs significant memory overhead. This is because both the task vectors and the trainable merging coefficients, which have the same dimensionality as the model parameters, have to be stored in memory. The memory cost scales linearly with the number of tasks. Besides, the optimization process requires to store the activations, gradients and optimizer states, further exacerbating memory overhead. This chal- lenge becomes particularly critical when merging large lan- guage models, which often contain billions of parameters.
To mitigate this issue, we propose the following surrogate to Objective 1. Instead of optimizing the global objective across all layers simultaneously, we adopt a progressive, layer-by-layer merging strategy. For each layer (1 I I < L). we minimize the feature distance between layer embeddings using the following objective:
5. Experiments
In this section, we present comprehensive experiment re- sults to evaluate the effectiveness of ProDistill across various settings.
min
D.
5.1. Setup
(5) ( 1)
We consider three main experimental setups: (1) Merging Vision Transformers (Dosovitskiy, 2020) on image clas- sification tasks; (2) Merging BERT (Devlin, 2018) and RoBERTa (Liu, 2019) models on natural language under- standing (NLU) tasks; (3) Merging LLAMA2 (Touvron et al., 2023b) model on natural language generation (NLG) tasks
(
(
- (0.0 (0))
(2)
Compared to Objective 1, this layer-wise formulation fo- cuses only minimizing the embedding distances in each layer, instead of all intermediate embeddings simultane- ously. Moreover, this objective introduces dual inputs by feeding different intermediate embeddings to the fine-tuned and merged models. Specifically. D. maintains pairs of embeddings (z1, 22) after the 1-th layer, where z is the em- bedding of the merged model, while 22 is the embedding of the fine-tuned model. These internal activations are cached and updated using the trained coefficients for each layer by
Tasks and Models: For image classification tasks, we fol- low the setting in Ilharco al. (2022) and use Vision Trans- former (ViT) models pre-trained on the ImageNet dataset and subsequently fine-tuned on 8 downstream datasets. For NLU and NLG tasks, we merge the BERT-base and RoBERTa-bass models fine-tuned on NLU tasks from the GLUE (Wang, 2018) benchmark, and perform pairwise
5

---

Scalable Model Merging with Progressive Layer-wise Distillation
Table Performance merging ViT-B-32 models across eight downstream vision tasks. ProDistils consistendly outperforms the baselines under different data availability, The results for Localize-and and- And-Stich are directly taken from He et al. (2024).
Method
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg
Individual
75.34
77.73
95.98
99.89
97.46
98.73
99.69
79.36
90.52
Task Arithmetic
55.32
54.98
66.68
78.89
80.21
69.68
97.34
50.37
69.18
RegMean
67.47
66.63
81.75
93.33
86.68
79.92
97.30
60. 6
79.15
Fisher merging
63.95
63.84
66.86
83,48
79.54
60.11
91.27
49.36
69.80
Localize-and-Stich
67.20
68.30
81.80
89.40
87.90
86.60
94.80
62.90
79.90
AdaMerging
63.69
65.74
77.65
91.00
82,48
93.12
98.27
62.29
79.28
ProDistill (Ours)
68.90
71.21
89.89
99.37
96.13
95.29
99.46
68.03
86.04
Individual
ProDistill (Ours)
Arithmetic
RegMean
Fisher merging
AdaMerging
Localize- Sallzeeandd-tich
SVHN
Figure 3. The I-SNE visualization of VIT-B-32 model trained by different merging algorithms, on the SVHN dataset. The features given by ProDistill are the most separated, resembling those of fine-tuned models.
merging of I- WizardMath- 13B and llama-2- 13b-code-alpaca models, following the setting in (Yu al., 2024b). Detailed information on the models and datasets can be found in Appendix B.I.
We also visualize the final-layer activations of the merged model using t-SNE (Van der Maaten & Hinton, 2008). The results are given in Figure and Appendix D.4. The visual- ization shows that the features given by ProDistill 11 are more separated compared to the baselines, closely resem- bling those of the fine-tuned models.
Baselines: For vision and NLU tasks, we compare ProDistill with wide range of baselines, includ- ing Task Arithmetic (Ilharco et al., 2022), Fisher merg- ing (Matena & Raffel, 2022), RegMean (Jin et al., 2022), AdaMerging (Yang et al., 2023) and Localize-and-Stic (He al., 2024). All methods, except Task Arithmetic, require few-shot unlabeled validation dataset, which is randomly sampled from the training set, with validation shot set to 64 per task. For NLG tasks, compare P ProDistil ill with Task Arithmetic (Ilharco et al., 2022), TIES-Merging (Ya- dav et al., 2024) and WIDEN (Yu et al., 2024a), due to scale constraints. A detailed discussion the baselines and their implementations is provided in Appendix B.2 and B.3.
5.3. Results on Merging Encoder-based I Language Models
Table 2 summarizes the results of merging RoBERTa models fine-tuned on the NLU tasks. The results BERT mod- els are deferred to Appendix D.I. Similar to the vision tasks, P ProDi st stilll achieves significant performance im- provements of 6.61% on the NLU tasks, outperforming all baselines across nearly all tasks.
Unlike vision tasks, the NLU tasks in the GLUE benchmark have small class numbers. For example, SUN387 dataset consists of 397 classes, while CoLA only has 2 classes. This class size disparity limits the performance of meth- ods that operate directly on the model output logits, such as AdaMerging and Fisher merging. Our method, along with RegMean, performs particularly well, emphasizing the importance of leveraging internal feature embeddings for effective model merging.
5.2. Results on Merging VIT models
Table presents the performance of merging ViT-B-32 mod- els across eight downstream vision tasks. The results for ViT-B-16 and ViT-L-14 are provided in Appendix D.I.
Our method consistently outperforms all baselines, yielding significant improvements in average performance. Specif- ically, broDistill achieves an average performance of 86.04%, surpassing the baselines by 6. 6.1. Notably, it is only 4% below the average performance of the individual fine-tuned models.
5.4. Results on Merging Large Language Models
We present the results of merging the WizardMath-133 and --ne-----BBB--- Lnd-aa2--11B- 3B-Codc---Cpccc models in Table 3, with ad-
6

---

Scalable Model Merging with Progressive Layer-wise Distillation
Table Performance merging RoBERTa models on the NLU tasks. ProDistill achieves superior performance across almost
all tasks.
Method
CoLA
SST-2
MRPC
STS-B
QQP
MNLI
QNLI
RTE
Avg
Individual
0.5458
0.9450
0.8858
0.9030
0.8999
0.8710
0.9244
0.7292
0.8380
Task Arithmetic
0.0804
0.8475
0.7865
0.4890
0.8133
0.7063
0.7558
0.6534
0.6415
RegMean
0.3022
0.9255
0.8183
0.5152
0.8176
0.7089
0.8503
0.6462
0.6980
Fisher merging
0.1633
0.7064
0.7264
0.1274
0.6962
0.4968
0.5599
0.5776
0.5068
Localize- and-Stich
0.0464
0,8922
0.7916
0.7232
0.7821
0.5709
0.7703
0,5632
0.6425
AdaMerging
0.000
0.8532
0.7875
0.5483
0.8086
0.7039
0.7247
0.6390
0.6332
ProDistill (Ours)
0.4442
0.9312
0.8464
0.6942
0.8134
0.7857
0.8900
0.7076
0.7641
ditional results provided in Appendix D.1 and generation examples provided in Appendix D.5. These findings demon- strate that our method effectively scales up to models with over 1OB parameters, and achieves superior performance compared to baselines.
6.2. Computation Efficiency
We evaluate the computational efficiency of ProDistill by varying the training epochs from to 100. We set the validation shot to 64, and choose learning rate from (0.1, 0.01}.
6. Further Analyses
The results, provided in Figure 4, highlight the rapid con- vergence of ProDi ProDistill still. With just epoch, the av- erage accuracy of the merged model has significant im- provement, rising from 69.8 to 80.6. After approximately 10 epochs, the accuracy is nearly identical to the final results. Thus, despite being a training-based algorithm, ProDistill demonstrates exceptional computational ef- ficiency with fast convergence.
In this section, we provide additional analyses on the effi- ciency of P ProDiss stillll in terms of data usage, computa- tional cost, and memory requirements.
Additionally, we conduct a broad range of comparisons and ablation studies to further evaluate ProDistill. The results can be found in Appendix C, including:
The computation efficiency be partially attributed to P ProDistill's ability to leverage large learning rates ef- fectively, which we hypothesize is due to its layer-wise training scheme. In contrast, algorithms such as AdaMerg- ing exhibit unstable convergence at high learning rates, as shown in the figure.
Analyses of merging coefficient granularity (Ap- pendix C.1)
Comparison with the end-to-end merging algorithm DistillMerge (Appendix C.2)
6.3. Memory Efficiency
Comparison with standard supervised training and knowledge distillation (Appendix C.3)
Next, we evaluate the memory efficiency of P ProDistii by profiling the maximum GPU memory consumption dur- ing training (excluding pre-processing and evaluation). The batch size is set to 32 for vision tasks and 16 for NLU tasks. The validation shot is set 64. We compare P ProDistill with its unoptimized direct training version DistillMerge and AdaMerging.
Ablation studies on the design of dual inputs (Ap- pendix C.4)
6.1. Data Efficiency
We first evaluate the data efficiency of ProDi itll by varying the number of validation samples, ranging from to 256. The results are presented Figure 4.
The results, shown in Figure 4, illustrate that P ProDistill has an almost negligible GPU memory footprint compared to the baselines. This difference particularly evident for models with a large number of layers, such ViT-L-14, since the memory cost of our method remains independent of the model's depth. Therefore, P roDistii offers sub- stantial ad advantages in memory efficiency and is scalable in resource-consee constraied environments.
For vision tasks, ProDistill achieves a performance im- provement of over 7% even with just 1-shot validation data per task. As the number of validation shots increases, the performance continues to improve, consistently surpassing that of AdaMerging. With 256 validation shot, the accu- racy reaches 88.05%, which is only 2% lower than that of individual checkpoints.

---

Scalable Model Merging with Progressive Layer-wise Distillation
Table Performance merging LLM models on Code and Math tasks. Our method demonstrates an improved performance and strong scalability. The results of TIES Merging and WIDEN are directly taken from Yu et al. (2024a).
Method
GSM8K
MATH
HumanEval
MBPP
Avg
Norm Avg
WizardMath-133
0.6361
0.1456
0.0671
0.0800
0.2322
0.6430
Llama-2- 1-----33--ddeapppee
0.000
0.000
0.2378
0.2760
0.1285
0.5000
Task Arithmetic
0.6467
0.1462
0.0854
0.0840
0.2406
0.6711
TIE--Merging
0.6323
0.1356
0,0976
0.2240
0.2723
0.7868
WIDEN
0.6422
0.1358
0.0976
0.0980
0.2434
0.6769
ProDistill (Ours)
0.6279
0.1424
0.1280
0.2239
0.2806
0.8288
Average Accuracy
Average Accurecy Epochs
Traing Memory Models
Prodketll (Dun))
AddHarging
Transinng Memory Cont (G.))
A100 Nenory Liik (80 GBI
Tisk de Arthhnct
62.5
20
40
Pro Misil Allmmagg
77.5
75.0
40
Pechabil (Gurs), Ir-O U21
12.5
Predusoll (Qurs), InwO
Adellergss, imnou11
20
AdaMerging Ine..
22
27
20
40
00
t0
100
1
VIL.14
REST
Fpoms
Figure 4. Analysis of Data, Computation and Memory Efficiency. Left: The average accuracy of ProDistill and AdaMerging across vision tasks, with different data availability. Our method demonstrates superior data efficiency. Middle: The average accuracy of ProDistill with different training epochs. Our algorithm achieves a fast convergence. Right: The training GPU memory cost of ProDistill, its unoptimized counterpart Disti Merge and AdaMerging. Our method has a significantly smaller memory footprint.
7. Related Works
2024; Huang al., 2024) propose dynamic model merging through task-specific routing and mixture-of-exxiee off-exeerss frame- works. Our method differs from them by preserving the original model architecture.
Model Merging via Weight Averaging. Weight averag- ing an effective and widely adopted approach in model merging (Izmailov et al., 2018; Wortsman et al., 2022; I1- harco et al., 2022). Researchers have developed various methods to improve the averaging approach. One line of work focuses on minimizing conflicts and promoting disentanglement between task vectors, through sparsifica- tion (Tang et al., 2023a; Yadav et al., 2024; Yu et al., 2024b; He et al., 2024; Wang et al., 2024b; Bowen et al., 2024; Deng al., 2024; Zhu et al., 2024; Davari & Belilovsky, 2025) or decomposition (Tam al., 2023; Xiong al., 2024; Stoica al., 2024; Wei et al., 2025; Gargiulo et al., 2024; Marczak et al., 2025; Yang et al., 2025), Another line of work (Ortiz- Jimenez. et al., 2024; Tang et al., 2023b) employs linearized training to explicitly enforce linearity. Some studies ex- plore methods for selecting optimal merging coefficients, using training-based (Yang et al., 2023; Gauthier- Caron al., 2024; Nishimoto et al., 2024) or training- maining-free (Matena & Raffel, 2022; Jin et al., 2022; Zhou et al., 2024; Wang et al., 2024a; Liu et al., 2024; Tang et al., 2025) approaches. Broadly speaking, our paper aligns with the former cate- gory. Other works (Qi et al., 2024; Lu et al., 2024; Zheng & Wang, 2024; Oh al., 2024; Zhang et al., 2024b; Osial et al.,
In addition to merging fine-tuned models, there is a broader field of research exploring more general setups, such as merging independently trained models (Singh & Jaggi, 2020; Ainsworth al., 2022; Navon et al., 2023; Horoi et al., 2024; Stoica et al., 2023; Xu et al., 2024) and merging models with different architectures (Avrahami et al., 2022; Wan et al., 2024a;b).
Distillation and Activation Alignment. Knowledge dis- tillation (Hinton, 2015; Romero et al., 2014; Yim et al., 2017) is well-established topic in machine learning where a student model is trained to mimic the behavior of teacher models. Some works leverage teacher-student activation matching merge models and do multi-task learning (Li & Bilen, 2020; Ghiasi al., 2021; Yang et al., 2022; Jin et al., 2022; Kong et al., 2024; Zhang al., 2024a; Nasery et al., 2024), which share similarities with our paper. For example, the Surgery algorithm proposed in (Yang et al., 2024b;c) introduces lightweight, task-specific modules to facilitate post-merging activation matching. However, these methods differ from ours as dynamic model merging approaches.
8

---

Scalable Model Merging with Progressive Layer-wise Distillation
very recent work Dai et al. (2025) matches the activations in each layer by solving linear equations of merging co- efficient. Their approach differs from ours in several key aspects. First, their merging coefficient granularity is layer- wise, whereas we consider element-wise coefficients. This distinction is crucial, as their method cannot be directly ex- tended to element-win wise merging without making the linear equations under-determined. Additionally, their layer inputs generated by pretrained models, in contrast the dual inputs approach used in our approach. The training-free algorithm RegMean (Jin et al., 2022) also shares similarity with our method. We leave its discussion to Appendix B.2.
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski,
H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 2021.
Avrahami, O., Lischinski, D., and Fried, O. Gan cocktail: mixing gans without dataset access, In European Confer- ence Computer Vision, pp. 205-221, Springer, 2022.
Bar-Haine HaimH,im, R., Dagan, I., Dolan, B., Ferro, L.., Giampiccolo, D., Magnini, B., and Szpektor, I. The second pascal recog- nising textual entailment challenge. In Proceedings of the second PASCAL challenges workshop on recognising textual entailment, volume 1. Citeseer, 2006.
8. Conclusion
Bengio, Y., Lamblin, P., Popovici, D., and Larochelle, H.
In this paper, we propose a novel model merging algorithm ProDistill which uses progressive layer-wise distilla- tion to efficiently merge large pre-trained models. Our theo- retical analysis shows the necessity of domain-specific data for effective merging. Empirical results demonstrate that ProDistill still outperforms existing methods across a vari- ety of tasks, achieving significant performance gains with reduced memory costs, which makes a scalable solution for merging large pre-trained models.
Greedy layer- wise training of deep networks. Advances neural information processing systems, 19, 2006.
Bentivogli, L.,, Clark, P., Dagan, I., and Giampiccolo, D. The fifth pascal recognizing textual entailment challenge. TAC, 7(8):1, 2009.
Bowen, T., Songning, L., Jiemin, w., Zhihao, S., Shiming, G., and Yutao, Y. Beyond task vectors: Selective task arithmetic based on importance metrics. arXiv preprint arXi::2411.16J39 2024.
Impact Statement
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
This paper introduces P ProDistill, a scalable and effi- cient model merging algorithm. Its potential applications could improve Al deployment in resource-constraing en- vironments, enabling the integration of specialized models across diverse tasks without significant retraining or compu- tational overhead.
Caruana, R. Multitask learning. Machine learning, 28: 41-75, 1997.
The broader societal impacts include reducing energy con- sumption and computational costs of training large mod- els, promoting more sustainable development. Addi- tionally, by improving the accessibility of advanced mod- els, P ProDist i 11 could help democratize Al capabilities, making them more adaptable and widely available across industries.
Cer, D., Diab, M., Agirre, E., Lopez- Gazpio, I., and Specia, I.. Semeval-2017 task 1: Semantic textual similarity- multilingual and tross-lingual focused evaluation. arXiv preprint arXiv: 001./00....00 00055, 2017.
Chaudhary, S. Code alpaca: An instraction-followinng llama model for code generation. GitHub repository, 2023.
There are potential ethical considerations, such as ensuring data privacy, mitigating biases, and preventing misuse of increasingly powerful AI systems. We emphasize the im- portance of responsibly managing the deployment of such technologies to minimize unintended consequences while maximizing their societal benefits.
Chen, M.. Tworek, J., Jun, H., Yuan, Q.. Pinto, H. D. O.,
Kaplan, J., Edwards, H., Burda, Y.. Joseph, N., Brockman. G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.033744 2021.
Cheng, G., Han, J., and Lu, X. Remote sensing image scene classification: Benchmark and state of the art. Proceed- ings of the IEEE, 105(10):1855-1833 2017.
References
Ainsworth, S. K., Hayase, J., and Srinivasa, S. Git re- re-basin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209. 0,436, 2022.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.

---

Scalable Model Merging with Progressive Layer-wise Distillation
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10..1110.444. 14168, 2021.
Gargiulo, A. A., Crisostomi, D., Bucarelli, M. S., Scar- dapane, S., Silvestri, F,, and Rodolå, E. Task singular vectors: Reducing task interference in model merging. arXiv preprint arXiv:2412.0008 2024.
Cortes, C. Support-vector networks. 1995.
Machine Learning,
Gauthier-Caron T., Siriwardhana, S., Stein, E., Ehghaghi, M., Goddard, C., McQuade, M., Solawetz, J., and Labonne, M. Merging in a bottle: Differentiable adaptive merging (dam) and the path from averaging automation. arXiv preprint arXiv:2410 08371, 2024.
Dagan, I., Glickman, O., and Magnini, B. The pascal recog- nising textual entailment challenge. In Machine learning challenges workshop, pp. 177-190. Springer, 2005.
Ghiasi, G., Zoph, B., Cubuk, E. D., Le, Q. V., and Lin, T.-Y. Multi-task self-training for learning general representa- tions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8856-8865, 2021.
Dai, R., Hu, S., Shen, X., Zhang, Y., Tian, X., and Ye, J. Leveraging submodule linearity enhances task arith- metic performance in LLMs. In The Thirteenth In- ternational Conference on Learning Representations, 2025. URL htttps://openreviiieee www.reeeiewweweeeeeeeeeeeeeeeeee forum? id=i FPcM6X5VV.
Giampiccolo, D., Magnini, B., Dagan, I., and Dolan, W.
The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop textual entailment and paraphrasing, pp. 1-9, 2007.
Davari, M. and Belilovsky, E. Model breadcrumbs: Scal- ing multi-task model merging with sparse masks. In European Conference on Computer Vision, pp. 270-287. Springer, 2025.
Goddard, C., Siriwardhana, S., Ehghaghi, M., Meyers, L., Karpukhin, V., Benedict, B., McQuade, M., and Solawetz, J. Arcee's 's mergekit: toolkit for merging large language models. arXiv preprint arXiv:2403. 2024.
Deng, W., Zhao, Y., Vakilian, V., Chen, M., Li, X., and Thrampoulidis, C. Dare the extreme: Revisiting delta- parameter pruning for fine-tuned models. arXiv preprint arXiv:2410.09334, 2024.
He, Y., Hu, Y., Lin, Y., Zhang, T., and Zhao, H. Localize- and-stitch: Efficient model merging via sparse task arith- metic. arXiv preprint arXiv:2408, 13656, 2024.
Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint 1OX0::181000885.. 2018
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Se- lected Topics in Applied Earth Observations and Remote Sensing, ((7))2217-22255 2019.
Dietterich, T. G. et al. Ensemble learning. The handbook of brain theory and neural networks, 2(1):110-13 125, 2002.
Dolan, B. and Brockett, C. Automatically constructing corpus of sentential paraphrases. In Third international workshop on paraphrasing ( (/WP2005), 2005.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.0:330 2020.
Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.1 11999, 2020.
Hettinger, C., Christensen, T., Ehlert, B., Humpherys, J., Jarvis, T., and Wade, S. Forward thinking: Building and training neural networks one layer at time. arXiv preprint arXiv: 1rXiv:1706.000 0708002400 2017.
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, B. Length-comtrolled alpacueval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404. 044455, 2024.
Hinton, G. Distilling the knowledge in a neural network. arXiv preprint 1503.0/5503.2553 2015.
Frankle, J., Dziugaite, G. K., Roy, D., and Carbin, M. Linear mode connectivity and the lottery ticket hypothesis. International Conference on Machine Learning, pp. 3259- 3269. PMLR, 2020.
Horoi, S., Camacho, A. M. O., Belilovsky, E., and Wolf, G. Harmony in diversity: Merging neural networks with canonical correlation analysis. In Forty-first International Conference on Machine Learning, 2024.
Ganaie, M. A., Hu, M.. Malik, A. K., Tanveer, M., and Suganthan, P. N. Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence, 115: 105151, 2022.
Huang, C., Ye, P., Chen, T., He, T., Yue, X., and Ouyang, W. Emr-merging: Tuning-fre free high-performance model merging, arXiv preprint arXiv:2405. 17461, 2024.
10

---

Scalable Model Merging with Progressive Layer-wise Distillation
Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing mod- els with task arithmetic. arXiv preprint arXiv:2212.04088, 2022.
Liu, S., Johns, E., and Davison, A. J. End-to-end multi-task learning with attention. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 1871-1880, 2019.
Iyer, S., Dandekar, N., Csernai, K., al. First quora dataset release: Question pairs. data. quora. com, 2017.
Liu, Y. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 364, 2019.
Izmailov, P., Podoprikhin, D., Garipov. T., Vetrov, D., and Wilson, A. G. Averaging weights leads to wider optima and better generalization. arXiv preprint 180300580 2018.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
Lu, Z., Fan, C., Wei, W., Qu, X., Chen, D., and Cheng, Y. Twin- Twin-merging: Dynamic integration of modular expertise in model merging. arXiv preprini urXiv:2406. 15479, 2024.
Jin, H., Son, S., Park, J., Kim, Y., Noh, H., and Lee, Y. Align- to-distill: Trainable attention alignment for knowledge distillation in neural machine translation. arXiv preprint arXiv:2403.01479, 2024.
Luo, H., Sun, Q. Xu, C., Zhao, P., Lou, J., Tao, C., Geng, X., Lin, Q., Chen, S., and Zhang, D. Wizard- math: Empowering mathematical reasoning for large lan- guage models via reinforced evol-instruct. arXiv preprint arXiv:2308. 0tt.:2338800983, 2023.
Jin, X., Ren, X., Preotiuc- Pietro, D., and Cheng, Data- less knowledge fusion by merging weights of language models. arXiv preprint arXiv:2212.098499 2022.
Karkar, S., Ayed, I., de Bézenac, E., and Gallinari, Module-wise training of neural networks via the minimiz- ing movement scheme. Advances in Neural Information Processing Systems, 36, 2024.
Lyu, and Li, J. Gradient descent maximizes the mar- gin of homogeneous neural networks. arXiv preprint arXiv: (056::/900008880 2019.
Marczak, D., Magistri, S., Cygert, S., Twardowski, B., Bag- danov, A. D., and van de Weijer, No task left behind: Isotropic model merging with common and task-specific subspaces. arXiv preprint arXiv:2502.049959 2025.
Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980 2014.
Kong, F., Zhang, R., Nic, Z., and Wang, Z. Rethink the evaluation protocol of model merging on classification task. arXiv preprint arXiv:2412. 22. /3526, 2024.
Matena, M. S. and Raffel, C. A. Merging models with fisher- weighted averaging. Advances in Neural Information Processing Systems, 15:17703-17716, 2022.
Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d object rep- resentations for fine- rree-graiedd categorization. In Proceed- ings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013.
Mirzadeh, S. I., Farajtabar, M.. Gorur, D., Pascanu, R., and Ghasemzadeh, H. Linear mode connectivity in multitask and continual learning. arXiv preprint arXiv:2010.00 04495, 2020.
Kulkarni, M. and Karande, S. Layer-wise training of deep networks using kernel similarity. arXiv preprint a7Xiv:1703.00 07115, 2017.
Misra, I,, Shrivastava, A., Gupta, A., and Hebert, M. Cross- stitch networks for multi-task learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 399439944444 4003, 2016.
Kurutach, T., Clavera, I., Duan, Y., Tamar, A., and Abbeel, Model-ensemble trust-region policy optimization. arXiv preprint
2018.
Nacson, M. S.. Lee, J., Gunasekar, S., Savarese, H. P., Srebro, N., and Soudry, D. Convergence of gradient descent separable data. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 3420-3428. PMLR, 2019.
LeCun, Y., Cortes, C., Burges, C., et al. Mnist handwritten digit database, 2010.
Li, W.-H. and Bilen, H. Knowledge distillation for multi- task learning. In Computer Vision-ECCV 2020 Work- shops: Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pp. 163-176. Springer, 2020.
Nasery, A., Hayase, J., Koh, W., and Oh, Pleas- merging models with permutations and least squares. arXiv preprint wxxiv:2407.02443 2024.
Liu. D., Wang, Z., Wang, B., Chen, W., Li, C., Tu, Z., Chu, D., Li, B., and Sui, D. Checkpoint merging via bayesian optimization in Ilm pretraining. arXiv preprint arXiv:2403. /9390, 2024.
Navon, A., Shamsian, A., Fetaya, E., Chechik, G., Dym, N., and Maron. H. Equivariant deep weight space alignment. arXiv preprint arXiv:2310. 13397, 2023.

---

Scalable Model Merging with Progressive Layer-wise Distillation
Netzer, Y., Wang, T.,, Coates, A., Bissacco, A., Wu, B., Ng. A. Y., al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 4. Granada, 2011.
Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013.
Stallkamp, J., Schlipsing, M., Salmen, J., and Igel, C. The german traffic sign recognition benchmark: a multi-class classification competition. In The 201 international joint conference on neural networks, pp. 1453-1460. IEEE, 2011.
Nishimoto, T., Hirose, Y., Kudo, Y., Yoshinari, N., Akizuki, R., Uchida, K., and Shirakawa, S. Differentiable dare- ties for neurips 2024 IIm merging competition. In LLM Merging Competition at NeurlPS 2024, 2024.
Stoica, G., Bolya, D., Bjorner, J., Ramesh, P., Hearn, T., and Hoffman, Zipit! merging models from different tasks without training. arXiv preprint arXiv:2305. 0.3000000033,. 2023.
Oh, C., Li, Y., Song, K., Yun, S., and Han, Adapting foundation models via training-free free dynamic weight in- terpolation. In Adaptive Foundation Models: Evolving Al for Personalized and Efficient Learning, 2024.
Stoica, G., Ramesh, P., Ecsedi, B., Choshen, L., and Hoff- man, J. Model merging with svd to tie the knots. arXiv preprint vrXiv:2410.1975:5 2024.
Ortiz-Jimenez. G., Favero, A., and Frossard, P. Task arith- metic the tangent space: Improved editing of pre- trained models. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Tam, D., Bansal, M., and Raffel, C. Merging by matching models in task subspaces. arXiv preprint arXiv: urXi::2312.043333 2023.
Osial, M., Marczak, D., and Zieliñssii, B. Parameter-efficient interventions for enhanced model merging. arXiv preprint arXiv:2412.17023 2024.
Tang, A., Shen, L., Luo. Y., Ding, L., Hu, H., Du, B., and Tao, D. Concrete subspace learning based interference elimination for multi-task model fusion. arXiv preprint arXiv:2312.06/73, 2023a.
Qi, B., Li, F,, Wang. 7.,, Gao, Li, D., Ye, P., and Zhou, B. Less is more: Efficient model merging with binary task switch. arXiv preprint arXiv:2412.0000 00005444 2024.
Tang, A.,
Shen, L., Luo, Y., Zhan, Y., Hu, H., Du, B., Chen, Y., and Tao, D. Parameter efficient multi-task model fusion with partial linearization. arXiv preprint arXi::2310.04742 2023b.
Rajpurkar, P. Squad: 100,000+ questions for machine com- prehension of text. arXiv preprint arXiv:1606.052550 2016.
Tang, A., Shen, L., Luo, Y., Hu, H., Du, B., and Tao, D. Fu- sionbench: A comprehensive benchmark of deep model fusion. arXiv preprint WxXiv:2406.03280 2024.
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text-to- image generation. In International conference on ma- chine learning, pp. 8821-8831. Pmlr, 2021.
Tang, A., Yang, E.,, Shen, L., Luo, Y., Hu, H., Du B., and Tao, D. Merging models on the fly without retraining: A sequential approach to scalable continual model merging. arXiv e-prints, pp. arXiv-2501, 2025.
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., and Bengio, Y. Fitnets: Hints for thin deep nets. arXiv 2014.
Team, G., Anil, R., Borgeaud, S., Alayrac, J.-.., Yu, Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 2023.
preprint 1rXiv:1412.655
Sakamoto, K. and Sato, I. End-to-end training induces information bottleneck through layer-role differentiation: A comparative analysis with layer-wise training. arXiv preprint arXiv:2402.0900 2024.
Touvron, H. Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprini urXiv:2302. htt.::233...199999 /3971, 2023a.
Sener, O. and Koltun, V. Multi-task learning multi- objective optimization. Advances in neural information processing systems, 31, 2018
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Singh, S. and Jaggi, M. Model fusion via optimal trans- port. Advances in Neural Information Processing Systems, 33:22045 21.22045-22255.... 2020.
Socher, R., Perelygin, A., WU, J., Chuang, J., Manning, C. D., Ng, A. Y., and Potts, C. Recursive deep models for semantic compositionality over sentiment treebank. In
Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
12

---

Scalable Model Merging with Progressive Layer-wise Distillation
Wan, F,, Huang, X.,
Cai, D., Quan, X., Bi, W., and Shi, S. Knowledge fusion of large language models. arXiv preprint arXiv:2401. 2024a.
Xu, Z., Yuan, K., Wang. H., Wang, Y., Song, M., and Song. J. Training-free pretrained model merging. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5915-5925, 2024.
Wan, F., Zhong. L., Yang. Z.,, Chen, R., and Quan, X. Fusechat: Knowledge fusion of chat models. arXiv preprint (tt::2408.079990 2024b.
Yadav, P., Tam, D., Choshen, L., Raffel, C. A., and Bansal, M. mies-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36, 2024.
Wang, A. Glue: A multi-task benchmark and analysis plat- form for natural language understanding. arXiv preprint arXiv: 10ll:/8800.000 0004400461,..0 2018.
Yang, E., Wang. Z., Shen, L., Liu, S., Guo, G., Wang, X., and Tao, D. Adamerging: Adaptive model merging for multi-task learning. arXiv preprint arXiv:3310.02575. 2023.
Wang, K., Dimitriadis, N., Favero, A., Ortiz-imennz,, G., Fleuret, F., and Frossard, P. Lines: Post-training layer scaling prevents forgetting and enhances model merging. arXiv preprint arXiv:2410.177 /7/46,, 2024a.
Yang, E., Shen, L.,, Guo, G., Wang, X., Cao, X., Zhang, J., and Tao, D. Model merging in Ilms, milms, and beyond: Methods, theories, applications and opportunities. arXiv preprint 2rXiv:2408.07666 2024a.
Wang, K., Dimitriadis, N., Jrmiz-iimenez, G., Fleuret, F., and Frossard, P. Localizing task information for im- proved model merging and compression. arXiv preprint arXiv:2405.078/1 2024b.
Yang, E., Shen, L., Wang, Z., Guo, G., Chen, X., Wang, X., and Tao, D. Representation surgery for multi-task model merging. arXiv preprint ar arXiv:240..02705, 2024b.
Warstadt, A., Singh, A., and Bowman, S. R. Neu- ral network acceptability judgments. arXiv preprint arXiv: 1805. 2018.
Yang, E., Shen, L.., Wang, Z., Guo, G., Wang, X., Cao, X., Zhang, J., and Tao, D. Surgeryv2: Bridging the gap be- tween model merging and multi-task learning with deep representation surgery. arXiv preprint arXiv:2410. /4389, 2024c.
Wei, Y., Tang. A., Shen, L., Xiong, F., Yuan, C., and Cao, X. Modeling multi-task model merging as adaptive projec- tive gradient descent. arXiv preprint arXiv:2501.01230, 2025.
Williams, A., Nangia, N., and Bowman, S. R. A broad- coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05444 2017.
Yang, J., Jin, D.. Tang, A., Shen, L., Zhu, D., Chen, Z.. Wang, D., Cui, Q.. Zhang, Z., Zhou, J., et al. Mix data or merge models? balancing the helpfulness, honesty, and harmlessness of large language model via model merging. arXiv preprint arXiv:2502. 06876, 2025.
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pp. 23965 23998. PMLR, 2022.
Yang, X., Ye, J., and Wang. X. Factorizing knowledge in neural networks. In European Conference on Computer Vision, pp. 73-91. Springer, 2022.
Yim, J., Joo, D., Bae, J., and Kim, A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4133- 4141, 2017.
Xiao, J., Ehinger, K. A., Hays, J., Torralba, A., and Oliva, A. Sun database: Exploring a large collection scene categories. International Journal of Computer Vision, 119:3-22, 22, 2016.
Young, L. D., Reda, F. A., Ranjan, R., Morton, J., Hu, J., Ling, Y., Xiang, X., Liu, D., and Chandra, V. Feature- align network with knowledge distillation for efficient denoising. In Proceedings of the IEFE/CVF Winter Con- ference on Applications of Computer Vision, pp. 709-718, 2022.
Xiong, F., Cheng, R., Chen, W., Zhang, Z., Guo, Y., Yuan, C., and Xu, R. Multi-task model merging via adaptive weight disentanglement. arXiv preprint arXiv:2411.18729, 2024.
Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., and Jiang, D. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. /2244, 2023.
Yu, L., Yu, B., Yu, H., Huang, F., and Li, Y. Extend model merging from fine-tuned to pre-trained large lan- guage models via weight disentanglement, arXiv preprint arXiv:2408 2.00022400. 2024a.
13

---

Scalable Model Merging with Progressive Layer-wise Distillation
Yu, L., Yu, B., Yu, H., Huang, F., and Li, Y. Language models super mario: Absorbing abilities from homol- ogous models free lunch. In Forty-first International Conference on Machine Learning, 2024b.
Zhang, F. Z., Albert, P., Rodriguez-Opazo. C., Hengel, A. d., and Abbasnejad, E.. Knowledge composition ing task vectors with learned anisotropic scaling. arXiv preprint arXiv:2407. 02880, 2024a.
Zhang, J., Liu, J., He, J., et al. Composing parameter- efficient modules with arithmetic operation. Advances in Neural Information Processing Systems, 36:12589- 12610, 2023.
Zhang. M., Liu, J., Ding, G.. Yu, X., Ou, L., and Zhuang, B. Channel merging: Preserving specialization for merged experts. arXiv preprint 2rXiv:2412.15283, 2024b.
Zheng. S. and Wang. Free-merging: Fourier transform for model merging with lightweight experts. arXiv preprint arXiv:2411.16811 2024.
Zhou, Y., Song, I.., Wang, B., and Chen, W. Metagpt: Merging large language models using model exclusive task arithmetic. arXiv preprint arXiv:2406. 1155, 2024.
Zhou, Z., Yang, Y., Yang, X., Yan, J., and Hu, W. Going beyond linear mode connectivity: The layerwise linear feature connectivity. Advances in Neural Information Processing Systems, 05:60853-608777 2023.
Zhu, D., Sun, Z., Li, Z., Shen, T., Yan, K., Ding, S., Kuang, K., and Wu, C. Model tailor: Mitigating catastrophic forgetting multi-modas large language models. arXiv preprint Arviv:2402. /2048, 2024.
14

---

Scalable Model Merging with Progressive Layer-wise Distillation
Appendix
A. Proofs
Theorem 3.1. There exist a task and loss function E, such that for any data-agnostic model merging algoritiu M, any pair of models ft + f2, and any E, C > 0, there exists two datasets D1, D2, such that fi, f2 have a near-zero loss on D and D2, respectively:
(DD. se, E(D2, f2) S.
but the merged model f M(fi, f2) has a constant loss on D1 UD:
(DD 2 C.
On the other hand, there exists ground truth model f that has near-zero loss on Dj and D2:
{() U D2, S.
Proof. We construct a hard instance of linear regression to prove the theorem. Let d denote the dimension of the data space. data point has the form x = (xc y). where a E R4 y E R, and the model is represented by a d-dimensional vector w E Rd. The loss function is the & loss É(2, w) = 1| w yl and ((D, w) Avg(E(z, w)).
Let w1, W2, i denote the weights of /1, f. Since fi * f2. we can assume, without loss of generality, that + w. Then we have the simple linear algebra fact that, for a large enough d, there exist (æ1, V1) and (æ2, 12), such that
21
I
2.
(m1),
2VC
91
3.
(() T
4. æ1 22 are not co-linear.
Let D1 = ((1) and D2 = {(x2, 12)3. This construction ensures that
1
((D1, f1)
x T
wi
y1112 0,
2
1
L(D2, f2) =
x2 W2
w2ll2 0,
2
(Di f)
x v - v1ll 2 = C.
On the other hand, since æ1, æ2 not co-linear, can find w" such that 2. " 8/1 xx. w" large enough. That is, we have
y/2 0, as long d is
((D: -)2
This completes the proof.
Theorem 3.2. There exist a task, loss function & and a learning algorithm L. such that for any data-agnostic model
merging algorithm M and any e, C > 0, there exist two adversarial datasets D1, D2, such that J1 = L(D1), f2 = L(D,) have a near-zero loss on D and D2 respectively:
(Di, ((D2, S
but the merged model f M(Ji,
(((1.2) f2) has a constant loss on D2 U Dz:
15

---

Scalable Model Merging with Progressive Layer-wise Distillation
((D1 UD,
On the other hand, the model learned on the merged dataset l" = L(Di UD2 that has near-zero loss on Dj and D2:
E(D1 U 12, f*) E.
Proof. Consider two-class linear-separable data classification problem in a d-dimensional space. Each data point consists of input E R label y E {1, (1). The hypothesis class is linear models, F = {f (w. b) w E {0), b E R. fxx w x + b}, where we exclude w = 0 to avoid degeneracy. For linear model f (wv, b) and data point == (z. v). the loss function is defined as {(z, = ( x + b), 0}. For a dataset 2, the loss function is defined as f) = Avg({(z, The algorithm outputs the &g normalized max- max--arginn classifier of training set with || = 1. which covers a wide range of practical algorithms including SVM and gradient descent algorithms that have max margin implicit bias.
Let d = 2. Consider four data points
2 (æ1, 91) = ((1,0).1) 1),
22 (x2, 12) ((-1,0), -1),
23 (I3, 333) = ((0, 1))
24 (24, V4) ((0, -1), -1),
Let D = {21, 22). D2 = {23, 21)-
easy to see that
L([21, f1 = ((1,0),0), L( 23, z1)) = =f2 = ((0, 1),0),
Consider merging S1 and f2. Let f = M((,, = ((vi), .22), Next we show that there exist such that
1. 0pp + u2 + b < -5C,
2. p 1 or 1.
We prove the fact by contradiction. If this claim does not hold, we know that for any p 1, E R and any q > 1,p E R. have dpp + uzq + i 2 -5C. This will give úū utg O, which contradicts the degeneracy of (w1, u2).
Let 25 = (as, ys) = ((p, 4). 1). If 1, we define
D1
{=1, 22, 20), D2 {z3, x1}.
If q > 1, we define
D1 (21, D2
{2s, 21, 26}.
This construction ensures that
1. C(Di) f1, ((Di, =0, 0,
2. ((D2) f2. ((,,, f2) =0 00
3. ((D) D2. f) - (wip+ + d22 6) C.
On the other hand, it is easy to see that Dj U D22 is linearly separable, due to the condition that p > 1 or g > 1. Therefore, the max- max-margin classifier fr = L(D, U D2) satisfies
((Di UD.2 0.
This completes the proof.
Remark A.I. The loss function in the proof does not reflect the classification accuracy. To take this into consideration, we can inject arbitrary number of adversarial data points like 2s, to make the classification accurate arbitrarily low.
16

---

Scalable Model Merging with Progressive Layer-wise Distillation
B. Experimental Setup
B.1. Dataset Details
For vision tasks, we follow the initial practice of (Ilharco et al., 2022) and build a vision benchmark consisting of eight datasets, including MNIST (LeCun al., 2010), EuroSAT (Helber et al., 2019), GTSRB (Stallkamp al., 201 1), SVHN (Netzer et al., 201 10, DTD (Cimpoi al., 2014), RESISC45 (Cheng al., 2017), Stanford Cars (Krause et al., 2013), SUN3977 (Xiao et al., 2016).
For natural language understanding (NLU) tasks, we follow the practice in Yu et al. (2024b) and use eight datasets from the GLUE benchmark (Wang, 2018), including CoLA (Warstadt et al.. 2018), SST-2 (Socher et al., 2013). MRPC (Dolan & Brockett, 2005). STS-B (Cer et al., 2017), QQP (lyer et al., 2017), MNLI (Williams et al., 2017), QNLI (Wang, 2018, 8; Rajpurkar, 2016), RTE (Wang, 2018; Dagan et al., 2005; Bar-Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009). We evaluate performance using accuracy for SST-2, QNLI, and RTE, matched accuracy for MNLI, the Matthews correlation coefficient for CoLA, the average of accuracy and Fl score for MRPC and QQP, and the average of Pearson and Spearman correlations for STS-B.
For natural language generation (NI.G) tasks, we follow the practice in (Yu et al., 2024a) and use WizardI MM-13B (Xu et al., 2023), WizardMath-Ie (Luo et al., 2023), (aama-2-1b-codd--lppae (Chaudhary, 2023) as Instruct, Math and Code expert models, respectively. Note that WizardLM-1331 model also has code generation abilities, and we include code benchmarks in its evaluation.
We use five datasets for evaluation, including AlpacaEval 2.0 (Dubois et al., 2024), GSMSK (Cobbe al., 2021). MATH (Hendrycks et al., 2020), HumanEval (Chen et al., 2021), and MBPP (Austin et al., 2021). AlpacaEval 2.0 measures performance using the win rate, defined as the proportion of instances where a more advanced large language model, specifically GPT-4 Turbo in this study, prefers the outputs of the target model over its own. GSM8K and MATH use zero-shot accuracy as the metric. HumanEval and MBPP use pass@) as the metric, representing the proportion of individually generated code samples that successfully pass the unit tests. In addition to reporting a simple average metric, we also provide the normalized average metric, calculated by dividing the metric of the merged model by the metric of fine-tuned models, ensure fair and consistent comparison across datasets.
B.2. Descriptions of Baselines
We evaluate several baseline methods for merging Vision Transformers (ViT) and encoder-based language models, which are outlined below:
Task Arithmetic (Ilharco et al., 2022): This approach constructs task vectors from model weights and merges them using arithmetic operations.
Fisher Merging (Matena & Raffel, 2022): This method uses Fisher-weighted averaging of model parameters merge models.
RegMean (Jin et al., 2022): RegMean determines merging coefficients by solving a linear equation to align internal embeddings, which shares similarity with ProDistill. One of the key differences is that RegMean relies on the linearity of the modules, and therefore can only be applied to linear modules. Our method is free of this limitation, since it a general distillation algorithm. Another primary difference from our method is that RegMean training- whereas P ProDiss s i 11 is training-based. This difference leads several consequences.
Performance:
The training-free nature of RegMean limits its ability to fully leverage the expressive power of neural network modules, resulting in lower performance compared to the training-based ProDistil still.
- -Computation: Although RegMean avoids the potential overhead of a lengthy training process, it still requires solving linear equations, which can become computationally intensive as model sizes increase. As demonstrated in Section 6.2, with just one epoch of training, ProDist ill achieves better merging results than RegMean, without incurring significant computational costs. Therefore, ProDistill does not suffer from high computational burden despite being training-based based method.
AdaMerging (Yang et al., 2023): This method adaptively selects merging coefficients by minimizing entropy on unlabeled test data. Like ProDistill, 1 it is both training- and data-dependent.
17

---

Scalable Model Merging with Progressive Layer-wise Distillation
Localize-and-Sticchh (He al., 2024): This approach identifies sparse masks to extract task-specific parameters from fine-tuned models and merges them back into the pretrained model.
We consider the following additional baselines for merging decoder-based large language models.
1. TIES-Merging (Yadav et al., 2024): TIES-Merging first trims task vectors to retain only the most significant parameters, then resolves the signs of the remaining parameters, and finally merges only those parameters that align with the resolved signs.
2. WIDEN (Yu et al., 2024a): WIDEN disentangles model weights into magnitude and direction components and merges them by considering their respective contributions. It is also applicable to merging independently trained models.
B.3. Implementation Details
We use the VIT checkpoints given by (Ilharco et al., 2022) for vision tasks. For NLU tasks, we fine-tune BERT-base-uncased and RoBERTa-base models for 10 epochs. The weight decay is set to 0.01. We use a learning rate of 10-5 with warm-up strategy.
For ProDistill ill and AdaMerging, train the merging coefficients using the Adam optimizer. The merging coefficients are initialized to 0.3 when merging eight models, and searched from {0.5, 1.0] when merging two models. The learning rate and training epochs are selected via grid search. For ViT models and LLMss, the learning rate is chosen from {0.1, 0.01 J; for Bert/RoBERTa models, the learning rate is chosen from {0.01, 0.001 The number of epochs is chosen from {50, 100, 200
For Task Arithmetic, we use a fixed merging coefficient of 0.3 when merging eight models, and search the coefficient from {0.5, 1.0) when merging 2 models. For RegMean, the scaling ratio to reduce its diagonal terms are selected from grid of {..7,0.8, 0.9, 1.0]. For Fisher Merging, the scaling coefficient is selected from a grid of {0.1, 0.3, 0.5, 0.7, 0.9, 1.0). For Localize-ande aattte--nnn-Stttchh we set sigmoid bias to 0.3, learning rate to 10", & regularization factor 10.0, sparsity level to 1% and epochs to 10.
For Vision and NLU tasks, the few-shot validation set is randomly sampled from the training set. For NLG tasks, the validation set is randomly sampled from the test set of AlpacaEval 2.0, GSM8K and MBPP, and we exclude these test data points in evaluation.
C. Ablation studies
C.1. Analyses of Merging Coefficient Granularity
This section analyzes the impact of different granularities on the merging coefficients. We consider three types of granularities:
1. Element-wise: Each element in the model's weights corresponds merging coefficient. In other words, the merging coefficients are tensors that have the same dimensions the model parameters.
2. Layer-wise: Each layer of the model is assigned scalar merging coefficient. The merging coefficient tensor, therefore, has the same number of elements as the number of model layers. In this paper, we slightly deviate from this naming convention by assigning a coefficient to each module in the network.
3. Task-wise: Each fine-tuned model has a scalar merging coefficient.
We conduct experiments using these three types of granularities, and present the results in Table 4. For our method, the element-wise granularity yields the best performance, owing to its higher expressive power. In contrast, for AdaMerging, the element-wise granularity performs worse than the layer-wise granularity. We hypothesize that this is due to the weak supervision power of entropy-based objective function in AdaMerging, which may lead to overfitting when the number of trainable parameters increased.
C.2. Comparison with DistillMerge
In this section, we compare the ProDistill algorithm, which uses the progressive training approach, with its unoptimized version Disti Merg, which uses the traditional end-to-end training approach and optimizes Objective 1 directly. To
18

---

Scalable Model Merging with Progressive Layer-wise Distillation
Val Shot
Granularities
Ours
AdaMerging
Val Shot
Granularities
Ours
AdaMerging
Element-wise
82.79
72.35
Element-wise
0.6980
0.5746
16
Layer-wise
73.81
75.92
16
Layer-wise 0.6340
0.6398
Task-wise
71.38
Task-wise
0.6406
Wlesent-wise
84.49
72.49
Element-wise 0.7473
0.5465
32
Layer-wise
73.50
78.28
32
Layer-wise 0.5996
0.6402
Task-wise
71.76
Task-wise
0.6403
Element-wise 86.04
73.11
Element-wise 0.7641
0.5415
64
Layer-wise
72.96
79.28
64
Layer-wise
0.5560
0.6332
Task-wise
71.69
Task-wise
0.6379
Table Impact of Granularity Merging Methods. Left: Accuracy results vision benchmarks using ViT-B-32. Right: Performance metrics on the NLU tasks using RoBERTa. The highlighted cells indicate the configurations used this paper.
ensure fair comparison, both approaches are evaluated using the same computational budget. The results, presented in Figure 5, indicate that P ProDisttll achieves better overall performance compared DistillMe rge. Therefore, progressive training also has performance advantage in the considered setup, in addition to its memory efficiency shown in Section 6.3.
VIT-B-32
RoBERTa
90
0.80
ProDistill (Ours)
ProDistill (Ours)
DistillMerge
DistillMerge
0.7641g.7612
8B
0.76
0.74730.7490
86.04
Avg Acc (%)
85.59
Avg Metric
0.74
0.72
0.6980
0.70
82.05
0.6774
82
0.68
0.66
80
16
Val Shot
Val Shot
Figure 5. Comparison between ProDistill and DistillMerge. Left: Accuracy results on 8 vision benchmarks using ViT-B-32. Right: Performance metrics on NLU tasks using RoBERTa. The results demonstrate performance improvement of progressive training in ProDi i 11, compared to end-to-end training in Di Merge, despite the latter being more resource-intensive.
C.3. Comparisons with Standard Training and Standard Distillation
We consider two additional baselines. The first, referred to Direct Train, follows standard supervised training approach. It assumes access to class labels and minimizes the standard cross-entropy loss:
min
Σ
Loe (v(0,z),)).
Dr
(2.))ED
where (b---) is the model output logits.
The second baseline, DirectDistill. is conceptually similar to DistillMerge, but does not utilize task vectors to scale the model weights. Mathematically, expressed as:
19

---

Scalable Model Merging with Progressive Layer-wise Distillation
ProDetii rours!
MestilMrgge
DirectDibbll
Dinec Thain
Figure 6. Ablation Results of DirectTrain and DirectDistill Traditional supervised training in Direct Train performs poorly in the few-shot setup. The standard feature-based distillation method, sirectDistill outperforms task arithmetic but still lags behind ProDistill 11 and Distil 1Megge, which incorporate task vector-based weight scaling.
Input Type
ViT-B-32
RoBERTa
Dual (in Alg 1) Merged Fine-tuned
86.04
0.7641
85.20 (-1.0%)
0.7303 (-4.4%)
84.98 (-1.2%)
0.5895 (-22.8%)
Table Performance different layer input configurations, dual activations approach used Algorithm achieves the highest performance, especially for NLU tasks.
min
11 (0,x) (0,, x)]2
D;
ØED,
The relationship between these algorithms can visualized as follows:
Distillation Loss
Task Scaling
DistillMerge Layer-wise Truining
DirectTrain
DirectDistill
ProDistill
We evaluate DirectTrain and DirectDisttll. on vision tasks using ViT-B-32. Compared to the main experimental setup, smaller learning grid of (1 10-5, 10-6, 10-7). The results, summarized in Figure 6, reveal several key findings:
1. Domain-specific data is crucial for model merging. With only 16-shot validation data, a vanilla distillation algorithm like DirectDistill can outperform Task Arithmetic.
2. The internal embeddings teacher models provide significantly richer supervision signals compared class labels alone, as reflected in the improved performance DirectDistill over DirectTrain
3. Scaling model weights using task vectors introduces an effective prior for model training, as reflected the improved performance of ProDistill over Di DiretDisttil.
C.4. Ablation Studies on Layer Inputs
P ProDistill ill maintains dual paths of internal activations as inputs to each layer: the activations from the merged models and those from the fine-tuned models. this section, we explore two altermative configurations: one using only fine-tuned activations and the other using only merged activations. The goal in these two configurations is to align the layer output under the same input, and can viewed as direct adaptations of traditional distillation loss to the layer-wise setting.
20

---

Scalable Model Merging with Progressive Layer-wise Distillation
(800,
p
p (0t
(
(
(
21
21
22
p/-
MSE
MSE
pi
MSE
De
22
θ
((1)
21
0 (1)
22
θ 0(1)
fo(0))
40 (0
Figure 7. Three configurations of layer inputs. Left: Dual activations (ours). Middle: Merged Activations. Right: Fine-tuned Activations. 21 is activation of merged model and 2g the activation of fine-tuned model.
We conduct additional experiments on these three setups, and present the results in Table The findings demonstrate a significant performance advantage of using dual activations over the two alternatives, particularly on the NLU tasks.
The performance gap can be attributed to the limited expressive power of single layer, which makes it impossible fully optimize Equation 4.2. Otherwise, if the loss in Equation 4.2 were zero, the three input configurations would become equivalent. Therefore, the proposed dual input strategy facilitates the progressive alignment of layer features by distributing the minimization of the distillation loss across all layers.
D. Additional Experiment Results
D.1. Additional Results on Vision, NLU, and NLG Tasks
This section presents the complete experimental results across various base models and tasks, including merging ViT-B-16 (Table 6), ViT-L-114 (Table 7) vision tasks, merging BERT-Base (Table 8) NLU tasks, merging L.MMs Instruct+Code (Table 10) and Instruct+Math (Table 9) tasks. The results demonstrate the consistent performance improvements achieved by our method across models of varying model sizes and different experiment setups.
Table 6. Performance of merging VIT-B-16 models across eight downstream vision tasks.
Method
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg
Individual
78.56
87.08
96.92
99.78
97.86
99.17
99.76
82.07
92.65
Task Arithmetic
62.07
66.14
74.00
76.48
88.02
73.79
98.52
52.50
73.94
RegMean
70.84
75.18
83.13
94.44
90.80
82.43
98.66
60.74
82.03
Fisher merging
66.78
70.49
72.17
80.19
88.33
68.14
96.60
48.46
73.89
Localize-and-Stich
67.38
69.23
82.38
90.37
88.84
83.58
97.24
74.10
81.64
AdaMerging
64.30
74.37
74.63
94.89
91.19
94.94
97.95
69.63
82.74
ProDistill, 16 shot (Ours)
71.47
79.99
88.06
96.15
96.37
93.52
99.58
65.00
86.27
ProDistill, 32 shot (Ours)
71.77
80.86
89.48
99.07
96.86
96.29
99.63
68.40
87.80
ProDistill, shot (Ours)
72.82
81.94
91.94
99.52
97.11
97.65
99.60
70.74
88.92
D.2. Additional Results on Data Efficiency
In this section, we provide further evaluations on the data efficiency of P ProDistill
The results for NLU tasks can be found Figure 8, which show that ProDistill has a performance decline in data-scares settings. However, still outperforms Task Arithmetic with as few as 4 data points per class. Overall, the results demonstrate that ProDist istill is highly data-efficient compared with the baselines.
21

---

Scalable Model Merging with Progressive Layer-wise Distillation
Table 7. Performance of merging VIT-L-14 models across eight downstream vision fasks.
Method
SUNN97
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg
Individual
82.32
92.35
97.38
99.78
98.11
99.24
99.69
84.15
94.13
Task Arithmetic
74.16
82.09
86.67
94.07
87.91
86.77
98.94
65.69
84.54
RegMean
74.04
87.22
88.52
98.15
92.89
90.22
99.27
69.84
87.52
Fisher merging
71.28
85.18
81.59
89.67
81.51
83.39
96.31
65.48
81.80
Localize-and-Stich
74.37
78.03
86.02
94.56
93.44
92.52
98.45
74.89
86.53
AdaMerging
75.96
89.42
90.08
96.59
91.78
97.52
98.91
77.61
89.73
ProDistill, 16 shot (Ours)
76.71
89.23
92.63
98.15
97.12
95.28
99.60
75.00
90.47
ProDistil1, 32 shot (Ours)
77.26
89.55
93.40
99.26
97.58
97.17
99.60
76.54
91.30
ProDistill, 64 shot (Ours)
77.73
90.04
94.43
99.48
97.71
98.26
99.63
78.24
91.94
Table 8. Performance of merging BERT models on the NLU tasks.
Method
CoLA
SST-2
MRPC
STS-B
QQP
MNLI
QNLI
RTE
Avg
Individual
0.5600
0.9243
0.8171
0.8754
0.8900
0.8402
0.9103
0.6282
0.8057
Task Arithmetic
0.0900
0.8383
0.7960
0.4897
0.7017
0.4919
0.6883
0.6354
0.5914
RegMean
0.3840
0.8842
0.7857
0.3155
0.7772
0.4799
0.7847
0.6173
0.6286
Fisher merging
0.1288
0.6995
0.6568
-0.3899
0.6698
0.3830
0.7150
0.5632
0.4283
Localize-and-Stich
0.0968
0.7878
0.7982
0.5645
0.6115
0.4475
0.5418
0.5776
0.5532
AdaMerging
0.2935
0.8085
0.7877
0.6607
0.4020
0.4311
0.5065
0.5235
0.5517
ProDistill 16 shot (Ours)
0.3055
0.8704
0.7853
0.5084
0.7788
0.5627
0.8212
0.6101
0.6553
ProDistill, 32 shot (Ours)
0.3369
0.8727
0.7858
0.5105
0.7782
0.6022
0.8320
0.6029
0.6652
ProDistill, 64 shot (Ours)
0.3881
0.8819
0.7951
0.5203
0.7811
0.6155
0.8413
0.6498
0.6841
Table Performance of merging L.M models on Instruct and Math tasks. The result of TIES-Merging and WIDEN directly taken from Yu et al. (2024a).
Method
AlpacaEval 2.0
GSM8K
MATH HumanEval
MBPP
Avg
Norm Avg
WizardL.M M-13B
0.1180
0.0220
0.0000
0.3659
0.3400
0.1692
0.6069
WizardMath-11
0.0117
0.6361
0.1456
0.0671
0.0800
0.1881
0.5036
Task Arithmetic TIES-Merging WIDEN
0.1015
0.6649
0.1420
0.2561
0.2960
0.2921
0.8902
0.1007
0.1577
0.0204
0.3780
0.3560
0.2026
0.7419
0.0945
0.6634
0.1358
0.2866
0.3040
0.2968
0,8907
ProDistill, 16 shot (Ours)
0.1131
0.6485
0.1358
0.2561
0.3040
0.2915
0.9009
ProDist 11, 32 shot (Ours)
0.1148
0.6441
0.1356
0.2805
0.2920
0.2934
0.9084
ProDisti 11, 64 shot (Ours)
0.1121
0.6518
0.1360
0.2683
0.3040
0.2944
0.9072
D.3. Analysis of Randomness
Although Algorithm itself is not inherently random, the randomness introduced by the sampling of validation data can influence the results, necessitating a careful analysis.
To evaluate the effect of randomness, we repeat the experiments on ViT-B-32 and RoBERTa using three different random seeds and present the results in the box plot shown in Figure 9, As expected, the variance in the metrics decreases as the number of validation shots increases. We also find that the VIT experiments show a small sensitivity to randomness. In contrast, for RoBERTa experiments, the impact of randomness can large in data-scarce setting, but this effect diminishes considerably once the number of validation shots exceeds a certain threshold (e.g., 16).
22

---

Scalable Model Merging with Progressive Layer-wise Distillation
Table Performance of merging LLM models on Instruct Code tasks.
result of TLEg-Merging and WIDEN is directly
taken from Yu et al. (2024a).
Method
AlpacaEval 2.0
HumanEval
MBPP
Avg
Norm Avg
WizardLM-13B
0.1180
0.3659
0.3400
0.2746
1.0000
Llama-2- Hamma-2-13b-aae 1-b-coode-alpace
0.0290
0.2378
0.276
0.1809
0.5691
Task Arithmetic
0.1035
0.3110
0.3200
0.2448
0.8894
TIES-Merging WIDEN
0.0727
0.000
0.000
0.0242
0.2053
0.0653
0.3170
0.3560
0.2461
0.8223
ProDistill, 16 shot (Ours)
0.1031
0.2929
0.3140
0.2367
0.8659
ProDistil,, 32 shot (Ours)
0.1056
0.3049
0,3036
0.2380
0,8737
ProDistill, 64 shot (Ours)
0.1042
0.3213
0.3232
0.2496
0.9039
ProDistill (Ours)
RegMean
0.75
Task Arithmetic
0.70
Avg
0.65
0.60
0
20
23
2
2
24
23
29
2
Validation Shot
Figure The average metric ProDistill and RegMean on the NLU tasks, with different data availability. Our method outperforms RegMean in data efficiency when more than 16 validation shots available.
VIT-B-32
RoBERTa
88
0.70
Aug Metric
82-
0.65
80
P
0.60
76
20
23
22
23
2a
22
20
2
Validation Shots
Validation Shots
Figure 9. The randomness Analysis Vision and NLU tasks. Left: Accuracy results of F FroDistill vision benchmarks using ViT-B-32. Right: Performance metrics of ProDistill on the NLU tasks using RoBERTa.
D.4. Additional Results on t-SNE Visualization
We provide the complete results of t-SNE visualization on all eight vision datasets in Figure 10,
D.5. LLM Generation Examples
We pick examples texts generated by merged model of WizardLM-133 and WizardMath-13B using ProDist and Task Arithmetic. The results are provided in Figure D.5 (Math,, Figure D.5 (Code) and Figure D.5 (Instruct).
23

---

Scalable Model Merging with Progressive Layer-wise Distillation
Individual
ProDistill (Ours)
Task Arithmctic
RegMean
Fisher merging
AdaMerging
Localize-and--ee
MNIST
EurosAt
CISIO
RESISC4S
Figure 10. The t-SNE visualization of the last layer embedding VIT-B-32.
24

---

Scalable Model Merging with Progressive Layer-wise Distillation
Prompt:
Below is instruction that describes task. Write response that appropriately completes the request. Instruction: Given ||v 5 and ||w|| 8, find the largest possible value of projw vll.
Response of ProDistill model:
Response of Task Arithmetic model:
Response: Let's think step step.
Response: Let's think step by step.
Step 1: Understand the problem We are
Step Understand the problem We are given the magnitudes of two vectors, and and we need to find the largest possible value of the magnitude of their projection.
given vectors and w with norms |||| 5 and || = 8, respectively. We need to find the largest possible value of the norm of the projection of v onto w.
Step 2: Recall the formula for projection The projection of vector onto vector w is given by:
Step 2: Define the projection The projection of v onto w is a vector that is as close as possible to w while still being a scalar multi- ple v. This vector is given by projw v www [|w]|2 w.
proj
||w||2
Step 3: Calculate the dot product The dot product of v and w is:
Step 3: Calculate the dot product The dot product of and w is v. w 5. = 40.
||||||||| cos
Step 4: Calculate the scalar multiple The scalar multiple of v that is closest to w is given by v-w
where @ is the angle between the two vectors. Step Substitute the given magnitudes We know that ||v|| 5 and || 8. Sub- stituting these values into the dot product equation, we get:
[w] w 5 w 5w.
Step 5: Caleulate the norm of the projection The norm of the projection of v onto w is given by || proj vll = || w|l 2 [||l| 58
Step 6: Find the largest possible value The largest possible value of the norm of the pro- jection of onto w projw v|| 5.
V 5- 8 cos 0 = cos 0
Step 5: Find the maximum value of the dot product The maximum value of the dot prod- uct occurs when the angle 0 between the two vectors is O, i.e., the vectors are aligned. In this case, the dot product is:
So, the largest possible value of the norm of the projection of v onto w is The answer is: 5. V
V = 40
Step 6: Calculate the projection Now, we can find the magnitude of the projection:
40
10
proj v/l
82
8
4
Step 7: State the final answer Therefore, the largest possible value proj v|| is 5 7.
The answer is: 4- X
Figure 1I. Comparison of LLM generation results the MATH benchmark.
25

---

Scalable Model Merging with Progressive Layer-wise Distillation
Prompt:
Below is an instruction that describes task. Write a response that appropriately completes the request. Instruction: Write a Python function to find the quotient of two numbers.
Test examples:
assert find (10, 3) == 3
assert find 2)
assert find
Response of ProDistill model:
Response Task Arithmetic model:
Response:
Response:
Here's a Python function to find the quotient of two numbers:
To find the quotient of two numbers, we can use division operator (/) Python. Here's simple function that takes two num- bers input and returns their quotient:
def find (dividend, divisor) : return dividend // divisor
def find (dividend, divisor) return dividend divisor
Now, let's test the function with the given examples:
Now, let's test the function with the given examples:
assert find (1
assert find (4, 2) assert find (20,5) 4
assert find (10,3)
assert find (4, 21 == 2 assert find (20, 5) -- 4
All the assertions are True, which means the function is working correctly.v
All three tests pass, which means the func- tion is working correctly. The output of the function for each test is:
3
So, the quotient of and 3, the quotient of 4 and 2 is 2, and the quotient of 20 and is 4. X
Figure 12, Comparison of LLM generation results on the MBPP benchmark.
26

---

Scalable Model Merging with Progressive Layer-wise Distillation
Prompt:
A chat between curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers the user's questions. Who created the Superman cartoon character?
USER:
Response of ProDistill model:
Response of Task Arithmetic model:
ASSISTANT:
The Superman cartoon char- acter was created by two American comic book writers, Jerry Siegel and Joe Shuster. They first introduced the character in Action Comics #1 in 1938. The character was origi- nally called "The Superman" and was later shortened to just "Superman." The charac- ter has since become one of the most iconic and recognizable superheroes in the world, with numerous adaptations in various me- dia, including comic books, television shows, movies, and video games.
ASSISTANT:
The Superman cartoon char- acter was created by two American comic book artists, Jerry Siegel and Joe Shuster. They first introduced the character in Action Comics #I in 1938. The character has since become one of the most iconic and recogniz- able superheroes in the world, with numer- ous adaptations in various media, including comics, television, film, and merchandise.
Figure 13, Comparison of LLM generation results on the AlpacaEval 2.0 benchmark.
27


Reviews:
[{"review_id": "kNIE8J4bc9", "reviewer": "Reviewer_ZoNc", "summary": "The paper introduces ProDistill, a model merging algorithm leveraging progressive layer-wise distillation. A key contribution is the use of merging coefficients that are the same size as the model weights, enabling a fine-grained control of the merging process through element-wise operations. ProDistill efficiently computes these coefficients by separately optimizing each layer, reducing memory requirements compared to traditional end-to-end methods. The authors show that this progressive layer-wise approach significantly improves merging performance, achieving state-of-the-art results.\n\n## update after rebuttal\n\nWhile the authors clearly addressed my question, I still find that the method lacks a compelling justification—especially given that it requires more computation and data, yet does not consistently outperform Task Arithmetic (while I acknowledge that other baselines also show similar limitations). Even after considering the rebuttal, this key concern remains unresolved. Therefore, I do not think the paper is strong enough for an accept at the level of a 4, and I will maintain my original score.\n\n(I’d like to note that I am not an expert in this domain, so I defer to the AC and other reviewers with more experience in this area when making the final decision.)", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's comments and valuable suggestions. **We conduct additional experiments to clarify the reviewer's question, with the results given in https://anonymous.4open.science/r/Experiments-for-Reviewer-ZoNc-9701.** We address the reviewer's questions in more detail as follows:\n\n>Q1: While the results of ProDistill are consistently strong on vision and small-scale NLP tasks, the performance gains are less clear and more nuanced in large language model (LLM) tasks...Task Arithmetic, which does not require a few-shot validation dataset or additional training—achieves better results than ProDistill on certain tasks. Why does ProDistill not show a clear performance advantage over Task Arithmetic in these cases?\n\n**A1:** This is a very good question. Task Arithmetic indeed has a good performance in our LLM experiments, which we summarize is due to the following reasons:\n1. Task Arithmetic is more effective for **larger models**. In our vision experiments, its gap to fine-tuned models is over 20% for small ViT-B-32 models but narrows to about 10% for larger ViT-L-14 models. This trend aligns with findings from [1], which suggest that larger models exhibit stronger kernel behavior during fine-tuning and are closer to linear models (see Section D.1 in [1]).\n2. Task Arithmetic works better for merging **a small number of models**. As shown in [2], increasing the number of models leads to greater task interference, reducing the effectiveness of methods that do not explicitly address interference. Our LLM experiments involve merging only two models, whereas our vision and NLP experiments merge up to eight models.\n3. Task Arithmetic can have a **less balanced accuracy across tasks**, which gives a false sense of high performance. For example, when merging WizardMath-13B and Llama-2-13B-Code-Alpaca models (results given in Table 3 in our manuscript), task arithmetic shows slightly better performance than ProDistill on math related benchmarks (0.6467 vs 0.6279 on GSM8k), but significantly worse performance on code related benchmarks (0.0840 vs 0.2239 on MBPP). In other words, although Task Arithmetic may perform better on some tasks, **ProDistill provides more consistent improvements**, as reflected in the normalized average metric.\n\n\n>Q2: Additionally, regarding Figure 4 (left), which illustrates a clear positive relationship between validation set size and performance in vision tasks, can we expect similar trends to hold consistently for LLM tasks? Specifically, how do we explain or interpret the anomaly in Table 9, where 32-shot performance is higher than 64-shot performance\n\n**A2:** Thank you for this question. We conducted additional experiments and found that the inconsistency arises from **randomness in the sampling of validation data**. To confirm this and further investigate data scaling, we repeated the experiments from Table 9 using three different random seeds, and **extend the validation shot to 128**. The averaged performance of ProDistill is given in the table below. The updated results demonstrate a clear positive correlation between validation shot count and merging performance.\n\n\n| | AlpacaEval 2.0 | gsm8k| hendrycks math | human eval |mbpp |Avg | Normalized Avg|\n|-|-|-|-|-|-|-|-|\n| ProDistill 16 shots   | 0.1124     | 0.6370\t    | 0.1302    | 0.2317    | 0.3120\t| 0.2847   | 0.8798  |\n| ProDistill 32 shots   | 0.1134\t    | 0.6387    | 0.1366    | 0.2317    | 0.3080\t| 0.2857   | 0.8885   |\n| ProDistill 64 shots    | 0.1136     | 0.6414    | 0.1338    | 0.2500      | 0.3030\t| 0.2884   | 0.8929   |\n| ProDistill 128 shots    | 0.1058     | 0.6624    | 0.1264   | 0.2866      | 0.3100\t| 0.2982   | 0.9002   |\n\nTo mitigate the effect of randomness, we **rerun all the LLM experiments for ProDistill** using three random seeds. The updated results can be found at https://anonymous.4open.science/r/Experiments-for-Reviewer-ZoNc-9701. \n\n>Q3: It would be helpful if the authors explained why a different set of baselines was chosen specifically for the LLM tasks compared to the other experiments.\n\n**A3:** We change the baselines for LLM tasks due to **scalability constraints**. Many of the baselines in vision and NLP tasks are computation or storage heavy, making them unsuitable for merging LLMs. For example, Adamerging requries storing each of the fine-tuned checkpoints in memory, and Localize-and-Stich requires solving an end-to-end optimization problems. Our method ProDistill is designed to be scalable and efficient, making it well-suited for merging LLMs without such constraints.\n\nWe once again thank the reviewer for their valuable feedback and insightful questions. Please let us know if any further clarifications are needed.\n\n**Reference**\n\n[1] Ortiz-Jimenez G, Favero A, Frossard P. Task arithmetic in the tangent space: Improved editing of pre-trained models\n\n[2] Yadav P, Tam D, Choshen L, et al. Ties-merging: Resolving interference when merging models", "claims_and_evidence": "While the results of ProDistill are consistently strong on vision and small-scale NLP tasks, the performance gains are less clear and more nuanced in large language model (LLM) tasks, which may indeed be the most significant scenario. For instance, in Tables 3, 9, and 10, the simplest baseline—Task Arithmetic, which does not require a few-shot validation dataset or additional training—achieves better results than ProDistill on certain tasks. Why does ProDistill not show a clear performance advantage over Task Arithmetic in these cases? Is it due to insufficient few-shot data (≤64 samples)? If so, how many validation samples would be necessary to clearly outperform this baseline?\n\nAdditionally, regarding Figure 4 (left), which illustrates a clear positive relationship between validation set size and performance in vision tasks, can we expect similar trends to hold consistently for LLM tasks? Specifically, how do we explain or interpret the anomaly in Table 9, where 32-shot performance is higher than 64-shot performance?", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria generally make sense and are well-justified for the problem at hand, except for the evaluation of Large Language Model (LLM) tasks, as previously discussed in the \"Claims and Evidence\" section.", "theoretical_claims": "There are no notable issues or concerns to highlight.", "experimental_designs_or_analyses": "The experiments utilize well-defined and widely recognized benchmark tasks and datasets.", "relation_to_broader_scientific_literature": "This work can directly inform future research on model merging, particularly in developing scalable methods under conditions with moderate unlabeled data availability.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "It would be helpful if the authors explained why a different set of baselines was chosen specifically for the LLM tasks compared to the other experiments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e6juPwlOu2", "reviewer": "Reviewer_CuV6", "summary": "Model merging is an emerging paradigm that combines multiple models into **a single, versatile model, eliminating the need for extensive retraining and substantial weight storage**. However, it is commonly observed that the performance of merged models degrades as the number of models increases. To mitigate this, existing research has proposed using **a small amount of validation data** from downstream tasks to improve performance.\n\nThis paper starts with a theoretical analysis of data-free merging algorithms, showing that their performance is arbitrary and often suboptimal. And this analysis highlights that a small amount of few-shot data is **essential** to achieve meaningful, non-trivial performance. Inspired by these insights, the authors reframe model merging as **a distillation process**. They propose aligning the activations of each layer individually to boost the merged model's performance. This approach offers multiple efficiency benefits, including **reduced data, computation, and memory requirements**. Empirically, the authors validate their framework across three distinct experimental setups, successfully demonstrating its effectiveness from various perspectives.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s valuable feedback. **We follow the reviewer's advice and conduct additional experiments, with the results provided at https://anonymous.4open.science/r/Experiments-for-Reviewer-CuV6-8476.** We address the reviewer's speficic questions as follows.\n\n>Q1: What happens if the fine-tuned models were trained under different configurations, such as varying architectures? ... How can this approach be extended to accommodate models with different embedding sizes?\n\n**A1:** Thank you for this insightful question. Our method can accommodate training differences such as varying hyperparameters; however, it is not directly applicable to merging models with different architectures, as activation matching strategies may fail. The broader challenge of merging models with differing architectures, which we highlight in the Related Works section, is a valuable research direction but is beyond the scope of this paper. Notably, most existing methods based on task vectors struggle with such settings, as weight averaging fails when architectures differ.\n\n>Q2: Can the authors justify how your method handles alignment before merging models in such scenarios?\n\n**A2:** Our method actually aligns the representation of each layer, which effectively addresses misalignment in the considered fine-tuning setup. In cases of severe misalignment, e.g. merging independently trained models, our method can be applied **in parallel to existing alignment algorithms**. Specifically, one can first apply an off-the-shelf alignment algorithm like Git-rebasin [1], and then apply our ProDistill to complete the merging process. \n\n>Q3: How does the proposed ProDistill method compare to multi-task learning (MTL) when both are trained on a validation dataset? What are the performance gap between these approaches?\n\n**A3:** We have already provided such an ablation study in **Appendix C.3**, where we study an algorithm termed as **DirectTrain**, which conducts supervised multi-task training on the few-shot validation datasets. Our method significantly outperforms DirectTrain algorithm, as indicated in Figure 6. \n\n>Q4: It is worth noting that existing literature, such as the work on model alignment prior to merging, has explored layer-wise alignment techniques.\n\n**A4:** Good point! The Git-rebasin paper [1] also explores a layer-wise activation matching strategy. The major difference is that \n1. they use activation matching to learn a **permutation matrix** that aligns the neurons\n2. we use activation matching to learn the **merging coefficients**,  which directly determine the final merged model.\n\nwe have cited [1] and will make further discussions in the revision. \n\n>Q5: How do the coefficients for each layer and downstream task evolve across training steps? \n\n**A5:** Thank you for this insightful question. We conduct additional experiments to track the change of merging coefficients . The results are provided in **Figure 1~3 in the link.**\n\nWe make several observations: \n1. The mean of merging coefficients remains stable around its initialization 0.3. \n2. The standard deviation of merging coefficicents keep increasing during training. \n3. The cosine similarities between coefficients keep decreasing during training.\n\nThese results indicate that ProDistill captures the **fine-grained (Obs 2) and task-specific (Obs3) variation within each module**, rather than merely adjust the overall scaling (Obs 1). Therefore, unlike previous approaches that use scalar coefficients, our element-wise coefficients carry richer information and are unlikely to be easily predictable from data or model alone.\n\n>Q6: How do these coefficients relate to the correlation or similarity between downstream tasks? \n\n**A6:** We provide additional experiments measuring the cosine similarity between merging coefficients across different tasks. The results, provided in **Figure 4 in the link**, reveal that the coefficients of different datasets have uniformly low cosine similarity of about 0.15. However, some correlation exists; for example, the coefficients for MNIST and SVHN (both digit classification tasks) have a slightly higher cosine similarity of 0.2.\n\n>Q7: Is it necessary to compute coefficients for all the layers, can we reuse several close layers' coefficients without computing them?\n\n**A7:** Thanks for the suggestions. We follow the advice and use vanilla task arithmetic for layers where task vectors have smallest $\\ell_2$ norm. The results are given in **Table 1 in the link**. It shows that skipping closer layers degrades performance. However, even when skipping 6 out of the total 12 layers, the average accuracy is still above 80%.\n\nOnce again, we sincerely thank the reviewer for their constructive feedback, and we are eager to engage in further discussions to clarify any concerns.\n\n**References**\n\n[1] Ainsworth S K, Hayase J, Srinivasa S. Git re-basin: Merging models modulo permutation symmetries[J].", "claims_and_evidence": "1. The claims in this paper are clear and well-written, and the experimental results appear **convincing**. The proposed method offers **numerous efficiency benefits**—such as reduced data, computation, and memory usage—further demonstrating its practical applicability.\n\n2. While the merging framework presented in this paper is promising, it consistently assumes that the models being merged share the same base architecture. What happens if the fine-tuned models were trained **under different configurations**, such as varying architectures, hyperparameters (e.g., batch size, learning rate), or training durations (e.g., total epochs)? **Misalignment prior to merging** could negatively affect performance, and this issue cannot be overlooked. Can the authors justify how your method handles alignment before merging models in such scenarios?\n\n3. Furthermore, how does the proposed ProDistill method compare to **multi-task learning (MTL)** when both are trained on a validation dataset? What are the performance gap between these approaches?", "methods_and_evaluation_criteria": "The benchmark datasets and experimental setup used in the paper are commonly used in the model merging field. The evaluation makes sense to me and the analysis of efficiency from multiple perspectives looks solid to me.", "theoretical_claims": "I have checked their proof in Appendix A. Both of them look correct to me.", "experimental_designs_or_analyses": "Yes. Their experimental design is valid to demonstrate their method's effectiveness.", "relation_to_broader_scientific_literature": "The primary contribution of this work compared to other model merging techniques lies in **reframing the merging process as a distillation task**, leveraging a few-shot validation dataset, and progressively optimizing coefficients across model layers **in an efficient manner**.", "essential_references_not_discussed": "There is no major reference to be discussed. However, it is worth noting that existing literature, such as the work on model alignment prior to merging (e.g., https://arxiv.org/abs/2209.04836), has explored layer-wise alignment techniques.", "other_strengths_and_weaknesses": "This paper is easy to read, and their experimental results and framework efficiency look very promising.", "other_comments_or_suggestions": "In Equation 2, the method minimizes the feature distance between layer embeddings, which assumes that the embedding sizes of the models being merged are **identical**. How can this approach be extended to accommodate models with different embedding sizes, and what strategies could mitigate this limitation?", "questions_for_authors": "How do the coefficients for each layer and downstream task evolve across training steps? Are there insights from the results that could enable us to predict these coefficients **without performing the full training process**? Additionally, how do these coefficients relate to the correlation or similarity between downstream tasks? And more, is it necessary to compute coefficients for all the layers, can we reuse several close layers' coefficients without computing them?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VU9cKGPVer", "reviewer": "Reviewer_sgse", "summary": "The paper presents a new method for model merging based on progressive feature alignment. It proposes to learn merging coefficients by progressively aligning the representation of the merged model and the constituent models (the finetuned ones) layer by layer. This reduces the computational requirements of the end-to-end implementation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude for the reviewer's helpful and positive comments. The suggestions provided have been instrumental in refining our work, and we will incorporate the necessary revisions accordingly. Below, we address each of the reviewer’s questions in detail.\n\n>Q1: Honesty, considering the success of data-agnostic methods and their empirical performance, I don’t think that these results are particularly motivating. ... Honestly, I think that these ablations and the results are much more insightful than the theoretical motivation. I suggest expanding them and finding space in the main paper.\n\n**A1:** Thank you for your thoughtful suggestion. We acknowledge that our theoretical results focus on worst-case analysis and may not fully capture real-world practice. However, we find these results delicate and interesting, as they highlight aspects that are not widely recognized in the community, which is why we included them in the main text.\n\nWe agree the ablations in the appendix convey much information that we would like to share with the readers. We will restructure the paper to incorporate these results into the main text in future revisions.\n\n>Q2: Missing comparisons in table 1,2 for data-agnostic methods is TIES merging [1], while for the data-driven approaches, it is important to include also MaTS [2]. \n\n**A2:** Thanks for the suggestions. We **add additional experiments to compare with TIES and MaTS**. For TIES, we use the same hyperparameter grid as in our LLM experiments. For MaTS, we use the RegMean Objective and RegMean Initialization, and choose the same hyperparameter grid as that of RegMean in our paper. \n\nThe results are given below. Our method ProDistill outperforms the new baselines. An interesting finding is that TIES merging has a bad performance on NLP tasks, which coincides with previous findings in [1]. \n\n|  | SUN397 |Cars |RESISC45| EuroSAT |SVHN |GTSRB |MNIST |DTD | Avg | \n|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|\n| TIES Merging    | 65.05     | 64.43     | 74.87     | 77.33     | 81.28     | 69.37     | 96.53    |54.52     |72.92    |\n| MaTS    | 67.65 | 67.23 | 81.76 | 93.30 | 86.76 | 80.08 | 97.39 | 60.21 | 79.30    |\n| ProDistill(Ours)    | 68.90 |71.21 |89.89 |99.37 |96.13 |95.29 |99.46 |68.03 |86.04|\n\n\n|  | CoLA| SST-2| MRPC |STS-B |QQP| MNLI| QNLI| RTE| Avg|\n|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|\n| TIES Merging    | 0.2050     | 0.8853    | 0.5427    | -0.2247     | 0.8282     | 0.7786     | 0.8142    | 0.6462     |0.5594    |\n| MaTS    | 0.3022 | 0.9243 | 0.8205| 0.5187 | 0.8178 | 0.7095 | 0.8506|  0.6498| 0.6992|\n| ProDistill(Ours)    | 0.4442| 0.9312 |0.8464 |0.6942| 0.8134| 0.7857| 0.8900| 0.7076|0.7641 |\n\n\n>Q3: I would have expected that DistillMerge would be an upper bound on the performance of ProDistill. Could you elaborate more on why it is not so? See Fig.5. \n\n**A3:** Thanks for pointing this out. We also find this result very intriguing. We hypothesize that ProDistill decomposes the overall training objective of DistillMerge, into more fine-grained layer-wise objectives. This decomposition makes it easier to minimize the objectives sequentially rather than optimizing the overall objective in one step.\n\n\n>Q4: Also in the right figure (Roberta exp) caption is DirectDistill. If this is not a typo, the caption should be updated.\n\n**A4:** Thanks for pointing this out. We will correct the typo in the revision. \n\n>Q5: Could you please give more details on DirectDstill? Am I understanding correctly that this baseline is just distillation where you fine-tune the model parameters of the merged model, instead of the lambda coefficients?\n\n**A5:** Yes exactly! The DirectDistill baseline uses $\\ell_2$ distillation loss to fine-tune the original model parameters, instead of lambda coefficients. \n\n\n>Q6: Could you give more details on the \"dual inputs\", I don't think I understood if this is something that is particular to your method or it just identifies the input of the merged and the fine-tuned models. The way is introduced is confusing as it seems a technical contribution, but to me this is just an implementation detail of your method.\n\n**A6:** Thanks for the question. The proposed \"dual inputs\" is an important implementation detail in our method, which has a big impact in the final accuracy (see Appendix C.4). Additionally, this design choice is often overlooked in activation-matching algorithms, and we highlight it to ensure awareness within the community.\n\nOnce again, we appreciate the reviewer's feedback and hope that our responses clarify the questions. We remain committed to improving the quality of our paper and welcome any further feedback.\n\n**References**\n\n[1] He Y, Hu Y, Lin Y, et al. Localize-and-stitch: Efficient model merging via sparse task arithmetic", "claims_and_evidence": "All empirical claims are supported by convincing experiments.  See below for the theoretical part.", "methods_and_evaluation_criteria": "Method and evaluation setting are sound. The benchmark used are standard in the literature.", "theoretical_claims": "The paper builds its motivation based on Theorem 3.1 and 3.2. These are the two worst-case analyses that the authors use in support of the motivation of their method, which instead is data-driven. Honesty, considering the success of data-agnostic methods and their empirical performance, I don’t think that these results are particularly motivating. This contribution seems irrelevant to the paper and the method itself.", "experimental_designs_or_analyses": "The experimental analysis is well conducted. Missing comparisons in table 1,2 for data-agnostic methods is TIES merging [1], while for the data-driven approaches, it is important to include also MaTS [2]. In general, performances are impressive. \n\n[1] Yadav, Prateek, Derek Tam, Leshem Choshen, Colin A. Raffel, and Mohit Bansal. \"Ties-merging: Resolving interference when merging models.\" NeurIPS 2023\n[2] Derek Tam, Mohit Bansal, Colin Raffel “Merging by Matching Models in Task Parameter Subspaces” TMLR 2024", "relation_to_broader_scientific_literature": "I think the papers connect to the right literature on model merging, both data-driven and data-agnostic, building on top of insights from other papers that show that representation alignment is crucial for model merging.", "essential_references_not_discussed": "see above.", "other_strengths_and_weaknesses": "The main strength is that this is an actual scalable method. \nMain weakness is that the theoretical motivation for this method seems irrelevant, considering the empirical performance of data-agnostic model merging methods, but I want to hear a better explanation from the authors.", "other_comments_or_suggestions": "see above", "questions_for_authors": "Could you give more details on the \"dual inputs\", I don't think I understood if this is something that is particular to your method or it just identifies the input of the merged and the fine-tuned models. The way is introduced is confusing as it seems a technical contribution, but to me this is just an implementation detail of your method.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3eAAzDbqin", "reviewer": "Reviewer_iWn3", "summary": "The paper introduces ​ProDistill, a ​progressive layer-wise distillation​ algorithm for merging multiple fine-tuned models into a single high-performing model. It theoretically demonstrates the necessity of ​task-specific data​ for effective merging and proposes a layer-by-layer distillation approach that minimizes activation distance between teacher (fine-tuned) and student (merged) models, reducing memory overhead.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude to the reviewer for their valuable feedback. The provided suggestions are extremely helpful and constructive, and we will revise the paper accordingly. We address the reviewer's questions as follows.\n\n>Q1: The experiments focus on vision and NLP tasks. Have you tested ProDistill on other domains, such as speech or reinforcement learning? If not, do you believe the method would generalize well to these areas?\n\n**A1:**  Thank you for the great question. Most existing works on model merging primarily evaluate their methods on vision and NLP tasks, with limited exploration in other domains. Actually, we **can not even find a commonly used model merging benchmark for speech or reinforcement learning in our pretrain-finetune paradigm**.  Constructing such a benchmark would require significant effort in dataset selection, model training, and evaluation setup, which is unfortunately beyond the scope of the limited time available for the rebuttal.\n\nHowever,  we believe ProDistill can generalize well across different domains. The key reason is that **ProDistill is inherently domain-agnostic**—it does not leverage specific properties of vision or language data for its effectiveness. The only requirement is a layer-wise network structure, which is a common standard in modern deep learning.\n\nBesides, we would like to highlight that the LLMs used in our experiments, such as WizardMath-13B, are actually **trained using RLHF-based algorithms**. This can partly give evidence that ProDistill applies to RL trained models. \n\n\n>Q2: Will you release the code and implementation details for ProDistill?\n\n**A2:** Yes, the code and implementation details are already provided in the zip file of the Supplementary Material part on the openreview console. \n\n>Q3: While layer-wise distillation reduces memory overhead, are there any trade-offs in terms of performance or training stability compared to end-to-end distillation?\n\n**A3:** Thank you for the question. We compare ProDistill with its end-to-end distillation counterpart, **DistillMerge, in Appendix C.2**. In our experiments, we do not observe degradation in performance or training stability when using layer-wise distillation.\n\nFinally, we thank the reviewer once again for the effort in providing us with valuable and helpful suggestions. We will continue to provide clarifications if the reviewer has any further questions.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "The key contributions of the paper are closely tied to and build upon ​broader scientific literature​ in model merging, knowledge distillation, and multi-task learning", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "It introduces a new method called ​ProDistill, which uses ​layer-wise distillation​ to merge models, addressing the performance drop issue in existing methods. The theoretical analysis on the necessity of task-specific data is convincing.\nIt works well with ​limited data​ (e.g., 1-shot) and scales to ​10+ billion parameters, making it suitable for real-world applications.\n\nExperiments are limited to vision and NLP tasks, with no validation in other domains like speech or reinforcement learning.", "other_comments_or_suggestions": "No", "questions_for_authors": "The experiments focus on vision and NLP tasks. Have you tested ProDistill on other domains, such as speech or reinforcement learning? If not, do you believe the method would generalize well to these areas?\n\nWill you release the code and implementation details for ProDistill? If not, what are the reasons?\n\nWhile layer-wise distillation reduces memory overhead, are there any trade-offs in terms of performance or training stability compared to end-to-end distillation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethics_expertise_needed": ["Other expertise"]}}]

Author rebuttal:

Meta-review: This paper explores a new technical perspective for model merging: leveraging a small amount of data from downstream tasks with layer-wise knowledge distillation to improve model merging performance. The paper initially got positive scores (3,3,4,3) by three knowledgeable reviewers and another reviewer, who mostly recognized the motivation, the idea and the basic performance of the proposed method. Meanwhile, the reviewers also raised some concerns about 1) less significant gains on LLMs compared to vision transformers; 2) unconvincing connections of some theoretical analysis (theorem 3.1 and 3.2) to the claimed motivation; 3) more experiments to study the generalization of the proposed method; 4) improving ablations in clarification and discussion.

The authors provided detailed responses to these concerns, which were most recognized by four reviewers. Finally, all reviewers maintained their positive scores. The AC read the paper, the reviews, the rebuttal and the reviewers' feedback, and agree with reviewers' assessment. Therefore, I recommend to accept this paper. The authors are encouraged to carefully consider the reviewers' comments/suggestions and their rebuttal in the final paper revision.
Decision: Accept (poster)