Foldable SuperNets: Scalable Merging of Transformers with Different Initializations and Tasks
Authors: Edan kinderman
Itay Hubara
Haggai Maron
Daniel Soudry
Venue: Accepted by TMLR ()

Abstract: Recent methods aim to merge neural networks (NNs) with identical architectures trained on different tasks into a single multi-task model. While most works focus on the simpler setup of merging NNs initialized from a common pre-trained network, we target the harder problem of merging large transformers trained on different tasks from distinct initializations. We show that traditional merging methods fail catastrophically in this setup, while Knowledge Distillation (KD) achieves much better results, though at a higher cost. However, KD is data-inefficient, as it does not exploit the original models' weights.  To solve this, we introduce "Foldable SuperNet Merge" (FS-Merge), which trains a SuperNet containing the original models (with frozen weights) using a feature reconstruction objective. After training, the SuperNet is folded back to the size of a single original model. FS-Merge is simple, data-efficient, has a computational cost comparable to KD, and is proven to have superior expressiveness over traditional merging methods. It achieves SOTA results when tested on MLPs and transformers across various sizes, tasks, modalities, and distribution shifts, especially in low-data scenarios.

Published in Transactions on Machine Learning Research (08/2025)
Foldable SuperNets: Scalable Merging of Transformers with Different Initializations and Tasks
Edan Kinderman
kinderman77@gaall.com
Electrical and Computer Engineering Deportment, Technion
Itay Hubara
ttazhuabora@@mmail.co
Habana Labs
An Intel company
Haggai Maron
heggasmaaron@gmaillcom
Electrical and Computer Engineering Deportment, Technion
NVIDIA Research
Daniel Soudry
doniec..ondry@@maai.c
Electrical and Computer Engineering Deportment, Technion
Reviewed on OpenReview: https: openreview. net/ net/ forum? id- 6FquLes tHu
Abstract
Recent methods aim to merge neural networks (NNs) with identical architectures trained on different tasks into a single multi-task model. While most works focus on the simpler setup of merging NNs initialized from a common pre-trained network, we target the harder problem of merging large transformers trained different tasks from distinct initializations. We show that traditional merging methods fail catastrophically in this setup, while Knowledge Distillation (KD) achieves much better results, though at higher cost. However, KD is data-inetts intfficiiii, as it does not exploit the original models' weights. To solve this, we introduce "Foldable SuperNet Merge" (FS-Merge), which trains a SuperNet containing the original models (with frozen weights) using a feature reconstruction objective. After training, the SuperNet is folded back to the size of single original model. FS-Merge is simple, data-efficient, has computational cost comparable to KD,, and is proven to have superior expressiveness compared to traditional merging methods MLP models. It achieves SOTA results when tested MLPs and transformers across various sizes, tasks, modalities, and distribution shifts, especially in low-data scenario
1 Introduction
Practitioners frequently train identical neural architectures for various tasks and share these models online, while the original training data is often unavailable due to privacy, proprictary, or other concerns. This led to an increased interest the field model merging (Akhlaghi Sukhov 2018 Goddard al. 2024). which aims to combine the weights, and sometimes the features, several models into a single new model (Figure Dp. This approach could allow the merging of multiple single-task models into single multi-tade model (Matena & Raffel 2022 Ilharco et al. 2023). eliminating the need to store and run multiple models via. Mixture-of-Fxperts Shazeer et al. (2017) ensembles (Ganaic et al. 2022).
Most existing merging methods have a strong restriction: they assume that the models were initialized from the same pre-trained model and subsequently fine-tuned. This encourages the models to stay aligned Ainsworth et al. 2023) and also to remain closer the weight space I]harco et al. (2023), and therefore easier fuse, for example by simply averaging their weights (Wortsman et al. 2022). However, this restricts the
Code available httpe://github. h..as//cmmddaniiideeaaaaaataa

---

Published in Transactions on Machine Learning Research (08/2025)
Table 1: Merging a pair of ViT-B-16, fine-tuned on Cars and CIFAR1O, using 100 original images and 800 augmented images from each datasct. The test accuracy is averaged on both tasks.
Method
Accuracy
Method type
Ensemble
89.27
Random guess
5.25
Average
5.56
Averaging-base
SLERP
4.80
Averaging-basoo
RegMean
6.58
Averaging-bes
Opt
6.32
Alignment-baa based
Distillation
75.81
Training-basee
FS-Merge (Ours)
85.83
Training-beae based
ability to merge models that do not share the same initialization. For example, consider the task of merging the weights of two unrelated models from an online repository (e.g,, Hugging Face or GitHub). These models were likely not fine-tuned from the same initial model, making most existing merging techniques inapplicable.
To overcome this, several studics have explored merging differently initialized models using alignment-basod methods [Eneeaari et al. 2022 Verma & Elbayad 2024 Ainsworth et. al., 2023 Stoica et al. 2024). This family methods. However, these approaches still use relatively simple morging rules and thus struggle with more complicated tasks such as merging transformers (Stoica et al. 2024). For a detailed discussion of related work, refer Appendix A.1
of methods is more resource-intensaee
compared to averaging-basee
To illustrate the limitations of these methods, we pre-trained two Vision Transformers (ViTs) (Dooovisskiy et al. 2021) with different initializations on ImageNet-1k (Deng et al. 2009), fine-tuned each on separate
tasks CCars Kranse et al. 2013) and CIFAR10 (Krizhevsky et al. 2009 ) and then merged them (Table
Trnaitional approaches like weight. averaging, SLERP (Shoemaks ke 1985), and RegMean (Jin et al. 2023
as well as the aligument merging method designed for transformers, 'Opt' (Imfeld et al.) 2023), all failed in this setting, resulting in a merged model with performance comparable to random guessing. Moreover, these methods show a significant accuracy gap compared to the model ensembling (Ganaie al.) 2022). ) method that averages the model outputs. Note that the ensemble is not valid merging method, as it uses the original models directly. These traditional merging methods rely on simple local merging rules, ensuring computational efficiency. However, they proved inadequate for our complex setting due to their simplicity. This pattern persisted across all other settings, tasks, and modalities we tested with transformers (see Section 3.2
These results indicate that merging large transformers from diverse initializations demands stronger and more resource-intensive techniques, such as Knowledge Distillation (KD) (Ba & Caruanal 2014 Hinton et al. 2015). In the multi-task setting, KD refers to single model trained to replicate the outputs of multiple models (Tan
et al. 2019 Vongkulbhisal et al. 2019). And indeed, KD significantly outperforms the previous methods (Table but still exhibits a large performance gap compared to the ensemble. Although KD shows promise, it often requires access to large parts of the original training dataset and labels (Clark et al. 2019 Liu et al. 2020) to achieve high accuracy, which may be problematic due to privacy or proprietary concerns. Moreover, it does not explicitly utilize the original models' weights, potentially overlooking valuable information.
Our approach. To leverage the hidden knowledge in the original weights and further improve accuracy beyond KD, propose the "Foldable SuperNet Merge (FS-Merge)) method. This approach utilizing "SuperNet""> a network in which the original models are embedded with frozen parameters. The SuperNet weights are then trained to minimize either local or global feature reconstruction loss. After training, the SeperNet weights are folded to produce a model with the same size as the original models (Figure Notably, this means that the additional computational cost occurs only once, during merging.
FS- FS-Merge simple, data-cfficient, and we prove that it has superior expressivity compared to the basclines (A [Appendix]] H). Tike other methods (Jin et al. 2023 Ainsworth et al. 2023). it requires only unlabeled
2

---

Published in Transactions on Machine Learning Research (08/2025)
(a) The Original models
(b) The Merged Model
Classification
Classification heads
Merged Model
Model A
A
A
Merged
Model B
B
B
Model
Figure 1: The model merging setting. (a) Consider models and B trained from different initializations and on different tasks, which create features processed by a classification head for predictions, (b) Merging methods fuse the models into new model of the same size while leaving the classification head untouched. The merged model generates leatures applicable to all tasks.
fraction of the original training data. We demonstrate its effectiveness by achieving SOTA results across diverse scenarios, architectures, model sizes, tasks, modalities, and distribution shifts.
We prove that FS-Merge offers greater expressiveness compared to traditional merging methods. Moreover, by leveraging the original weights, FS-Merge outperforms KD, especially in data-limited regimes, significantly reducing the gap with the ensemble (Table and even surpassing it some cases. While this work focuses on transformers, FS- MerMesss's versatility allows it to be exlended to other architectures such as RNNs (Sherstinsky 2020) and CNNs (He al. (2166). This adaptability sets FS-Merge apart from alignment- based methods, which are often designed for specific architectures.
2 Method
Problem formulation. For simplicity, outline the merging problem for two models A and B with identical widths, trained distinct tasks with unique initializations and separate classification heads (Figure [a). Using unlabeled subsets of each task's training set, D and DB our goal is to construct a new model with the same architecture, that minimizes the losses for tasks A and B. Be aware that the classification heads are not merged, meaning we retain a separate head for each task (Figure p)) Note that FS-Merge can be easily extended to merge any number of models or models with varying widths. We denote D D DDB
In this section, introduce FS-Merge, our proposed approach, first for merging MLPs and then for the more complex task of merging transformers.
2.1 Warmup: merging Multi-Layer Perceptrons
Let consider the I-th layer in the Multi-Layer Perceptron (MLP) model A. The features this layer, denoted by f Rd, can be expressed as follows:
= - + bi f/ o(211))
(1)
Here, WA E B4xd-11 and b/ E R4, are the weights and biases of the current linear layer, respectively. f6 = x denotes the MLP input, di denotes the width of the I-th layer, a represents a non-lincar clement-wise activation function, and zz E pd are the pre-activation features.
Our method has two versions: local FS-Merge and global FS-Merge. Each has its own Foldable SuperNet and reconstruction optimization problem, neither requiring true labels. The parameters learned during FS-Merge optimization are highlighted in red, and the notation represents feature's reconstruction attempt of the Foldable SuperNet.
3

---

Published in Transactions on Machine Learning Research (08/2025)
(a) Local FS
Merge
L
W
H
WA
ft
M
O
U
M
U
wP
WP
ft
(b) Global FS Merge
L
W1
W
WL
ft
M
U
M
M
U
W
W
W
ft
Folded after optimization
The FS-Merge for MLP. (a) Local FS-Merge: Mi and Uj are optimized separately for each layer I to reconstruct fA (b) Global FS-Merge: composing the M and U matrices for each one of the L layers of the MT.P to reconstruct SAIIS, the features before the classification head. In both versions, after optimization, the Foldable SuperNet is folded create the merged model. Red represents the SuperNet weights, blue represents the original frozen weights, gray represents features, and green represents the activation function.
Figure 2:
Local FS-Merge. In the case of merging the 1-th linear layers of models A and B, the local Foldable SuperNet (Figure 2 is defined as follows:
M (M 11
(2)
The input is concatenation of the original pre-activation features z1 =| E 2-2 Mr E Rdd222d ("Merge") is used to merge the original features into a lower-dimensional space (from 2 di to di), and Uz E g2-di ("Unmerge") is used to approximately reverse the merge operation, attempting to reconstruct the original post-activation features f/ fI e g2-d, with ft 22d,, Another way to express ft is by f f...
For optimization, is used extract the models features JA and ff. Then, Mj and Un are optimized separately for each layer I, the following reconstruetion optimization problem:
Mf, Ui -argmin Egen || 11 , 2- Mi24
Global FS-Merge. In the global version (Figure 2b), the Foldable SuperNet is created by composing the M and U matrices for each one of the 1, layers of the MLP. In the forward pass, the I-th layer of the Foldable SuperNet uses the reconstructed pre-activation features of the previous layer z & as inputs. Observe that this does not include the classification head, which not being morged. Then, all those matrices are optimized logether on the following global optimization problem:
Mr, U. MJ, argmin E | - F2())
(4)
M.U,,...
where A E IRd are features from the last representation layer (L-th layer) of model A, applied the input x. The output of the Foldable SuperNet is defined as fr e ?Rd which attempts to reconstruct f Iff e 22d2, As we will demonstrate later, the global problem allows us to deal with more complicated architectures such as transformers. Note that we have a slight abuse of notation, as fr denotes reconstructed features from both the local and global Fe-Merge.
4

---

Published in Transactions on Machine Learning Research (08/2025)
W04
84
W
R
M
W
4
aa
x
U
M
Uo
MO
Qa
woo
M
g
WK
K
Original frozen weights
SuperNet's weights
WV
4
Features
Softmax
Figure 3: The Foldable SuperNet for merging two attention blocks (4 and B). Only the red components are trained. M matrices are used to merge the models' features into merged features (X*), ) and U are applied to the merged features to reconstruct the original ones (XA BB) After training, the Foldable SuperNet is folded to produce a merged attention block with the same size as the original blocks.
Folding. After optimizing the M and matrices all layers (using the local or the global version), we can "fold" the Foldable SuperNet in order create the merged model. The "folding" operation is defined follows:
W7 Mi (W/
(WA
0
UL bj Mī WBB
(5)
Where Uo = (1)- Intuitively, this folding operation creates a merged model which "under the hood" reconstructs the original features from the previous layer (using Ut-1), applies the original weights, and then merges those features again (Mi). This occurs with the same complexity as each of the original models, since Wi E Bddxdd-..
The Foldable SuperNet can be initialized in various ways (full details in Appendix G.1 In the simplest approach, the M and U matrices may be initialized randomly. We found that the best approach is to initialize M and U using the "first" initialization method (as described in Eq, G where only the weights of the first model are selected. This implies that, the weights are folded at this point, we obtain W7 = W/A For an ablation study in the matter, see Appendix G.T and Appendix G.2
Initialization.
Mi (1 0) U1 (
(6)
Relation with ZipIt (Stoica et al.) 2024). This "folding" operation proposed by ZipIt (Stoica et al. 2024), which is closely related to work, and inspired it. This method also merges models from various initializations and tasks, targeting MT.Ps and CNNs. In the MLP context, ZipIt represents special case of FS-Merge, where M is chosen to average highly correlated pairs in a hard-coded way, and U = 2M T For other layers, such as skip connections and normalizations, ZipIt employs various heuristics that cannot be easily extended to transformers and may harm performance. For more details, see Appendix A.2
2.2 Merging transformers
Merging transformers (Vaswani et al. 2017 is a much more challenging problem than merging MLPs, due to their larger scale and more complicatod structure. Creating a naively Foldable SuperNet as described in the MLP Section 2.1 not feasible for transformers because it fails to account for their skip connections, layer normalization
[Ba et. al. 2016), and multi-head attention.
5

---

Published in Transactions on Machine Learning Research (08/2025)
To tackle these issues, develop new Foldable SuperNet architecture, and train it using the global objective (Eq. 4). We found that the global objective is essential for merging transformers, as training the SuperNet locally (Eq.. 3) significantly reduces the accuracy of the resulting merged model (reasons for this are discussed in Appendix G.3
Foldable SuperNet for attention blocks. In this section, we briefly describe the Foldable SuperNet for the attention block (Figure 3) as it is the most complex component of the transformer. For full technical details the SuperNet for this block, well as for the other transformer blocks (MLP and pre-processing), please refer to Appendix B.1 The optimized parameters at this stage are highlighted in red, with X represents reconstruction attempt of our method, and X represents the merged features.
Suppose there are two Iransformers trained on distinct tasks A and B. The Foldable SuperNet for their attention blocks receives as input X; E RTxd the merged features from the previous SuperNet, where T is the sequence length and the embedding size. It then applics a layer norm (which is also Icarnod, as proposed in other merging works (Jordan et al. 2023 ), and uses learned matrix Ur RRx22-d to reconstruct. the original model features X/" !
Next, the Foldable SuperNet calculates the queries, keys, and values for A and B using the original frozen weights. The queries of both models are then concatenated and merged using a learnable matrix M, R2-dxd, and similarly for the keys and values with MKK, MY- Following this, the Foldable SuperNet performs multi- head attention with the merged queries, keys, and values; uses UP E Idd22d to reconstruct the original multi-head attention output; applies the aggregation frozen weights of the original models; uses M E a to compress it once more; and applies the skip connection. This process results in merged features of dimension T x d, serving as the input for the subsequent Foldable SuperNet of the MLP block at layer 1.
Parameterizing M and U. Utilizing full-rank M, E Rdxxdd and Uj E Ra xd; for merging n models introduces a number of parameters that increase quadratically with the layer width di In large models like transformers, this leads to a very high demand for resources, hinders the optimization process, and cause overfitting. To miligate this, we adopt a parameterization strategy akin to LoRA (Hu el al., 2022), using a sum of a low-rank matrix and a concatenation of diagonal matrices. For instance, in My:
Mi Mpling + M, MF.
When r is the inner rank, M1 E Rdxxr, M² E grxwdd and M E gdiwndd a concatenation of a n diagonal matrices, each with di learnable parameters. A similar structure is proposed for Uy. This ensures the number of learnable parameters is linear with the layer width dj We found that adding M,ling is crucial for FS- FS-Merge, as it enables initializing the Foldable SuperNet with strong initializations such "first" (Eq. 6
FS-Merge seq. To address the high costs of merging a large number of models, we introduce a more efficient variant called FS-Merge Seq., which merges the models sequentially. It starts with the first two models, then continucs by merging the resulting merged model with the third model, and so on. Full details can be found in Appendix B.4.
Data and Augmentation. This work addresses realistic setting, involving limited subset of unlabeled samples used for merging. For transformer merges, augmentations (Zhang et al. 2018) are employed to cxpand this subset, as commonly done in regular training. Appendix G.1 studies the ellect of using augmentation on accuracy.
Exprcssive power. As shown in Section 3.2 FS-Merge significantly surpasses traditional merging methods. We explain this with tcal results demonstrating its superior expressivity over the baselines.
Theorem 2.1 (Informal). FS-Merge can ezactly reproduce any target MLP model when merging models with sufficient rank conditions. Moreover, FS-Merge strictly generalizes previous merging methods like RegMeon and Git Re- Re-Basin, in the context of merring MLPs.
See Appendix H for the formal statement and conditions.
6

---

Published in Transactions on Machine Learning Research (08/2025)
Table 2:
We merged pairs of MLPs, each initialized differently and trained on distinct halves of the MNIST dataset. These MLPs have hidden width of 128 neurons, with the number of hidden layers varying from 1 to 6. Each experiment was replicated five times. We present the average per-task accuracy on the test set, along with the standard deviation. Model
Number of Hidden Layers
1
2
3
Two models
96.83 + 0.13
96.51 + 0.19
96.46 + 0.23
95.86
0.40
96.81 + 0.58
Ensemble
94.70 + 0.95
95.14 + 1.12
95.73 + 0.22
0.12 95.58
95.78 + 0.73
average
94.36 + 0.76
85.90 + 4.46
78.78 + 6.72
61.76 + 6.36
25.12 + 2.70
RegMean
95.90 + 0.37
92.97 = 2.71
92.11 + 1.90
87.79 + 3.77
81.55 = 2.49
ZipIt
96.35 + 0.17
95.75 = 0.58
95.43 - 0.50
94.59 0.50
94.03 = 2.27
Distillation
93.35 + 1.07
93.13 + 1.74
93.71 0.39
93.38 0.67
90.79 1.89
FS-M
95.89 + 0.03
95.68 = 0.19
95.37 = 0.33
94.94 + 0.43
94.89 = 0.89
FS-M, ZipIt init
96.62 = 0.08
96.29 + 0.24
96.18 + 0.20
95.63 = 0.52
96.22 + 0.84
3 Results
We evaluate our method on MLPs (Section 3.1 Vision Transformers (Section 3.2 and Text Transformers (Section 3.3 and show it achieves SOTA results.
Baselines. We compared with "Original Models", representing the average accuracy of the models to be merged; and Ensemble (Ganaie et al. 2022) which averages the models outputs and then applies classification heads. Note that these are not valid merging mothods as they use the original models directly. For legitimate merging techniques, comparisons were made with weight averaging (Wortsman al. 2022); RegMeaa (Jin)
et al. 2023) which applies closed-form linear regression solution to each layer; and distillation (Hinton et al.
2015 which trains single model to mimic the pre-classification layer features. Ziplt (Stoica et al. 2024 averages highly correlated ncurons, used only in the MLP experiments, as it is inapplicable for transformers.
"Opt" " (Imfeld el al. 2023), uses optimal transport for aligning transformers, and "SLERP" (Shoemake) 1985 utilized only on the ViT case, as they introduced for transformers.
Data. When merging the models, a small unlabeled subset of their original training data used. In the case of merging ViTs, this subset is further extended using augmentations. For a fair comparison, all baselines use the same amount of original and augmented data.
Metrics. Following ZipIt (Stoica et al. 2024). we evaluate multi-task merged models using per-task and joint accuracy, Per-task accuracy is calculated by evaluating each task individually with its relevant classification head and reporting the mean accuracy across all tasks. Joint accuracy is calculated by making predictions for each task based on the maximum score across all classification heads and reporting the mean accuracy across tasks.
3.1 Warm-up: Merging Multi-Layer Perceptrons
Setting. MNIST (LeCun 1998) was split into two subsets: images with labels 0-4 and images with labels 5-9. These subsets were further divided into training, validation, and test sets. Two MLPs were trained separately these subsets, varying in the number of layers and widths, each initialized with different seed. The goal was to merge these pairs of models, excluding the last linear layer, which serves the classification head.
FS-Merge. Our method was tested in two variants: the local version of FS-Merge (Eq. 3), which trains a Foldable SuperNet for each layer independently; and FS-Merge Ziplt, which initializes the Foldable SuperNet's M and U as the solutions of ZipIt (Stoica et al. 2024), and then optimize them using the local version. Data. All merging methods, excluding "Average", use features from the original models. Thus, we sampled 64 images from each dataset's training set to generate the necessary features.
7

---

Published in Transactions on Machine Learning Research (08/2025)
Table 3: Merging pairs of ViT-B-16 using 16 original images from each training set and 800 augmented images from each dataset. The per-task and joint accuracy the test set reported
Merging
DTD, EuroSAT
CIFAR100, SVHN RESISC45, SVHN
Methods
Learnable Parameters
Per-task
Joint
Per-task
Joint
Per-task
Joint
Original models
81.55
91.19
95.12
Ensemble
78.64
74.11
88.26
57.44
93.21
75.07
Average
11.48
1.15
3.71
0.84
6.23
3.68
X
SLERP
7.99
1.27
4.10
0.65
6.67
3.24
X
RegMean
8.40
1.69
6.17
1.27
6.44
0.74
X
Opt.
4.51
0.75
4.52
0.88
7.01
2.07
X
Distillation
57.31
52.61
62.93
48.12
66.18
63.16
V
Distillation, LoRA r
57.23
51.83
58.81
49.83
62.17
57.04
V
Distillation, LoRA 12
55.07
48.28
59.06
47.59
64.16
58.66
/
FS-M,
62.64
57.42
66.50
53.98
73.77
69.26
V
Table 2 presents the per-task accuracy the test set, for merging MLPs trained on half of the MNIST dataset. We merged ML.ss with 128 hidden widths, and hidden layers varying from 1 to 6. Each experimental condition was replicated five times with 5 different sceds. The same hyperparameters were used for all those experiments. Full information about the setting and hyperparameters are available in Appendix E.1
Our results indicate that merging deeper models more challenging, consistent with previous studies (Jordan ct al. 2023). Employing the ZipIt initialization, our method establishes new SOTA for both per-task and joint accuracy, oulperforming ensemble in many cases, and nearly matches the accuracy of "Original Models", For results more tasks and FS- FS-Merge versions, Appendix F.1
3.2 Merging Vision Transformers
Next, evaluated our method merging Vision Transformers (VIT) initialized differently and trained distinct tasks, much more challenging problem.
Models and Data. Several VTT-B-16 and ViT-L-14 models (Dosovitskiy et al. 2021 Touvron et al. 2021
were pro-trained on ImageNet-1K (Deng et al. 2009 using distinct random sceds. Then, each differently pre-trained ViT was fine-tuned on rownstream tasks. Following (Ilharco et al. 2023 we fine-tuned on Cars (Krause et al. 2013) DTD (Cimpoi al. 2014), FuroSAT (Helber et al.] 2019 GTSRB (Stallkamp al. 2011), MNIST [LeCun 1998). RESISC45 (Cheng al. 2017), SVHN Netzer et al.) 2011), CIFAR1O, and
CIFAR100 (Krizhevsky et al. 2009 For extended details regarding the setting refer to Appendix D
In these experiments, the KD baseline aims to reconstruct the features from the last layer of the models to be merged, using the "first" initialization, meaning that the student model was initialized from the first model. We tested additional KD variants, such as a baseline that attempts to mimic the internal features of the models (Appendix G.5), and baselines where the student model is initialized using different merging methods (such as RegMean) before continuing training (Appendix G.2.. Neverthcless, we found that our KD baseline, which uses only the last-layer features with the "first" initialization, outperforms all other variants. The KD LoRA is identical to the KD baseline, except that it trains only LoRA Hu et al. (2022) adapters with low rank r for each linear layer in the model.
FS-Merge. The global version of FS-Merge was used (Eq. with the Foldable SuperNet's M and U parametrized as a concatenation of diagonal matrices plus low-rank matrices (Eq. 7 The Foldable SuperNet was initialized using the "first" initialization (Eq. G), which showed the best performance (as demonstrated in Appendix G.1). FS- -S-Merge Seq. (Appendix B.4] memory and compute-eficiint variant of FS- FS-Merge, designed for merging large number of models. In Table 3 pairs of ViT-B-16 models fine-tuned different.

---

Published in Transactions on Machine Learning Research (08/2025)
Table 4: Merging groups of 4 ViT-B-16 using 100 original and 1000 augmented images per training dataset.
(a total of 400 original images and 4,000 augmented images). We will denote: C = Cars, D DTD, E EuroSAT, G = GTSRB, M MNIST, R RESISC45, S SVHN, C10 CIFARIO, C100 CIFAR100.
Merging Methods
R. C10, S, G D. E.
C. M, C100, E
Learnable
Per-task
Joint
Per-task
Joint
Per-task
Joint
Paramcters
Original models
96.47
88.72
92.61
Ensemble
86.11
46.80
76.81
52.96
82.04
63.92
Average
5.40
1.04
3.66
1.35
4.55
0.38
X
SLERP
5.55
1.04
4.88
0.87
7.27
0.86
X
RegMean
6.38
0.61
4.78
0.61
5.60
0,52
X
Opt
5.79
0.24
3.70
0.38
5.75
2.58
X
Distillation
82.09
67.60
67.31
57.67
45.75
40.62
Distillation, LoRA r 4
72.92
54.73
57.91
46.36
75.76
70.34
Distillation, LoRA r = 12
77.43
60.18
63.56
52.81
75.52
69.29
Distillation, LoRAA 24
78.58
61.47
63.61
52.93
68.75
64.31
FS-M, 24
84.58
70.92
68.41
56.40
79.71
73.98
V
FS-M seq., r = 16
83.90
71.07
68.96
58.48
73.57
68.31
Table 5: We merge pairs of ViT-B-16 models using 16 original and 800 augmented images per dataset. Accuracy reported on the corrupted test sets of CIFAR100-C, with mean and standard deviation computed over the five severity levels.
Corruption type
CIFAR100 ViT-B-16
CIFAR100, RESISC45
CIFAR100, EuroSAT
Distillation
FS-Merge
Distillation
FS-Merge
Snow
71.9 + 5.1
63.4 + 5.5
71.2 + 5.2
64.1 + 5.2
69.6 + 5.6
Frost
69.6 + 8.8
61.2 + 8.9
69.0 + 8.9
62.2 + 9.2
64.6 + 10.9
Fog
75.7 + 11.2
66.4 + 13.1
75.2 + 11.2
68.5 + 12.4
74.4 + 11.3
Brightness
82.8 = 2.9
75.4 3.6
82.4 3.0
76.4 3.4
81.8 = 3.0
Gaussian noisc
24.9 16.8
22.8 14.3
24.2 17.0
22.4 14.8
21.5 15.8
Shot noise
33.7 21.9
30.3 + 18.6
33.0 22.0
30.6 19.2
30.0 21.1
Impulse noise
30.0 20.3
21.8 + 15.9
28.6 + 19.8
20.3 17.2
26.3 + 20.5
Contrast
69.3 = 20.7
69.2 20.5
62.9 21.1
66.4 + 22.3
Elastic transform
70.5 = 12.1
60.1 12.0
69.9 =
1
69.3 + 12.2
Pixelate
62.9 22.0
58.1
18.4
61.5 + 22.7
58.4 19.9
61.4 + 21.8
Jpeg compression
60.2 = 6.7
51.3 6.2
59.8 + 6.7
53.1 6.2
59.0 = 6.8
Defocus blur
76.3 10.7
13.3
1.3318 13.3
75.9 + 10.7
75.3 + 10.7
Motion blur
69.2 8.6
5..7
68.9 8.6
60.9
67.8 8.6
Zoom blur
72.9 + 6.1
61.5 1.557.00
72.6 + 6.1
64.5 7.0
71.9 + 6.1

---

Published in Transactions on Machine Learning Research (08/2025)
Test Accurrad (Cars, U.S...NIST USIIA
Ted Accurecy 1DTD, CHIMR10)
85
60
Ensemala, Pestusk
Dittin Pentesk
66
F5 Pereer. Parczsk
Envemale, Jont
60
Datt listion jo
FS Norgs, joint
50
Original Images
Original Images
Original Images
Figure We used Ensemble, Distillation, and FS-Merge to merge pairs models trained on RESISC45 and GTSRB (left), Cars and MNIST (center), DTD and CIFAR10 (right). We varied the number of original images per dataset and added augmentation images so the total number of images per dataset would be 1024. We present the per-task and joint accuracy.
tasks were merged in a low-data scenario of using only 16 original images and 800 augmented images per datasct. To study the impact of merging larger number of models, Table shows results for merging groups of four B6 models. For additional results and hyperparameter details, refer to Appendices P.2 and E.2
As can be seen, FS-Merge outperforms all other merging methods, and even surpasses ensembles in some cases. This holds for both per-task and joint accuracy across all scttings. Additionally, it is evident that all local and simple methods (such as Average, SLERP, RegMean, and "Opt") completely fail to effectively merge VIIs in this challenging setting, resulting a merged model that performs comparably to random guess. Similar results were observed when merging larger ViTs (Appendix F.2 and ViTs trained with different. pretraining strategies (Appendix F.3
Robustness evaluation. In the next section, the robustness of merging methods under distribution shifts was evaluated. The ViT-B-16 model fine-tuned on CIFAR100 was merged with other ViT-B-16 models using KD and FS-Merge. The merged models were then tested on CIFAR100-C (Hendrycks Dietterich, 2019). a dataset that introduces various corruptions at five severity levels to the CIFAR100 test set. For each corruption type, the mean and standard deviation across the severity levels were calculated (Table 5). The performance of the original ViT-B-16 model trained on CIFAR100 is also included as upper bound for the merged models.
As demonstrated, FS-Merge consistently outperforms KD across corruption types and severity levels, achieving performance close to the original ViT-B-16 model trained on CIFAR100. This highlights its effectiveness in producing signficantly more robust merged models.
Number of original training images. We examine the impact of varying [D], the number of images from the training datasets of the models to be merged ("original images"), which are used to create features. [D] was varied from 16 to 1024. Augmented images datasot reached 1024, maintaining consistent datasot size. Pairs of ViT-B-16 models were merged using Ensemble, KD and FS-Merge. The per-task and joint accuracies on the test set are presented in Figure For additional details and experiments, see Appendix F.8
created to ensure the total number of images per
As observed, Distillation underperforms with few original images, while FS-Merge excels. Increasing original images enhances both techniques, reducing the performance gap. With enough data, merging methods can sometimes surpass ensemble performance, which is often considered "gold-standard method" in merging and multitask articles.
3.3 Merging Text Transformers
In this series of experiments, we aimed to evaluate FS-Merge on a different modality: merging text transformers. We fine-tuned differently initialized BERT models on distinct classification task from the GLUE dataset (Wng et al. 2019). Then, applied KD and FS-Merge merge pairs of them, utilizing 200 data points
10

---

Published in Transactions on Machine Learning Research (08/2025)
Table 6: Merging pairs of BERTs with 200 data points from the training set of each dataset. We report the test set per-task accuracy.
Original models
Tasks
Distillation
FS-Merge
RTE, QQP
79.49
65.00
68.88
MNLI, MRPC
85.73
76.87
78.29
MRPC, QNLI
89.02
80.67
79.33
SST-2, RTE
80.15
75.45
76.00
MNLI, SST-2
88.39
82.81
83.24
RTE, QNLI
79.75
66.40
69.69
QNLI, QQP
91.01
82.40
84.75
QQP, RTE
79.49
73.30
75.62
SST-2, QNLI
91.68
82.92
84.39
MRPC, RTE
77.49
68.70
68.37
QNLI, MNLI
87.99
71.83
73.79
QQP, SST-2
91.41
87.49
87.94
MNLI, QQP
87.7
78.74
79.22
RTE, MRPC
77.49
63.46
66.05
from each training set to create features. In Table 6 we report the per-task test accuracy of these experiments. We observe that FS-Merge outperforms KD, similar to the ViT case. For details and additional results, see Appendix F.7
4 Discussion
Merging Complexity. FS-Merge and KD are more computationally intensive than standard merging methods such as Averaging and RegMean, which fail catastrophically in our setting. When merging two models, FS-Merge and KD have comparable resource usage. However, when merging multiple models, the resource usage gap between FS-Merge and KD becomes more significant. To address this, we propose FS-Merge seq,, which is comparable to KD's resource use while outperforming it in terms of accuracy. Therefore, FS-Merge is recommended for optimal test accuracy, given sufficient computational resources and limited data. If resources are more limited, F'SMMeree seq. should be the method of choice. Refer to Appendix for the full details.
Why does FS-Merge work well? We attribute FS-Merge's superior performance three key aspects. First, FS-Merge and KD offer greater expressiveness than traditional methods (Appendix HH). Second, in contrast to the traditional methods, FS-Merge and KD use global objective, which our experiments show is crucial in this challenging setting (see Appendix G.3 Lastly, we believe FS-Merge generalizes better than KD because it ha s an useful inductive bias that constrains the merged model to be a low-rank weighted average of the original models' weights (as determined by the Foldable SuperNett..
Limitations. FS-Merge requires a small unlabeled subset of the original training data, similarly to most previous merging mothods. Additionally, FS-Merge, like KD, more computationally intensive than standard merging methods; however, these methods completely failed in more challenging seltings. Moreover, FS-Merge has fewer learnable parameters than KD, but they increase linearly with the number of models and hidden width; however, FS-Merge seq. solved the first issue. Lastly, one cannot naively merge two models of different. depths using our method; we believe this could be solved in future work.
Summary. In this work, we address the challenging task merging transformers from different initializations and tasks into a unified multitask model using a small subset of unlabeled data-a sotting in which traditional methods fail. Our proposed FS-Merge, which employs feature reconstruction approach train Foldable SuperNet, is simple, data-efficient, and more sophisticated merging rules compared to other baselines.
11

---

Published in Transactions on Machine Learning Research (08/2025)
FS-Merge outperforms traditional methods and achieves SOTA results across various scenarios, model sizes, datasets and modalities.
FS-Merge addresses key challenges in Al development. The ability to merge models from different initializations and tasks offers an efficient alternative to using an ensemble of these models. This approach enables greater resource efficiency and reduces the model's carbon footprint. Moreover, FS-Merge can alleviate privacy concerns. For instance, in cases where multiple users train models on private datasets, FS-Merge allows these models to be combined into single multi-task model without accessing the full private datasets or any labels.
Impact Statement.
12

---

Published in Transactions on Machine Learning Research (08/2025)
Acknowledgements
The rescarch of DS was Funded by the European Union (ERC, A-B-C-Doep; 101039436). Views and opinions expressed are however those of the author only and do not necessarily reflect those of the European Union the European Research Council Executive Agency (ERCEA). Neither the European Union nor the granting authority can be held responsible for them. DS also acknowledges the support of the Schmidt Career Advancement Chair in AI. HM is the Robert J. Shillman Fellow, and is supported by the Israel Science Foundation through a personal grant (ISF 264/23) and an equipment grant (ISF 532/23).
References
Samuel Ainsworth, Jonathan Hayaso, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations, 2023.
URL htttp:://pperreiiiett http:/tteee.orrroiidd/ooo/aamll
Aditya Kumar Akash, Sixu Li, and Nicolás García Trillos. Wasserstein barycenter- based model fusion and lincar mode connectivity of neural networks. arXiv preprint arXim:2210.060% 06066611, 2022.
Milad Akhlaghi and Sergey V Sukhov. Knowledge fusion feodforward artificial noural networks. Neural Processing Letters, 48(1):257 272, 2018.
Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. arXiv preprint (03::22033.1187, 2024.
Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? Aduances in neural information processing systems, 27, 2014.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv:1607.06450 2016.
arxiv preprinl
Vijay Badrinarayanan, Bamdey Mishra, and Roberto Cipolla. Understanding symmetrics deep networks arXiv preprint Xrxiv:1511.0 1.0002,, 2015.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. pood-101-miningg with random forests. In Computer vision EECVV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13, pp. 446-461. Springer, 2014.
discriminative components
Xingyu Cai, Jiaji Huang, Yuchen Bian, and Kenneth Church. Isotropy in the contextual embedding space: Clusters and manifolds. In International conference on learning representations, 2021.
Sahil Chelaramani, Manish Gupta, Vipul Agarwal, Prashant Gupta, and Ranya Habash. Multi-task knowledge distillation for eye disease prediction. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pp. 3983-3993, 2021.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664 16678, 2022.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and stale of the art. Proceedings of the IEEE, 105(10):1865 1883, 2017.
Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. Fusing finetuned models for better pretraining. arXiv preprint arXie:2204.000 03044, 2022.
Mircca Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vodaldi. Describing textures the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2014.
13

---

Published in Transactions on Machine Learning Research (08/2025)
Kevin Clark, Minh- Minh-Thang Luong, Urvashi Khandelwal, Christopher D Manning, and Quoe V Le. Bam! born-again multi-task networks for natural language understanding. arXiv preprint arXiv: 1907. 04829, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference on compuler vision and pollern recogrilion, pp. 248 255. leee, 2009.
Jacob Devlin, Ming-Wer Chang, Kenton Lec, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. orXi preprint ar urXiv:1810.04800 2018.
Shachar Don- Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, Yoar Katz, and Leshem Choshen. Cold fusion: Collaborative descent for distributed multitask finetuning. arXiv preprint or Xxx:2212.01377 2022.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghann, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. Tn International Conference Learning Representations, 2021. URL http::///ppearrreeeeeeeeeerrrrrr
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference machine learning, pp. 1309- 1318. PMLR, 2018.
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. In International Conference on Learning Hepresentations, 2022. URL https://openrevier http://rrrrreeeeeee/oooorooaaaaa
Jonathan Frankle, Gintare Karolina. Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 3259 3269. PMLR, 2020.
Mudasir A Ganaie, Minghui Hu, Ashwani Kumar Malik, Muhammad Tanveer, and Ponnuthurai N Suganthan. Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence, 115:105151, 2022.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31, 2018.
Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee's mergekit: A toolkit for merging large language models. arXiv preprint XrXiv:2403. 18257, 2024.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789 1819, 2021.
Cheng Han, Qifan Wang, Sohail A Dianat, Majid Rabbani, Raghuveer M Rao, Yi Fang, Qiang Guan, Lifu Huang, and Donglang Liu. Amd: Automatic multi-step distillation of large-scale vision models. In European Conference on Computer Vision, pp. 431-450 Springer, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026 1006-104,11 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770 778, 2016.
Robert Hecht-Nielsen. On the algebraic structure of feedforward network weight spaces. In Advanced Neural Computers, pp. 129-135. Flsevier, 1990.
14

---

Published in Transactions on Machine Learning Research (08/2025)
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. TEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217 2226, 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXie preprint Stt::993.12266. 2019.
Bycongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 1921-1930, 2019a.
Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 3779-3787, 2019b.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXie preprint arXiu: 503. 02531, 2015.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parametor-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790 2799. PMLR, 2019.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL / net/ pttt:ooorrn/.ddddeeeeeeeeeee
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang. Chao Du, and Min Lin. Lorahub: Efficient. cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.1322 2023.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference Learning Representations, 2023. URL httpps://openreview wttp://wwwwwweeeeeeeeeeeeeeeeeee mett/forrmmi.dtttttkffef-
Moritz Imfeld, Jacopo Graldi, Marco Giordano, Thomas Hofmann, Sotiris Anagnostidis, and Sidak Pal Singh. Transformer fusion with optimal transport. arXiv preprint arXiv:2310.05719, 2023.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803. 05407, 2018.
Geethu Miriam Jacob, Vishal Agarwal, and Björn Stenger. Online knowledge distillation for multi-task learning. In Proceedings of the LEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2359 2368, 2023.
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietre and Pengxiang Cheng, Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representations, 2023. URL
Keller Jordan, Hanic Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. REPAIR: REnormalizing permuted activations for interpolation repair. In The Eleventh International Conference on Learning Representations, 2023. URL htttps://ppenrevier netr//oorrmm/dddddssss32ggg
Simran Khanuja, Melvin Johnson, and Partha Talukdar. Mergedistill: Merging pre-trained language models using distillation. arXiv preprint ar arXiw:2106.02834, 2021.
Philip A Knight. The sinkhorn knopp algorithm: convergence and applications. SIAM Journal on Matrix Analysis and Applications, 30(1):261-277 2008.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fino-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013.
15

---

Published in Transactions on Machine Learning Research (08/2025)
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun. The mnist database of handwritten digits. http://yann. Lecun. com/exdb/mnist//
1998.
Chang Liu, Chenfei Lou, Runzhong Wang, Alan Yuhan Xi, Li Shen, and Junchi Yan. Deep neural network fusion via graph matching with applications to model ensemble and federated learning. In International Conference Machine Learning, pp. 13857-13869. PMLR, 2022.
Yuang Liu, Wei Zhang, and Jun Wang. Adaptive multi-teacher multi-level knowledge distillation. Neurocom- puting, 415:106-113, 2020.
Ekdoep Singh Lubana, Eric J Bigelow, Robert P Dick, David Krueger, and Hidenori Tanaka. Mcchanistic mode connectivity. In International Conference on Machine Learning, pp. 22965 23004. PMLR, 2023.
Michael S Matena and Colin A Raffel. Merging models with weightee averaging. Advances in Neural Information Processing Systems, 55:17703-17716, 2022.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication- efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273-1282 PMLR, 2017.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasenzadeh. Linear mode connectivity in multitask and continual learning. In International Conference on Learning Representations, 2021. URL htttps://oeenreee wwtp:/nee/fooomidfftoo... F fQYUejf
Aviv Navon, Aviv Shamsian, Ethan Fotaya, Gal Chechik, Nadav Dym, and Haggai Maron. Equivariant deep weight space alignment. arXiv preprint arXiv:2310. 139977 2023.
Yuval Netzer, Tao Wang, Adain Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep leorning and unsupercised feature learning, volume 2011, pp. 7. Granada, Spain, 2011.
Behnam Neyshabur, Russ R Salakhutdinov,, and Nati Srebro. Path-sgd: Path-normalized optimization in deep neural networks. Advonces in newral information processing systems, 28, 2015.
Guillermo Ortiz-Jimencz, Alessandro Favero, and Pascal Frossard. Task arithmotic the tangent space: Improved editing pre-trained models. Advances in Neural Information Processing Systems, 36, 2024.
Seo Yeon Park and Cornelia Caragea. Multi-task knowledge distillation with embedding constraints for scholarly keyphrase boundary classification. In The 2023 Conference Empirical Methods in Natural Language Processing, 2023. URL https:// net/foorum/1ddBBE2WW5
ScongUk Park and Nojun Kwak. Food: Feature-level ensemble for knowledge distillation. arXiv preprint arXiv:1909, 1et...1994.7 2019.
Cuong Pham, Tuan Hoang, and Thanh-Toan Do. Collaborative multi-teacher knowledge distillation for learning low bit-width deep neural networks. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 6435-6443, 2023.
Mary Phuong and Christoph H Lampert. Functional vs. parametric equivalence of relu networks. 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. Tn International conference on machine learning, pp. 8748-8768 PMTR, 2021.
Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamoujy, patrick gallinari, and Matthicu Cord. Diverse weight averaging for out-of-distribution generalization. In Alice H. Oh, Alckh Agarwal, Daniclle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL httt:p/s///gaeeeeeeeeeee/////eed J_MqU
16

---

Published in Transactions on Machine Learning Research (08/2025)
Bharat Bhusan Sau and Vineeth N Balasubramanian. Deep model compression: Distilling knowledge from noisy teachers. arXiv preprint Xin: 1ix:160000..... 0.000000, 2016.
Thibault Sellam, Steve Yadlowsky, lan Tenney, Jason Wei, Naomi Saphra, Alexander D'Amour, Tal Linzen, Jasmijn Bastings, Iulia Raluca Ture, Jacob Eisenstein, Dipanjan Das, and Ellie Pavlick. The multiERTTs: BERT reproductions for robustness analysis. In International Conference on Learning Representations, 2022. URL http:://ppaaeeeeeeeeeeeetrrrraaa FOgFDgA
Aviv Shamsian, Aviv Navon, David W Zhang, Yan Zhang, Ethan Fetaya, Gal Chechik, and Haggai Maron. Improved generalization of weight space networks via augmentations. arXiv preprint arXiv:2402.4408. 2024.
Noam Shazeer, Azalia Mirhoscini, Krzysztof Maziarz, Andy Davis, Quoc Le, Gooffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. orXio preprinl. WrXii:1101.0633 2017.
Alex Sherstinsky. Fundamentals of recurrent neural network (rnm) and long short-term memory (Istm) network. Physica D: Nonlinear Phenomena, 404:132306, 2020.
Ken Shoemake. Animating rotation with quaternion curves. In Proceedings of the 12th annual conference on Computer graphics and interactive techmiques, pp. 245 254, 1985.
Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Advances in Neural Informalion Processing Systems, 2922044--20555 2020.
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: a multi-class classification competition. The 2011 international joint conference on neural networks, pp. 1453-1460. IEEE, 2011.
Samuel Don Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew Gordon Wilson. Does knowledge distillation really work? In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan ), Advances in Neural Information Processing Systems, 2021. URL https://openrrevee het/forum?id=7J-- fKoXiReeA
Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. Transactions Machine Learning Research, 2022. ISSN 2835-8856. URL http:////ppeeeeeeeeeeeefoooouone 4nPswriKeP
George Stoica, Daniel Bolya, Jakob Brandt Bjorner, Pratik Ramesh, Taylor Hearn, and Judy Hoffman. Zipit! merging models from different tasks without training. In The Twelfth International Conference on Learning Representations, 2024. URL / wetpiiewweeeeeeeeeeeeeeeeeeeeeee wttp://wwww..t.tetotre.eeeeeeeea aome//orron//////kdduuug
Yi-Lin Sung, Linjie Li, Kevin Lin, Zhe Gan, Mohit Bansal, and Lijuan Wang. An empirical study of multinodal model merging. In The 2023 Conference on Empirical Melhods in Natural Language Processing, 2023. URL https://openrevicee net www./oouuniddwwdaggee....
Xu Tan, Yi Ren, Di He, Tao Qin, and Tie- Yan Liu. Multilingual neural machine translation with knowledge distillation. In International Conference on Learning Representations, 2019. URL https://openrrevev
Bet/forrum?id=S1guseea
Norman Tatro, Pin-Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai. Optimising mode connectivity via neuron alignment. Advances in Neural Information Processing Systems, 33:15300- 153111, 2020.
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMI.R, 2021.
17

---

Published in Transactions on Machine Learning Research (08/2025)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Ncha Verma and Maha Elbayad. Merging text transformer models from different initializations. arXiv preprinl Xiv:2400 00986, 2024.
Jayakorn Vongkulbhisal, Phongtharin Vinayavekhin, and Marco Visentii-Sccazaaeella Unifying heterogeneous classificrs with distillation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 3175 3184, 2019.
Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge fusion of large language models. arXiv preprint or 0rxi::4011.1044 2024.
Alex Wang. Amanprect Singh, Julian Michacl, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019. URL. httpps:/openzreview net/forummid==rr4kmmm
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging, In International Conference Learning Representations, 2020. URL
https://openrrevie http:////pppaaeeeeeeeees//////ap
Mitchell Wortsman, Gabriel IIharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontije-Topes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pp. 23965 23998.. PMLR, 2022.
Chuhan Wu, Fangzhao Wu, and Yongfeng Huang. One teacher is enough? pre-trained language model distillation from multiple teachers. arviv preprint Xix: 01023, 2021a.
Siyue Wu, Hongzhan Chen, Xiaojun Quan, Qifan Wang, and Rui Wang. Ad-kd: Attribution-driven knowledge distillation for language model compression. arXio preprint orXiv:2305.10010 2023.
Yimeng Wu, Mehdi Rezagholizadeh, Abbas Ghaddar, Md Akmal Haidar, and Ali Ghodsi. Universal-kd: Attention-basee output-grounde intermediate layer knowledge distillation. In Proceedings of the 2021 conference empirical methods in natural language processing, pp. 7649-7661 2021b.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking 2017.
machine learning algorithms. arXiv preprint
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to 2oo. In 2010 IEFE compuler sociely conference compuler vision and pattern recognition, pp. 3485 3492. IEEE, 2010.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pp.
PMLR, 2020.
Prateck Yadav, Derck Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36, 2024.
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openrevieet wttpe/vwwwwwwweeeeeeeeeeeee http:///oorrr//////ooooggaa
Hao Yu and Jianxin Wu. Compressing transformers: features are low-rank, but weights are not! In Proceedings of the AAAT Conference on Artificial Intelligence, volume 37, pp. 11007-11015, 2023.
18

---

Published in Transactions on Machine Learning Research (08/2025)
David Yunis, Kumar Kshitij Patel, Pedro Henrique Pamplona Savarese, Gal Vardi, Jonathan Frankle, Matthew Walter, Karen Livescu, and Michael Maire. On convexity and linear mode connectivity in neural networks. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop), 2022. URL
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via allention transfer. In International Conference on Learning Represenla- tions, 2017. URL httpp::/ http:////ppppaeeeeeeeeeeoroaaaaa Sks9_ajex
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-eflcient fine-tuning for transformer-basee masked language-models. ar preprint (XXi:2106.10199 2021.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018. URL https://openrevierr
net/forrumiid=<1Ddppp
Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. Llama-adapter: Efficient fine-tuning of large language models with cero-initialized attention, In The Twelfth International Conference on Learning Representations, 2024.
Zhanpeng Zhou, Yongyi Yang, Xiaojiang Yang, Junchi Yan, and Wei Hu. Going beyond linear mode connectivity: The layerwise linear feature connectivity. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL http:://pparrroii.ss.eee/rrrr/es
19

---

Published in Transactions on Machine Learning Research (08/2025)
A Related work
A.1 Related work
Mode Connectivity. A pair of models has mode connectivity when there exists a simple path between them in the loss landscape with low loss (Garipov et al. 2018 Draxler et al.) 2018 Lubana et al., 2023
Frankle et al. (2020) demonstrated that this path can be lincar (LMC) when the models originate from the
same initialization. Tatro et al. (2020); Ainsworth et al.
Entezari et al. (0022) showed that it is feasible to establish LMC between MLPs and CNNs from different initializations using permutations. Jordan
(2023):
et al. (2023) enhanced these solutions by addressing their variance collapse. |Wang et al. (2020) used these permutations to match and then merge models within a federated learning framework. [mfeld et al.] (2023); Verma & Elbayad (2024) used permutations to align two transformers with different initializations that were trained on the same task. MMrzadeh et al. [2021] leveraged LMC for multitask and continual learning. Yunis et al. (0022) expanded LMC to more than two models, discovering a high-dimensional convex hull of low loss. Zhou et al. (2023) introduced Layerwise Linear Feature Connectivity (LLFC), demanding that the features of the models be lincarly connected. [Singh & Jaggi (2020); Akash et al. (2022) addressed a similar challenge of neuron aligmment using optimal transport (Knight 2008), and Liu et al. (2022) generalized it as a graph-matching task. Navon et al. (2023) solved the neuron aliguments problem by training an equivariant deep weight space network. It is important to note that previous works which merged models from different initializations using permutations primarily focused on merging pairs of models trained on the same task, with many of them concentrating on MLPs and CNNs. In contrast, our method is capable of merging larger groups of models, with different initializations and tasks, and is specifically designed to handle transformers.
Model merging. Model merging technique (Goddard et al., 2024 has gained increasing interest in the past years, allowing the creation of stronger single-task and multi-task models. Akhlaghi & Sukhov (2018) showed that averaging the weights of multiple simple neural networks maintains their performance. Model soups (Wortsman et al. 2022) proposed averaging multiple models trained on the same task from identical initializations to enhance task accuracy. In addition, Weight averaging has been employed for various purposes, including improving optimization through checkpoint averaging (Izmailov et al. 2018). federated learning [MeMahan et al. 2017), developing superior pre-trained models (Choshen et al.
2022 Don-Yehiya et al.,
2022). improving
Distr--Dittiitttion Generalization (Rame et al. 2022), achicving success in unseen tasks Huang el al. 2023), and as augmentations for weight space networks (Shamsian et al. 2024). In many cases, spherical linear interpolation (SLERP) (Shoemake] [985) is used to average the models' weights. Matena & Raffel (2022) utilized WeiihteWei Averaging to fuse multiple models from the same initializations but trained on diverse tasks, resulting in multi-task model. RegMean (Jin et al. 2023) addressed similar scenario and proposed a closed-form solution that solves a local linear regression problem for each linear layer in the model. [Iharco et al. (023) defined task vector by subtracting the parameters of a fine-tuned model from those of the pre-trained model, and used it to fuse models finc-tuned from the same pre-trained model. Yadav el al. (2024); Ortix-Jimenez el. al. (2024); Yang el al. (2024); Akiba el al.] (2024) analyzed and proposed more merging methods based on task vectos. Sung et al. (2023) fused pairs of models trained on different modalities. Ziplt (Stoica et al.] (024) merged models from various initializations and tasks, focusing MLPs and CNNs, by averaging pairs of highly correlated neurons between and within the models (see
Appendix A.2 for more details). In contrast, our work focuses on MLPs and transformers, and can use much more complicated merging schemes.
Distillation. In Knowledge Distillation (KD), a small student model is trained mimic the outputs of one or more larger teacher models (Ba & Caruana 2014 Hinton et al.) 2015 Stanton et al. 2021 Gou et al, 2021). Sau & Balasubramanian (2016) implemented noise- based methodology to simulate learning single task from multiple teachers. Tan et al. (019 Khanuja et al. (2021) used several teachers, each translating between a specific language pair, to (rain a singular multilingual student. Clark et al. (2019); Park & Caragea (023): Chelaramani et al. (2021) utilized multiple models and true labels to instruct a multi-task model. Pham et al. (2023) employed numerous quantized tcachers trained on the same task to teach a quantized model, also leveraging true labels. Wu et al. (2021a) co-fine-tuned multiple teachers in downstream tasks with shared layers to instruct student. Jacob et. al. (023) simultaneoully trained multiple single-task models
20

---

Published in Transactions on Machine Learning Research (08/2025)
with a single multi-task student to facilitate the student's optimization. Similar to our distillation bascline, Vongkulbhisal et al. (2019) utilized KD to unify knowledge from various models with different targets into a single model, relying solely on unlabeled data. [Wu et al. (2021a); Zagoruyko & Komodakis (2017 Heo et al. (2019b a focused on using a single teacher's outputs and inner features to train a single model. Park & Kwak (2019); Liu et al. (2020) applied multiple teachers trained on the same lask, their inner features, and the true labels to train a single-task student. Inspired these works, we tried to merge transformers using a loss function that takes into account the inner features. However, this approach proved ineffective for both our method and distillation. See Appendix G.5 for details. Wan et al. (2024) generates next-token probability distributions from multiple LLMs on a shared corpus, merges them into a single fused probability matrix, and uses it to transfer knowledge into a single LLM. Crucially, their setting does not involve models trained on different tasks. In contrast, our setting involves classification models with a shared architecture but trained on different tasks. Therefore, using a shared corpus to generate a fused probability matrix would not enable effective knowledge merging our case.
Adapter modules. FS-Merge also resembles adapter- based methods (Houlsby et al. 2019 Chen et al., 2022, Zhang et al., 2024], which perform Parameter-Efficient Fine-Tuning (Hu et al. 2022 Zaken et al. 2021 of large models by injecting a small module ('adapters") into the original model. During fine-tuning, the original weights of the pre-trained model are frozen, and only the adapter weights are optimized. Similarly, FS- FS-Merge injects new parameters into the model, which are the only ones optimized, though there are key differences. First, FS-Merge is used to merge the knowledge of multiple models, in contrast to adapter-basod methods, which fine-tune a single model. Additionally, at the end of the training phase (or "merging"), the trainable parameters of FS-Merge are folded into the original weights, whereas in many adapter methods, the adapters remain in the finc-tuned models, adding complexity.
A.2 Comparison to Ziplt
Our work strongly relates to ZipIt (Sticaa et al. 2024). which also merges models from various initializations and tasks, with focus on MLPs and CNNs. We will briefly explain the algorithm and its limitations. ZipIt uses the original training data to extract features from the models to be merged, A and B. Then, for each layer I, it employs the features to identify pairs of highly correlated neurons using a greedy algorithu. Following this, it builds Mi E R4x2.d; which is zero except for entries corresponding to a matched pair (i.j)
indexed by p, where Mi[p, i] = Mrpp = 1. The purpose of this matrix is to merge features by averaging pairs of highly correlated neurons. Then, U, = 2M, is used to reposition the merged features back to their original locations. After building Mr and UI, ZipIt proposes folding them with the original weights (Eq. 5 Lo create the weights of the merged model.
Like other current model merging methods (Ainsworth et al. 2023 Jin et al. 2023 Infeld et al. 2023 ZipIt is efficient and focuses only on simple fusion schemes. Tt limit itsell to pairing similar neurons, ad addressing only the permutation symmetries of neural networks (Helst-Nielsen 1990). Permutation symmetries mean that it is possible to swap any two neurons of a hidden layer in a neural network without altering its functionality. However, ZipIt falls short in handling the scale symmetries of neural networks (Neyshabur et al.) 2015 Badrinarayanan et al. 2015 Phuong & Lampert 2019). or in considering more complicated merging rules.
Probably due to these limitations, Ziplt underperforns on a large scale (such as ResNet-50 trained on datasets with 200 categories). Furthermore, this merging method is formulated as local problems, merging one layer at time, and relies on heuristics to handle more complicated layers (such as batch normalization and skip connections). This makes it difficult to generalize to more complex architectures like transformers, where ZipIt struggles with sell-attention and skip connection structures.
Our work aims to solve a similar problem, but adopts a more expressive approach without relying on houristics. Moreover, the global version of our method allows the merging of any architecture by simply constructing a Foldable SuperNet that is suitable for it.
21

---

Published in Transactions on Machine Learning Research (08/2025)
B Merging Vision Transformers with FS-Merge
B.1 Foldable SuperNet for Vision Transformers
Assuming there are two Vision Transformers (ViTs) (Dosovitskiy et al. 2021 trained on two distinct tasks, A and B, we aim to define a Foldable SuperNet that combines the original weights of the ViTs with new learnable parameters M, U. This structure is designed so that all skip connections and layer normalizations (Ba et al. 2016 Xiong al. 2020) operate on the merged dimension, and that there is a lincar layer before alter every M U matrix, allowing them to be folded afler training. It important to highlight that layer norms possess significantly fewer learnable parameters compared to other layers in ViTs. Therefore, we can initiate their parameters from good starting point (e.g,, the parameters of the first model to be merged) and proceed to optimize the parameters as usual, similar to strategies employed in previous merging works (Jordan et al. 2023 Stoica et al. 2024
The method described here for merging two models can be readily extended to any number of models. The parameters optimized this stage are highlighted in red. The notation X represents the outputs of layer norm, X represents feature reconstruction attempt after using the U matrix, and X represent the merged features.
B.1.1 Pre- processing
First, each ViT creates patches from the input image and reshapes them into a series of vectors Tproj E RRx-1xd using Wit E gdu xd project these vectors into tokens. It then concatenates the CLS e RR token and adds emb R Txd which is the positional encoding. For example, for model A:
z Concat|/proj Win.A, CLSA + +emb
The Lokens from both models are then concalenated to form (2^1|2B) E RTx22,, and the Foldable SuperNet merges them using a learned matrix Min E 2dd to produce Z' E R'xd
2 = (Z^1l²))M..
Subsequently, and like the ViT, the Foldable SuperNet applics layer normalization, with paramcters 7-,,8"1 E Rd, to generale the input for the transformer. These parameters will be optimized.
X; -I. INN,,,82, (z*)
B.1.2 The attention block
The Foldable SuperNet of the attention block at layer (Figure 3 receives merged features Xi E Rxd as inputs, and applics layer normalization. It's parameters ni 8 E R will be Icarned. Additionally, a learnable matrix Uj Rdx2dd is introduced reconstruct the original features from the merged ones.
XAILEP = LN,; o (X))U
After the layer normalization, the queries, keys, and values of each ViT are calculated. Then the Foldable SuperNet concatenates these components, from all heads and models, and merges them using learnable matrix. Taking the queries as example:
@ = (Xi (Xx W..A B W.Q.B) RTx2d
Qi =0:(0AP)
(Q.,,...QH..)))
The weights W.O.A W.Q.B E gdxd generate queries from the embeddings. The matrix M E 2xddd merges these features, which are then divided into H heads, where each head Qi, has dimensions T 4, Similarly, the keys and values are created by WKK W.V E Rddd and compressed using the matrices MK MY e 2 respectively.
22

---

Published in Transactions on Machine Learning Research (08/2025)
Following this, and similar to the ViT, the Foldable SuperNet executes multi-head attention with the merged queries, keys and values and concatenates the features from the heads. In our Foldable SuperNet, this step also includes adding UP E [_dx2d to reconstruct the original multi-head attention outputs. Then each ViT utilizes W E Rd to aggregate those outputs. We also use MP R2ddxd to compress it once more. This is followed by skip connection.
(oibbia vd
up (W/
(WO. WOBB) MP.
Y1 = X + Concat, softmax
Vin...]
This process results in YJ" Tx serving the input for the subsequent MLP block at layer I.
B.1.3 The Multi-Layer Perceptron block
The Foldable SuperNet of the / MLP block receives Y E ggd d input, which are the merged features of the previous attention block. It learns the layer norm parameters of the MLP block of 07 E Rd, which, usual, acis on the compressed dimension:
= LNa,, (Y7).
After the layer norm, the VIT's MLP block applies a sequence of operations: a linear layer, an activation function, and another linear layer. Our Foldable SuperNet mimics this process and uses M and U matrices both compress and reconstruct the features each stage, akin to the approach described in Section 2.1 After these operations, skip connection is applied the compressed dimension.
W7/h Xi+ = YY; to Y
W?2 0
(0
W.B.B
Mt
U?
W2BBB M².
0
Where U/, U? E Rdx2d MP, M2 E 2dddd block.
X44 E Rxd then serves as the input for the + 1 attention
B.2 Training the Foldable SuperNet
In the of merging two ViTs A and B, DA and DB are defined small subsets of training data from tasks A and task B respectively.
Our objective is to define global optimization problem for training the Foldable SuperNet. As in the case of linear layers, we aim to reconstruct the features from the last representation layer (just before the classification head) of the original ViTs. Hing represents an input image from task & and f (Jimg) E I2d represents the Teatures from the last representation layer of the original model fine-tuned on lask k, created from the input Ling. Observe that f the CLS token after being processed by the transformer and various post-processing stages that should also be merged (for instance, final layer normalization and linear projection layer).
We will define the output of the Foldable SuperNet (r.(m E RR2-d, which is reconstruction atlempt for fA (4mgg) (Tugg) e Rd2 Also, fi (Amgg) [k] will note the reconstruction attempt for model & features. Then the loss function will bo:
Lout Σ Eya ~D ( (z |+11
This implics that for the input Hing belonging to task k we will only learn from the features the model trained on this task, and not for example from the fcatures that the model j created F (T This loss differs from the one used in the MLP case, where each layer attempts to reconstruct the features that both models create from the input T regardless of the task it belongs to. This method was adopted for both FS- FS-Merge and KD when merging Vils, as we found it performed better.
In the case of merging ViTs, this global approach worked much better than addressing a series of local problems for each block, done in the ML. case. For more details, please refer to Appendix G.3
23

---

Published in Transactions on Machine Learning Research (08/2025)
B.3 Folding the Foldable SuperNet
Our next slep after learning involves folding the Foldable SuperNet. This procedure aims to create a merged VIT that operates within the same dimensionality as the original models. The layer norm parameters acquired through our optimization process will be directly used in the merged model, as they already work in the merged dimension.
The folding operation (Eq. 5 follows the methodology outlined in Section 2
For instance, within the pre-processing block, the new merged projection weights and positional embeddings will be defined as follows:
Win, (Win, || Mi.
emb (emb ||emb B) )
Also, taking the attention block at layer I as an example, the merged query weights will be:
w Ui (WQ.A
w.O.B MP.
The other weights will be folded in a similar manner. Intuitively, this folding operation creates merged model that, "under the hood", uses U reconstruct the original features from the previous layer, applies the original weights, and then uses M to merge those features again, all with the same complexity cach of the original models.
B.4 Merge tasks sequentially with FS- FS-Merge seq.
Using the global version of FS-Merge on large models like transformers comes with significant resource cost. As we have shown, the number of learnable parameters is smaller than in the distillation case, due to our modeling of the M and U matrices as a concatenation of diagonal matrices plus a low-rank matrix (Eq, 7 However, we still need to retain the frozen original weights of all models memory, leading to increased memory and compute resource demands when merging multiple models.
To address this issue, we introduce FS-Merge Seq,, which merges the models sequentially, For example, if we wish to merge models, we start by merging the first two models using the global FS-Merge (Eq, E with the features of these two original models as targets. The M and U matrices are still modeled as a concatenation of diagonal matrices plus low-rank matrix, and we apply the "First" initialization (initialized from the first model).
After merging the first two models, we use global FS-Merge again to merge the resulting model (capable of solving the first two tasks) with the third model. In this phase, we use the features of all models seen so far (the first, second, and third models) targets, and initialize from the merged model obtained in the previous step. This process is repeated, merging the previous merged model with a new original model at each step, until all n models are merged. Note that In FS-Merge Seq,, at cach phasc, we only merge and load the weights of two models (the previous merged model and new original model), even though all features and models are utilized.
Experiments merging groups of four and five ViTs demonstrate that FS-Merge Seq. requires significantly less memory and compute resources, and merges models faster compared to regular FS- (S-Merge. While this approach results in slightly lower accuracy compared to regular FS-Merge, FS-Merge Seq. still achieves better performance than distillation in most cases.
C Time and memory complexity analysis
For memory complexity analysis, let's consider the merging of n fully connected layers with weights W e Raxd to one layer. When using distillation to merge these layers, d² learnable parameters are required as we train a single weight matrix for all n models.
In the FerMerge case, we will have 2nd learnable parameters in the M and U matrices. Additionally, we must hold ndl frozen weights in memory, which comes with reduced cost compared to learnable parameters
24

---

Published in Transactions on Machine Learning Research (08/2025)
Table 7: Measuring the total time and the number of optimized parameters, while merging group of four ViT-B-16 with 100 original images and 1000 augmented images from each dataset. The per-task test accuracy is also reported. The merged models are the models fine-tuned on RESISC45, EuroSAT, CIFAR1O, and MNIST.
# Parameters Optimized
Method Merging time
Accuracy
Average
4 Seconds
8.33
SLERP
4 Seconds
0
8.69
RegMean
3 Minutes
0
8.33
Opt
~ 18 Minutes
0
8.76
Distillation
1.9 Hours
111M
86.86
FS-Merge diagonal
3.2 Hours
900K
87.35
FS-Merge low rank
3.6 Hours
60M
91.63
F-rMergee seq. 2.2 Hours
18M
91.10
(as we do not need to compute gradients for these matrices, and the optimizer does not need to save their moments). Nevertheless, this 'vanilla' version of FS-Merge is much more memory-intensive compared Distillation.
To mitigate this issue, suggest two additional versions of FS-Merge. In the first onc, parameterize the M and U matrices a concalenation of diagonal matrices plus low-rank matrix with a rank of (Eq. 7 resulting in 2(nrd + nd + rd) learnable parameters, As demonstrated in the transformer case, small values of r are sufficient to outperform distillation, which also results in fewer learnable parameters compared to distillation. However, we still need to retain ndf frozen weights in memory.
Our final and most efficient version is FS-Merge soq. (Appendix B.4 which merges pair of models each stage. Thus, we the previous calculation with n = 2, which results in 6rd 4d learnable parameters and 2d2 frozen weights in memory at each stage. FS-Merge seq. comes with small cost to performance, but still outperform distillation (and see Section 3.2 Appendix F.2
Table 7 presents the total time and the number of optimized parameters when merging a group of lour ViT-B-16 models with 100 original images and 1000 augmented images from each dataset. FS-Merge seq. used with a low rank of 16. The per-task accuracy is also shown.
Pre-training Vision Transformers
D.1 Training Vision Transformers from scratch
We pre-trained ViT-B-16 and ViT-L-14 (Dosovitskiy et al. 2021 Steiner et al. 2022 Touvron et al. 2021 on ImageNet- 1K (Deng et al.] 2009). These models were initialized from distinct random seeds and exposed to training data in different orders. Following a sctting similar to other merging works [[harco et al. 2023 Stoica et al. 2024), frozen classification head derived from CLIP's (Radford et al. 2021 label embeddings was used, in order to make the outputs space of the ViTs similar. Training and merging ViIs with learned classification heads are left for future research. It is important to mention that the classification heads are not being merged.
We pre-trained the ViTs following common practices (Touvron et al. 2021 such as augmentations, MixUp (Zhang et al. 2018), distillation using resnet152 (He al. 20166), entropy loss, and cosine scheduler with single cycle and warmup.
D.2 Datasets and fine-tuned Vision Transformers
After obtaining different pre-trained ViTs, we fine-tuned each ViT on a different downstream task. Following the approach outlined in (Tharco et al. 2023 we fine-tuned on range of datasets including Cars (Krause)
25

---

Published in Transactions on Machine Learning Research (08/2025)
Table 8: Dataset details and the test accuracy of the fine-tuned ViT-B-16 and ViT-L-14.
ViT-B-
ViT-L-14
Dataset
Train size
Val size
Test size
# Classes
16
test acc.
test acc.
EuroSAT
18,700
3,300
5,00
10
99.06
98.10
GTSRB
22,644
3,994
12,630
43
98.32
97.88
Cars
6,923
1,221
8,041
196
86.12
93.32
CIFAR-10
42,500
7,500
10,000
10
97.33
98.9
CIFAR-100
42,500
7,500
10,000
100
85.61
92.74
DTD
1,880
1,880
1,880
47
64.04
64.57
MNIST
51,000
9,000
10,000
10
99.66
99.68
RESISC45
18,900
6,300
6,300
45
93.46
93.95
SVHN
62,269
10,988
26,032
10
96.78
96.63
al. 2013). DTD Cimpoi al. 2014). EuroSAT (Helber al 2019 GTSRB (Stalkkamp et al. 2011 MNIST (LeCun 1998). RESISC45 Cheng et al. 2017). SVHN (Netzer et al. 2011), CIFARIO, and CIFAR100
Krizhevsky et al. 2009). We used the existing datasets' training set, validation set, and test set. If there wasn't t a validation set, one was created by using 15% of the training set.
All models were finc-tuned with a batch size 256, a Icarning rate le-5, cross-entropy loss, and cosine scheduler using a single cycle with warm-up phase. As done in previous works (Ilharco al. 2023 Stoica et al., 2024), the classification head was frozen and obtained from CLIP embeddings (Radford et al. 2021 of the label names. In Table g we present the dataset details and the test accuracy of the fine-tunet models their respective tasks. The models' accuracies on tasks they were not fine-tuned for are as good as random guess.
E
Merging experiments details and hyperparameters
E.1 Merging Multi-Layer Perceptrons
We divided the MNIST dataset (LeCun 1998) into two subsets: the first containing labels to 4 and the second containing labels 5 to 9. Each subset was further split into training, validation, and testing sets, with the validation set comprising 10% of the original training set. similar approach was adopted for the Fashion-MNIST dataset. (Xiao et al. (017)).
We trained MLPs on each of the MNIST subsets, utilizing SGD with a learning rate of 0.01, a batch size of 10, and cross-entropy loss. The training duration set to 1 epoch. However, if the number of hidden layers exceeded four, or the hidden dimension surpassed 256, the training was extended to 2 epochs. For the Fashion MNIST dataset (Xiao et al., 2017), the same hyperparameters were used, with adjustments made only to the number of epochs, increased to 10, and the learning rate, reduced to 0.001.
Data. All the merging methods, with the exception of simple weight. averaging, need features generated from the MIPs. Those were crcated by using unlabeled data from the training sets. 64 images from each split dataset were utilized, resulting in total of 128 images. Additionally, we normalized the target features of the two models to the same scale in FS-Merge and in distillation, because it led to an improvement in accuracy. Note that this will not hurt the performance of the merged model during inference, as the scale of the last layer's features does not change the prediction.
Hyperparameters for the merging methods. The hyperparameters for the merging methods were determined soparately for MNIST and Fashion MNIST. We chose the hyperparametors that maximize the per-task task accuracy on the validation set when merging two MLPs with two hidden layers and a hidden width of 128. We then used those hyperparameters for merging MLPs with different depths and widths.
26

---

Published in Transactions on Machine Learning Research (08/2025)
We will now outline the hyperparameters grid used for the hyperparameter search in each merging method. We used GD optimizer in FS-Merge and Distillation (so the batch size is 128, the whole data). The step learning rate scheduler employs two learning rate drops with y = 0.9, whereas the Cosine scheduler utilizes a single cycle with a warmup length of 20 epochs. FS-Merge.
initialization type: "random"
num epochs: [1k, 5k, 10k, 15k (MNIST), 20k (Fashion MNIST), 25k]
learning rate: [0.3, 0.1, 0.03, 0.01, 0.003]
momentum: [0.9, 0.8]
scheduler: (step Ir, cosine]
FerMerge global.
initialization type: ["First", "Average" (the average of the original models), "random")
num epochs: [200, 400, 1k (MNIST), 1.5k (Fashion MNIST), 5k, 10k, 15k]
learning rate: [0.3, 0.1, 0.03, 0.01, 0.003]
momentum: [0.9, 0.8]
scheduler: (step lr, cosine]
FS- global, ZipTt init.
initialization type: [*ZipIt"]
num epochs: [200, 400, 1k, 1.5k, 5k, 10k, 15k]
learning rate: [0.3, 0.1, 0.03, 0.01, 0.003]
momentum: [0.9, 0.8]
scheduler: (step lr, cosine]
Distillation.
initialization type: [*First", "Average" (the average of the original models), "random")
mum epochs: [200, 400, 1k (MNIST), 1.5k (Fashion MNIST), 2k, 5k]
learning rate: [0.3, 0.1, 0.03, 0.01, 0.003]
momentum: [0.9, 0.8]
scheduler: [slep Ir, cosine)
RegMean.
[1.0, 0.9 (MNIST), 0.8 (Fashion MNIST), 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1]
27

---

Published in Transactions on Machine Learning Research (08/2025)
Table 9:
The different experimental settings. "#Origiaal Images" refers the number of original images taken from the training set per datasct. "#Augmented Images" refers the number of augmented images created per dataset. The fine-tuned models refer to the models used for the hyperparameter search. When C10 = CIFAR1O, G GTSRB, R = RESISC45, S SVHN.
#Original #Augmented Images
fine-tuned models
Setting
What is merged
Total # #Images
Images
2 ViT-B-16
16
800
1,632
R, C10
4 ViT-B-16
100
1000
4,400
R. C10, S, G
2 ViT-L-14
100
1000
2,200
R, C10
E.2 Merging Vision Transformers
Data and augmentations. We took a set of images from the training set and expanded it using augmenta- tions. Specifically, these augmentations were used: Random Crop; Random Horizontal Flip; Random choice between grayscale, Solarization, and GaussianBlur; and MixUp (Zhang et al. 2018). Then this dataset is used to create features, where features of the model fine-tuned on task A were generated only from images of this task (and their augmentations). For efficiency and reproducibility, we saved these features and then used them for all our merging methods. In the case of F'S-Meree and distillation, a single epoch means using the entire features dataset once, including the features created by augmentations.
Hyperparameten Three types of hyperparameter experiments were conducted: merging pairs of ViT-B-16 in a low-data scenario, merging groups of four ViT-B-16, and merging pairs of ViT-L-14. For each experimental sctting, a specific group of finc-tuned models was selected and hyperparameter scarch performed. The hyperparameters that maximized the per-task validation accuracy for this group were chosen and then applied when merging other model groups in this setting. The settings can be seen in Table
For all methods that require training (FS-Merge and distillation), a batch size of 128 was used, along with a cosine scheduler that utilized a single cycle with a warmup phase, an ADAMW optimizer with a weight, decay of 0.001, and initialization from the first model ("First"). Similar the MLP experiments, for the distillation baseline, the larget features were scaled to an L2 norm of 0.5. However, in this specific setting, this scaling proved unhelpful for FS-Merge and was therefore not utilized. The hyperparameter grid used for the hyperparameter search will now be outlined.
FS-Merge, concatenation of diagonal matrices, without a low rank matrix.
epochs: [30, 100 (c), 200 (a, b), 300, 400]
Ir: [0.1, 0.01 (c), 0.001 (a, b), 0.0001]
FerMerge, concatenation of diagonal matrices + low rank matrix.
epochs: [30, 100, 200, 300, 400]
Ir: [0.1, 0.01, 0.001, 0.0001, 0.00001]
FS-Merge seq. (Appendix B.4 "epochs" refers to the number of epochs used for all iterations except the last one. "last iteration epochs" denotes the number of epochs applied in the last iteration, which involves merging the final model with the model obtained from the previous itcration.
epochs: [10, 50, 100]
last iteration epochs: (50, 100, 200, 300]
Ir: [0.0001]
28

---

Published in Transactions on Machine Learning Research (08/2025)
Distillation.
epochs: [30, 100 (a, c), 200, 300 (b), 400]
Ir: [0.1, 0.01, 0.001, 0.0001, 0.00001]
2023 a is a factor which decrease the non-diagonal items of the inner product matrices RegMean (Jin et al.
in the MeaM solution.
a: [1.0, 0.9 (a, c), 0.8, 0.7 (b), 0.6, 0.5, 0.4, 0.3, 0.2, 0.1]
"Opt" [Imfeld et al. 2023 The hyperparameters include the filter type (which determines the feature tokens used by the optimal transport solver) and X, a regularization term for the solver SSnkhorn-Knapp algorithm). Lower values of À result in harder alignment.
filter: [Only CLS, Full (c), Window 2, Window 4, Window 6, Window 8 (a, b), Window 10, Window 14]
A: [0, 0.08 (a.c), 0.2 (b), 0.5]
F
Additional results
F.1 Merging Multi-Layer Perceptrons
Additional results were obtained by merging pairs of MLPs, using 64 images from each lask to create features. Each experiment was replicated five times with different random seeds. Note that we do not merge the last lincar layer (the classification head).
Fe-Merge. Our method Foldable SuperNet for each layer / independently, using the original models' pre-activation features as inputs 2f, FS-Merge global is the global version of our method (Eq. 4h, training Foldable SuperNet for all the layers together to reconstruct the features of the final representation layer fA.ff. For both of these versions, we also tested a variant where we initialized the Foldable SuperNet's M and U with the solutions of Ziplt (Stoica et al. 2024), and then continued to optimize them (FS--Merge Ziplt init and F--M global ZipIt init). FS-Merge no cross compresses each of the two MLPs (A and B) individually half of their widths using a local FS-Meeree and then concatenates those two compressed models. This means that neurons between these two models cannot be merged.
tested in five variants. FS-Merge is the local version (Fq. 3), which trains
We also tried local FS-Merge version where the 1-th Foldable SuperNet layer uses the reconstructed features from the previous layer 2/ 2/3 as inputs, but it achieved the same accuracy as the regular local F'--Merge.
Table 10 presents the per-task accuracy for merging MLPs trained on half the MNIST dataset (LeCun 1998). with varying hidden width. Table 11 and Table 12 present the joint accuracy for fusing ML.Ps trained half of the MNIST dataset, with variations in depth or hidden width, respectively. Similarly, Table 13 and
Table 14 present the per-task accuracy for merging MLPs trained half of the Fashion MNIST dataset
Xiao et al. 2017) again varying by depth or hidden width. The joint accuracy for these models are detailed
in Table 15
As shown in the experiments, our method, especially when using ZipIt initialization, demonstrates SOTA results across all settings and outperforms ensembles some cases. also appears that FS FS-Merge achieves better per-task accuracy, while global FS-Merge achieves better joint accuracy. Furthermore, ZipIt stands out as strong baseline.
F.2 Merging Vision Transformers
For this section, C Cars, C10 CIFAR10, C100 CIFAR100, D DTD, E EuroSAT, GTSRB, M MNIST, R RESISC45, S SVHN.
29

---

Published in Transactions on Machine Learning Research (08/2025)
Table 10: Merging pairs of MLPs, each initialized differently and trained on distinct halves of the MNIST dataset. These MLPs have single hidden layer, with the Hidden width varying from 16 to 1024. Fach experiment was replicated five times. We present the average per task accuracy on the test set, along with the standard deviation.
Merge Method
Hidden width
16
64
128
512
1024
Original Models
96.34 + 0.3
96.77 + 0.1
96.8 +
97.8
98.0 0.1
Ensemble
84.47 + 4.3
93.19 + 1.8
94.0 + 0.6
96.4 0.3
96.5 0.3
average
83.88 + 3.9
93.05 + 1.7
94.1 + 0.4
96.6 + 0.1
96.8 + 0.1
RegMean
88.12 + 3.6
95.06 + 1.2
95.2 + 0.2
96.7 + 0.1
96.9 + 0.2
ZipIt
91.84 + 1.9
96.06 = 0.12
96.3 + 0.2
97.6 0.1
97.7 0.1
Distillation
82.69 + 4.7
91.41 + 2.5
93.0 + 0.7
95.7 + 0.4
96.0 + 0.3
FS-M
76.7 + 29.0
95.87 + 0.1
95.8 + 0.2
96.1 + 0.2
95.9 + 0.2
FS-M, Ziplt init
95.32 + 0.4
96.50 + 0.1
96.6 + 0.1
97.5 + 0.1
97.8 + 0.1
FS-M no cross
63.8 11.2
96.18 + 0.1
96.1 0.1
96.6
96.4 0.1
FS-M global
11.67 + 4.2
95.66 + 0.3
95.7 = 0.1
96.3 0.2
96.4 + 0.2
FS-M global, ZipIt init
11.84 = 4.6
96.44 0.1
96.6 0.1
97.6 0.1
97.8 0.1
Table 11: Merging pairs of MTPss, each initialized differently and trained on distinct halves of the MNIST dataset. These MLPs contain 128 neurons per hidden layer, with the number of hidden layers varying from 1 to 6. Each experiment was replicated five times. We present the average joint accuracy the test set, along with the standard deviation.
Merge Method
Number of Hidden Layers
2
Ensemble
80.20 + 1.7
83.25 + 3.3
84.6 + 2.1
83.9 1.9
83.1 1.5
average
78.96 + 0.7
69.83 + 6.2
57.4 + 5.2
38.2
10.2
11.5 2.9
RegMean
82.81 + 1.5
80.38 + 4.6
75.3 + 2.8
74.7 - 2.2
63.0 - 7.0
ZipIt
85.33 + 1.1
84.49 + 1.9
81.5 + 1.4
80.0 + 1.4
80.2 + 3.2
Distillation
77.32 + 1.5
79.2 3.96
79.6 + 2.1
78.2
3.6
74.4 2.0
FS-M
84.44 + 0.6
84.92 + 1.0
82.7 + 2.2
79.5 + 3.1
79.1 3.1
FS-M, Ziplt init
86.18 + 0.6
86.57 + 1.4
83.9
79.7 - 3.8
80.9 - 3.6
FS-M cross
85.39 + 0.6
85.88 + 1.3
86.0 1.0
84.6 - 1.1
83.8 + 0.4
FS-M global
83.67 + 0.7
84.12 + 1.1
83.9 1.0
82.8
1.1
81.6 + 0.7
FS-M global, ZipTt init
86.18 + 0.7
86.40 + 1.2
86.3 1.2
84.7
1.3
84.1 + 0.9
30

---

Published in Transactions on Machine Learning Research (08/2025)
Table 12: Merging pairs of MLPs, each initialized differently and trained on distinct halves of the MNIST dataset. These MLPs have a single hidden layer, with the hidden width varying from 16 to 1024. Each experiment was replicated five times. We present the average joint accuracy on the test set, along with the standard deviation.
Merge Method
Hidden width
16
64
128
512
1024
Ensemble
58.86 + 7.6
77.60 + 5.5
82.1 + 1.8
85.7 + 0.5
87.0 0.7
average
61.65 + 8.0
77.16 + 4.1
81.0 + 2.2
84.4 + 1.4
86.2 + 0.6
RegMean
66.73 + 7.7
81.54 + 3.2
84.0 + 1.6
85.2 + 1.1
87.1 + 1.0
Ziplt
66.89 + 2.9
85.21 + 1.0
83.9 + 1.8
87.7 + 1.5
88.0 + 1.5
Distillation
57.15 + 7.8
74.98 + 4.9
79.6 + 1.8
83.4 + 0.6
85.0 0.6
FS-M
59.1 + 26.2
84.30 + 1.0
83.9 + 1.0
85.6 + 0.9
85.5 + 0.5
FS-M, ZipIt init
77.52 + 4.2
85.91 + 1.1
86.3 + 0.7
88.5 1.0
89.3 - 0.5
FS-M no cross
54.00 + 9.3
4 1.5
85.0 1.2
86.5 0.8
86.5 + 0.4
FS-M global
9.46 + 0.1
83.72 1 1.6
$3.5
85.0 = 1.0
85.7 0.5
FS-M global, Ziplt init
9.63 + 0.1
85.99 + 1.3
86.1 + 0.9
88.6 0.9
89.4 0.4
Table 13: Merging pairs of MLPs, each initialized differently and trained on distinct halves of the Fashion MNIST dataset. These MLPs contain 128 neurons per hidden layer, with the hidden depth varying from to 6. Each experiment was replicated five times. We present the average per task accuracy on the test set, along with the standard deviation. Model
Number of Hidden Layers
2
3
Original Models Ensemble
90.45 + 0.14
90.53 + 0.1
90.4 + 0.1
89.9 = 0.1
83.0 + 2.2
87.31 + 1.93
88.68 + 0.6
86.2 + 1.9
86.9 = 2.0
77.7 4.0
average
86.04 + 2.40
78.20 + 5.1
58.9 10.3
47.8 5.4
22.4 + 3.0
RegMean
89.29 + 0.49
86.29 + 0.2
81.3 + 3.3
76.7 4.4
64.1 + 5.5
ZipIt
89.24 + 0.44
87.40 + 0.5
85.4 + 2.9
83.2 = 1.4
70.2 + 8.5
Distillation
86.94 + 1.49
87.84 + 0.8
83.0 + 1.6
83.2 2.4
69.2 3.9
FS-M
89.86 + 0.13
89.80 + 0.1
89.1 + 0.2
88.4 + 0.5
73.1 + 4.1
FS-M, Ziplt init
90.20 + 0.12
90.28 + 0.1
90.0 + 0.1
89.7 + 0.2
80.0 + 2.0
FS-M no cross
90.03 + 0.13
90.20 + 0.1
99.9
89.3 = 0.2
82.4 + 2.2
FS-M global
89.85 + 0.15
89.81 + 0.2
8.1
88.5 0.2
78.5 + 2.4
FS-M global, ZipIt init
90.04 + 0.08
89.95 + 0.1
8.2
88.4 = 0.2
62.5 2.4
31

---

Published in Transactions on Machine Learning Research (08/2025)
Table 14: Merging pairs of MLPs, each initialized differently and trained on distinct halves of the Fashion MNIST dataset. These MLPs have single hidden layer, with the hidden width varying from 16 to 1024. Each experiment was replicated five times. We present the average per task accuracy on the test set, along with the standard deviation. Merge Method
hidden width
16
64
128
512
1024
Original Models
90.31 + 0.06
90.43 + 0.1
90.4 + 0.1
90.6
90.7 0.1
Ensemble
72.71 + 4.72
85.21 + 3.3
86.9 + 1.0
88.9 0.4
87.0 1.0
average
70.33 + 6.56
83.90 + 4.2
86.2 + 1.4
88.5 = 0.4
86.8 + 0.8
RegMean
74.90 +: 8.31
87.33 + 2.5
88.7 + 1.5
89.1 = 0.3
87.1 + 1.3
Ziplt
77.39 = 7.60
88.73 = 0.7
89.4 0.6
89.8 = 0.2
88.7 + 0.2
Distillation
74.06 + 5.12
85.49 + 2.9
86.7 1.3
88.4 = 0.6
86.5 + 1.0
FS-M
89.06 + 0.64
89.90 + 0.1
89.8 + 0.1
89.5 = 0.1
89.1 0.1
FS-M, ZipIt init
90.10 + 0.19
90.25 + 0.1
90.2 + 0.1
90.3 + 0.1
90.3 + 0.1
FS-M no cross
49.2 12.63
90.09 + 0.1
9.1
89.8 = 0,1
89.5 0.1
FS-M global
13.98 + 4.88
89.87 + 0.1
8.
89.6 0.1
8.4
FS-M global, ZipIt init
11.98 = 3.97
89.98 + 0.1
90.1
90.2 = 0.1
0.11
Table 15: Merging pairs of MLPs, each initialized differently and trained on distinct halves of the Fashion MNIST dataset. These MLPs contain 128 neurons per hidden layer, with the hidden depth varying from to 6. Each experiment was replicated five times. We present the average joint accuracy the test set, along with the standard deviation. Model
Number of Hidden Layers
2
3
Ensemble
55.94 + 4.14
56.28 + 2.0
57.7 + 2.9
56.9 = 2.2
53.0 0.9
average
54.71 + 4.04
42.76 + 6.4
33.6 + 5.7
21.8 1.8
8.9 2.0
RegMean
56.94 + 1.24
54.78 + 2.1
54.7 + 4.3
50.0 4.5
51.6 + 5.1
ZipIt
59.26 + 1.66
60.01 + 1.5
60.9 + 2.4
59.6 = 2.5
48.3 + 7.3
Distillation
55.70 + 2.49
54.33 + 2.5
54.0 + 3.9
52.6 4.0
45.5 3.9
FS-M
54.40 + 0.29
49.91 + 1.1
49.1 + 1.5
47.3 1.6
42.2 + 2.3
FS-M, Ziplt init
54.13 + 0.28
48.71 + 1.0
4.1
47.3 = 1.2
44.3 10
FS-M no cross
56.81 + 0.23
56.19 0.9
1.3
56.7 1.0
53.0 - 0.8
FS-M global
56.63 + 0.27
55.64 + 0.9
55.8 1.4
56.1 = 0.9
51.9 0.9
FS-M global, ZipIt init
60.46 + 0.94
59.31 + 0.3
60.1 + 0.5
61.7 + 2.3
40.5 + 3.4
32

---

Published in Transactions on Machine Learning Research (08/2025)
Table 16: Merging pairs of ViT-B-16 with 16 original images from the training set and 800 augmented images from each datasct. We report the per-task and joint accuracy the test set.
Merging Methods EuroSAT, CTFAR100
Cars, MNIST
RESISC45, CIFAR10
Per-task
Joint
Per-task
Joint
Per-task
Joint
Original models
92.33
92.89
95.39
Ensemble
90.25
83.72
90.44
89.63
92.17
86.06
Average
5.635
1.94
4.96
4.87
5.61
1.66
SLERP
5.88
2.87
7.12
4.34
4.82
1.76
RegMean
4.45
0.95
5.18
0.27
8.45
5.44
Opt
5.60
0.90
5.37
5.16
6.32
5.32
Distillation
72.02
66.34
80.14
63.28
65.47
59.64
FS-M diagonal
69.53
63.98
85.88
73.10
72.11
65.68
FS-M low rank
70.86
65.23
87.92
76.24
76.35
70.96
Baselines. Our goal was merge Vision Transformers (ViTs) that were initialized differently and trained various tasks. We compared method against "average" (Wortsman et al. 2022). simple weight averaging technique; "SLERP" (Shoemake] 1985), spherical linear interpolation: RegMean (Jin et al. 2023
which offers a closed-form solution by solving linear regrossion problem for each lincar layer; "Opt" [Infeld
et al. 2023), which uses optimal transport (night 2008) to align transformers, and can be viewed as a generalization of neuron alignment methods (Ainsworth et al. 2023) because it can find soft alignments as well as hard ones; and distillation bascline (Hinton al. 2015), which trains a single VIT to mimic the features of the last representation layer of the original models.
FS-Merge. We examined three versions of our method: FS-Merge low rank, where the M and U matrices were parameterized which is as "FS-Merge low rank" but with low rank of O; in the experiments of merging groups of four and five ViTs, we also used FS-Merge seq- (Appendix B.4). with a low rank of 16.
concatenation of diagonal matrices, plus a low rank matrix; FS-Merge diagonal,
Additional results of the experiment involving the merging of pairs of ViT-B-16 can be seen Table 16 This experiment examined our ability merge models with a very low number of only 16 original images per dataset. An additional 800 images per dataset created using augmentations. FS-merge low rank used with low rank of 12.
Table 17 presents experiments on merging groups of three ViT-B-16 models, each fine-tuned on different tasks, using 100 original images per dataset and 1000 augmented images per dataset. The FS-M low-rank method was applied with a rank of 28. All merging methods were used with the hyperparametors chosen for the experiment involving the merging of groups of four ViT-B-16 models.
Additional results of the experiment involving the merging of groups of four VIT-B-16 can be seen in Table I8 and Table 19 FS-merge low rank was used with low rank of 32. In Table 20 we merged groups of five ViT-B-16, where FS-merge low rank was used with a low rank of 32. Note that solving five different tasks via merging is an extremely difficult challenge, and even ensemble struggles with it.
Table 21 Table 22 and Table 23 present additional results from the experiment involving the merging of ViT-I.-14 pairs. This experiment investigated how the merging methods scale to larger models, with FS-Merge Low Rank applied using a rank 32.
F.3 Merging Vision Transformers with different pre-trained strategies
The next section investigates the ability merging methods to merge vision transformers that used different pre-training strategics. ViT-B-16 models were pro-trained from scratch using supervisod training on the
ImageNet- 1k dataset (Deng al.) 2009) (the pre-training strategy used throughout the rest of this work).
These models then fine-tuned RESTSC45 (Cheng et al. 2017). DTD (Cimpoi et al. 2014). SVHN
33

---

Published in Transactions on Machine Learning Research (08/2025)
Table 17: Merging groups of 3 VIT-B-16 with 100 original images from the training set and 1000 augmented images from each dataset (resulting in a total of 300 original images and 3,000 augmented images). We report the per-task and joint accuracy on the test set. "Paramcters Optimized" refors to the mumber of lcarnable parameters. We will denole: Cars, C10 = CIFARIO, C100 = CIEARIO0, D DTD, = EuroSAT, G GTSRB, M MNIST, RESISC45, S SVHN.
Merging Methods
R, C100,
C100, C, M
D, M, E
Parameters Optimized
Per-task
Joint
Per-task
Joint
Per-task
Joint
Original models
91.95
90.46
87.58
Ensemble
84.28
48.49
83.92
61.67
82.71
63.02
Average
4.02
0.34
3.77
0.25
7.76
3.39
SLERP
3.88
0.69
3.69
0.08
8.84
5.11
RegMean
3.81
0.83
4.76
0.20
7.17
0.71
Opt
3.95
0.52
4.01
0.13
7.22
2.36
Distillation
65.03
52.36
59.79
49.64
79.04
76.76
111M
FS-M, diagonal
63.77
45.32
57.54
38.97
79.06
59.70
500K
FS-M, low rank
67.88
56.05
62.65
33.16
83.10
81.10
42M
Table 18: Merging groups VYT-B-16 with 100 original images from the training and 800 augmented images from each dataset (resulting in total of 400 original images and 3,200 augmented images). We report the per-task and joint accuracy on the test set. We will denote: C Cars, C10 CIFARIO, C100
CIFAR100, D = DTD, E = EuroSAT, G = GTSRB, M MNIST, R = RESISC45, S = SVHN.
Merging Methods
G, M, C100,
E, G, C10, S
R, E, C10, M
Per-task
Joint
Per-task
Joint
Per-task
Joint
Original models
95.09
97.87
97.37
Ensemble
85.98
40.31
91.97
58.64
89.41
50.56
Average
5.12
0.05
8.57
1.44
8.33
0.68
SLERP
6.20
0.33
7.81
1.12
8.69
0.43
RegMean
6.32
0.25
11.29
1.54
8.33
1.23
Opt
5.17
0.16
11.53
3.90
8.76
1.62
Distillation
76.18
42.67
44.65
40.71
86.86
75.90
FS-Merge, diagonal
68.87
34.59
79.30
67.62
87.35
72.17
FS-Merge, low rank
73.26
41.33
86.44
78.78
91.38
79.12
FS-Merge seq.
74.10
40.76
84.85
75.86
90.93
80.29
34

---

Published in Transactions on Machine Learning Research (08/2025)
Table 19: Merging groups of 4 VTT-B-16 with 100 original images from the training set and 800 augmented images from each dataset (resulting in a total of 400 original images and 3,200 augmented images). We report the per-task and joint accuracy on the test set. We will denote: C = Cars, C100 = CIFAR100, D DTD, E EuroSAT, GTSRB, M MNIST,
RESISC45, = SVHN.
Merging Methods
C, C100, R, E
D, M, C100
D,
# Parameters Optimized
Per-task
Joint
Per-lask
Joint
Per-task
Joint
Original models
91.06
86.52
89.05
Ensemble
77.56
64.78
73.72
29.98
78.58
42.25
Average
5.19
0.26
5.24
0.14
7.11
0.68
0
SLERP
3.15
0.19
5.37
0.25
7.47
0.59
0
RegMean
4.03
0.27
5.94
0.25
7.11
0.41
0
Opt
4.17
0.23
5.92
0.26
7.94
1.67
O
Distillation
59.28
53.04
66.22
36.50
53.52
45.85
111M
FS-Merge, diagonal
60.35
46.83
60.69
29.01
56.51
46.76
600K
FS-Merge, low rank
71.84
65.43
68.12
38.11
68.31
60.84
60M
FS-Merge seq.
71.71
61.21
67.75
38.14
66.19
56.64
18M
Table 20: Merging groups VYT-B-16 with 100 original images from the training and 800 augmented images from each dataset (resulting in total of 500 original images and 4,000 augmented images). We report the per-task and joint accuracy on the test set. We will denote: C = Cars, C10 CIFARIO, C100
CIFAR100, D = DTD, E = EuroSAT, G = GTSRB, M MNIST, R = RESISC45, S = SVHN.
Merging Methods
R, M, S, C10
C, M, C100, E, E, S, C, C10, G
Per-task
Joint
Per-task
Joint
Per-task
Joint
Original models
90.25
92.78
95.52
Ensemble
75.06
35.98
76.77
50.06
82.69
65.32
Average
7.22
1.65
4.63
0.37
6.57
0.10
SLERP
7.40
0.40
6.33
0.62
6.34
0.59
RegMean
7.08
1.91
4.64
0.67
9.16
1.57
Opt
7.67
0.65
3.98
1.69
9.18
3.18
Distillation
74.87
63.99
65.72
56.22
77.04
64.27
FS-Merge, diagonal
73.30
59.82
67.16
50.49
70.27
59.00
PS-Merge, low rank
77.05
60.77
75.97
65.95
80.83
65.50
PS-Merge seq.
76.82
55.30
72.80
62.51
78.24
60.36
35

---

Published in Transactions on Machine Learning Research (08/2025)
Table 21: Merging pairs of ViT-I-114 with 100 original images from the training set and 1000 augmented images from each dataset. We report the per-task and joint accuracy on the test set.
Merging Methods
DTD, EuroSAT
CIFAR100, SVHN
Cars, MNIST
Per-lask
Joint
Per-lask
Joint
Per-lask
Joint
Original models
81.33
94.68
96.50
Ensemble
77.71
71.54
94.25
77.90
96.23
96.22
Average
6.36
1.40
9.18
8.03
5.98
0.08
SLERP
5.21
1.58
5.20
2.59
8.48
5.44
RegMean
8.66
4.21
5.64
0.47
10.97
0.13
Opt
10.33
3.11
4.68
2.45
5.95
5.67
Distillation FS-M
78.51
75.84
90.91
85.78
91.82
90.58
78.60
74.68
91.78
87.67
95.80
94.31
Table 22: Merging pairs of ViT-I-144 with 100 original images from the training set and 1000 augmented images from each dataset. We report the per-task and joint accuracy on the test set.
Merging Methods
RESISC45, CIFAR10
GTSRB, RESISC45
Cars, EuroSAT
Per-task
Joint
Per-task
Joint
Per-task
Joint
Original models
96.42
95.91
95.71
Ensemble
95.52
94.12
94.05
93.35
93.07
92.95
Average
11.21
3.83
1.92
1.10
9.36
1.97
SLERP
16.14
4.48
2.42
1.43
8.23
5.71
RegMean
13.14
8.61
4.56
2.69
6.91
6.04
Opt
10.89
3.06
3.60
2.46
3.07
0.28
Distillation
84.37
82.20
82.62
80.64
89.11
86.98
F'-M diagonal
82.93
77.91
76.94
73.83
91.14
82.54
FS-M low rank
88.02
85.46
80.54
77.13
93.85
90.61
Table 23: Merging pairs of ViT-I-14 with 100 original images from the training set and 1000 augmented images from each dataset. We report the per-task and joint accuracy on the test set.
Merging Methods
RESISC45, SVHN
DTD, GTSRB
CIFAR100, EuroSAT
Per-lask
Joint
Per-task
Joint
Per-task
Joint
Original models
95.29
81.22
95.42
Ensemble
93.53
90.37
78.22
77.02
91.13
90.46
Average
8.32
1.07
2.66
1.09
9.87
0.82
SLERP
5.97
1.65
1.80
0.72
10.48
6.29
RegMean
12.95
3.32
4.83
2.98
13.14
8.61
Opt
5.14
4.26
4.03
2.95
3.54
0.52
Distillation
87.66
77.28
67.20
65.84
92.39
86.62
FS- FS-Merge diagonal
85.93
77.27
63.79
61.36
91.99
84.80
Me-Me low rank
88.11
78.32
69.09
67.14
93.56
88.20
36

---

Published in Transactions on Machine Learning Research (08/2025)
Table 24: Merging pairs of ViT-B-16 models, where the first model is pre-trained using CLIP and the second model is pre-trained ImageNet. The merge is performed using 16 original images from the training set and 800 augmented images from each dataset. We report the per-task and joint accuracy on the test set.
Merging Methods
Cars, RESISC45
GTSRB, EuroSAT
CIFARIO, DTD
Per-task
Joint
Per-task
Joint
Per-task
Joint
Original models
90.08
98.99
81.26
Ensemble
82.40
81.94
97.72
95.18
80.58
79.88
Average
1.76
1.49
6.39
2.74
6.26
4.95
SLERP
1.50
1.14
6.11
2.60
5.89
2.26
Opt
1.87
1.20
5.75
2.18
6.10
3.61
Distillation
51.10
46.62
63.96
61.26
56.36
55.48
FS-Merge
60.36
59.52
84.52
84.23
58.56
58.11
Table 25: Merging pairs of ViT-B-16 models, where the first model is pre-trained using ImageNet and the second model is pre-trained on CLIP. The merge is performed using original images from the training set and 800 augmented images from each dataset. We report the per-task and joint accuracy on the test set.
Merging Methods RESISC45, GTSRB
DTD, MNIST
SVHN CIFAR100
Per-task
Joint
Per-task
Joint
Per-task
Joint
Original models
96.19
81.86
93.83
Ensemble
92.96
91.14
83.01
69.70
82.38
72.74
Average
2.79
1.73
3.10
1.30
8.19
0.83
SLERP
2.45
1.43
6.59
1.43
8.08
0.48
Opt
1.86
1.02
4.75
1.32
7.21
0.47
Distillation
60.08
57.63
67.62
67.35
50.25
43.12
FS-Merge
64.55
62.43
73.90
70.97
51.74
40.06
Netzer et al. 2011), and EuroSAT (Helber al.) 2019). Additionally, the vision encoder from a pre-trained CLIP model (Radford et al.) 2021), which is also ViT-B-16, was used. This VIT pre-trained from scratch using contrastive learning approach that leverages pairs of similar and dissimilar image-caption pairs. Subsequently, this ViT was fine-tuned on Cars (Krause et al. 2013). GTSRB (Stalkkapp et al. 2011
MNIST LLCun 1998), CIFAR10, and CIFAR100 (Krizhevsky et al. 2009
Table presents experiments in which a CLIP pre-trained model was merged with an ImageNet pre-trained model, using the CLIP model as the "first" " initialization. Table 25 shows similar experiments, using the ImageNet pre-trained model as the "first" initialization. Note that the order of the models is important due to the "first" initialization used by both FS-Merge and distillation. For merging, 16 original images from the training set and 800 augmented images were used from each dataset (a total of 1,632 images per merge). The merged model was then evaluated on the fine-tuned datasets. FS-Merge was used with low rank of 12.
As shown, even with the new pre-training strategy, our main conclusions from the other experiments remain consistent: local and simple merging methods, such as Average and Opt, fail in this setting, while FS-Merge continues outperform distillation by a significant margin most cases.
F.4 Merging Vision Transformers fine-tuned from the same initialization
To explore FS-Merge's capabilitics, we investigate its performance the more common sotting found in merging method literature: merging models fine-tuned on different tasks from the same pre-trained initialization. We fine-tuned the ViT-B-16 model on GTSRB (Stallkamp et al. 2011), MNIST (LeCun) 1998
37

---

Published in Transactions on Machine Learning Research (08/2025)
Table 26: Merging ViT-B-16 models, fine-tuned on different tasks from the same pre-trained initialization. The merge is performed using 100 original images from the training set and 1000 augmented images from each dataset. The per-task and joint accuracy on the test set are reported. We will denote: C10 CIFAR1O, C100 CIFAR100, G GTSRB, M MNIST, R RESISC45.
Merging Methods
G, R
R. C10, G
C100, R, M, G
Per-task
Joint
Per-task
Joint
Per-task
Joint
Original models
135
96.68
94.45
Ensemble
92.66
86.85
86.15
67.88
80.40
41.91
Average
82.24
72.43
59.00
37.71
41.42
12.20
SLERP
82.23
72.42
57.08
45.68
38.76
30.82
RegMean
93.56
90.35
87.51
76.29
75.72
46.73
Distillation
91.76
90.37
90.85
86.44
82.41
63.11
FS- Merge
93.30
91.41
91.62
86.52
84.51
58.43
Table 27: Merging ViT-B-16 models, fine-tuned on SUN397, Food101 and CIFAR100. The merge is performed
using 100 original images from the training set and 1000 augmented images from each datasct. The test accuracies for individual lasks, along with the per-task and joint accuracies on the lest set, reported.
Merging Methods
SUN397
Food101
CIFAR100
Per-task accuracy Joint accuracy
Original models
65.37
81.93
85.61
77.63
Ensemble
46.91
69.13
74.96
63.66
55.61
Average
0.10
0.95
0.50
0.51
0.05
SLERP
0.15
0.95
0.74
0.61
0.32
Opt
0.12
0.90
0.65
0.55
0.24
Distillation
44.20
20.46
35.19
33.28
27.17
FS-Merge
47.71
26.43
38.87
37.67
32.82
RESISC45 CCeng et al. 2017), CIFAR1O, and CIFAR100 (Krizhevsky et al. 2009 all starting from the same pre-trained model. Subsequently, we morged groups of these finc-tuned models in varying sizes.
The results, shown in Table 26 utilize F--Merge and distillation with the "average" initialization, as this configuration yielded better accuracy on the validation set. As demonstrated, FS-Merge outperforms other note that in this sctting, FS-Merge is less dominant, its margin over other baselines, such as distillation and RegMean, is relatively small. Moreover, when the models are fine-tuned from the same pre-trained model, traditional merging methods, such RegMean, achieve performance very close to FS-Merge while using significantly fewer resources. Thus, despite its success in this scenario, we conchude that FS-Merge's true strength lies in more challenging scenarios involving models with different initializations.
basclines in most cases. However,
F.5 Merging Vision Transformers large datasets
To evaluate the merging methods on larger datasets, different pre-trained models were fine-tuned on SUN397 Xiao et al. (2010), Food101 [Bosaard et al. 2014), and CIFAR100 (Krizhevsky et al.) 2009). Combined, these three datasets contain over 290,000 images and 598 classes. This represents the most challenging merging attempt known to us in the literature under our setting (merging differently initialized transformers with only small unlabeled subset of the training data). Table 27 presents the results of merging these models using 100 original images and 100 augmented images per dataset. FS-Merge was applied with low rank of 32. As shown, FS-Merge out performs the baselines in this challenging scenario.
38

---

Published in Transactions on Machine Learning Research (08/2025)
F.6 Merging large number of Vision Transformers
To investigate the ability of FS-Merge Seq. in the extremely challenging setting of merging large number of models, we merged all our existing ViT-B-16 models, which were trained from different initializations different tasks-a total of cight models. We used a relatively small amount of data: 100 original images and 1000 augmented images per task. The models were trained on Cars, DTD, CIFAR100, GTSRB, RESISC45, MNIST, EurOSAT, and SVHN, which together comprise around 5000 classes.
As shown in Table 28 this task is difficult, even for the ensemble method, which has direct access to all eight models. Mere Seq. outperforms the baselines and even achieves higher joint accuracy than the ensemble, demonstrating its superiority.
However, we note that key challenge in this scenario is the computational cost of merging such a large number of models: FS-Merge was not evaluated due to its high memory requirements for storing the frozen weights of all models. Additionally, the KD process took around 4.0 hours, while FS-Merge Seq. required approximately 5.3 hours. In contrast, traditional merging methods execute much faster: averaging and SLERP take only few seconds, RegMean takes around minutes, and Opt requires approximately minutes. However, all of these methods are inadequate in this challenging setting, resulting in merged models that perform no better than random guessing.
Table 28: Merging eight ViT-B-16 models trained on different tasks using 100 original and 1000 augmented images per task. We report per-task and joint accuracy on the test set.
Metric
Ensemble
Average
SLERP
Opt
RegMean
Distillation
FS-Merge seq.
Per-task accuracy
70.53
4.44
5.45
4.78
5.03
63.72
68.81
Joint accuracy
42.37
0.23
0.17
0.11
0.17
42.98
46.00
F.7 Merging text Transformers
In this series of experiments, we aimed to evaluate F'-MMere on a different modality: merging differently initialized text transformers. Following (Verma & Elbayad 2024), we used five bert-base-uneased models from the MultiBERTs reproductions (Devlin et al. 2018 Seelam et al., 2022 each trained from different. random initialization and with different data ordering. We finc-tuned each model on distinct classification lask from the GLUE dalasel (Wang et al., 2019), using six tasks: RTE, QQP, MNLI, MRPC, SST-2, and QNLI. The same pre-trained BERT was fine-tuned on QQP and MRPC, so their merge was not evaluated. It is worth noting that ViT is a pro-LN Transformer, while BERT is a post-LN Transformer (Xiong et al.) 2020 Moreover, unlike the ViT experiments where models were fine-tuned with frozen classification heads derived from CTII embeddings, these experiments used trainable classification heads.
Basclines. We compared our method against two basclinos: "average" Wortsman et al. 2022). a simple weight averaging technique; and distillation (Hinton et al. 2015). which trains a single BERT to mimic the features of the last representation layer of the original models. We also reported the performance of the "original models", representing the average accuracy of the models to be merged; and ensemble (Ganaie et al.) 2022), which averages the models' outputs and then applics classification heads. Note that these last two are not valid merging methods as they use the original models directly.
Hyperparameters. We performed a hyperparameter search similar to the VIT case (Appendix E.2) We selected a pair of tasks, QQP and MNLI, and created a validation set for each task from the training set. We then fine-tuned pair of differently initialized BERT models, one for each task. These two models were used to conduct a hyperparameter scarch for both distillation and FS-Merge using the same hyperparameter grid. The hyperparameters that maximized the average per-task validation accuracy for this pair were chosen and then applied when merging the other BERT pairs. For FS-Merge, the chosen hyperparameters were 400 epochs, learning rate of 0.0001, weight decay of 0, "first" initialization, and low rank of 12. For Distillation, the chosen hyperparameters were 300 epochs, learning rate of 0.0001, weight decay of 0, and "first" initialization. Both methods used batch size of 128.
39

---

Published in Transactions on Machine Learning Research (08/2025)
200 data points were taken from each training set to create features for the merging methods. In Table 24 report the per-task test accuracy of these experiments. We can see that, similarly to the ViT case, traditional merging methods like "average" fail on the challenging task of merging text transformers from different initializations. FS-Merge achieve SOTA results, outperforming the ensemble in some cases.
Table 29: Merging pairs of BERIs with 200 original texts from the training set of each dataset. FS-Merge used with low rank of 12. We report the per-task accuracy on the test set.
Tasks
Original models
Ensemble
Average
Distillation
FS-Merge
RTE, QQP MNLI, MRPC
79.49
71.13
41.87
65.00
68.88
85.73
67.17
50.52
76.87
78.29
MRPC, QNLI
89.02
87.19
40.53
80.67
79.33
SST-2, RTE
80.15
71.64
51.06
75.45
76.00
MNLI, SST-2
88.39
84.78
43.75
82.81
83.24
RTE, QNLI
79.75
74.60
48.94
66.40
69.69
QNLI, QQP
91.01
71.63
56.32
82.40
84.75
SST-2, QNLI
91.68
90.44
49.38
82.92
84.39
MRPC, RIE
77.49
68.41
57.83
68.70
68.37
QNLI, MNLI
87.99
71.83
41.62
71.83
73.79
QQP, RTE
79.49
71.13
41.87
73.30
75.62
MRPC, MNLI
85.73
67.17
50.52
62.90
62.81
QQP, SST-2
91.41
86.59
55.48
87.49
87.94
MNLI, QQP
87.7
78.69
36.09
78.74
79.22
QNLI, MRPC
89.02
87.19
40.53
82.97
82.98
RTE, SST-2
80.15
71.64
51.06
69.20
73.39
QNLI, SST-2
91.68
90.44
49.38
87.27
88.09
RTE, MRPC
77.49
68.41
57.83
63.46
66.05
SST-2, MNLI
88.39
84.78
43.75
66.44
67.29
QQP, QNLI
91.01
71.63
56.32
82.16
82.79
F.8 Number of original training images Vision Transformer
Here present additional experiments which examine the effect of the number of original images versus augmented images on the ViT merged model. Original images, defined those sourced from the training datasets of the models to be merged, were varied in number: 16, 64, 128, 256, 512, 1024. For each scenario, augmented images were generated to bring the total images per dataset to 1024, ensuring uniform datasct size across all cases. These images were then used to create features for the merging processes.
Ensemble, Distillation, diagonal FS-Merge (low rank of 0), and FS-Merge with low rank of 24 were applied to merge pairs of ViT-B-16 models. It should be noted that Ensemble is not considered legitimate merging method. The per-task and joint accuracy on the test set were reported, employing the optimal hyperparameters identified in earlier chapters. The outcomes of this experiment are presented in Figure 5 and Figure i Note that in these experiments, different features were used compared to the old experiments, so the results may vary.
F.9 Number of training data points BERT
We examined the impact of the amount of data points taken on the merged BERT model. Training data points, sourced from the training datasets of the models to be merged, were adjusted to sizes of: 16, 64, 128, 256, 512, 1024. These samples were used to generate features for the merging process. Unlike the ViT case, we did not employ data augmentation, resulting in non-uniform datasct sizes in each case. To compensate, an additional 200 data points from each dataset were used to implement early stopping, preventing overtraining with smaller datasets.
40

---

Published in Transactions on Machine Learning Research (08/2025)
4
88
Text Accuraty (EuresAt
70
Accurary (CIFAR100,
85
80
800
Fasemale, Peotask
50
Distilation, Perzast
Acturary
FS-M Diagonal Pertask
FS-M Diegonal +-o----owwoee Per-tesk.
joint
30-
Distillatonn
FS-M Diagenal. Joint
20
FS M Diagonal Low rank joint
10a
10'
207
207
Original Images
Original Images
Original Images
Figure 5: We used Ensemble, Distillation, and FS-Merge to merge pairs of models trained on EuroSAT and Cars (left), CIFAR100 and SVHN (center), RESISC45 and EuroSAT (right). We varied the number of original images per dataset and added augmentation images so the total number of images per dataset would be 1024. We present the per-task and joint accuracy.
100
Test Accuracy (CFFAR100. Euro5ATI
90
Texs (DTD. SAYIN)
60
80
Ensemole,
Accuracy
Distilation, Pentask
FS-M Diagenal. Pertask
[S-M Diegonal tank, Persekk
15
Distilation,
06
FS-M Dtegonal Joint
PS-M Dtagonsl Low rank, jonn
10r
Original Images
Original Images
Original Images
Figure 6: We used Ensemble, Distillation, and FS-Merge to merge pairs of models trained on GTSRB and MNIST (left), DTD and SVHN (center), CIFARI00 and EuroSAT (right). We varied the number of original images per dataset and added augmentation images so the total number of images per dataset would be 1024. We present the per-task and joint accuracy.
41

---

Published in Transactions on Machine Learning Research (08/2025)
82.5
Ensemble Disiliation FS-Me
Test accuracy (ONLI MAFC)
Test Accuracy (MNLI OCP)
70
77.5
Text Accuracy
82-
70.3
03.0
16?
Figure 7: Merging of BERT model pairs trained on RIE and SST-2 (left), QNLI and MRPC (center), MNLI and QQP (right) using Ensemble, Distillation, and FS-Merge. The graph shows per-task test accuracy across varying training data sizes. Three runs were conducted, cach with different random seed, to generate error bars and verify the statistical siguificance of the results.
We applied Ensemble, Distillation, and FS- MerMe with a low rank of 12 to merge pairs of BERT base models. It important to note that Ensemble is not considered a valid merging method. We reportod per-task accuracy the test. set using optimal hyperparameters identified previously. The results of this experiment. are shown in Figures 7
G Ablation studies
G.1 FS-Merge ablation study
Table 30 shows an ablation study of merging a group three ViT-B-16 using FS-Merge. This study involved three groups of models. We evaluated both the per-task average accuracy and joint accuracy on the test set.
We first used FS-Meeree diagonal, where the M,U were parameterized as concatenation of diagonal matrices. We used random initialization, and created features using only the original 100 images from each dataset. without augmentations (so total number of 300 images). Then, 1000 more augmented images were added for cach dataset. Then, "average initialization" for the M. was tested, meaning the Foldable SuperNet initialized the merged model from the average of the original ones; and also first initialization was tested, so the Foldable SuperNet initialized the merged model from the first original model. "Low rank" stand for FS-Merge with M, U parametrized as a concatenation of diagonal matrices plus low-rank matrices (Eq. 7). with low rank of 24.
As can be observed, the "first" initialization leads to better merged model compared to the average random initialization. Additionally, this experiment shows the significance of creating more images through augmentations and using low rank in the M, U matrices of the Foldable SuperNet.
Table 31 displays the per-task accuracy for each task of the merged model across RESISC45, CIFAR10, and EuroSAT: . can be seen that the "first" initialization improves not only the first task's accuracy but also the accuracy across all tasks. This behavior recurs in every other group of models that we merged.
Our experiments suggest that in the VIT case, initialization is extremely important for training the Foldable SuperNet. When initialized randomly, FS-Merge does not converge, even with the addition of augmented data (and see Appendix G.4 discussing this issue). Only Foldable SuperNet initialized from the average first model allows it to converge into a successful merged model.
G.2 Knowledge Distillation ablation study
Table 32 presents an ablation study on merging group of three VIT-B-16 models using distillation. This study involved the same three groups of models used in the FS-Merge ablation study (Table 30 We followed
42

---

Published in Transactions on Machine Learning Research (08/2025)
Table 30: Ablation study comparing the effectiveness of different initialization and augmentation strategies
the merging of groups of three ViT-B-16. Only our method, FS-Merge, the test set. "Aug" stands for Augmentations, and "Init" stands for Initialization. We will denote: C = Cars, C10 = CIFAR1O, C100 = CIFAR100, D = DTD, E = EuroSAT, G = GTSRB, M =
used. We show the per-task
and joint accuracy
MNIST, R = RESISC45, $ = SVHN.
Setting
R, C10, E
S, E,
C10, G,
FS-Merge
Init
Aug
Per-task
Joint
Per-task
Joint
Per-task
Joint
Diagonal
Random
X
7.43
0.77
11.05
1.18
7.71
1.18
Diagonal
Average
X
25.87
10.15
21.42
12.19
14.71
4.25
Diagonal
First
X
76.75
55.44
66.87
50.20
67.74
51.38
Low rank
First
X
83.40
73.76
77.48
65.69
80.77
64.93
Diagonal
First
86.04
73.47
82.15
71.18
76.45
57.13
Low rank
First
89.17
79.34
85.96
75.38
86.41
66.84
Table 31: Merging three VIT-B-16 models, fine-tuned RESISC45, CIFAR10 and EuroSAT, using diagonal FS-Merge. We are examining the effects of initialization and augmentation on the per-task accuracy of the lest set.
FS-Merge details
Original tasks
Initialization
Angmentations
Average Acc
RESISC45
CIFAR10
EuroSAT
Average
X
25.87
3.73
22.94
50.94
First
76.75
89.4
56.24
84.62
Average
37.51
7.3
33.59
71.64
First
86.04
87.86
79.76
90.52
Appendix G.1 and created features using 100 original images and 1000 augmented images from each dataset (resulting in total of 3,300 images).
We aimed to investigate how different initializations affect the performance of the distillation merge, also examining traditional merging baselines as initializations (such as average and RegMean). Additionally, we explored the impact augmentations, We evaluated both the per-task average accuracy and the joint accuracy on the test set.
Table 32: Ablation study comparing the effectiveness of different initialization and augmentation strategies on the merging of groups of three ViT-B-16. Only distillation merge was used. We show the per-task and joint accuracy on the test set. We will denote: C = Cars, C10 = CIFAR1O, C100 = CIFAR100, D DTD, E
EuroSAT, G GTSRB, M MNIST, R RESISC45,
SVHN.
Distillation
R, C10, E
S. E, G
C10, G, M
Initializatinn
Augmentations
Per-task
Joint
Per-task
Joint
Per-task
Joint
Random
33.20
25.93
29.00
25.75
27.56
19.77
Random
39.64
30.83
34.39
30.13
33.17
24.69
Average
X
38.16
30.22
37.78
33.49
37.40
25.68
Average
41.78
32.76
55.35
46.20
52.84
39.54
RegMean
49.44
39.61
66.50
55.54
57.80
46.31
RegMean
58.53
47.84
77.18
66.42
72.55
57.42
First
81.56
71.32
77.24
65.02
78.38
67.76
First
84.14
73.98
84.46
73.19
84.30
70.92
43

---

Published in Transactions on Machine Learning Research (08/2025)
Table 33: Comparing Distillation and FS-Merge (both with "first" init), with and without augmentations,
while morging groups of three ViT-B-16. Features were created using 100 original images and 1,000 augmented images per dataset. We show the per-task and joint accuracy on the lest set. We will denote: C = Cars, C10
CIFAR1O, C100 CIFAR100, D DTD, E EuroAAT, G GTSRB, M MNIST, R RESISC45, S
= SVHN.
Setting
R, C10, E
S, E, G
C10, G, M
Method
Aug
Per-task
Joint
Per-task
Joint
Per-task
Joint
Distillation
X
81.56
71.32
77.24
65.02
78.38
67.76
FS-Merge, Low rank
X
83.40
73.76
77.48
65.69
80.77
64.93
Distillation
84.14
73.98
84.46
73.19
84.30
70.92
FS-Merge, Low rank
89.17
79.34
85.96
75.38
86.41
66.84
As observed, the "first" initialization leads to superior merged model compared to all other initializa- tions, including other merging basclines such as average and RegMean. Morcover, augmentations enhance performance in all cases.
In Table 33 we compare the best distillation and FS-Merge versions from the ablation studies (i.e., with "first" initialization), showing that FS-Merge outperforms distillation with and without augmentations.
G.3 Merging Vision Transformers: Local VS. Global perspective
Observe that in the MLP merging method from Section 2.1 the local version of FS-Merge was used, which involves training a Foldable SuperNet individually for each layer. In the ViT case, local FS- Merge involves training each block of the Foldable SuperNet separately. For example, training a Foldable SuperNet that merges the first atlention blocks, then training a Foldable SuperNet that merges the first MLP blocks, and so on. We found empirically that this approach leads to a poor solution in the VIT case, resulting in a dysfunctional merged model with accuracy nearly as poor as a random guess. Instead, we found that the global version of FS-Merge much more effective in this case, involving training the entire Foldable SuperNet of the ViT to reconstruct the features of the last representation layer of the original models, fL.
A few explanations exist for this issue. First, training the Foldable SuperNet in local manner for the VIT, meaning block-wise, must be performed on very unnatural blocks, which "break" the transformer blocks. This is necessary because M and U matrices must be placed before or after a linear layer to allow them to be folded after training. Moreover, the attention score computation, layer normalization, and skip connections must be performed on the merged features (with the lower dimensionality)). These conditions forced the "breaking" of existing ViT blocks, and, for example, required teaching the Foldable SuperNet of the attention block to reconstruct fcatures that are within the next MLP blocks. This complicated structure probably have hindered the optimization process.
Second, the Foldable SuperNet consistently uses the merged features in the skip connection. This means that when learning block-wise, the features forwarded via skip connection to the next block are dramatically changed. These new merged features are very different from the original ones, which likely severely affected the optimization of the next Foldable SuperNet block.
G.4 Merging Vision Transformer with randomly initialized Foldable SuperNet
In our experiments, we found that smart initialization of the Foldable SuperNet is crucial for FS-Merge. As common in NNs, we first tried to initialize the M, U matrices using a random Gaussian distribution dependent on the hidden dimension (He et al. 2015)..
In the MLP case, a random initialization can work, but better results are achieved when using ZipIt as the initialization method for the Foldable SuperNet (and see Section 3.1.. In the ViT case, We tested multiple
44

---

Published in Transactions on Machine Learning Research (08/2025)
scales for the random initialization, but could not find setup that allowed the FS-Merge converge into functional merged model.
This led us to study smarter initializations, such as "average". When merging K models, a Foldable SuperNet that creates an average merge the weights is achieved by setting all the M, matrices as follows:
K U U-
M
1)
K
When 7 is the identity matrix. In the case of ViTs, the "first" initialization proved to be the most effective, involving initializing the Foldable SuperNet so it exclusively selects the weights of the first model. It can be achieved by setting all the M, U matrices as follows:
U = (1 1)
By the end the training, the "first" initialization results in a merged model with improved accuracy across all tasks, not just the task of the first model (see Appendix G.1 for more details). Surprisingly, averaging initialization also performed well, despite the fact that the average merge is not an effective merging method when combining ViTs trained from different initializations.
This effectiveness is the reason the Foldable SuperNet's M, matrices were modeled as sum of low-rank matrices plus a concatenation of diagonal matrices (rather than just a low-rank matrix in LoRa (Hu et al., 2022
The concatenation of diagonal matrices enables the initialization of the M, U matrices using those successful methods ("first", "average").
G.5
Using inner features when merging Vision Transformers
In line with several distillation studies Wu et al. 2021a Zagoruyko & Komodakis 2017 Heo et al. 2019b aj
Park & Kwak, 2019 Liu al. 2020 Wu et al. features of the models to be merged as regularization for FS-Merge and for distillation. We focused on the features obtained after the MLP block or after the atlention block. The attention features in the block of model k, created from the imput ting. can be written as fhan (4inn) E 7'xxd, Then, the "inner loss" for the global Fe-Merge, when handling two tasks A and B, can be defined as follows:
2021b
2023 Han et al. 2024). tried to use the inner
== Toout + A Σ
J DD || (Timu) - fr,ate [
IEC
E 1mg
fl.aee (T [B]12²
[
Where Lout is the regular global reconstruction loss, which attempts to reconstruct the features of the two original models from the layer preceding the classification head (Appendix B.2 f.au(() are the reconstructed attention features of our Foldable SuperNet block for model k. is the set of blocks from which we decided to extract features, and the regularization coefficient. D and DB defined as small subsets of data from the training data of tasks A and task B respectively.
This can casily be rewritten for distillation, only that for each k we compare fhan with flatt. which are the inner features of the student model in the matching block. Note that in the distillation method, this regularization forces the inner features flat to resemble the inner features of all the models to merged (in our example, both A and B). In contrast, FS-Merge, after applying the U matrix, the reconstruction of features from both models and B are obtained, and each of these features will be compared with its corresponding ground truth.
45

---

Published in Transactions on Machine Learning Research (08/2025)
Table 34: The effect of inner loss on distillation, FS-Merge low rank, and FS-Merge full rank is demonstrated. Pairs of ViT-B-16, fine-tuned RESISC45 and CIFARIO, were merged using 100 original images and 1000 augmented images per task. The per-task accuracy the test set is presented.
Method
Inner loss details
Per-task Accuracy
Distillation
86.41
Distillation
A = 0.1, (5)
83.22
FS-Merge, rank 12 FS-Merge, rank 12 À = 0.1, (5)
=
89.78
86.39
FS-Merge, full rank
No regularization
40.20
FS-Merge, full rank
X = = {5}
52.77
FS-Merge, full rank
À 0.1, [3,5,7,9]
60.41
FS- TS-Merge, full rank
0.5, n = (3,5,
70.52
FS-Merge, full rank
1, = (3,5,7.9)
70.11
We tried multiple À values, and various C' sets, for features from the MLP or attention blocks. Yet, it seems to only detriment the performance of FS-Merge and distillation. Table shows some of those experiments, merging pairs of ViT-B-16, fine-tuned on RESISC45 and CIFARIO, using 100 original images and 1000 augmented images per task.
It should be observed that specifically in the case of FS-Merge using full rank M and U matrices, the inner loss seems to improve the results. We conclude this because, in the full M, U case, there is very large number of learnable parameters, the ViT case is not recommended due to the very high memory and time complexity, and even with this regularization, it underperforms compared to Low rank FS-Merge.
stronger regularization is needed. However, using full rank M, U matrices in
46

---

Published in Transactions on Machine Learning Research (08/2025)
H FS-Merge expressive power
As shown in Section 3.2 Section 3.3 and Appendix F.2 FS-Merge significantly outperforms traditional
merging methods, such as weight averaging and RegMean, when merging transformers trained from different initializations and tasks. order to explain this, we present theoretical results that highlight FS-Merge's superior expressive power compared to the baselines.
We examine the case of merging pair of Multi-Layer Perceptron (MLP) models, A and B. Let us consider the 1-th layer in the MLP model A. The features at this layer, denoted by S RRxN,, can be expressed follows:
(8) Here, W& E Rdxd are the weights of the current linear layer, d denotes the width, N denotes the number of data points, a represents the ReLU activation function (a(x) = max(0,x)), and 2f E RdxN are the pre-activation features. And similarly for model B.
x/ = ( o(
Definition H.1. A matrix A E RRd. is said to have rank S d1, d2 if the dimension of its column space (or equivalently, its row space) is r. It implies that the maximum number of linearly independent columns (or rows) in A is r.
Definition H.2. Let x be a data point sampled from the distribution D, and let fi())) denote the features of the 1-th layer an MLP computed from these inputs. We define the Teatures fi as having a low rank of (z) has a rank of at most for any x D.
We will assume low-rank features in several of the following sections, it has been found that neural network features are often effectively low-rank Cai et al. (202I); Yu & Wu (2023). Such low-rank features imply that it is possible to use low-rank weights to achieve the same function (i.c., the input-output relation) of the model, even if the original weights are high-rank. In other words, if fi is a set of features with rank 9, even if Wi is full rank, we can find a different matrix Wi of rank r such that
Wifi Wifi.
This holds because the action of linear transformation (such as Wi) subspace can always be reproducod by another transformation of the same rank as the subspace.
Our main theoretical result, presented in Theorem H.3 demonstrates the significant expressive power of both full-rank and low-rank FS-Merge, surpassing that of the two popular merging methods, Git Ro-Basin Ainsworth et al. (2023) and RegMean Jin el. al. (2023
Theorem H.3. A summary of our theoretical resulls:
1. Given two MLP models, A and B, where at each layer at least one has an invertàble weight matria, a full-rank FS-Merge con reconstruct any target MLP model of the same size from these models. The same holds for a low-rank FS-Merge, assuming that the target MLP features are low-rank.
Any merged MLP obtained using the RegMean or Git Re-Basin merying methods can also be constructed using full-rank FS- FS-Merge, even without assuming inverlible weights. Moreover, there are scenarios that FS-Merge can solve which RegMean Git Re-Basin cannot. The some holds for low-rank FS-Merge, given the additional assumption of low-rank features.
As shown in Theorem full-rank FS-Merge has greater expressiveness compared to low-rank FS-Merge. However, we argue that in the context of merging large models with a limited amount of data, low-rank FS-Merge is preferable, as it introduces fewer learnable parameters and thus enhances generalization. This claim is supported by our experiments, which show that low-rank FS-Merge outperforms full-rank FS-Merge (Appendix G.5 and Knowlodge Distillation (Appendix 3.2 when merging transformers.
Proof. Part 1 of Theorem H.3 is proved in Appendix H. below.
Part 2 of Theorem H.3 follows from Proposition H.4 and Lemma H.5
which are proven in Appendix H.2 and
Appendix H. respectively.
47

---

Published in Transactions on Machine Learning Research (08/2025)
Proposition H.4. Given the linear layers of two models, A and B:
Any merged model obtained using RegMean or Git Re-Basin can also be constructed using full-ronk FS- MerMeree.
2. Assuming that the features of at least one model have low rank of then for any solution created by Git Re- Re-Basin, low-rank FS-Merge with rank of can create equivalent solution.
3. Assuming that the features of both models have a low rank of r, any merged model obtained using RegMean can also be constructed using low-runk FS-Merge with rank of 2r.
Lemma H.5. Consider the scenario of two linear layers that share the same. backbone weight matris, with each applying a different feature selection, permutation, and scaling matrix. Then, full-rank and low-rank FS-Merge merge these layers into single layer in manner that allows them to reconstruct the original Seatures from the merged one. In contrast, RegMean and Gil Re-Basin cannol perfectly merge these two layers.
H.1 Proof of Part of Theorem H.3: FS-Merge expressive power
The Foldable SuperNet for two linear layers is defined as follows:
M = (M )|
(10)
where the input is a concatenation of the original pre-activation features =/ = e 1R2d and f E R2-dxN are the features reconstruction attempt. Intuitively, FS-Merge learns to compress and reconstruct the features of two different models, while also taking into account the activation function.
In the full rank vesion of FS-Merge, Mi Rdx2-d and Ui R2-dxd are the learnable parameters of the Foldable SuperNet. In the low rank version of FS-Merge, the Mi and Uj matrices are parametrizod as of low-rank matrix and concatenation of diagonal matrices. For example, in the case of Mr:
Mi M,lian + M2 M²
(11)
When r is the rank, M/ E Rdxr, M2 E Rrx2d, and Mf E pdx22d is concatenation of two diagonal matrices, each with d learnable parameters. A similar structure is proposed for U. In both version of FS-Merge, the merged layer is created as follows:
WA
0 WBB
Wi Wi= = Mi
U-1
(12)
0
Next, we wish to prove part 1 of Theorem H.3
Proof. Let / be an arbitrary layer in the MLPs. Given two linear weight matrices W.A, W.B E Rdxd we will assume without loss of generality that W,A is invertible. Let Wr denote the target MLP weights. In the case where the input features for the weight W are full rank, full-rank FS-Merge capable of reconstructing the target weights using:
1
M (WT(WA))
(13)
As the merged weight will be
W.A
(1) = W
W, = (W/(W
0)
(14)
WBB
1
48

---

Published in Transactions on Machine Learning Research (08/2025)
In the case where the imput features for the weight W? have low rank of r, there exists equivalent weight W with rank that can generate the same features (i.e. Wi Wi ft). In this case, low-rank FS-Merge with rank r can reconstruct these equivalent weights using a similar solution:
Mi = (WT(WA)- Ul U (
(15)
This solution can be achieved using a low-rank FS-Merge, as Uj =(1 can be reconstructed using its diagonal component Udias, Additionally, the matrix (WIAW))- 0) has rank lower, meaning it be reconstructed using the low-rank component M1 Mt.
This result happens generically, since we only require that for each layer, one of the models (either A or B) has full-rank weight matrix. This happens almost surely with respect to the Lebesgue measure, as almost. all randomly chosen weight matrices will be full rank (i.e., singular matrices have measure zero).
H.2 Proof of Proposition H.4 FS-Merge expressiveness compared to RegMean and Git Re-Basin
Here aim to demonstrate that full-rank FS-Merge generate any solution produced by the popular merging methods Git Re-Basin Ainsworth el al. (2023) and RegMean Jin et al. (2023), even without assuming an invertible weight matrix. The same applies to low-rank FS-Merge, with few additional assumptions.
Git Ro-Basin Ainsworth et al. 2023 an alignment- based merging method that aligns lincar layers using a permulation matrix before averaging them. It defines the activation matching objective for the I-th layer as:
Pt argmin
|| P.
(16)
PiESa
Where Sa the of all permutation matrices size x d. A permutation matrix a square binary matrix that has exactly one entry of in each row and each column, with all other entries being 0. It then merges the weights as follows:
Wi = ク W. + PRWPPI...
(17)
Intuitively, Git Re-Basin searches for a permutation matrix Pt Sa that makes the features of the linear layers fi and ff similar possible (in terms of the Frobenius norm). Applying this permutation to the lincar layer weights W.8 is equivalent to rearranging the neurons in the layer so that neurons generating similar features are aligned in the same positions.
RegMean Jin et al. 2023 is a strong averaging-baat merging method, targeting the following objective:
W = argmin - || - 1/
(18)
Wi
which has the next closed form solution:
W7 = ( +1L02) +12,0
(19)
Next, we wish to prove Proposition
H.4
Proof. Given that Git Re-Basin converged to permutation matrix Pt, full-rank FS-Merge produce the same solution using
Mi = ( (P). U-1 = (1/)) (PI.)
(20)
49

---

Published in Transactions on Machine Learning Research (08/2025)
By additionally assuming that the features fl have low rank of r, there exist projection matrices Ct Ci E Raxd, both of rank r, such that:
P.f = -PCC.f.
(21)
And similarly for f Thus, a low-rank FS-Merge can reconstruct the next solution:
M 1 (CP(C) UnI ((LL)-1C)))
(22)
Which is equivalent the Git Re-Basin solution. This valid solution, as the identity matrices can be reconstructed using the diagonal components U Mp while PCC are matrices with rank lower.
In addition, full-rank FS-Merge can produce the same solution as Reg.Maa using
(EUD))) UL1 (1)
Mi (AAWAA + (LU0E)
(23)
By additionally assuming that the features fi f have low rank of r, low-rank FS-Merge with rank 2r can also produce the solution in Eq. 23 This is because U1-1 can be reconstructed using its diagonal component U and Mt is a matrix with rank 2r or lower.
H.3 Proof Lemma H.5) Expressive power case study
We examine a simple scenario where traditional merging methods fail due their reliance on basic merging rules. In contrast, we prove here that both full-rank and low-rank Fe-Merge can handle this scenario.
In the next section, we wish to prove Lemma H.5
In more detail, scenario (Eq. 24 involves two linear models, A and B, which generate features for two different tasks. We assume that both models share the same backbone weight matrix, W* E Rdxd, and that X E R represents the inputs. Each model then selects subset of shared features using VA/B E Rdxd which are diagonal matrices with 0/1 entries, each having a rank of 5 The two matrices do not have 1s in the same positions. Next, the two models apply a permutation matrix pA/B gdxd and diagonal matrix with positive entries that scale the fcatures, DA/B E Rdxd, The ReLU function a is used to generate the post-activation features f T E BRdN which are later used by the classification head to make predictions.
A = A= = = B=D =
(24)
We will also denote
= (VA B W x, f =0(2")
(25)
Next, we prove Lemma H.5 in Appendix H.3.1 Appendix H.3.2 and Appendix H.3.3 demonstrating that low-rank FS-Merge can achieve zero-loss solution, while Git Ro-Basin and RegMean cannot.
H.3.1 FS-Merge
Low-rank FS-Merge is capable of reconstructing f*, using
=() U U= ))
M = ((A))
(26)
50

---

Published in Transactions on Machine Learning Research (08/2025)
As the obtained features will be:
j = o=) =o
-00MMM W' 0 WB
[UX]==
UX
((DA)) (DB)-()))-1) (DApAYAW* How Depaveww) (+)x) o(2)) = f (27)
Note that Eq. 26 a valid solution, M has rank of most r, and is a concatenation of diagonal matrices, and therefore, they can be parameterized by low-rank FS-Merge with rank of
Moreover, the low-rank Foldable SuperNet perfectly reconstruct the features of both linear layers, achieving zero loss, using
M = (VA(PA))((A))) (D U (DApAyA)
(28)
Note that the operation of DA/B and a is commutative, and thus
=A= DAAAAA
(29)
The same holds for p. Using it, the Foldable SuperNet output is
=0= = Uo (M (M(2))) (34) (B) (=a))= (M(2²)
= (DAPAyA) DBBBBB) a (DA)))) (D(D))
(pApaTAw*))
(Dapryew-x)
DAPAVA -DSPBVB)) a ((V4 VB) (veW*)) DBBBBB o(z") (DSpayeT.) 1-1 (30)
Moreover, this is a valid solution, as both M and U from Eq. 28 are matrices with rank of at most and therefore, they can be parameterized by low-rank FS-Merge with rank of r.
We note that full-rank FS-Merge can generate any solution obtained by low-rank FS-Merge since can use matrices of any rank. Therefore, full-rank F'S-Merge capable of handling these scenarios as well.
H.3.2 RegMean
Under our assumptions, RegMean's 's Jin et al. 2023 objective is:
W argmin |W DAPAVAW x |WX - D
(31)
Which uses the Frobenius norm. For RegMean to achieve zero loss, it is required that WX DAPAYAW** X. Thus
W = =
(32)
Where we denote t as the Penrose-Moore pseudo-invens inverse, Similarly, to achieve zero error, we must also ensure that W D BBw.BBBBBBBXXX xxt. Using both results we obtain:
(33)
Note that Eq. 33 is a set of rational equations, and therefore it can be transformed to set of polynomial equations. These polynomials are not identically zero, due to the assumption r >>> 0. Therefore, the set of solutions of these equations a zero-measure set (under the Lebesgue measure). This means the chances of randomly selecting parameters (from some continuous distribution) that satisfy this equation are zero. Therefore, in general, RegMean cannot find parameters that satisfy this condition, so lacks the expressive power to handle the case being studied.
51

---

Published in Transactions on Machine Learning Research (08/2025)
H.3.3 Git Re-Basin
Under our assumptions, Git Rasin'sn'
Ainsworth et al. (023 objective is:
P argmin PES.
- PD PD
(34)
For RegMean achieve zero loss, it is required that
BB DAPAv - BB fr = 0.
(35)
Eq. 35 is a polynomial, and thus we can apply the same reasoning as in the previous section, leading to the conclusion that. the probability of this equation holding general is zero. Therefore, Git Re-Basin lacks the expressiveness needed to handle this case.
52


Reviews:
[{"review_id": "dB25c7l8XQ", "reviewer": "Reviewer_kq3s", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThe motivation of this paper is sound, the paper experimentally shows that current approaches are insufficient.\nThe experiments are mostly comprehensive, see weakness for additional comments.\n\nWeakness:\n1. The implementation of KD needs further clarification. For example, is the KD here supervises only the final layer. Currently, there are lots of approaches that exceed the final layer supervision (e.g., intermediate layer/feature supervision [1-3]. They should be in detailed discussed and highlight the novelty of the proposed method). Some of them are specifically designed for Transformer-based architectures.\n2. I notice a recent paper mentioning language-based Transformer fusion [4], please highlight the technical differences between them.\n\n[1] Universal-kd: Attention-based output-grounded intermediate layer knowledge distillation\n\n[2] Ad-kd: Attribution-driven knowledge distillation for language model compression\n\n[3] AMD: Automatic Multi-step Distillation of Large-scale Vision Models\n\n[4] Knowledge Fusion of Large Language Models", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This is an interesting work. It targets a harder problem of merging large Transformer-based models trained on different tasks from distinct initializations. Traditional merging suffers from catastrophic forgetting. Knowledge distillation achieves much better results with the price of a higher cost. In light of this view, the authors propose FS-Merge.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see above weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IGjBx4TlHs", "reviewer": "Reviewer_pvuc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- FS-Merge is well-motivated, with clear architectural design choices for both MLPs and transformers. The use of a feature reconstruction loss and learnable compression/reconstruction modules is elegant and justified.\n- The paper includes a meaningful theoretical result showing that FS-Merge generalizes and subsumes existing methods (e.g., RegMean, Git Re-Basin) under appropriate assumptions. This enhances the credibility of the method's expressivity claims.\n- The experiments are extensive and cover multiple settings: MLPs on MNIST/Fashion-MNIST, ViTs on nine image classification datasets, and BERT on GLUE tasks. \n- FS-Merge Seq. is a thoughtful engineering adaptation that addresses the growing resource cost when merging many large models. This improves the method’s applicability in real-world systems.\n- Unlike prior work that assumes a common pre-trained initialization, FS-Merge handles models trained from entirely distinct random seeds and fine-tuned on different tasks.\n- To keep the number of learnable parameters linear in layer width (rather than quadratic), the paper adopts a LoRA-style decomposition (low-rank plus diagonal) for the merging matrices.\n# Weaknesses\n\n1. **High computational and memory requirements:** Despite being more expressive, FS-Merge (and even KD) is significantly more expensive than simple averaging or alignment methods. Although the sequential variant (FS-Merge seq.) alleviates this somewhat, it still incurs substantial resource use and introduces a small accuracy drop (Table 7).\n\n2. **Dependence on availability of unlabeled data**: FS-Merge requires access to “a small unlabeled subset of the original training data,” which may not be available in privacy-sensitive or proprietary settings—undermining its applicability when no data can be shared\n\n3. **Architecture constraints (identical depths/widths only)**: The method only handles models of the same architecture and width. “One cannot naively merge two models of different depths using our method,” and no solution is provided. Thre  authors list this as future work, yet many practical cases (e.g., ViT-B with ViT-S) are depth-mismatched.\n\n4. **Lack of theoretical guarantees for transformers**: The expressive-power result (Theorem 2.1) is stated only for MLPs (“FS-Merge can exactly reproduce any target MLP model…”), with no formal counterpart for transformers—leaving its theoretical foundations in the transformer setting unaddressed\n\n5. **Limited evaluation scope**: Experiments focus on small-scale classification splits (e.g., MNIST halves, 9 vision tasks with ViT-B/16, GLUE pairs). There is no evaluation on larger, more realistic multi-task benchmarks (e.g., many simultaneous tasks or production-scale multitask settings). Further, the paper compares only to simple averaging, SLERP, RegMean, “Opt,” and KD. It omits several recent, advanced alignment or merging techniques (e.g., REPAIR, equivariant weight alignment) that could be competitive (Related Work, Appendix A.1)\n\n6. **Hyper-parameter tuning is heavy** and carried out on validation data, risking unfair advantage over baselines. Grid search over 40+ settings per method (detailed in Appendix E.2) versus single default settings for others like SLERP/average.\n\n7. **Hyperparameter sensitivity and tuning complexity**:  FS-Merge introduces many hyperparameters (learning rates, ranks r, initialization schemes), but the main text offers almost no guidance on their selection or sensitivity; all details are relegated to the appendix (Appendix E).\n\n8. **Sequential merging trade-offs underexplored**: While FS-Merge seq. reduces memory, it is only briefly described and empirically shown to incur a minor accuracy drop. The paper does not analyze how the order of sequential merges affects final performance or whether better schedules exist (§2.2 and Appendix B.4).\n\n9. **Task Head Assumption**: FS-Merge assumes task-specific classification heads remain separate, which simplifies merging but and may inflate reported gains. Further, it limits end-to-end deployment for some multitask systems (real multi-task setups normally train or merge the heads). A discussion on merging or adapting classification heads would be valuable.\n\n10. **High Cost of Full-Rank M and U** While theoretically more expressive, using full-rank M and U matrices in the ViT case is not recommended due to very high memory and time complexity.\n\n11. **Simplicity of the Method for Transformers**: While the authors state \"FS-Merge is simple\", and the core concept is indeed straightforward, its application to complex architectures like transformers (Figure 3, Appendix B.1 ) is quite involved with multiple M and U matrices at different stages of the attention and MLP blocks. \n\nIn summary, most prominent messages in this paper: simplicity, SOTA status, theoretical guarantee for transformers, and robustness/generalisation advantages—are not backed by sufficiently comprehensive or clear evidence. Additional experiments, broader baselines, quantitative cost analysis and an extended proof are required for the evidence package to be convincing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work addresses the challenging task of merging transformer models trained on different tasks with distinct initializations, where traditional merging methods often fail. The authors propose **Foldable SuperNet Merge (FS-Merge)**, a novel technique that embeds the original models with frozen weights into a SuperNet, which is trained using a feature reconstruction loss and then folded into a single model of standard size. FS-Merge is extended to support the complexity of transformers and is further scaled through FS-Merge Seq., a sequential variant that reduces computational overhead. The method is theoretically shown to be more expressive than prior approaches and achieves superior empirical performance across MLPs, Vision Transformers, and Text Transformers, particularly in low-data and multi-task scenarios.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. **Add missing state-of-the-art baselines (Critical)**: Compare FS-Merge to 2024 weight-merging / PEFT methods such as TIES-Merging, AdaMerging, Task-Arithmetic 2.0, BitFit/Adapter fusion and Rank-Free SLERP so the SOTA claim is substantiated.\n\n2. **Clarify or relax the “no-data” assumption (Critical)**:  State explicitly that FS-Merge needs *unlabeled* task data **or** demonstrate a variant that works with zero data / public proxy data to align with the stated motivation.\n\n3. **Extend the theoretical guarantee to transformers (Critical)**: Either prove expressivity/convergence for attention blocks or clearly bound the limitation to MLPs.\n\n4. **Equalise hyper-parameter budgets across methods (Critical)**: Cap search trials identically (or use a common auto-tuner) and document settings in the main text.\n\n5.  **Scale experiments to 10 + merged models (Would strengthen)**: Merge a larger set of vision or language tasks and plot accuracy versus number of models.\n\n6. **Broaden robustness testing (Would strengthen)**: Add ImageNet-C, Stylized-ImageNet and language perturbation suites; compare to all baselines.\n\n7.  **Use trainable or merged classification heads (Would strengthen)**: Replace frozen CLIP label embeddings in at least one experiment; discuss the effect.\n\n8. **Address depth-mismatched models (Would strengthen)**: Provide possible solution or emphasise this limitation in the conclusion.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TUV8FjCEaE", "reviewer": "Reviewer_JgDq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths:\n\n1. FS-Merge addresses a significant gap in model merging by making it possible to merge models from different initializations, which is a practical and challenging problem.\n\n2. The method is computationally efficient and data-efficient and therefore can be extended to real-world applications where large training data availability is scarce.\n\n3. The paper provides strong empirical evidence by performing experiments on a wide range of neural architectures and tasks, showing the efficacy of the proposed method.\n\n4. The theoretical analysis is sound and forms a good foundation for the technique, showing that FS-Merge is more expressive than traditional methods. \n\n5. The flexibility of the suggested method in handling different model architectures (e.g., MLPs, ViTs, and BERTs) is a strong point. \n\n6. The thorough comparison with baseline methods, including averaging, alignment, and distillation-based methods, is comprehensive and easy to understand.\n\n# Weaknesses\n\n1. The scalability of the method to a large number of models is discussed but not fully evaluated in large-scale settings.\n\n2. Although FS-Merge outperforms KD and standard merging methods, computational costs in merging large sets of models can be further investigated. \n\n3. The performance may be affected by the initialization method of FS-Merge, and a more thorough ablation on initialization methods would be beneficial.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel and efficient technique for merging neural networks with various initializations and tasks, referred to as Foldable SuperNets (FS-Merge). FS-Merge aims to merge individually trained massive transformer models as its primary objective, exceeding the limitation of existing methods based on shared initialization. The paper addresses the challenging scenario where models trained under various tasks from diverse initializations must be merged effectively without any performance loss. The new FS-Merge method does this by creating a Foldable SuperNet that wraps the original models with frozen weights and trains a feature reconstruction goal to best combine them. The combined model is then folded back down to the original model size, maintaining computational efficiency comparable to Knowledge Distillation (KD) while improving expressiveness and data efficiency by a significant margin.\n\nThe key contributions are:\n\n1. Introduction of the FS-Merge framework to merge transformers with different initializations and tasks efficiently.\n\n2. Creation of the Foldable SuperNet method that inserts models with frozen weights and trains using a reconstruction loss.\n\n3. Empirical validation on a variety of architectures, sizes, tasks, and modalities, with SOTA performance, especially in low-data regimes.\n\n4. Theoretical validation of FS-Merge being more expressive than conventional merging techniques.\n\n5. Demonstration of robustness in diverse settings, including vision and text transformers, highlighting the method's adaptability.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Expand the scalability analysis to include combining larger quantities of models, especially in practical settings where multiple models are combined.\n\n2. Add an ablation study on initialization methods to see how they impact combining quality more profoundly.\n\n3. Offer more understandable descriptions during the theoretical sections to enhance content readability by readers with diverse mathematical backgrounds.\n\n4. Compare the computational cost of FS-Merge with other merging methods when expanding the number of models.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)