Model merging with SVD to tie the Knots
Authors: George Stoica
Pratik Ramesh
Boglarka Ecsedi
Leshem Choshen
Judy Hoffman
Venue: ICLR 2025 Poster (2025)
Keywords: model merging; lora PEFT; computer vision;
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We align LoRA PEFT models to improve the application of existing merging methods.

Abstract: Recent model merging methods demonstrate that the parameters of fully-finetuned models specializing in distinct tasks can be combined into one model capable of solving all tasks without retraining. Yet, this success does not transfer well when merging LoRA finetuned models. We study this phenomenon and observe that the weights of LoRA finetuned models showcase a lower degree of alignment compared to their fully-finetuned counterparts. We hypothesize that improving this alignment is key to obtaining better LoRA model merges, and propose KnOTS to address this problem. KnOTS uses the SVD to jointly transform the weights of different LoRA models into an aligned space, where existing merging methods can be applied. In addition, we introduce a new benchmark that explicitly evaluates whether merged models are general models. Notably, KnOTS consistently improves LoRA merging by up to 4.3% across several vision and language benchmarks, including our new setting. We release our code at: https://github.com/gstoica27/KnOTS.

Published as a conference paper at ICLR 2025
MODEL MERGING WITH SVD TO TIE THE KNOTS
George Stoica' 1+
Pratik Ramesh" 1a
Boglarka Ecsedi'
Leshem Choshen
Judy Hoffman'
Georgia Tech
IBM Research, MIT
Correspondence emails:
ABSTRACT
Recent model merging methods demonstrate that the parameters of fully-fineun sully-finetuned models specializing in distinct tasks can be combined into one model capable of solving all tasks without retraining. Yet, this success does not transfer well when merging LoRA finetuned models. We study this phenomenon and observe that the weights of LoRA finetuned models showcase a lower degree of alignment compared their fully-finetuned counterparts. We hypothesize that improving this alignment is key to obtaining better LoRA model merges, and propose KnOTS to address this problem. KnOTS uses the SVD to jointly transform the weights of different LoRA models into an aligned space, where existing merging methods can be applied. In addition, we introduce new benchmark that explicitly evaluates whether merged models are general models. Notably, KnOTS consistently improves LoRA merging by up to 4.3% across several vision and language benchmarks, including our new setting. We release our code at: /tttps:// com wttpghtww..oggtt/ota///////////e
INTRODUCTION
Model merging (Garipov et al. 2018 Draxler et al.) 2018 Wortsman et al. 2022a Choshen et al.
2022) is an increasingly popular technique that can surprisingly create a single general model by combining weights of task-specitt models. This allows creating multi-task models by accumulating skills (Stoica et al,, 2024; Iharco et al.] 2023] Yaday et al., 2023) Ortiz-Jimenez et al.] 2024),
desirable trait in various scenarios such recycling models shared on hubs (Choshen al. 2023).
patching model weaknesses (Cai et al., 2023 Zaman et al., 2023) and collaborating to improve
models (Don-Yehiya et al., 2023).
Model merging approaches have found substantial success when merging models that are full-rank finetuned (i.e., all parameters are tuned with maximum rank, we denote it FFT) to solve distinet
tasks from the same pretrained checkpoint, into one model capable of solving all (Ilharco et al. 2023
Matena & Ratfel [022]] Yaday al. [023] Ortiz-Jimenez al. 2024 Wortsman et al.) 2022b). In
many cases, performing a linear sum over the finetuned model weights without further training, can
achieve strong performance wootssna et al., 20226 Ilharco et al.) 2023 Yadav al. [0023).
Interestingly, existing merging approuches do not always transfer well when applied to models finetuned with parameter efficient finetuning (PEFT) strategies (Tang et al.] (024), such as the extremely popular Low-Rank Adaptation (LoRA) (Hu et al. 2022). We probe into this phenomenon ($ by comparing the pairwise centered kernel alignment (CKA) (Kornblith al., 2019) between FFT models and their equivalent LoRA finetuned counterparts. While observe that FFT models have very high CKA ind-iiniia that the finetuning-updates they apply to the respective pretrained weights (we call these "task- uudats"" have high alignment, LoRA models exhibit considerably lower CKA. This suggests that the task- susttess between different LoRA models process inputs through misaligned subspaces. We hypothesize that aligning these task-updates crucial strengthening model merging operations, and propose a data and gradient free approach to do this.
Our method, termed KnOTS (Knowledge Orientation Through SVD), builds upon singular value decomposition (SVD) to transform the task-updates of different LoRA models into a shared space, where merging methods can be applied (Fig. Dp. Notably, KnOTS is simple and readily slots into
"Equal Contribution & Corresponding Authors.

---

Published as a conference paper at ICLR 2025
SVD
AWIR
Σ
VIU
V/R
Merge
Select & Concat Models
The KnOTS
U
Viregged
Alignment & Merge
Figure 1: The KnOTS method for merging "lask-updates" from an arbitrary layer-j of different models. Each upight- is denoted by Aw(() where i is the task-update of the ith model. KnOTS first concatenates the updates together and applies the SVD, to obtain U, E and a set of concatenated (1() matrices that each correspond to a particular task. KnOTS then merges the V's into single (mmerged) matrix. Finally, KnOTS multiplies the U, E and (merged) to obtain merged-update to be added to the pretrained model.
many existing merging methods. KnOTS first concatenates all the task-updates for layer, and then decomposes the result with the SVD to obtain: UVVT (see $4). We observe that in this construction, U consists of vectors that form the orthonormal basis for a shared representation space between all task updates, the non-negative diagonal values Σ represent the scale associated with each basis vector in the aligned space, and / is a set of concatenated matrices (one for each task-update) that are all aligned to the common U. KnOTS then applies existing merging methods, such as those of Ilharco et al. 2023);; Yadav et al. (003)), the V task-matrices and obtains (mergedd) matrix that is still aligned to the common U. After merging. KnOTS constructs the merged into one matrix at every layer, and adding it to
model by multiplying the resulting VVVV(errge the corresponding parameter matrix from the pretrained model. We validate KnOTS several model merging settings spanning vision and natural language, demonstrating that KnOTS strengthens existing model merging approaches by up to 4.3% ($
In addition, we introduce new variant of the popular eight- vision task merging benchmark originally proposed by Ilharco et al. (2023), aimed at explicitly evaluating merging method's 's ability to create general models. Specifically, we transform the benchmark to a "joint-evaluation" setting. This consists of evaluating the performance of arbitrary merged models the union of all inputs and labels across all eight datasets, without providing the model with any information regarding the dataset particular example comes from. It contrasts the existing setting where only labels pertaining the dataset an example stems from are given to the models to classify. We argue that the boint-evaluation setting step towards simulating the ability to create general merged models, capable of preserving the union of all skills captured by their base models ($ 5.3 Notably, KnOTS further improves merging performance in this challenging setting.
RELATED WORK
Landscapes between models. The geometric landscape of non-convex loss functions used to train neural networks remains largely uncharted. However. Draxler al. (2018); Garipov et al.) (2018); Simsek et al.) (202T); Frankle & Carbin (2019) reveal that the parameter values of independently trained neural networks can be interpolated without increasing the test loss, a phenomenon known as mode-connectivity. Neyshabur et al. (2020), shows that models are often mode-connected when they are finetuned on the same task from the same pretrained initialization. In parallel, it has become common practice to finetune models using pretrained weights (Dosovitskiy et al. 202 1: Huang et al.) 2023, Oquab et al., 2024 Hsu et al., 2021 Touvron et al.] 2023) Radford et al.) (002)). Together with recent discoveries Wortsman al. (2022b); Matena & Raffel (2022) that even the parameters of models finetuned on different tasks from the same initialization may be merged to create strong multitask models, there has been surge of model merging approaches for finetuned models.
2

---

Published as a conference paper at ICLR 2025
Model merging methods. Several approaches improve robustness scores by averaging finetuned weights (Choshen et al. 2022; Wortsman et al. 2022b] Rame et al. [022] 2024). Task arithmetic (TA) ([hharco et al. 2023) first subtracts the parameter values of the pretrained model from those of the finetuned models, creating a set of "task-vectors." These are then linearly summed to create merged model, without any gradient calculations retraining. Ortiz-Jimenez et al. (2024) theoretically grounds TA, by showing that its success is dependent on the task-vectors governing disjoint regions in the pretrained model's function space-a a concept known as "weight disentanglement" In practice, they find that the weights of different finetuned models heavily conflict, and propose finetuning strategy that disentangles finetuned models to merge better with TA. Daheim et al. (2024); [Shah et al.) (2024) also find that TA-style merging can improve with finetuning and access to large amounts of training data. TIES (Yadav et al. 2023) improves TA by resolving the parameter interference between models when merging. Specifically, it prunes low-magnitude weights and then only averages weights which share the dominant sign. DARE (Yu et al. 2024) further explores this issue by randomly dropping fine-tuned weights and rescaling the remaining ones to create sparse task-vectors.
Merging LoRA models. LoRA (Hu et al. 2022) has quickly become a very popular finetuning
technique. However, existing merging methods (Ilharco et al. 2023 Yadav al. 2023 Yu et al.
2024) do not transfer well to merging LoRA models (Tang et al. 2024). Tang et al. (2024) suggest this is due to increased similar to Ortiz-Jimenez et al. (2024) which improve the performance of each merging method. To
between the models, and propose to a finetuning method
the best our knowledge, we present the first framework capable of achieving strong merged LoRA models without fineluning.
SVD-based LoRA Approaches. Employing the SVD on LoRA models is not a new idea for purposes other than merging and aligning weights. [Meng et al. 2024) initialize the parameters of LoRA models using the SVD of the weights of the corresponding pretrained model, yielding improved finetuning performance. Zhang al. (023) use the SVD improve the gradient updates in parameter efficient finetuning. The popular Hugging Face library (Mangrulkar et al. 2022) includes several merging methods for LoRA models that are suffixed by "-svd" (e.g., "TIES-svd"". These first merge the LoRA parameters of different models according solely to the prefix merging-method (e.g.. "TIES"). Afterwards, they use the SVD to decompose the merged parameters back into LoRA weights. KnOTS is unrelated to these approaches. Instead, it significantly improves merging performance by aligning LoRA models with the SVD.
BACKGROUND AND MOTIVATION
Problem setting. We study gradient-free multitask model-merging (Ilharco et al.) 2023 Stoica et al. 2024; Yadav et al.) 2023) Jin et al. 2023). Suppose that we have a set of models with the same architecture, that have each been finetuned with -ank Adaptation (LoRA) (Hu et al. 2022) on distinct task (c.g., different image classification problems) from the same pretrained model. Let ((2), f()) denote the finetuned models and f(rtt) be the pretrained model. Model-merging methods fuse the parameters of these finetuned models into a single unified model capable of solving all tasks. We study gradient-free approaches they are both data-efficien and training free, making them lightweight tools for merging models on-the-fly (Wortsman et al.) 2022a).
We study models finetuned with LoRA due to its wide usage and because recent work observes notable challenges when applying existing gradient-free methods on such models (Tang et al. 2024).
Model definitions. Let a pre-trained model f(vt) have I layers, each containing weights and an optional bias. Assume all biases are concatenated in respective weights, and let weight of the 5th layer be represented as wr(rt) Let g(pt) be the collection of all model weights, such that w/(20). Finetuned models are obtained by applying task updates to each weight in g(re), We denote these updates by (()) = (A
g(rt) = (w(p*) ..... w-(pt) J
AW ...., (w,"), where (w is the update corresponding to the jth layer in the th model. As LoRA constrains each update to be low-rank, we let all updates have a rank of r << min (T, O), where I and O represent the input and output dimension. Every AW(") ROX! transforms input features (1) TR! to output features ( R follows: (()) AW (1) 200, Due the low-rank constraint, AW can only act on
3

---

Published as a conference paper at ICLR 2025
subspace of size << min (I, (I) over y (Hu et al. 2022). Finally, the parameters of f())
given by ((ed) + 7()) = (W + Aw W + Aw w( + AW,
Merging models. Existing methods merge models by first defining a function (e.g., identity in (lhaacco et al. 2023)) that transforms the task-updates {-(").... ......-(m), (()). Second, each update is uniformly scaled via a unique "scaling coefficient", denoted by (*) 2 0. Third, these updates are summed to create a single set of merged updates, which are then added to the pretrained model weights to obtain the final merged model. Optimal values for (i) 2 are selected according the performance of the merged model on any available data Ilharco et al. (2023); ; Yadav et al. (2023); Yu et al. (2024). Although LoRA updates factorize into smaller matrices and B, we utilize the full-matrix representation Aw = BA in all merge operations. We explain this decision in App. A
3.1 LORA MODELS ARE DIFFICULT TO MERGE
Existing full-rank finetuned (FFT) model merging approaches perform well without additional data, even on different tasks (llharco et al. 2023 Yadav et al.] 2023). However, Tang et al. (2024) observe that the same merging methods encounter significant challenges when merging LoRA finetuned models, even when these are finetuned on the same tasks. Previous works (Iharcco et al.) 2023).
Ilharco et al. (2023): Yadav et al. (223); Tang et al. (2024) use "task-vector orthogonality" as a proxy for understanding when merging is difficult. This consists of flattening the updates (r(1).... (()) into vector, and computing the pairwise cosine similarity between them. The intuition is that when the cosine similarity between vectors is near zero (i.e., they are orthogonal), their weights are disentangled and lie in distinct subspaces enablinn them to be merged without interference.
Tesk-vector orthogonality may not reliably measure merge potential. To see why, consider simple toy example. Let (x) w."" (eLRRLLLU((") 2) and f(2) W ReLU(W! x) be two classification models that transform inputs x E IR to some y( (0) ER respectively, where w,"), W w w E R. Let W((") 1, w( = 1 w: that each weight was initialized with Here, ,() classifies all x E R+ as positive (i.e., y > 0) and all IR- as negative (i.e., y(2) < 0), while f(2) does the opposite. Calculating the task- of (() and (() give [1, and [-1, respectively-making them orthogonal. However, it is trivial to see that merging their model weights catastrophically affects the performance of at least one model. Any scaled sum over their weights with an (9) will preserve one model's predictions while flipping the other's. Despite having orthogonal task- merging the weights of these models is catastrophic. Thus, additional measures may be helpful to understand when models can be merged.
1, and w 1, and assume
Merging is easier when model layers share similar activations. Entezari et al. (022)); Ainsworth et al. (2023) Jordan et al. ( Stoica et al. (2024) extensively show that models whose layers extract similar intermediate activations for the same inputs, are easier to merge- even when trained on distinct tasks (Stoica et al. 2024). This often occurs when the weights across models extract the same kinds of meanings in a similar order from the inputs (Entezari et al.) 2022 Stoica et al. 2024), and are thus aligned. We posit that understanding model alignment from this perspective can serve as a helpful tool for understanding the challenges behind merging. One way to measure the possibility of this is using the centered kernel alignment (CKA) measure Kornblith et al. (2119). CKA measures the similarity between the intermediate activations of two models, at each layer. CKA values close to one indicate that model parameters can be transformed to process information similarly and thus transform inputs in a functionally aligned manner. When CKA is low, the reverse is true.
FFT models have very aligned CKA representations. Fig. 4b shows the average pairwise CKAS between the task-updates of the FFT models presented in Ilharco et al. (202)). We observe very high CKA, indicating that the updates between models process information in a way that ensures the intermediate feature activations between their layers are well aligned. This suggests that their updates extract similar information from inputs at each layer, and that merging them is less likely to cause significant disruption seen by [lharco et al. 2023).
LoRA models have considerably less aligned CKA representations. Conducting the same CKA analysis LoRA variants of the same models from (Ilharco al. 2023) shows that the models
4

---

Published as a conference paper at ICLR 2025
Cars
1.00
0.81
0.76
0.81
0.78 0.63
0:85
0.76
1.30 0.23 0.24
0.23
0.24
0.22
0:25
0.22
1.00 0.54 0.54
0.55
0.52
0.53
0.56
0.49
1.0
DTD
0.81
1.00
0.79
0.82
0.79
0.
0.84 0.78
0:23 1.00
0.20
0:21
0.15
0.30
0:28 0.17
00 0.54
1:30
0.52
0.54
0.49
0.57
0.57
0.49
EuroSAT
0.76
1.00
0:80
0.77 0.63
0.79 0.78
0 24 0.20
1.00
0:20
0.20
0:21 0.19
0 04
0.52
1.00
0.51
0.52
0.49
GTSRB
0.81 0.82
0.80
1.00 0.84 0.84
0:84 0.87
0. 0.21
0.20
1.00
0.19
0.17
0:21
0.25
05 0.54
0.52
1.00
0.52
0.52
0.54
0.52
MNIST
0.78 0.79 0.77 84
100
81
0:79 0 82
0 24 0.15
0.20
0
19
1.00
0.74
0
0.25
00 0/ 0:49
0.51
0.52
1.00
0.48
0.51
0.53
RESISC45
0.83
0.63 0:84 0.87 1.00
86
0.78
0 22 0.30 0.20
0:17
1.00
0.14
0.53 0.57 0.52
0:48
1.00
0.44
0.4
SUN397
0:85 0.84 0.79 0 84 0. 79 0.66
00 0. 79
0. 25 0.28 0.21 0:21 0.18 0.27
100
0.18
0 55 0.57 0.53 54 0.51 0.57
1.00
0.47
SVHN
0.76 0.78
0.78 0.87 0.82 0.78
0.79 1.00
0 22 0.17 0. 25 0.25 0.14
0:18
1000
0 4949 0.46 0.49 0.52
0.53 0.44
0.47
1.00
0.2
Cars
010 EuroSAT
GTSRB
MHIST
RESIS445
SUN397
SVMN
Cars
D10 EurSAAT
GTSRB
MNIST
RESISCAS
SUN397
SYHI
Cais
OTD
EuroSAT
GTSRB
MNIST
RESISG4S
SUN097
SVNN
(a) Fulk finetuned model alignments (b) LORA finetuned model alignments (c) LoRA finetuned model alignments with KnOTS
Figure 2: Finetuning strategy impacts representation alignments between models trained on
different tasks. The figure shows the average pairwise centered kemel alignment (CKA Kornblithh
et al. ) between the outputs solely given by finetuning updates (e.g.. AW across every attention layer, from models finetuned on different tasks with different strategies (defined in $5.2).
High CKA A indicates that the task-updates of different models are aligned. (a) Full-rank finetuned
models exhibit high CKA and alignment. (b) LoRA finetuned models are drastically less aligned. (c)
However, they are dramatically more aligned with KnOTS.
exhibit dramatically lower average CKA between task-updates (Fig. Ap). This suggests that they are functionally misaligned: from the same imputs each LoRA extracts unrelated features. |Entezari et al.) (2022): ; Jordan et al. (2023); Stoica et al. (2024) observe that merging models in these circumstances can yield poor performance. We speculate that this phenomenon may be due to the low-rank constraint imposed on LoRA models. Specifically, each task-update can only act subspace of the input and output activation space. Since each LoRA model is finetuned on a different dataset, must carefully pick the subspace it uses to extract information. Thus, it is unlikely that the task-updates across different models extract the same kind of information, leading to low activation alignment between updates and poor merges.
Our hypothesis. Coupling the CKAs of FFT model task-updates with those of LoRA models, we hypothesize that aligning LoRA task-updates is a crucial first step improving their mergeability, and the performance of existing merging methods. We seek to design such an alignment method without assuming access to data or gradients.
METHOD: KNOTS
We propose a data-free and gradient-free method for aligning LoRA models, to improve the perfor- mance of existing model merging methods. Our method, termed KnOTS (Knowledge Orientation Through SVD), uses singular value decomposition (SVD) to jointly align the representation spaces between LoRA models (see illustration in Fig.
Subspace alignment with the SVD. KnOTS aligns the updates across different LoRA models layer-wise. Thus, it suffices to consider how we align the updates for an arbitrary layer j. Recall that these updates are denoted by: Aw(") Aw " Aw("), of information. For notational ease, we drop the subseript j for the remainder of this section. Most often SVD is used with square matrices, aligning vectors in them. Instead, we propose align these updates into a shared space where they can extract similar information, by concatenating the updates across tasks and deconstructing the result with SVD:
each extracting different kinds
(2 SVD Aw(2), ....;
Aw(m) :..:;:(("))
UUVT
(I)
-UY [(1), ((2)...;
(2)
where, U E B Σ Diag R [varr ....., [()]') R*x1 < k
min (I,O, nr). With the SVD,
Here, AW()) US| AW U AW(n) Unl
5

---

Published as a conference paper at ICLR 2025
each task-update is decomposed into a shared UE term defining the orthogonal basis for the activations from all task-updates-aiee along with task specific components, [V('] By construction, each [V()]T's from any task-update acts to transform the same information (given by UE) with any input, thereby increasing their alignment. We visualize this effect in Fig. E Specifically, we compute the CKA between the intermediate activations obtained from passing the same inputs through each [V(")T ..... ..... [V(a]]" of our LoRA models, and averaging over all layers to be merged. We observe that the CKAs are significantly higher compared to Fig- 2p indicating that the task-updates between
LoRA models are substantially more aligned under KnOTS.
Applying merge methods. In many cases, we can directly apply existing merging methods on the aligned [vOn] .... [V(N)TT without modifications. Specifically, we can apply some merging function (e.g., weighted sum a-la-(Ilharce et al. 2023)) over these parameters to obtain single merged [V (mmergadd)) We can then construct the merged update with the remaining SVD components: AW, merged))
UE[V(merged)]] which is finally added the pre- pre-trained layer weights y..
EXPERIMENTS AND RESULTS
We validate KnOTS across diverse benchmarks spanning both vision and language domains. We first evaluate KnOTS on the popular "per-task" setting across both vision and language tasks ($/5.2).
Here, models finetuned distinct datasets are merged and then evaluated on each dataset separately.
KnOTS consistently improves the capabilities of existing merging methods across all experiments. Second, we study the capabilities of merging methods building general models by introducing new benchmark ($ (.3)). Despite being very challenging, KnOTS is still capable of uplifting existing merging approaches. Third, we conduct extensive analysis on different facets of KnOTS ($ 5.4).
5.1 EXPERIMENTAL DETAILS
Models. We use VIT-B/32 or VIT-L/16 (Dosovitskiy et al. (021) for all our vision experiments.
These are two variants of the CLIP vision encoder (Radford et al. 2021) that are finetuned separately on variety of tasks. Our natural language experiments are conducted with LLama3-8B model (Aí 2024) across different natural language inference (NLI) tasks. As in (Hu et al., 2022), we only apply LoRA on the weight matrices in the attention layers: namely the key, query, value and output layers.
Unless otherwise specified, each LoRA has a rank of 16. Further training details are found in App. D
Merging methods. As discussed in we restrict our scope to gradient-free merging methods in this work. To the best of our knowledge, there are only four such merging methods for our setting: RegMean (Jin et al., 2023), Trik-mrithietic (TA) (Ilharco al. 2023). TIES (Yadav al. (002)). and DARE (Yu et al. 22024). RegMean (Jin et al., 2023) is a merging approach that also aligns the weights of each model. does this by solving a closed-form locally linear regression problem at every model layer. This results in obtaining transformation matrix for each models' task- updates, that when applied aligns the updates between models. In contrast, TA, TIES and DARE assume that the task updates between models stemming from shared pre-trained checkpoint are functionally aligned and directly merge them.
TA (Iharcoo al. 2023) merges models finetuned different tasks with the same pretrained checkpoint by linearly summing their parameters. They apply a summation-weight (termed "scaling coefficient") to the parameters of each model that achieves the best merged model performance over a held-out validation set. Instead of tuning a unique scaling co-efficient for each task-vector which may be expensive and does not scale well, we only tune a single scaling co-efficient for all models as recommended by Ilharco et al. (2023): Yadav et al. (2023). TIES (Yadav et al. 2023) extends TA by reducing noise and conflicts between parameters when merged. It reduces noise by pruning k parameters with the lowest magnitudes, and reduces parameter conflicts by also pruning those that lie in outlying directions-a step called "sign-resolution" TIES then linearly sums the model parameters using the scaling coefficients and & for pruning, that achieve the best merged model over the held-out set. DARE (Yu et al.) 2024) reduces noise by randomly pruning parameters following Bernoulli distribution with probability p. The remaining parameters by 1/11 - p) to account for scale lost from pruned parameters. Afterwards, DARE typically employs sign-resolution of TIES (specified by DARE-TIES). To account for its random pruning step, run
then rescaled

---

Published as a conference paper at ICLR 2025
Table Eight models per-task results. We merge eight ViT-B/32 models finetuned with LoRA on different image classification datasets. "Finetuned" refers to the accuracy of each finetuned model on the dataset it was trained on, We report the per-task (including the average) normalized-accuraciio across other merging baselines. These describe how close they get to the "Finetuned" accuracy. KnOTS-TIES improves over baselines by up to 4.3% average accuracy.
Datasets
Method
Cars
DTD
EuroAAT
GTSRB
MNIST RESISC45 SUN397
SVHN
Avg
Per-Task Absolute Accuracies (%)
Finetuned
74.0
58.3
99.0
92.7
99.3
88.4
64.5
96.2
84.1
Per-Task Accuracies of Merged Models Normalized Against Finetuned Models (%)
RegMean
80.2
71.3
37.9
47.3
43.1
70.5
93.9
43.0
60.9
TA
82.0
73.6
48.8
42.1
53.1
71.5
97.5
41.2
63.7
TIES
82.2
72.8
50.0
36.8
56.8
69.4
96.9
44.6
63.7
DARE-TIES
81.4
74.5
50.8
39.2
55.0
70.7
97.6
40.1
63.7
KnOTS-TIE
82.7
73.7
49.3
48.9
68.9
70.9
95.5
53.8
68.0
KnOTS-DARE DARE-TIES
81.8
75.9
50.7
40.3
53.2
70.2
97.9
41.0
63.9
all DARE experiments across three separate seeds, and report the best performance. KnOTS can be coupled with merging methods such as these latter three by applying them directly on aligned [V[]] parameters. Note that we make exception for methods like TIES Yadav al.) (023) that involve magnitude-based model pruning. While model-scale is absent in [V (its row-wise magnitudes are < 1 by definition), it is retained in Σ. Thus, we may apply such operations on ())" but merge using the newly pruned [V s. Additionally, we observe that applying KnOTS on TA reduces to performing TA: Σin ( AW Σ a.U'NV TA can be considered generalization of TA: KnOTS reduces to the standard merging method when its alignment is not leveraged. We refer to all methods we apply KnOTS to by prefixing their names with KnOTS (e.g.. KnOTS on TIES is denoted by KnOTS- TIES). Inspired by the recommendations made by each merging baseline, we tune hyperparameters such as caaling-cofffcceent top-K and as described in [ Note: KnOTS does not require any new hyperparameter to be tuned apart from the ones used by the original merging methods.
UΣ (, Thus, KnOTS on
In some experiments, we add the "Ensemble" as a baseline, which consists of all models used in particular merging evaluation. The ensemble passes an input in parallel to all models, and predicts its class according to the highest confidence prediction across all models.
Metrics. We report the absolute accuracy of all individual finetuned models on their respective datasets, and utilize these for merging. Similar to Ilharco et al. (2023); ; Yadav al. (2023).
compare our merging methods via noormalized-accurayy" wherever applicable. This is obtained by dividing the performance of a merged model on a task (c.g.. Cars (Krause al. 2013) by the performance of the original model finetuned on the task. For instance, the normalized accuracy for a given task-i is computed as Accuracy of merged model on lask-i Accuracy finctuned model task-i This metric shows how close the merged model gets to the original finetuned model for each task. Certain experiment settings study different generalization properties and thus have different metrics. For the remainder of this paper, we assume all accuracy measurements are normalized, and define other metrics in their relevant sections.
5.2 PER-TASK EVALUATIONS ON VISION AND NLP SETTINGS
We first compare KnOTS's ability improve existing merging methods in the popular "per-task" experiment settings (Iharco et al. 2023 Yadav et al. 2023 Tang et al.) 2024). These settings comprise of merging a set of models independently finetuned on different datasets into single model, and then evaluating it each dataset independently-use
only its examples and labels. This way, the merged model judged on ability to preserve the individual skills of each original model.
Merging eight ViT-B/32 models finetuned on image classification datasets. We follow the image classification benchmark from Ilharco et al. (2023) and merge models finetuned on eight different datasets: Cars (Krause et al. 2013), DTD (Cimpoi et al. 2014), EuroSAT (Helber et al. 2019).

---

Published as a conference paper at ICLR 2025
GTSRB Sta Staplkamp et al. (2011), MNIST (LeCun 1998,, RESISC45 Cheng et al. 2017), SUN397 (Xiao et al. 2016) and SVHN (Netzer et al. 2 D. Like Ilbarco et al. (2023) ; Yadav et al. (2023).
report normalized per-tas task accuracies and their average. Tab. D reports merging performances. Interestingly, nearly all merging methods achieve comparable performance. However, applying KnOTS on both TIES and DARE-TIES elevates their respective performances, with KnOT'S- strengthening TIES by 4.3% average normalized accuracy. Note, we also compare against a popular gradient-baeed approach- Fisher weight averaging (Matena & Raffel 2022) in App. and achieve considerable gains, though gradient-based based merging is not our focus.
KnOTS scales to VIT-L/14 models. We also eval- uate how KnOTS is affected by merging larger vision models. Specifically, we merge eight ViT- L/14 models finetuned with LoRA the same datasets as our ViT-B/32 counterparts. We then evaluate the merged model's normalized per-task accuracies over each dataset. Results for several merging methods are summarized in Tab. 2
Table 2: Normalized per-task avg. image clas- sification results. We merge eight ViT-L/14 models finetuned with LoRA eight vision datasets. We report the average normalized accu- racies against average absolute accuracy of the finetuned models: 92.3%. KnOTS is best.
KnOTS
TA
TIES
DARE-TIES
TIES
DARE-TIES
While performance of all merging methods im- prove with the larger models, KnOTS notably con- sistently outperforms the baselines in both KnOTS- TIES and KnOTS CETIS-DAR-TIEES Additionally, LnOTS-TIES is still capable of improving TIES by 3%, showing its ability to maintain performance with scale.
74.4
75.2
74.7
78.2
75.6
KnOTS performs well on LLMs. We also eval- uate KnOTS in the NLI setting, by merging six PEFT 1amma38B (AI 2024) models finetuned on
Table 3: Normalized per-task avg. NLI results. We merge six LLama3-8B models finetuned with LoRA on different NLI datasets. All numbers are normalized against the absolute average per- task accuracy of the individual finetuned models: 92.9%. KnOTS performs best.
SNLI (Bowman et al. 2015), MNLI (Williams
cet al.
2018), SICK (Marelli et al. 2014), QNLI, Waag et al.] 2019), and SCITAIL (Khot) 2018). Each of these tasks involves perform- ing 3-way classification to determine whether given "hypothesis" is true (entailment), false (contradiction) or inconclusive (neutral) compared to given "premise." QNLI, RTE and SCITAIL. only employ two of the three classes, so we simply mask the missing label when finetuning and evaluating on these datasets. Tab. 3 shows the results of merging these models using TA, TIEs, DARE-TIES, KnOTS-TIES and
RTE
et al.
KnOTS
TA
TIES
DARE-TIES
TIES
DARE-TIES
90.2
90.0
90.9
92.9
91.1
Overall, we observe that KnOTS-TIES sig- nificantly outperforms baseline merging methods by up to 2.9% average normalized accuracy, and KnOTS- DEEIS--DRE--E DARE-TIES further improves DARE-TIES by .2%. This demonstrates the power of utilizing KnOTS to align LoRA models even dramatically larger models (8B parameters), and its robustness across modalities.
5.3 A NEW BENCHMARK TOWARDS BUILDING GENERAL MODELS
We further introduce a new experimental setting the eight vision per-task benchmark in Tab. [ that we call the "joint-task." This task differs from the "per-task" regime in that it evaluates merged mod- els over the union of all inputs and labels across every vision-task. The joint-task is significantly more challenging than the per-task counterpart as it explicitly examines whether merged model is a general model: whether it can classify any input to any label.
Table 4: Eight models joint-task results. We
merge eight VIT-B/32 models finetuned with LoRA on different datasets. We report the joint- task "Union" performances for several merging methods. KnOTs-TIER performs the best.
Meuric
Method
KnOTS
Ensemile
TA
TIES
DARE-TIES
TIES
DARE-TIES
40.7
43.6
43.6
44.0
46.8
HESSIO
63.1
65.3
65.3
66.4
68.1
66.9
HERSaG
72.6
74.0
73.9
75.1
763
75.3
After aggregating the labels across all eight tasks
and removing duplicates (e.g., MNIST (LeCun 1998) and SVHN (Netzer et al.) (011 have the same
labels), we obtain 748 unique labels over which to classify all the images across all datasets, As some labels in task hyponyms of those in another (e.g., "islet" in SUN397 (Xiao et al. (016) and
8

---

Published as a conference paper at ICLR 2025
"island" in RESISC45 (Cheng et al.
2017)) making it challenging to distinguish labels, report performance using "Hits @ k." This refers to the number of times the expected label is within the top-k predictions of a model (e.g., Hits is accuracy). Tab. 4 shows the results over the "Union" of all images and labels across all tasks, when merging our ViT-B/32 LoRA models. Please see App. for performances broken down by dataset. Overall, KnOTSS significantly outperforms every baseline at all Hits @ k levels on the extremely challenging "Union" evaluation-by up to 3.2% on also observe that the ensemble performs partieularly poorly in the joint setting. We posit this is due to certain models making over-confidently incorrect predictions on data from tasks they are not finetuned on, notable issue in ensemble models (Kardan et al.) (022). We argue that the joint-task is important benchmark for assessing a merged model's s generality, and we hope future work expands upon
Hits Interestingly,
5.4 ADDITIONAL ANALYSIS & ABLATIONS
We also conduct three analysis experiments understand different facets of KnOTS. We conduct our analysis with KnOTS-TIES consistently achieves the best performance across all settings.
Normalized Accuracy (%)
KIOTS-TIES
85
TIES
80
KnOTS scales better with the number of mod-
els. Following Ilharco et al. (2023); ); Yadav et al. (2023), we evaluate the performance of KnOTS on the eight per-task vision benchmark with our ViT-B/32 models as we increase the number of tasks being merged. Fig. 3 illustrates the same measuring performance with average normalized accuracies the merged model, where the dots represent the performance of model evaluated only on the tasks included in the merge, while the bars indicate the 95% confidence intervals based on random subselection of 28 task combinations.
70
65
3
# Models Merged
Figure 3: KnOTS boosts performance with scale. KnOTS-TIES continues see gains, out- performing original TIES (Yadav et al. 2023)
and Task Arithmetic (TA) (Ilharco et al.. 2023)
when merging an increasing number of tasks in the per-task evaluation vision setting $ [.2
Notably, KnOTS-TIES achieves significantly bet- ter merging results compared to the baselines, with performance gap that remains consistently 4% (for >2 tasks being merged), underscoring its ro- bustness the number of tasks increases.
Performance is the average normalized accuracy with 95% confidence intervals over merging dif- ferent combinations of the tasks
70
Normalized Accuracy (%)
KnOTS is robust to different LoRA ranks. Fig. shows the performance of KnOTS-TIES
68
(+.0%)
66
against TIES (Yadav et al.) (023) in merging our ViT-B/32 models trained at various LoRA ranks
64
(+6.0%)
(4, 16, 64, 256, 768} on the eight per-task vi- sion benchmark. Note that these VITs each have feature dimension of 768, making rank 768 setting "full rank". The results consis- tently demonstrate that KnOTS-TIES outperforms TIES across all rank settings. In the very low rank setting (e.g., rank four), KnOTS improves TIES by 6% (64.6% normalized accuracy vs. 58.6%). The performance uplift continues with increasing LoRA rank, with KnOTS improving TIES by 4% in the full rank setting. This experiment highlights how KnOTS is both robust and scalable to merging models finetuned on diverse tasks with arbitrary LoRA rank.
62
LoRA
60
LnOTS-TIES
TIES
58
16 64
256
Full Rank
LoRA Rank (log scale)
Figure 4: KnOTS boosts performance across LoRA ranks. KnOTS- consistently outper- forms TIES (Yadav al. 2023) across varying LoRA ranks the per-task evaluation vision set- ting. Performance is reported in terms of average normalized accuracy across eight vision tasks.
The way task-updates are concatenated matters. KnOTS concatenates task-updates column- wise before computing the SVD. In this experiment, investigate the effects of concatenating them

---

Published as a conference paper at ICLR 2025
row-wise instead. We conduct analysis using the same vision models from Tab, I and merge them with KnOTS-TIES, We observe that applying KnOTS-TIES after concatenating the rows of the task task-updates achieves average normalized accuracy of 65 65.4%. It performs 2.6% worse than KnOTS- TIES with column-wise task-update concatenation. We posit this may be due to the difference in the deconstructed task-update terms when concatenating them row-wise. Specifically, each update would denoted by AW/" = U EV, where EV are shared across all updates with different U+)), In this way, the shared EV would act distinct (+ containing different information- cirrmattoonwwichh which in turn may lower alignment. These results suggest that concatenating the updates column-wis wise crucial to obtaining strong performance with KnOTS.
CONCLUSION
In this paper, study merging LoRA models sharing the pretrained checkpoint and finetuned on different tasks without additional finetuning. We find that prior work does not transfer well in this setting, and observe that this is due to parameter misalignment between LoRA models. We introduce KnOTS to tackle this problem by using the singular value decomposition (SVD) to map the parameters of each LoRA model into shared representation space with aligned elements, prior work can be applied to merge the parameters. In addition, we introduce a novel benchmark designed to evaluate whether merged models can be general. Notably, KnOTS improves prior work by up to 4.3% across both vision and language settings, including our new setting.
which
7 REPRODUCIBILITY STATEMENT
We describe the procedure for finetuning all our models, including compute resources, datasets, and hyperparameter searches in App- D App. describes our hyperparameter searches for all merging methods. App. B explains how we calculated our centered kernel alignment (CKA) visualizations in Fig. In addition, App. provides extensive details on how we created our novel benchmark.
ACKNOWLEDGEMENTS
This work partially sponsored by NSF awards #2403297 and #2144194, NSF-GREP, and Google. All views and conclusions expressed in this work are those the authors and not a reflection of these sources.
We would like to thank Simar Kareer for his valuable insights and discussions that contributed to the brainstorming process of this work.
REFERENCES
Meta AI. Meta Ilama 3.
2024.
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In ICLR, 2023.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. large annotated corpus for learning natural language inference. In EMNLP, 2015.
Ruisi Cai, Zhenyu Zhang, and Zhangyang Wang. Robust weight signatures: gaining robustness as easy as patching weights? In JCML, 2023.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. IEEE, 2017.
Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. Fusing finetuned models for better pretraining. arXiv:2204.03044 2022.
Leshem Choshen, Elad Venezian, Shachar Don- Yehiya, Noam Slonim, and Yoav Katz. Where to start? analyzing the potential value of intermediate models. In EMNLP. 2023.
10

---

Published as a conference paper at ICLR 2025
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2014.
Nico Daheim, Thomas Möllenhoff, Edoardo Ponti, Iryna Gurevych, and Mohammad Emtiyaz Khan. Model merging by uncertainty-based gradient matching. In ICLR, 2024.
Shachar Don-Ychiya, Elad Venezian, Colin Raffel, Noam Slonim, and Leshem Choshen. ColD fusion: Collaborative descent for distributed multitask finetuning. ACL, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In ICML, 2018.
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. In ICLR, 2022.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2019.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. NeurlPS, 2018.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: novel dataset and deep learning benchmark for land use and land cover classification. JSTARS, 2019.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. /EFE/ACM TASLP, 2021.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv:2307. 00.007./3266 2023.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In ICLR, 2023.
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. ICLR, 2023.
Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. Repair: Renor- malizing permuted activations for interpolation repair. In ICLR, 2023.
Navid Kardan, Ankit Sharma, and Kenneth O Stanley. Towards consistent predictive confidence through fitted ensembles. In IJCNN, 2021.
Tushar Khot, Ashish Sabharwal, and Peter Clark. Scitail: A textual entailment dataset from science question answering. In AAAI, 2018.
Simon Kornblith, Mohammad Norouzi. Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In ICML, 2019.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In IEFE 3D Representation and Recognition Workshop ICCV, 2013.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/nmist// 1998.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In ICLR, 2017.
11

---

Published as a conference paper at ICLR 2025
Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the of-the-art parameter-efficien fine-tuning methods. https://github. com/ huggingface/peft, 2022.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. A SICK cure for the evaluation of compositional distributional semantic models. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerek, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), LREC, 2014.
Michael Matena and Colin Raffel. Merging models with fisher-weighted averaging. In NeurlPS, 2022.
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. NewrlPS, 2024.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, al. Reading digits in natural images with unsupervised feature learning. In Deep learning and unsupervised feature learning workshop NeurlPS, 2011.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is heing transferred in transfer learning? NeurIPS, 2020.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez. Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. TMLR, 2024.
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. NeurlPS, 2024.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurIPS, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy. Patrick Gallinari, and Matthieu Cord. Diverse weight averaging for out-of-distribution generalization. NeurIPS, 2022.
Alexandre Rame, Nino Vieillard, Leonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. WARM: On the benefits of weight averaged reward models. In ICML, 2024.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In ENLP, 2019.
Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, and Varun Jampani. Ziplora: Any subject in any style by effectively merging loras. In ECCV, 2024.
Berfin Simsek, François Ged, Arthur Jacot, Francesco Spadaro. Clement Hongler, Wulfram Ger- stner, and Johanni Brea. Geometry of the loss landscape in overparameterizee neural networks: Symmetries and invariances. In ICMI, 2021.
Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: multi-class classification competition. In IJCNN, 2011.
12

---

Published as a conference paper at ICLR 2025
George Stoica, Daniel Bolya, Jakob Brandt Bjorner, Pratik Ramesh, Taylor Hearn, and Judy Hoffman. Zipit! merging models from different tasks without training. ICLR, 2024.
Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. Parameter-efficient multi-task model fusion with partial linearization. In ICLR, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. ar.iv:2330.. 13971, 2023.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL, 2018.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lppes Ari Morcos, Hongseok Namkoong, Farhadi, Yair Carmon, Simon Kornblith, al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. ICMI., 2022a.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. CVPR, 2022b.
Jianxiong Xiao, Krista A. Ehinger, James Hays, Antonio Torralba, and Aude Oliva. SUN database: Exploring a large collection of scene categories. IJCV, 2016.
Chen Xu, Weiwei Xu, and Kaili Jing. Fast algorithms for singular value decomposition and the inverse of nearly low-rank matrices. NSR, 2023.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Ties-merging: resolving interference when merging models. NeurlPS, 2023.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In ICML, 2024.
Kerem Zaman, Leshem Choshen, and Shashank Srivastava. Fuse to forget: Bias reduction and selective memorization through model fusion. arXiv:2311.07682 2023.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. ICLR, 2023.
13

---

Published as a conference paper at ICLR 2025
A
WHICH LORA REPRESENTATION IS BEST FOR MERGING?
LoRA is a popular Parameter Efficient Finetuning (PEFT) method that finetunes models by applying low-rank updates to the pretrained parameters. For instance, let W AW, Rox! With
LoRA, each Aw is rank << min(O.1). and can be factorized into two low-rank matrices A R2'1, BB1) R with AW, _ A The existence of A/", and B3" for each AW(") may suggest that merging should be done on them separately, and the results multiplied to achieve the merged-updand AW((), However, this immediately leads to issues when applying existing merging methods. For instance, let us apply task-arithmetic [Iharroo et al. (0033) on a single arbitrary layer (e.g., the jth layer) across f(i) ....,. f(r), First, TA merges the factorizations
of AWi"... ..... AW(")) by computing Ei= 1 A/ E 1 \0B B Multiplying each to obtain A Aw(m) m) yields,
n2
AW,
o
BJS
Σ 100
B
Σ
1 ( B Ag00
Products of aligned factorizations
Products of misaligned factorizations.
Notably, Aw((m) consists of two terms: one composed of B and Al" matrices from the updates of the same model, and one composed of (( and Ala)) matrices from updates of different models. Unfortunately, there is no guarantee that = expects the same input representation as the output of
A (4) when * k, and including these terms in the merged model can incur significant drop in model performance (Soica et al. (2024). Similarly, this same issue persists as well when applying Yaday
et al. 2023). Thus, we only conduct merging on the original update representations: Aw of each model. This trivially avoids mismatches when applying merging methods.
B COMPUTING CENTERED KERNEL ALIGNMENTS ACROSS TASKS
We create the pairwise centered kernel alignment (CKA) (Kornblith al. 2019) matrices presented in Fig. [ using models that are finetuned from the same pretrained checkpoint on the different tasks shown Tab. Each matrix is generated using a set of unlabeled heldout data from all eight datasets. Specifically, this heldout set consists of the validation data of the respective dataset when it exists and otherwise randomly samples 20% of the test set. Note that in situations where we sample 20% of dataset's 's test split, we always evaluate any merged model on the remaining 80% of examples. For fair comparison, all CKA plots are created using the same data.
Pairwise CKA on full-rank finetuned models. We compute Fig. set of eight full-rank finetuned (FFT) ViT-B/32 (Dosovitskiy et al. 202T) models released by Ilaarco al. (203)) along with their shared pretrained model. We then collect the intermediate outputs of every key, query, value and projection layer across all nine models over all the data from our heldout set. This yields nine different intermediate activations for each layer across all models. We then subtract the activations of the pretrained model from the eight activations of the finetuned models, at every layer. We then compute the CKA (following (Kornblith et al.) (019)) between activations of two different models at every layer, across all model pairs. Finally, we average the results over all layers.
follows. We first take the
Pairwise CKA on LoRA finetuned models. We compute Fig. as follows. We first take the set of eight LoRA models we merge in in Tab. [ and extract the intermediate activations over all LoRA layers of each model and over the same data used in our FFT CKA plots. Specifically, these layers are every key, query, value and projection layer of each model. Since LoRA finetuning adds new parameters pretrained model, there are no layers from the pretrained model to subtract. Thus,
14

---

Published as a conference paper at ICLR 2025
directly compute CKA over the activations at the same layer across LoRA model pairs, and average over all layers.
Pairwise CKA LoRA finetuned models aligned with KnOTS. We compute Fig. 3 follows. This procedure is extremely similar to computing the CKAs for LoRA finetuned models, with one small difference. Specifically, rather than computing the CKAs from the output activations of LoRA layers, we instead calculate the CKA over the outputs of different [V(]] for the same layer of different LoRA models (using the same data again). Afterwards, we average the results across all layers.
C TUNING THE HYPERPARAMETER FOR DIFFERENT MERGING BASELINES
Scaling co-efficient: A single scalar scaling co-efficient is tuned across the range [0.1, 0.2, 0.3,.., 1.0]
top-k Like Yadav al. (2023), we define top-k the percentage of task-update elements retained when merging. It is tuned across the range [10, 20, 30.... 100]
DARE-pruning co-efficient (p): [0.99, 0.9, 0.8, .... 0.1]
DARE-Seeds: We evaluated DARE over the following five seeds: [420, 421, 422, 423, 424).
For methods such as TIES Yadav al. 2023) and DARE (Yu et al., 2024) which involve tuning more than one scaling co-efficient we tune by performing a lincar-search by first tuning the scaling co- efficient and then the corresponding pruning hyperparameter i.e top-k for TIES and DARE-pruning co-efficient for DARE. The default value used for TIES top-k is 30 and for the DARE-pruning co-efficient is 0.9 as recommended by their respective baselines.
D TRAINING DETAILS
Training Vision models. We make use of the CLIP (Radford al. 2021) based ViT-B/32 and ViT- L/14 models from Hugging Face (HF). These models are then LoRA fine-tuned using HF's PEFT LoRA library. Across all our experiments we initialize the LoRA layers across the query, key, value and output layer. Note that these are the only learnable layers. We set the LoRA rank to be 16, LoRA alpha to be 16, LoRA dropout to be 0.1 and disable the use of bias parameters. All models trained using the AdamW (Loshchilov & Hutter 2019) optimizer, with cosine learning rate scheduler (Loshchilov & Hutter 2017) using Cross-Entrope loss.
The VIT-B/32 models were fine-tuned on the 8 vision tasks using standard learning rate of 1e-5, weight decay of le-1 and label smoothing set to 0.
The ViT-L/14 models were fine-tuned the 8 vision tasks using standard learning rate of 3e-4, weight decay of le-4 and label smoothing set to 0.
Across all experiments the text encoder in the CLIP-model remains frozen and the text embeddings are obtained by passing the class labels through the text encoder.
Training LLMs NLI tasks. For the setting in $ 5.2 We initialize the model using the SowoMooelForSScuenneClsssss
the LLama-3 SB parameter model.
model class from Hugging Face's "odt classify across classes. The models are LoRA fine-tuned using the PEFT LoRA library. We initialize rank- 16 LoRA weights across the weights of the attention layer only i.e the query, key, value and output layer. The models were trained using Adam W (Loshchilov & Hutter 2019) optimizer using linear learning rate scheduler, with learning rate of 3e-5 and warm steps set to 6% of the total number training steps. For datasets like QNLI (Wang et al. 2019). RTE (Wang al. 2019) and SCITAIL (Khot et al. 2018) which only employ two of the three classes, simply mask the missing label when finetuning and evaluating on these datasets.
During merging only merge the Llama back-bone with the LoRA layers and use task-specific head when evaluating across each dataset. Even for the merged model we use the merged backbone and task-specific head during evaluation.
15

---

Published as a conference paper at ICLR 2025
Table AI: Eight models per-task results including Fisher weight averaging. We merge eight ViT-B/32 models finetuned with LoRA on different image classification datasets. "Finetuned" refers to the accuracy of each finetuned model on the dataset it was trained on. We report the per-task (including the average) normalized-accraciie across other merging baselines. These describe how close they get to the "Finetuned" accuracy.
Datasets
Method
Cars
DTD
EurSATT
GTSRB
MNIST
RESISC45 SUN397
SVHN
Avg
Per-Task Absolute Accuracies (%)
Finetuned
74.0
58.3
64.5
96.2
84.1
Per-Task Accuracies of Merged Models Normalized Against Finetuned Models (%)
RegMean
80.2
71.3
37.9
47.3
43.1
70.5
93.9
43.0
60.9
Fisher
84.5
72.4
44.4
55,8
47.8
70,9
96.1
39.2
63.9
TA
82.0
73.6
48.8
42.1
53.1
71.5
97.5
41.2
63.7
TIES
82.2
72.8
50.0
36.8
56.8
69.4
96.9
44.6
63.7
DARE-TIES
81.4
74.5
50.8
39.2
55.0
70.7
97.6
40.1
63.7
KnOTS-
82.7
73.7
49.3
48.9
68.9
70.9
95.5
53,8
68.0
KnOTS-DARE: 81.8
75.9
50.7
40.3
53.2
70.2
97.9
41.0
63.9
Compute resources. All our experiments were conducted on machines with one Nvidia A40 with 48GB of VRAM, and CPU that has 8 workers. We trained all our models across these machines, and also applied every merging algorithm in this environment. KnOTS is capable of running entirely on the CPU. We compute the SVD using the Pytorch (Paszke al. 2019) "torch, finalg.svd" solver. However, we note that more efficient SVD algorithms can easily be employed with our approach, such the recent Fast SVD (Xu al. 2023) which is designed for low-rank matrices.
Datasets and licenses. This paper uses the following datasets and associated licenses. Cars (Krause) al. 2013) 3) and GTSRB (Stallkamp et al. (01T) both use the Creative Commons License. EuroSAT (Heleer et al. 2019) is under the MIT license and MNIST (LeC'nn 1998) is under the Gnu General
Public License. We could not find the licenses of DTD (Cimpoi et al. 2014), RESISC45
(Cheng
al. 2017), SVHN (Netzer al. 201T) and SUN397 (Xiao al. 2016).
E
COMPARISON WITH A FINETUNING BENCHMARK
We also compare KnOTS against the popular finetuning benchmark Fisher Weight Averaging (Matena Raffel 2022) in the same setting as Tab. [ We classify this method as finetuning because adds learnable parameters to every model that are optimized via gradient descent to obtain the best merged model. We select the best reported Fisher model according to the hyperparameter configuration which achieved the hest performance on the same held-out validation data as all methods in this setting. Our hyperparameter search consisted of two ranges. First, the number of examples used to compute the Fisher information matrices: [256, 512, 1024, 2048] (selected from the training data of each dataset). Second, the scaling term to merge model parameters: 1, I, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]. The best performing configuration
our held-out validation dataset used scaling coefficient of 1.0 and required 256 examples to compute the Fisher weights. Tab. Tab.AAT summarizes the results. Overall, we observe that while Fisher matches TIES, it is substantially outperformed by KnOTS method that doesn't require any training.
F JOINT TASK FULL PERFORMANCES
Tab. A2 shows performances for each merging method across all datasets on the Joint-task. Each dataset column shows results images only from the dataset, but with using the joint labels-set of 748 labels.
F Finding synonymous labels. After removing duplicates from the joint label space, are left with 748 total labels. However, a manual search over these labels reveals that some may be synonyms/hy- ponyms of each other. To get better idea of how many there are, encode each label using
16

---

Published as a conference paper at ICLR 2025
Table A2: Eight models joint-task results. We merge eight ViT-B/32 models finetuned with LoRA on different datasets. We report the joint-task performances for several merging methods. KnOTS- TIES improves over baselines by significant margins across across nearly every evaluation, often only trading best performances with the "Ensemble."
Method
Metric
Joint: Task Performances (8)
Cars
DTD
EumSAT
GTSRB
MNIST
RESISC45
SUNN297
SVHN
Union
Hit:21
58.5
16.9
29.5
35.8
54.2
62.4
25.1
40.7
Ensemble
Hit.583
83.6
61.4
31.4
59.5
58.0
78.2
84.0
44.3
63.1
91.0
72.4
40.0
73,0
69.1
85.9
89.5
55.2
72.6
TA
Hit.83
84.9
63.7
23.0
66.1
48.4
83.6
83.9
50.1
65.2
Hit
92.0
74.0
31.0
77.9
55.7
90.2
89.9
61.6
74.0
43.6
TIES
Hit:23
84.9
61.9
21.9
63.3
48.2
82.8
83.7
53.6
65.3
Hit:585
92.1
72.4
29.0
75,3
$4.0
89.4
89.9
64.1
73.9
60.8
39.3
12.8
33.7
34.3
57.5
60.4
35.5
44.0
DARE-TISS
18.1
63.6
57.8
66.4
Hit
92.6
73.0
20.9
74.66
55.7
89.0
89.1
69.5
75.1
61.7
40.5
16.2
44.2
39.1
$9.0
60.6
36.7
46.8
KHOTS-TIES
Hits25
92.6
74.5
31.1
79.8
58.4
90.4
89.1
68.5
76.3
HiT
60.4
40.3
15.9
41.9
34.6
58.4
60.4
34.8
45.2
KOOTS-DARE-TIS
85.0
63.5
68.4
50.0
83.8
56.5
66.9
92.2
74.5
26.6
78.9
55.9
90.4
88.9
67.4
75.3
model taken from the Sentence Transformers library (Reimers & Gurevych] 2019), and compute the pairwise cosine-similarities between each label. We then take all label pairs with cosine-similarity 2 0.8 to be "synonyms," and reproduce them at the bottom of this section in dictionary form where keys and values are tuples containing "(label, dataset origin)." In total, find 111 synonyms, leading to average of 0.15 synonyms per-label and 1.95 synonyms per-labels that have synonyms. Thus, measuring performance on the joint-task using "Hits@k" with h (1,3,5) appears to be suitable choice.
pretrained Bestilbert-base-eee
Below we print all synonyms found by our automatic search.
Found_synonyns.
('lake natural', 'sun397'): [('lake', 'resisc45' )}, ('forest', 'resisc45'):
('tree house', 'sun397'), ('rainforest', 'sun397'), ('forest broadleaf', 'sun397'), ('forest road', 'sun397'), ('forest needleleaf', 'sun397'), ('forest path', 'sun397')
},
('athletic field outdoor', 'sun397') ): (''ground track field', 'resisc45') ),, ('lake sea', 'eurosat'), ): [('lake', 'resisc45')], ('rainforest', 'sun397') ): [(' forest' 'resisc45')] )],
('iceberg', 'sun397'): [[ ('snowberg'' "resisc45") ], ('meadow', 'resisc45'):
('field wild', 'sun397'), ('park', 'sun397'), ('field cultivated', 'sun397'),
('pasture land', 'eurosat') ('yard', 'sun397'), ('pasture', 'sun397') 1,
( ('pond', 'sun397'): [('lake', 'resisc45") ))], d'ground track field', 'resiscu5'): ('athletic field outdoor', 'sun397'), ), ('track outdoor', 'sun397')
I,
( ('desert vegetation', 'sun397'): [('desert', 'resisc45'). )
('marsh', 'sun397'), ): [('wetland', 'resisc45')) ( ('freeway', 'resisc45") )): [(' (Chighway', 'sun397')], ('islet', 'sun397'): [('island', 'resisc45')) )], (' ('permanent crop land', 'eurosat'): ('rectangular farmland', "resiscu5'), ('field cultivated', 'sun397')
},
('track outdoor', 'sun397') ): [('ground track field', 'resisc45" )
('swamp', "sun397') ): let'''''' 'resiscu5')
17

---

Published as a conference paper at ICLR 2025
('sea ice', 'resisc45"): [(' lice shelf', 'sun397'), ('ice floe', 'sun397') )}, ('desert', 'resisc45') ('desert vegetation', 'sun397'), ('desert sand', 'sun397')
J, ('snowberg', 'resisc45') ('iceberg', "sun397'), ('mountain snowy', 'sun397') ), ('snowfield', 'sun397')
1,
('railway station', 'resisc45'):
('train railway', 'sun397'), ('railroad track', 'sun397'), 'train station platform', 'sun397')
}, ('street', 'sun397'):
('commercial area', 'resisc45'), ) l'intersection', 'resisc45')
1,
('rectangular farmland', 'resisc45") ): [( permanent crop land', 'eurosat )},
('field wild', 'sun397'); t('meadow', 'resiscus)),, ('thermal power station', 'resisc45"): [''electrical substation', 'sun397' )],
('lake', 'resisc45'):
('lake natural', 'sun397'), ('lake or sea', 'eurosat'), ), ('pond', 'sun397')
1,
(' ('runway', 'sun397'): [('airplane', 'resisc45'), ), ('airport', 'resisc45') )
('baseball field', 'sun397') ): [(' ('baseeall diamond', 'resisc45')], ( ('industrial area', 'sun397')> ):
('industrial buildings or commercial buildings', 'eurosat')
],
('ice shelf', 'sun397'), ) [('sea ice', 'resisc45" )},
('airplane', 'resisc45'): [( (('runway', 'sun397'), ('airplane cabin', 'sun397')], )], ('park', 'sun397')s ): [( forest', 'resisc45'), ('meadow', 'resisc45')], ('thermal power station', 'resisc45'): [( ('eelectrical substation', 'sun397')], ))] ('electrical substation', 'sun397'): [( ('theeral power station', 'resisc45" )},
('field cultivated', 'sun397'):
('meadow", 'resisc45'), ('permanent crop land', 'eurosat') ('circular farmland', 'resisc45"), ('pasture land', 'eurosat')
},
('patio', 'sun397'), ): [( ('terrace', 'resisc45" )}, ('shopfront', 'sun397' ): [(' commercial area', 'resisc45')1, ('highway', 'sun397'): [('freeway', 'resisc45"), ('highway or road', 'eurosat ]
('circular farmland', 'resisc4s'):
('field cultivated', 'sun397'), ('pasture land', 'eurosat'), )
('pasture', 'sun397')
,,
('residential buildings or homes or apartments', 'eurosat'). ): ('residential neighborhood', 'sun397')
1,
('pasture land', 'eurosat'):
('meadow', 'resisc45'), ('field cultivated', 'sun397'), ) ('circular farmland', 'resisc45'), ('yard', 'sun397') ('pasture', 'sun397')
J,
('desert sand', 'sun397'); [('desert', 'resisc45') }},
('train railway', 'sun397'). [[ [('railway', 'resisc45'). )], ('wetland', 'resisc45') ): [[ (('marsh', 'sun397'), ('swamp', 'sun397')] )], ('commercial area', 'resisc45'):
('street', 'sun397'), ('shopfrant', 'sun397'), ('industrial buildings commercial buildings', 'eurosat')
}, ('industrial buildings commercial buildings', 'eurosat'); ('industrial area', 'sun397'), ) ('commercial area', 'resisc4s')
,,
18

---

Published as a conference paper at ICLR 2025
('airport', 'resisc45'): [(' [('airport terminal', 'sun397'), ) ( ('runway', 'sun397') )],
('tennis court outdoor', 'sun397'); ): [('tennis court', 'resisc45") )),
('baseball diamond', 'resisc45'); [
('baseball field', "sun397'), ('stadium baseball', 'sun397') ) ('batters box', "sun397')
J, ('railway', 'resisc4s'): [
('train railway', 'sun397'), ('railroad track', 'sun397'), )
('train station platform', 'sun397')
J.
('highway or road', 'eurosat'): [('highway', 'sun397')]
('yard', 'sun397'): [('meadow', 'resisc45'), ('pasture land', 'eurosat') )},
('raitroad track', 'sun397') ):
('railway station', 'resisc45'), ('railway', 'resisc45')
1,
('tennis court', 'resisc4s'): ): ('tennis court outdoor', 'sun397'), ('tennis court indoor', 'sun397')
J,
(' ('residential neighborhood', 'sun397') ): [
('residential buildings or homes or apartments', 'eurosat')
1,
('island', 'resisc45') )) [('islet', 'sun397')],
('train station platform', 'sun397') ): [('railway station', 'resisc45 )],
('pasture', 'sun397'):
('meadow', 'resisc45'), ('circular farmland', 'resisc45'), )
('pasture land', 'eurosat')
1, ('terrace', 'resisc45' ): [
('courtyard', 'sun397') ), ('promenade deck', 'sun397'),
('pavilion', "sun397'), ) ('patio', 'sun397'),
('carrousel', "sun397'), ('balcony exterior', 'sun397') )
('veranda', 'sun397')
1,
19


Reviews:
[{"review_id": "86T8BfNnts", "reviewer": "Reviewer_RYg4", "summary": "The paper titled \"Model Merging with SVD to Tie the Knots\" explores the challenge of merging Low-Rank Adaptation (LoRA) finetuned models. While model merging has shown success in combining fully-finetuned task-specific models, the same methods often fail when applied to LoRA finetuned models due to misaligned weight structures. To address this, the authors introduce KnOTS, a technique that employs Singular Value Decomposition (SVD) to align LoRA model weights, thereby improving the effectiveness of existing merging methods. KnOTS demonstrates up to a 4.3% improvement in merging performance across vision and language benchmarks. Additionally, the paper presents a new benchmark for assessing the generality of merged models, which evaluates their performance on a joint dataset combining multiple tasks.", "questions": "Please check the weakness.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  The concept of using SVD to align model weights for improved merging is novel and practical, addressing a previously unexplored limitation in merging LoRA models.\n2. The paper is generally well-structured, with clear descriptions of both the KnOTS methodology and experimental setups.", "weaknesses": "1. It would be beneficial if the authors could validate the effectiveness of the method on larger LLMs, such as LLaMA and Qwen2, for more in-depth evaluation.\n2. Although the method is effective, the improvements are limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aonPlE3vjj", "reviewer": "Reviewer_c5rA", "summary": "The paper \"Model Merging with SVD to Tie the Knots\" proposes KnOTS, a method to improve merging of LoRA-finetuned models, which traditionally struggle with alignment compared to fully-finetuned models. KnOTS uses SVD to align task-specific updates into a shared space, making it easier to merge LoRA models effectively with existing techniques. The authors also introduce a benchmark to test whether merged models generalize across tasks, showing that KnOTS boosts merging performance by up to 4.3% across various benchmarks in vision and language.", "questions": "See in Weaknesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* Model merging, specifically for LoRA models, is an interesting and cutting-edge field, with related techniques being widely proposed and explored in recent years.\n* KnOTS shows excellent performance across both vision and language tasks, enhancing merged model effectiveness and enabling better generalization on the newly introduced benchmark for multi-task data.", "weaknesses": "* As far as I know, merging LoRA models using SVD is not a new technique; implementations have long been available in some open-source libraries and are widely used. Therefore, the innovation in this paper is questionable and appears limited. I'd like to know what are the differences and advantages of the techniques proposed in this paper compared to the SVD-based merging techniques in these libraries?\n\n* The experimental work in this paper is insufficient and does not meet a certain standard; more experimental data is needed to support the authors' announced conclusions. Additionally, the writing quality needs improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F7fpdsoxLg", "reviewer": "Reviewer_AJPw", "summary": "This paper introduces a new technique to merge the parameters of different LoRAs onto the same model weights. The authors first show that CKA representations better align with the ability to merge model weights, both for fully fine-tuned approaches as well as LoRA-based ones. Then, they propose to use SVD to align the subspaces of different LoRAs. After doing so, the resulting matrices can be merged into a single model by applying previous techniques. The authors show that their approach better merges LoRA weights into a single model, for both vision tasks and language tasks.", "questions": "1. Can you analyze and discuss potential reasons for the discrepancy in performance gains between KnOTS-TIES and KnOTS-DARE-TIES?\n2. Why is the better CKA alignment given by KnOTS not resulting in better performance with KnOTS-DARE-TIERS? Are there additional analyses you could run to better understand the relationship between CKA alignment and performance across different merging method?\n3. Can you provide more insights into why row-wise KnOTS does not work?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors show that CKA representations seem to align with model merging abilities, without some limitations given by orthogonality approaches.\n2. The authors propose to merge the LoRA weights after SVD, and showcase better performance than using existing full-rank approaches.\n3. The authors propose to evaluate merged models on a multi-task benchmark that they obtained by combining the individual datasets in Ilharco et al. (2023).", "weaknesses": "1. While the performance of KnOTS-TIERS is usually significantly better than TIERS, it is not the case for DARE-TIERS. This is not discussed in the paper, and it would be good to understand this behavior.\n2. While the CKA alignment given by using KnOTS is significantly better than by the original LoRA weights, the performance improvements (e.g. on DARE-TIERS) are less pronounced, leaving the reader wondering whether CKA is indeed a good-enough metric for weight alignment.\n3. The authors should down-weight their “novelty” contributions towards creating a new benchmark by simply putting together the datasets of a previous benchmark. The benchmark itself is a contribution of the paper, but not one that the reviewer feels should be stressed as much. For example, you could present it as a useful extension rather than a major novel contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vYMtpi1TFi", "reviewer": "Reviewer_6fFE", "summary": "This paper introduces a method called KnoTs which provides a strategy for merging LoRA modules. First the authors show that centered kernel alignment (CKA) between activations of LoRA modules is not as high as that of fully fine-tuned models, demonstrating the need for developing methods specially for merging LoRA modules. They propose to perform SVD on the task updates and perform merging operation on V matrix in the SVD which is known to contain task specific information. First they show that CKA on these V matrix is high and this would eventually help in improving merged model performance. They run experiments on both vision and language tasks to demonstrate the effectiveness of the method and their method achieves better normalized accuracy in comparison to other SOTA methods specially designed for merging fully fine-tuned models.", "questions": "* What is the motivation for performing merging operation on V matrix from SVD?\n* Does the author believe that merging methods would have a greater negative impact if the base model’s zero-shot performance were lower?\n* Is it possible to compare merged models performance to multi-task model performance on all the datasets?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* CKA analysis provided in the beginning to show misalignment of weights in LoRA provides good motivation to develop specialized method for merging LoRA trained models.\n* Proposes a merging method for combining LoRA modules using SVD decomposition on weight update matrix. Shows improvements over different merging baselines for both vision and language tasks. \n* KnoTs demonstrates superior scalability compared to other methods, particularly as the number of merged models increases.", "weaknesses": "* This method doesn't allow merging of LoRA modules with different ranks.\n* The datasets used in this work are selected to ensure strong baseline performance, where models like ViT and Llama already demonstrate high zero-shot accuracy. This high initial performance makes it challenging to quantify the specific gains achieved through the proposed method.\n* Multi-task trained performance should be considered as one of the baseline to understand how far is the merging performance from multi-task performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)