Parameter-Efficient Multi-Task Model Fusion with Partial Linearization
Authors: Anke Tang
Li Shen
Yong Luo
Yibing Zhan
Han Hu
Bo Du
Yixin Chen
Dacheng Tao
Venue: ICLR 2024 poster (2024)
Keywords: model fusion
parameter-efficient fine-tuning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: we propose to partially linearize parameter-efficient fine-tuning models to improve the multi-task model fusion capability.

Abstract: Large pre-trained models have enabled significant advances in machine learning and served as foundation components.
Model fusion methods, such as task arithmetic, have been proven to be powerful and scalable to incorporate fine-tuned weights from different tasks into a multi-task model. 
However, efficiently fine-tuning large pre-trained models on multiple downstream tasks remains challenging, leading to inefficient multi-task model fusion.
In this work, we propose a novel method to improve multi-task fusion for parameter-efficient fine-tuning techniques like LoRA fine-tuning.
Specifically, our approach partially linearizes only the adapter modules and applies task arithmetic over the linearized adapters.
This allows us to leverage the the advantages of model fusion over linearized fine-tuning, while still performing fine-tuning and inference efficiently.
We demonstrate that our partial linearization technique enables a more effective fusion of multiple tasks into a single model, outperforming standard adapter tuning and task arithmetic alone.
Experimental results demonstrate the capabilities of our proposed partial linearization technique to effectively construct unified multi-task models via the fusion of fine-tuned task vectors. 
We evaluate performance over an increasing number of tasks and find that our approach outperforms standard parameter-efficient fine-tuning techniques. The results highlight the benefits of partial linearization for scalable and efficient multi-task model fusion.

Published as a conference paper at ICLR 2024
PARAMETER EFFICIENT MULTI-TASK MODEL FUSION WITH PARTIAL LINEARIZATION
Anke Tang', Li Shen?*, Yong Luo'; Yibing Zhan', Han Hu', Bo Du', Yixin Chen', Dacheng Tao WWhhan University, China JJ Explore Academy, China 'Beijing Institute of Technology, China *ashingtonn University, USA 'Nanyang Technological University, Singapore
(anketang, luoyong, dubo) @whu. edu.
meaathshenii@gmie com, zyb 2yb jyemail wuw.ustcc.... edu. en ihhu@bit edu. cn cchnncsse. wustl... edu ddcchnng. tao edu.sg
ABSTRACT
Large pre-trained models have enabled significant advances in machine learning and served as foundation components. Model fusion methods, such task arith- metic, have been proven to be powerful and scalable to incorporate fine-tuned weights from different tasks into a multi-task model. However, efficiently fine- tuning large pre-trained models multiple downstream tasks remains challeng- ing, leading to inefficient multi-task model fusion. In this work, we propose novel method to improve multi-task fusion for parameter-efficient fine-tuning techniques like LoRA fine-tuning, Specifically, our approach partially linearizes only the adapter modules and applies task arithmetic over the linearized adapters. This allows us to leverage the the advantages of model fusion over linearized fine- tuning, while still performing fine-tuning and inference efficiently. We demon- strate that our partial linearization technique enables a more effective fusion of multiple tasks into single model, outperforming standard adapter tuning and task arithmetic alone. Experimental results demonstrate the capabilities of our proposed partial linearization technique to effectively construct unified multi-task models via the fusion of fnne-tund task vectors. We evaluate performance over an increasing number of tasks and find that our approach outperforms standard parameter-efficieet fine-tuning techniques. The results highlight the benefits of partial linearization for scalable and efficient multi-task model fusion.
INTRODUCTION
Pre-trained models play a crucial role in machine learning systems, serving as foundational com- ponents. In order to optimize their performance for specific downstream tasks (Ilharco al., 2022; Wortsman et al., 2022b; Matena & Raffel, 2022), address biases or undesired behavior (Santurkar al., 2021; Ribeiro Lundberg, 2022; Murty et al., 2022), align them with human preferences (Ouyang et al., 2022; Ribeiro & Lundberg, 2022), or incorporate new information (Mitchell et al., 2022a;b), is often necessary to further customize or edit these models after pre-training
Multi-task model fusion is a powerful approach to extracting knowledge from models fine-tuned on different downstream tasks, allowing us create unified model that performs well across multiple tasks. This approach proves to be helpful when only the fine-tuned model can be obtained but the data remains private and inaccessible (Wu et al., 2019; Lou et al., 2020; Tang et al., 2023); besides, of the multi-task model, since compared to the pre-trained model,
it can expedite the we have a better starting point from which to fine-tune (Kaddour, 2022; Sanyal et al., 2023). In recent studies, researchers have introduced many powerful methods for editing pre-trained models and merging speciffe fine-tuned models. We further introduce this field in Section 2.
The vast parameter size of pre-trained models poses challenges in terms of computational efficiency and memory usage during the fine-tuning process; these difficulties further lead to inefficient multi- task model fusion. Fine-tuning large-scale models requires significant computational resourees and
"Corresponding authors.

---

Published as a conference paper at ICLR 2024
memory. making the process inefficient. To address this concern, many parameter-efficient fine- tuning (PEFT) techniques are proposed, these approaches significantly reduce the number of pa- rameters that need to be fine-tuned, meanwhile achieving comparable performance to full parameter fine-tuning. However, naively combining models that were fine-tuned in a parameter-efficien man- can more readily result in representational interference between tasks, which makes model fu- sion algorithms suboptimal. While some research has explored fusing parameter-efficient fine-tuned models for multi-task model fusion (Chronopoulou et al., 2023; Zhang et al., 2023; Huang et al., 2023), performance still lags considerably behind fusing fully fine- fine-tuned models. Therefore, the key challenge is performing PEFT while also preventing negative interference between task specific representations. Motivated by these concerns, we aim to enhance the multi-task model fusion capa- bilities of parameter-efficien fine-tuned models.
In this work, we present a novel approach to improve the multi-task fusion capability of parameter- efficient fine-tuning models. Recent advances in understanding task arithmetic and weight disen- tanglement have demonstrated that linearizing the entire model and fine-tuning the corresponding tangent model in tangent space can enable more effective task arithmetic (Guillermo Ortiz-Jimenez et al., 2023). While promising, completely linearizing large mre-trained model can be computa- tionally expensive. Typically, this approach requires two to three times the computational resources needed for fine-tuning and inference. Our key insight is that we can perform efficient fine-tuning and disentangle task representations by only linearizing a subset of parameters appended to fixed pre- trained backbone. essence, we are proposing hybrid approach that leverages parameter-efficicee fine-tuning for efficiency, while locally linearizing the adaptable modules to attain enhanced disen- tanglement and improved multi-task fusion capabilities.
Our experiments on image classification and natural language processing tasks demonstrate that our partial linearization technique enables more effective model fusion, achieving superior performance across tasks compared to conventional PEFT methods and model fusion algorithms alone. In some cases, our proposed method is even comparable to full fine-tuning. In addition to the direct compar- ison of multi-task model fusion performance, we also visualize the weight disentanglement gain of our method on different downstream task pairs. The results show that our method can effectively improve the weight disentanglement of parameter-efficien fine-tuning models, which is the key to improving the multi-task fusion capability of parameter-efficien fine-tuning models.
To summarize, contributions are follows:
We propose a novel partial linearization method for parameter-efficicet fine-tuning models in order to improve the multi-task fusion capability of fine-tuned task-specific models with a low computational cost overhead.
We apply our method to the LoRA modules to construct Linearized LoRA (L-LoRA) modules and conduct extensive experiments on seven tasks from the GLUE benchmark to demonstrate that method is effective in improving the task fusion capability of fine-tuned task-specific models.
We present an extension of weight disentanglement property and weight disentanglement error for parameter-efficient fine-tuning models to analyze the impact of the linearization process on parameter-efficient modules. We evaluate fine-tuned models to visualize and analyze the weight disentanglement gain of L-LoRA on downstream tasks.
2 RELATED WORK
Model Fusion. In practical training scenarios, is common practice to train multiple models with various hyperparameter configurations. Subsequently, the top-performing individual model is se- lected, or an ensemble of models is constructed attain optimal performance (Dietterich, 2000). However, deploying ensemble of models incurs additional computing expenses during infer- ence. Mcanwhile, selecting a single top top-performing model typically entails discarding the remaining trained models, resulting in neglecting their encoded knowledge.
Model fusion integrates the knowledge from multiple models into a single unified model (Li et al., 2023). There two main works for model fusion. The first line of work focuses the fusion of entire models. Interpolating the entire weight space of multiple models by taking their average or weighted combination can be effective for model fusion as demonstrated in previous work (Garipov
2

---

Published as a conference paper at ICLR 2024
et al., 2018; Ilharco et al., 2022; Matena & Raffel, 2022; Wortsman al., 2022a; Ilharco et al., 2023; Liu & Soatto, 2023). When the models are not well aligned or lie in different loss basins, techniques have been proposed to align the feature representation before fusion (Liu et al., 2022a; Wang et al., 2022; Ainsworth et al., 2023; Jin et al., 2023). This feature alignment helps matching the models to similar behaviors transforming to same loss basin. Fusing the full model can leverage knowledge from all layers but can be computationally expensive.
The second line of work focuses on fusing parts of the model. For example, (George Stoica al., 2023) aligns and fuses a large portion of the models to combine models trained on disjoint tasks and supports partial zipping up to a specified layer create multi-head model. (Yadav et al., 2023) removes redundant and potentially interfering values and resolves sign conflicts before doing weight interpolation. Other researches explore merging parameter-effiient modules as more lightweight approach (Chronopoulou et al., 2023; Huang et al., 2023; Zhang et al., 2023; Wu et al., 2023).
Parameter-efficint fine-tuning (PEFT). The use of transfer learning from pre-trained models has become widely accepted practice, resulting in impressive performance across various domains (De- vlin et al., 2019; Radford et al., 2019). In addition to fine-tuning all the model parameters, parameter- efficient fine-tuning techniques have been proposed to reduce the number of parameters that need to be fine- tunet such as adapter tuning (Houlsby et al., 2019), prefix tuning (Li & Liang, 2021), prompt tuning (Lester al., 2021), LoRA (Hu et al., 2021) and (IA)3 Liu et al. (2022b) al. These methods only update a small subset of (extra) parameters, lowering computational costs and memory requirements compared to full fine-tuning, while retaining comparable performance.
RETHINKING MODEL FUSION IN PEFT SETTING
In this section, we first introduce the preliminary formulas of parameter-efficient fine-tuning and define task vectors for parameter-efficient fine-tuning models. Then, we introduce the weight disen- tanglement property and weight disentanglement error in PEFT setting.
3.1 PRELIMINARY
Under the framework of supervised learning, assume we have a set of tasks T = (71, T2,".. ... Tnl and pre-trained language model feo parameterized by 8g which is trained massive dataset. Each task 7; is associated with a dataset Dr, = ((x%)) (1) )) where 8; is dataset size of D (1.)1 )5=1 is the input data, and (u()) )イ= the output data.
Given a specific task ri the pre-trained model fa, is fine-tuned on the task-specific dataset Dr, to obtain task-specific model. We consider full parameter fine-tuning and parameter-efficien fine- tuning methods. If the model fa, fully fine-tuned on task Tr the parameters Ou are updated to 0 by minimizing the loss function 0; arg ming L(7;; 0) on the task-specific dataset Dr,- Otherwise, the model fo, is parameter-efficientl fine-tuned on task т;, the original parameters 0g are fixed and some extra added parameters (the number of parameters of is much less than fg. i.e., | < 1) are updated to o; by minimizing the loss function é; = arg ming [(T;; lo Φ) on the task-specific dataset D,.- Where L typically the Cross-Entropy loss function maximize conditional posterior probability p(ulz), more specifically
C(T;; 0. Φ) Σ LCe (Jeo (x) 2
E RR.
Task vector. We define the task vector as the difference between the fine-tuned trainable parameters and their initial state. The task vector is denoted as v; = - 9n for full fine-tuning or P; = o; do for parameter-efficient fine-tuning. We elaborate on this definition in Appendix A, discussing the rationale behind this definition and its associated benefits.
3.2 WEIGHT DISENTANGLEMENT FOR PARAMETER FARAMETER-EFRICIEN FINE-TUNING
In this subsection, discuss and present extension of the weight disentanglement property and weight disentanglement error to parameter-effict efffcien fine-tuning models, which are original intro- duced in (Guillermo Ortiz-Jimenez al., 2023).
3

---

Published as a conference paper at ICLR 2024
Cani DTD
GTSRB
-TSRB
Jy (C
Az (DTD)
Au (GTS8B)
(GTSKB)
(a)
(b)
(C)
Figure 1: Loss landscape visualization. Here, we visualize the loss landscape 8) + [(:); for CLIP model on combinations of three downstream image classification tasks by interpolating on the 2D plane. = 8g + Σ Ao(0; 0o), where 9g are the pre-trained weights, 0, are the task-specific full fine-tuned weights for task 7.. From these heatmaps, we observe that task-specific models reside in the same loss basin when evaluated on the joint task.
Weight disentanglement for parameter-efffcient fine-tuning. The weight disentanglement is de- fined by the function outputs of the merged model. Consider a PEFT model denoted as fe, (2; do) whose tunable weights initialized Φo. We define this model to possess weight disentangle- ment characteristics in relation to a specific set of task vectors {wi|es = Φ; ΦoJ and their if the following condition is satisfied: corresponding support datasets (D,, DiEM!
folæ; Ø + Σ (pps) = Σ (x; Aprs) + 90(x),
(2)
where (x;;; (,Ps)) = O for x & Dr, and i 1,2,..,, and (o()) = 0 for x UiE(M) Dr.- This condition ensures that the model fo(z; o) can be expressed as linear combination of individual terms 9: (z; \u4)), incorporating the respective task vectors and an additional term (v()). Through adhering to this disentanglement condition, the model demonstrates the desired capability of effec- tively separating and disentangling the weights associated with the function outputs, enhancing its capacity capture task-specific information.
However, important to highlight that weight disentanglement characteristic specific to the function outputs and is not directly linked to their performance. In other words, model may exhibit weight disentanglement property yet still perform weakly. Since weight disentanglement of function outputs does not guarantee task success in itself, other factors, such the evaluation metrics, can be non-linear to determine the overall performance. Therefore, it is crucial to consider weight disentanglement alongside other performance metrics when analyzing model behavior and effectiveness across different tasks.
Weight disentanglement error in PEFT setting. Given two task vectors {vi we define the extension of disentanglement error for model fo, (z; Φo) as follows:
{()1,22) Σ E(DP(D.,)) [dist( (o: 20 + Avi)), for do daTy A222)))
(3)
Where dist(::)) represents any chosen distance metric between two vector outputs. Taking classifi- cation tasks as an example, dist can be chosen as the prediction error, i.e., dist(y) }/) = 1 * y2)- Intuitively, this measures how much the prediction changes when adding each task vector individ- ually versus adding both together. If task knowledge is well-disentangled, adding the specialized vectors independently or together should yield similar predictions, minimizing &. The disentangle- ment error quantifies the degree of interference between task vectors. Lower values indicate task knowledge that can be combined with less destructive interaction during model merging.
3.3 GEOMETRIC INTUITION FOR MODEL FUSION
In previous work, it is generally believed that the closer the orthogonal between task vectors, the less interference between tasks and the better the effect of weight disentanglement (Ilharco et al., 2023; Chen et al., 2023; Guillermo Ortiz-Jimenez et al., 2023). The intuition is that orthogonal
4

---

Published as a conference paper at ICLR 2024
Cup
130
1.04
6:20
0.15
0.15
OL13
Caraeli 1:00
520
6:12
0.9
0.30
6.99
033
836
0.30
BTD
6:5
D1D-X11 [:9
0.59 0.34
0:20
1.20
0.5
GTNRR
020
0.20
GTTRR
6.12 8:19
110
020
GTSER 0:55 0:45 676
out
RES18043 Au19
0.21
011
622
825
637
0:22
1350
RESISCAR -010 0.14 0,20
1.00
0.
SUND33
Au19 0.20 6:7 021
0.13
100 011
SUN977-8.38
624
0:1
033
1:33
SENN 2331 1:14 one aa5
-02
SVEEN AU1
pow
Con mm
Greea
C4
mm
Can
DID-
(a) full fine-tume line-tuning
(b) LoRA fine-tuning
(c) L-LoRA fine-tuning
Figure 2: Similarity heatmaps. These figures show heatmaps of the cosine similarity between task vectors from task-specifie CLIP models (Radford et al., 2021) fine-tuned on different tasks. (a) Cos similarity matrix of task vectors when using full fine-tuning of the entire model. (b) Task vector similarities when using LoRA. (c) Cos similarity of task vectors when using L-LoRA, our proposed partial linearization approach that linearizes PEFT modules and fine-tunes in tangent space.
task vectors indicate that the specialized knowledge captured for each task lies in distinct subspaces with minimal overlap or redundancy. This enables better preservation of specialized knowledge and avoids destructive interference during model merging.
We visualize the loss landscape of the joint tasks in Figure 1 (Li et al., 2018), where we interpolate between the pre-trained weights 6g and two task-specific fine-tuned weights f1, 02 for CLIP mod- els (Radford et al., 2021). The 2D heatmaps show the loss values [(T], 0) + [(T2, 0) evaluated on the joint tasks. Notably, we observe that the task-specific models reside the edge of the same low-loss basin, with no significant barriers or discontinuities in between. It provides geometric intu- ition for why a simple linear arithmetic operation of task specific parameters 01 and 8g can produce effective merging for multitask learning, as empirically observed.
The results in Figures 2(a-b) and 8(a-b) show the cosine similarity between task vectors from CLIP and Flan-T55 (Chung et al., 2022), which are fully fine-tuned or LoRA fine-tuned on image classifi- cation tasks and natural language processing (NLP) tasks respectively. We observe that vectors from full fine-tuning are closer to orthogonal than those from LoRA, which indicates that models fine- tuned with full fine-tuning are more independent than those in LoRA. This finding is consistent with the discussion about task addition in (Ilharco et al., 2023), the experimental results from Figures 4 and also support our statement. The experimental details are described in Appendix C.
Remark 3.1 At first glance, it may seem unfair to compare the task vectors from full fine-tuning to those from parameter-efficien fine-tuning methods, since full fine-tuning has access to much more truinable parameters. In fact for full fine-tuning we have fa(r) [o(2; Φ0) so that
(Vi,V))
{[0: 0, 0] [0 0o. 0))
{TA: Gg. Øu Øul, [0, Gn du - Φol)
|1 11312 01 6, 01 10, Po 0ll 11 0o do Φ 0, 0 Φ0 Φ0
on the other hand, for parfmeter-efffcient fine-tuning we have
(g),
(0, Φ. Qul, 10, 65 du))
(0o O d. Φul, 10 0u のb Øul
| 0, Φ. 20ll22 11, ΦÚ 0oll2 00 G, Φi - Φ 0. Go Φ3 Φ 112
Therefore, the comparison between full fine-tuning and parameter-efficint fine-tuning methods can be made fair by viewing them as updating different subsets of the joint parameter space (0, +).
METHODOLOGY
Based on the above observations, we propose novel partial lincarization method for parameter- efficient fine-tuning models in order to enhance the weight disentanglement and improve the multi- task fusion capability of fine-tuned task-specific models with low computational cost overhead.
4.1 LINEARIZED GARAMETER-EFFICIEE FINE- FINE-TUNING
To monitor the progressive of trainable parameters & throughout the training trajectory, we treat ø as function of the training time i, denoted ((t). Where ranges from 0 to T. By employing
5

---

Published as a conference paper at ICLR 2024
Thinable Parameters
Fixed Paramcters
I Linearized Part
Feed Farvar
Foal Fanard
Feod Faevad
Fool Farari
x Lave Nor
Ad Lager horn
Laver
Layer Norm
Alaxtian
Alzatien
(b)
(d)
Figure 3: Four types of fine-tuning paradigms. (a) Full parameter fine-tuning. (b) Full-model linearization. (c) Parameter-efficient fine-tuning, (d) Linearized parameter-efficien fine-tuning. In this paper, explore LoRA fine-tuning and linearized LoRA (L-LoRA) fine-tuning.
first-order Taylor expansion, we can streamline the dynamies of network learning as (Weng, 2022):
fao (2; (()) (2: ((t)) fao (2; (()) V Vofe, (z; 6(0)) (Ø(t) =60)).
(4)
The linearized model, described by (4) also referred to as a tangent model, approximates the be- havior of the neural network fe, (z,, (()) at a specific time t, where 0u, fe, (z; (0)), fo (z; (0)) are all constants. Consequently, the lincarized function far (d((); x) is a linear function of o(t).
Grounded in the key result detailed in Appendix B, where we characterize the evolution of model outputs. We hypothesize that by partially linearizing only subset of parameters within the adaptable modules, the model can gain from a more disentangled representation. This partial linearization facilitates the separation of task-specific knowledge; hence, model fusion becomes more robust against the negative transfer effects and task interference in multi-task learning settings.
Figure 3 illustrates four different types of fine-tuning paradigms. The first three are existing meth- ods, while the fourth is our proposed partial linearization fine-tuning method. (a) The full fine- tuning paradigm, all parameters f are updated during fine-tuning. (b) The full-model linearization paradigm, but instead we fine-tune the tangent model in the tangent space. It is worth noting that al- though the Jacobian-vector products can be computed in a single forward pass (Pearlmutter, 1994), training and inference in this paradigm are usually twice or three times as expensive as full fine- tuning, quantitative comparison is shown in Tables 3 and 4. (c) The PEFT paradigm, only a small number of parameters & are updated while 0 is fixed at 0g- (d) the linearized PEFT (L-PHFT) paradigm, which similar to PEFT fine-tuning, where á is updated while On is fixed. However, in L-PEFT, only the linearized PEFT modules are fine-tuned in the tangent space. This approach incurs only a fraction of the training and inference costs compared to standard PEFT fine-tuning. In this paper, we explore LoRA fine-tuning and linearized LoRA (L-LoRA) fine-tuning.
From Figures and 8, we observe that task vectors from L-LoRA fine-tuning are closer to or- thogonal than those from LoRA, which indicates that models fine-tuned with L-LoRA are more lask-independent than those in LoRA. This is because the trainable parameters & with L-LoRA are fine-tuned in the tangent space, which is linear space, while LoRA are fine-tuned in the orig- inal ambient space, which is a non-linear space. Besides, in some cases, the cosine similarity of task vectors from full fine-tuning are more orthogonal than those from L-LoRA. This may due to the larger space of trainable parameters available to full fine-tuning, providing an extremely high- dimensional space to encode specialized task knowledge. So even though L-LoRA tunes parameters in linear subspace, the total capacity is restricted compared to full fine-tuning.
4.2 THE PARAMETER-SCALIN LAW FOR WEIGHT DISENTANGLEMENT
Previous findings in (Ilharco et al., 2023; Guillermo Ortiz-Jimenez et al., 2023) also suggest an empirical parameter-scaling law wherein weight disentanglement improves with increased number of trainable model parameters. Intuitively, (1) The larger the number of parameters, the stronger the expressive ability of the model, which can better learn the correlation between different tasks, thus facilitating weight disentanglement. (2) The paramtteriziai brought about by a large number of parameters can provide more degrees of freedom to learn the feature representation required for

---

Published as a conference paper at ICLR 2024
fdl-inan: SeepleAAvzapee
RPT man
LetA mon
L-ARA
L-LORA man
Arthhentic
Ties-Meping
0.75
6.75
0.75
0.50
0.30-
0,50
0.50-
0.25
0.25-
0.25
01.25
0.0
0.5
6.0
Tak
Tack
Bodk1
Taok
(a) CLIP-ViT-B-I6 model on image classification tasks.
full-immm::
FFT man
LoKA
Lell man
L-LakA
L-LoXA ment
Sample Average
Task Artthentic
Ties-Merping
LomHub
0.75
0.75
0.50
0.50
0.50
0.50
0.35
0.5
0.24
0.25
0.00
(b) Flan-T5 Maan-T--Base model on NLP tasks.
Figure 4: Pairs of model fusion. These figures show scatter plots demonstrating the performance of different model fusion techniques pairs of tasks. Each plot corresponds to different fusion method. The x and y axes in each plot denote the normalized scores on the two tasks. Points indicate the performance of specific instances. Dashed lines represent the average performance per task for each method. (a) Image classification tasks. (b) NLP tasks.
each task and reduce parameter competition or interference between different tasks. Figure 5(a) shows that even though L-LoRA fine-tuning has far fewer trainable parameters compared to full fine-tuning (<1%), it achieves comparable average normalized scores to full fine-tuning.
5 EXPERIMENTS
In this section, conduct experiments on diverse tasks spanning both vision and natural language domains. We employ CLIP and Flan-T5 models on vision and language domains, respectively. For detailed descriptions, please refer Appendix C.
Throughout the following experiments, we compare the performance when models are fine-tuned using full fine-tuning. LoRA (Hu al., 2021), and our proposed L-LoRA fine-tuning. more detailed experimental description of model fine-tuning is provided in Appendix C.
Baseline methods. We compare our fine-tuning method combined with four baseline multi-task model fusion methods: simple average, task arithmetic (Ilharco et al., 2023; Zhang et al., 2023), ties-merging (Yadav et al., 2023), and LoraHub (Huang al., 2023). Please refer to Appendix E. for more detailed description of these methods and hyper-parameter settings in our experiments.
5.1 MULTI-TASK MODEL FUSION ON VISION AND LANGUAGE DOMAINS
We extensively investigate the multi-task model fusion by considering all possible subsets of the tasks, amounting to a total of 2" - (n++1) subsets (excluding subsets with single task and an empty set). We employ various model fusion algorithms accomplish this. Subsequently, we assess the performance of the final model across the downstream tasks in both the vision and language domains. Detailed experimental settings are provided Appendix
Figure 4 shows scatter plots demonstrating the performance of four fusion methods: simple averag- ing, task arithmetic, ties-merging, and LoraHub. Each plot corresponds to one fusion approach ap- plied to two tasks. This analysis of pairs of tasks highlights distinctions between fusion techniques. For image classification tasks, we observe that L-LoRA tine-tuning surpasses LoRA fine-tuning with all four model fusion algorithms, with task arithmetic and LoraHub exceeds full fine-tuning. As for NLP tasks, we observe comparable results between LoRA fine-tume tuning and L-LoRA fine-tuning, although not strong as full fine-tuning in some cases.

---

Published as a conference paper at ICLR 2024
fell-ineta::
LoRA
L-LoRA
Simple Average
Task Arihmeric
Ties Merging
LoraHub
Noeegge Nomaltzze Score
0.8
8.6
Taak
Taska
Taskk
(a) CLIP-ViT-B-16 model on image classification tasks.
fill-frmmm:
LoRA
I-LoRA
Simple Average
Task Arithmetic
Ties Merping
LoraHuh
Aeetage Nornolond Score
0.2
0.1
Nanber Tades
Nunba
Number
(b) Flan-T5-Base model on NLP tasks.
Figure 5: Multi-task model fusion. we construct multi-task models by utilizing task vectors spe- cific individual tasks, employing various model fusion algorithms. In the evaluation, the x-axis represents the number of task vectors used in building the multi-tasi task model, while the y-axis repre- sents the average normalized scores of the resulting models across all seven downstream tasks. The lines on the plot represent the average normalized scores of all multi-task models when considering a fixed number of tasks, and the shaded area corresponds to the standard deviation.
Table 1: The average of the normalized scores for all combinations (avg. + std.).
Fine-tuning method
Simple average
Task Arithmetic
Ties-merging
LoraHub
CLIP-VIT-B-16
full fine-tuning
0.65 + 0.04
0.69 + 0.06
0.69 = 0.06
0.61 = 0.03
LoRA fine-tuning
0.58 + 0.18
0.64 + 0.03
0.63 = 0.03
0.65 + 0.04
L-LoRA fine-tuning
0.66 + 0.02
0.77 + 0.05
0.73 + 0.04
0.77 + 0.05
Flan-T5-Ban 0.32 + 0.08
full fine-tuning
0.19 + 0.08
0.28 + 0.08 0.08
0.24 = 0.06
LoRA fine-tuning
0.06 + 0.07
0.21 + 0.07
0.21 + 0.08
0.23 + 0.07
L-LoRA fine-tuning
0.05 + 0.07
0.26 + 0.05
0.24 + 0.06
0.28 + 0.08
Figures 5, 10 and 11 show the performance of multi-task models constructed using different fine- tuning methods and fusion algorithms over an increasing number of task vectors. Table 1 shows the average over all combinations. In line with (Ilharco al., 2023), we compare the normalized scores of merged multi-task models. The normalized score is defined as the absolute score divided by the score the corresponding single-task model. These results highlight the capability model fusion methods to effectively leverage knowledge into single unified multi-task model.
In Figure we also notice that for full fine-tuning the ties mersimg performs better than LoraHub on visual tasks, while this is not the case on NI.P tasks. Ties- introduces several innovations address the challenges of parameter interference. This indicates that the degree of interference between visual tasks is higher than that between NLP tasks. Compare the cosine similarities matrix between Figure 2(a) and Figure 8(a), indeed, the task vectors on vision domain have higher cosine similarity, which is consistent with this model fusion phenomenon. Since the higher cosine similarity between task vectors implies greater redundancy and overlap the knowledge captured. This results in more destructive task interference when naively merging the specialized models.
For both image classification tasks and NLP tasks, we observe that the normalized scores of L- LoRA surpass LoRA fine-tuning in most cases, and I.-LoRA even exceeds full fine-tuning with task
8

---

Published as a conference paper at ICLR 2024
COLA-MRIC
COLA-RTE
MRPCC-TTE
MRPC-SST2
LoRA
10
93
0.0
-1
0 MRPC-RTE
COLA-MRIC
COLA-RTE
MRPC-SST2
20
155
A2
1.0
A1
A1
A2
Figure 6: Weight disentanglement error visualization (LoRA vs L-LoRA): The heatmaps demon- strate the disentanglement error, denoted as E(A1, 22), for both a LoRA fine-tuned flan-t5-base mod- els and their L-LoRA counterparts across various NLP task pairs. In these heatmaps, lighter regions indicate lower weight disentanglement error, indicating stronger weight disentanglement. Within the heatmaps, the red box outlines the specific search space utilized to calculate the optimal À hy- perparameters for our multi-model fusion experiments (refer to Section 5.1 and Appendix E).
arithmetic and ties-merging. In addition, LoRA is more effective than L-LoRA for a smaller number of tasks some cases, but as the number of task vectors increases, L-LoRA surpasses LoRA. This phenomenon also suggests the effectiveness of L-LoRA fine-tuning for multi-task model fusion, we further discuss it in Appendix E.2.
A more in-depth analysis comparison between LoRA and L-LoRA is provided in Figures 10 and 11. For further elaboration and supplementary details, please refer to Appendix E.
5.2 WEIGHT DISENTANGLEMENT VISUALIZATION
To analyze the impact of the partial linearization process PEFT modules weight disentangle- ment, we evaluate LoRA fine-tuned models and their L-LoRA counterparts to visualize in Figure 6.
The heatmaps allow us compare the error landscapes in the interpolation space. There are clear differences between the LoRA and L-LoRA heatmaps. The results indicate that the LoRA models exhibit negligible disentanglement error confined narrow region in proximity their pre- trained model. On the contrary, the L-LoRA models showcase low-error distribution across wider area. These findings strongly suggest that the proposed partial linearization process employed in L-LoRA fine-tuning offers notable advantages in effectively disentangling the weights learned for each specific task, thereby enhancing its ability to capture task-specific information and improve performance across diverse tasks.
CONCLUSION
In this work, we propose novel partial linearization method to improve multi-task fusion for parameter-effiiiet fine-tuning techniques. Specifically, our approach partially linearizes only the adapter modules and applies model fusion algorithms over the linearized adapters. This allows us to leverage the the advantages of model fusion over linearized fine-tuning, while still performing fine- tuning and inference efficiently. Our experiments conduct on both vision and language domains extensively. The results demonstrate that partially linearized LoRA models enable a more effective fusion of multiple tasks into unified multi-task model, outperforming standard LoRA fine-tuning and model fusion algorithms alone.
Future work could further explore partial linearization, where some parameters fine-tuned in linear space while others remain nonlinear. This hybrid approach could trade-off between special- ized single-task performance and weight disentanglement capability.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work is supported in part by the National Key Research and Development Program of China under No. 2023YFC2705700, the Special Fund of Hubei Luojia Laboratory under Grant 220100014, the National Natural Science Foundation of China (Grant No. 62225113, U23A20318 and 62276195), and CCF-Zhipu AI Large Model Fund OF 202224.
REFERENCES
Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git Re-Basin: Merg- ing Models modulo Permutation Symmetries, March 2023. URL http:/ http://arxiv http://axivvvvv/ abs/2209. 04836. arXiv:2209.04836 [cs] Read_Status: Read Read_Status_Daii 2023-08- 31110:29:51.9972.
Hong-You You Chen, Jike Zhong, Mingda Zhang, Xuhui Jia, Hang Qi. Boqing Gong, Wei-Lun Chao, and Li Zhang. Federated Learning of Shareable Bases for Personalizatioo-FFrendyy Image Clas- sification, April 2023. URL http::/aax..v.rrg/aaas////////// 204.. 07882. arXiv:2304.07882 [cs].
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote Sensing Image Scene Classification: Beneh- mark and State of the Art. Proceedings of the IEEE, 105(10): 865-1883, October 2017. ISSN 0018-9219, 2256. doi: 10. 1..11000K00..22...255599999 URL http://arxiv.orrt [cs].
abs/1703 00121.
Alexandra Chronopoulou, Matthew E. Peters, Alexander Fraser, and Jesse Dodge. AdapterSoup: Weight Averaging Improve Generalization of Pretrained Language Models, March 2023. URL http: /ttt:://axxie org/ als 0 arXiv:2302.07027 [cs] Read Status: Read Read Status_Date: Date: 2023-08-311009-554553777
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling Instruction- Finetuned Language Models, December 2022. URL http: org/ abs/2210 11416. arXiv:2210.1 14416 [cs].
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing Textures the Wild. In 2014 IEEE Conference on Computer Vision and Pattern Recogni- tion, pp. 3606-3613, Columbus, OH, USA, June 2014. IEEE. ISBN 978-1-4799-5118-5 doi: 10. 220CC. URL https: Leeexploreeeeeeee ieee wwwpeeeorroooooooeeoeeeoeeooooee http///ccccntttt999eee
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. May 2019. URL http:://axixi
org/abs//i 04805. 1eXiv:1810.0490 [cs].
Thomas G. Dietterich. Ensemble methods in machine learning.
In Multiple classifier systems, pp. 1-15, Berlin, Heidelberg. 2000. Springer Berlin Heidelberg. ISBN 978-3-540-45014-6.
Read Status: Read Read Status Date:
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson
Loss Surfaces, Mode Connectivity, and Fast Ensembling DNNs, October 2018. URL http:
/ //aaxivv. org.vvorrg//ooo abs/ 10025. 6. 1eXiv:1802.10022 [cs, stat].
George Stoica, Daniel Bolya, Jakob Bjorner, Taylor Hearn, and Judy Hoffman. Ziplt! Merging Models from Different Tasks without Training, May 2023. URL
2305. 03053. arXiv:2305.033: .03053 [cs].
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task Arithmetic in the Tangent Space: Improved Editing of Fre-Trained Models, May 2023. URL http://// //tp///aiivv..// org/ abs/2300 12827. arXiv:2305. 12872 [cs] Read Status: Read Read Status Date: 2023-08- 31T08: 04:00.477.
10

---

Published as a conference paper at ICLR 2024
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Introducing eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. In IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium, pp. 204-201 IEEE, 2018.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruma Morrone, Quentin de Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Garameter-Efficiee Transfer Learning for NLP, June 2019. URL http://arxiv/ org 9922 00751. arXiv:1902.00751 [es, stat].
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models, October 2021. URL http://an /tttp://rxxi..../ org/ abs 2tss/2000 22001216..0 09685. arXiv:2106.0968. [cs].
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lo- raHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition, July 2023. URL http: /arxiv org/ abs 2307 13269. arXiv:2307.1226. [cs] Read_ Status: Read Read_Staaus Date: 010-000--1109957311555555
Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannanch Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights, October 2022. URL arXiv:2208.05592 [es].
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing Models with Task Arithmetic, March 2023. URL http:// /ttp///aavvvv/.e org/ abs 2212 04089. arXiv:2212.0400 [cs].
Arthur Jacot, Franck Gabriel, and Hongler Clement. Neural Tangent Kernel: Convergence and Generalization in Neural Networks. Advances in Neural Information Processing Systems 31 (NeurIPS 2018), 2018.
Weisen Jiang, Baijiong Lin, Han Shi, Yu Zhang, Zhenguo Li, and James T. Kwok. Effective and Parameter-Efficient Reusing Fine-Tuned Models, October 2023. URL WRRphhtt:///xxi..o. abs/ 2310. 01886. 6, 2.Xi::331000182 [cs] version: 2.
Xisen Jin, Xiang Ren, Daniel Preotiuc-Piete and Pengxiang Cheng, Dataless Knowledge Fusion by Merging Weights of Language Models, April 2023. URL http::::/wwww...og///////o/ 09849. arXiv:2212.0968 [cs].
Jean Kaddour. Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averaging, October 2022. URL http:// arxiv. org abs/2209. 14981. arXiv:2209. 14981 [cs, stat].
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D Object Representations for Fine- Grained Categorization. In 2013 IEEE International Conference on Computer Vision Workshops, pp. 561, December 2013, doi: 10. 10.10C//CCC. URL https: /ttt://eeeeeeeeeeeee// leee. org/ wttp://.ccc/////oo/// 675559466577559
Brian Lester, Rami Al-Rfou, and Noah Constant. The Power of Scale for Parameter- Efficient Prompt Tuning, September 2021. URL http://arxiv. abs/2104. 08691. arXiv:2104.0869) [cs].
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the Loss Landscape of Neural Nets, November 2018. htLhttp:// //aptt::///xxi...//// arxiv. wttpii.oog/ab//11222.999999 arXiv:1712.099) [cs, stat].
Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep Model Fusion: A
Survey, September 2023. URL http: http:::///a.i...aaeea wwtiiiiorrg/bbb/ /2309. 21699.1698.......
Xiang Lisa Li and Percy Liang. Prefix-Tuning: Optimizing Continuous Prompts for Generation, January 2021. URL htt.tttpp///xxie wwwp://wwwgg//////////o/oo 001 90. mwXiv:2101.011 [cs].

---

Published as a conference paper at ICLR 2024
Chang Liu, Chenfei Lou, Runzhong Wang, Alan Yuhan Xi, Li Shen, and Junchi Yan. Deep Neural Network Fusion via Graph Matching with Applications to Model Ensemble and Federated Learn- ing. In Proceedings of the 39th International Conference on Machine Learning, pp. 13857-13869. PMLR, June 2022a. URL https: / httnttsss//pooeeeeeeeeee ppoceed ngs. ml www.prsss/see h /1lk22k html.
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning, August 2022b. URL htLhttp:// //axivv. org/ abs///b////// /2005 05638. arXiv:2205.05638 [es].
Jialin Liu, Antoine Moreau, Mike Preuss, Baptiste Roziere, Jeremy Rapin, Fabien Teytaud, and
Olivier Teytaud. Versatile Black-Box Optimization, April 2020. URL MtRLat::///axi..co abs/2004 4.004.000 14014. 2XXv:2004.110114 [cs].
Tian Yu Liu and Stefano Soatto. Tangent Model Composition for Ensembling and Continual Fine- tuning, July 2023. URL http: pett:://rxxtr //ariiivv.. wwtp:xivv.rraaaeeeoe http//bs/23007 08114. arXiv:2307.081 [cs] Read_Status: Read
Date: 2023-08-31T09::5:40.385
Yihang Lou, Ling Yu Duan, Yong Luo, Ziqian Chen, Tongliang Liu, Shiqi Wang, and Wen Gao. Towards Efficient Front-End Visual Sensing for Digital Retina: A Model-Centric Paradigm. IEEE Transactions on Multimedia, 22( 1):3002-3013, 2020. ISSN 19410077. 2020.2966885. ISBN: 1901014738.
doi: 10.
Michael Matena and Colin Raffel. Merging Models with Fisher- Weighte Averaging, August 2022. URL http:://rxiia org/ abs/ /abs/2111 2111. 09832. arXiv:2111.099 [cs].
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. Fast Model Editing at Scale, June 2022a. URL http://axxvv.orroe arXiv:2110.11309 [es].
11309.
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, and Chelsea Finn. Memory- Based Model Editing Scale. Proceedings of the 39th International Conference on Ma- chine Learning, pp. 15817-15831. PMLR, June 2022b. URL htlhttps:/// http:::://///ceeingge..s. mlr. html. ISSN: 2640-3498.
press/vv62/mitchhee
Shikhar Murty, Christopher Manning, Scott Lundberg, and Marco Tulio Ribeiro. Fixing Model Bugs with Natural Language Patches. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11600-11613, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 18653//v11 1220///eemeeelpmmma... https: //ahaaalaatthoocco ogy. org.org///222 2022 emn1p-ma i 797.
URL
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading Digits Natural Images with Unsupervised Feature Learning. 2021.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, March 2022. URL http: : aawiviiiiii. org/ abs 220.. 02155. urXiv:2203.02155 [cs].
Barak A. Pearlmutter. Fast Exact Multiplication by the Hessian. Neural Computation, 6(1):147-160, January 1994. ISSN 0899-7667, 1530-888X. doi: 10. wwtp:///ccco.99........... 1994.6.1.147. URL https:
mit edw wttp://eeeeeeeeeeeeeeeeeee///e/e 11t //di //direct... .oneet..i....m
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. 1:9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, February 2021. URL http:/// wwwp//o..////oooooeoooooooo.osss 03. 00020. arXiv:2103.00M [cs].
12

---

Published as a conference paper at ICLR 2024
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the Limits of Transfer Learning with a Uni- fied Text-to-Text Transformer, July 2020. URL http:/ /ttt:://axiiy org/ wws//911...............aata.e 10683. arXiv: 1910. 10683 [cs, stat].
Marco Tulio Ribeiro and Scott Lundberg. Adaptive Testing and Debugging of NLP Models. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 3253-3267, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 1o..8665//2222...llloomm222333 230.
URL https: ptttps://aclanttol..oo org/
Shibani Santurkar, Dimitris Tsipras, Mahalaxmi Elango, David Bau, Antonio Torralba, and Aleksander Madry. Editing classifier by rewriting its prediction rules. In Advances in Neural Information Processing Systems, volume 34, pp. 23359-23377 Curran Asso- ciates, Inc., 2021. URL https: c4 64 8.489a2000 2d5a 9 9ecfc53b17666 092 6ddd- Aastdbtttttraathtt......
Sunny Sanyal, Jean Kaddour, Abhishek Kumar, and Sujay Sanghavi. Understanding the Effee- tiveness of Early Weight Averaging for Training Large Language Models, June 2023. URL http:://axiiai org/ abs/ abs/aa0s/552 2306. 03241. arXiv:2306.03241 [es].
Stallkamp. M. Schlipsing. Salmen, and C. Igel. Man vs. computer: Benchmarking ma- chine learning algorithms for traffic sign recognition. Neural Networks, 32:323-332, Au- gust 2012. ISSN 0893-6080. doi: 10.
URL https://ww/
sciencedirect http:///ciiieeeeaaaaae//////////
Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black Box Tuning for Language-Mode -s-a--las-aSServicc, June 2022. URL httt:://arae www.sooorr//bba.. a20aaabss/22..2 2201 03514. arXiv:2201.036 003514 [cs].
Anke Tang, Yong Luo, Han Hu, Fengxiang He, Kehua Su, Bo Du, Yixin Chen, and Dacheng Tao. Improving Heterogeneous Model Reuse by Density Estimation. volume pp. 4244-4252, August 2023. doi: 10. 10.24963/ijcai, 2023//477... URL https: /wwww//w.////////////////// 2023 472. ISSN: 1045-0823.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Pro- ceedings of the 2018 EMNLP Workshop BlackboxVLP: Analyzing and Interpreting Neural Ner- works for NLP, pp. 353-355, Brussels, Belgium, 2018. Association for Computational Linguis- tics. doi: 10. 10.18653/v1/ 5446. URL http: acl wlwww. org/ anthology 818-5446.
Zhenyi Wang, Xiaoyang Wang, Li Shen, Qiuling Suo, Kaiqiang Song, Dong Yu, Yan Shen, and Mingchen Gao. Meta-learning without data via Wasserstein distributionalll--ooust model fu- sion. In Proceedings of the Thirty-Eighuth Conference on Uncertainty Artificial Intelligence, pp. wwi.weng.githhee teub.hu.i httbbb.o/pssss/2222222 09-08-
PMLR, August 2022. URL https:: h/tpp::////cccednngg. mlr ppess/vlBes wang22a html.
Lilian Weng. Some math behind neural tangent kemel. Lil'Log Sep 2022. URL https://
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves aceu- racy without increasing inference time, July 2022a. URL http:://taxiii orw. abs/ 2203.. 05482. rate: 4 Stad_Statts: Read Sead_Status. Date: 2003-08-31/11::29555.99.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael GontijoLopes, Hannanch Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models, June 2022b. URL http: /aaaaaivvvv... org/ abs/2109. 01 903, arXiv:2109.0190 [es].
13

---

Published as a conference paper at ICLR 2024
Chengyue Wu, Teng Wang. Yixiao Ge, Zeyu Lu, Ruisong Zhou, Ying Shan, and Ping Luo. Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolation, May 2023. URL http: /aarivvaaaiiva///aaa/eaeeeeeiiia wwwpeiiii.orgaeoooooooe abs
MIXIv:2304.14388 [cs].
Xi Zhu Wu, Song Liu, and Zhi Hua Zhou. Heterogeneous model reuse via optimizing multiparty multiclass margin. 36th International Conference on Machine Learning, ICML 2019, 2019-June: 11862-11871 2019. ISBN: 9781510886988
rate: 0.
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Patlern Recognition, pp. 3485-3492, San Francisco, CA, USA, June 2010. IEEE. ISBN 978- 278-1-4244-6984- doi: 10. 1001109/CVPR.22000555533 leeexplore ieee. wwtp:/ogggoooooeee.eeee /btp//mmmtt/5339//.....
URL http://
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Resolving Inter- ference When Merging Models, June 2023. URL http: //tp/i//aiivv///////ee/eeeeeeaee www.vv.orggbbb/////eee 01708. arXiv:2306.01700 [cs] Read Status: Read Read Status_Datee 2023:08-31108::1244777..
Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He. Composing Parameter-Efficient Mod- ules with Arithmetic Operations, June 2023. URL htRp:tt://aae URL.tttp//aarrx...rr/a////// 144700 arXiv:2306. 14870 [cs].
14

---

Published as a conference paper at ICLR 2024
A DEFINITION OF TASK VECTOR
In this section, we provide insights into the rationale behind defining the task vector and the as- sociated advantages. Our definition of the task vector for full fine-tuning is consistent with that introduced in (Ilharco et al., 2023). However, for parameter-efficient fine-tuning, we define the task vector as Φ; @u on the trainable parameter space instead of the change of on the original model parameter space as done in (Jiang et al., 2023; Zhang et al., 2023). This deliberate decision carefully considered and corresponds to the nature of our fine-tuning approach.
Take LoRA as an example, when partially linearizing a LoRA model, we focus on the dynamics of the trainable parameters and compute the Jacobian- vector product for all trainable parameters, specifically the LoRA parameters denoted by Φ. The original model parameters, dg. remain constant throughout the fine-tuning process and serve as fixed foundations or buffers that enable the adaptable components of the model, the LoRA parameters, to manifest their effects.
From a mathematical standpoint, we perform a first-order Taylor expansion exclusively on the train- able parameters. This expansion does not include the parameter merging operations of LoRA:
S8 Φ) = foo (2; Φo) VaJa (2); do) (Ø Φ0)
(5)
It is essential to emphasize that the application of this Taylor expansion is restricted to the parameters explicitly designated as trainable. These are the parameters that undergo active adjustments during the fine-tuning phase, representing the modifiable components of the model.
What's more, while our experiments have primarily utilized LoRA fine-tuning, such a definition proves beneficial for generalizing our method to other parameter-efficient fine-tuning techniques, such as adapter-tuning.
B MODEL LINEARIZATION FROM A NTK PERSPECTIVE
In this section, we show that a linearized model is more capable to separate the function outputs in different directions in weight space, thus leading to enhancement of weight disentanglement and offering advantages over non-linear models for model fusion.
The linearized model, described by Eq.(4) approximates the behavior of the neural network Sag (x, (()) at specific time I, where Oo, So, (I;; ((0)), VaJa, (1; 6(0)) are all constants. Con- sequently, the linearized function (o(1); a)) is a linear function of ((t).
Assuming the trainable parameters are updated by gradient descent on task Tr, hence
V d 0o, ②) = E Jr, So, (I o) Lo
(6)
((Ai) (00) - oL (Tx; 0n. (6)) = -- D., VV ( V.)),.
(7)
(8)
where y7 denotes the learning rate and Lee is the cross-entropy loss. Consequently,
flin (x; ol ((At)) fa, (x; (0))
(9)
V Vafa, (z; o (0)) (d(At) 6(0)
(10)
([rre) Dr,
V Vofo, (z; ( (444 o(0)), (())),..))
(00)) (
(11)
- Dr. K (z, x,, (0)) Cat
vr,
(12)
where K is known neural tangent kernel (Jacot al., 2018), which is defined K K( x';
Jop Ao (x;; Φ), Vola, (x'; ()) R| x Rll RlfXi|fl Since the NTK term is fixed throughout the training. the change in the model output A Aii (z; d(At)) is solely determined by the gradient of the loss V VfLce(fon (# ((0)). V..). This establishes the connection between the neural network dynamics and kernel methods, allowing us to analyze the model through the lens of NTK.
Connection between linearization and weight disentanglement. Integrating the concept of lin- carized model, we can express how the linearized model's 's output for a given input sample x evolves from its initial state (0) over time follows: For any small time increment At, the difference in
15

---

Published as a conference paper at ICLR 2024
output from the linearized model fig at the state ((At) compared to the original model fo, at the initial state ((0) is calculated by projecting the gradient of the model output with respect to its initial parameters Vaf(x; (0)) onto the parameter change (Aat) - ((0). Symbolically, this relationship
is denoted as:
f (2; (b Sn (z; (0)) = V a - - (0)),
(13)
which further simplifies to a term involving the expectation over the dataset DT., capturing the interaction between the kernel representation K of the input samples and the gradient of the loss function y LCB. Mathematically, this term is represented as (q.(
(E,, [K (2, _ (())
(Ix, (()), (+)))
(14)
Building upon this derivation, arrive fundamental equation that depicts the proportional relationship between the expected task-specific gradient of the loss function and the modification in parameters for the linearized model:
((+t) (0)) (o,,,,))) Dr,)-D, (Kx R ( (x,, (00),
(15)
In essence, this formula underscores the direct correspondence between the changes in weights of linearized model and the gradient of the loss. hints at the underlying mechanisms of weight dis- entanglement, revealing that the adjustment in parameters is essentially directed by the aggregated gradient information from the task-specific data. This understanding is crucial as it provides insight into how model's weights evolve in response the learning process.
C MODEL FINE-TUNING DETAILS
All of our experiments were performed using the same hardware consisting of cight NVIDIA GTX 3090 GPUs with 24GB video memory. We used PyTorch 2.0 and Python 3 throughout all experi- ments.
Model, Evaluation Tasks, and Metrics. We conduct experiments on diverse tasks spanning both vision and natural language domains. The model, evaluation datasets and metrics settings as follows:
For vision tasks, we utilize CLIP (Radford et al., 2021) as our pre-trained model. For fine- tuning, we employ the CLII--VITB-11 models on seven image classification tasks derived with the same random seed 42 to initialize the parameter-efficient models. These tasks are Stanford Cars (Krause et al., 2013), DTD (Cimpoi et al., 2014), EuroSAT (Helber et al., 2018), GT- SRB (Stallkamp et al., 2012), RESISCA5 (Cheng et al., 2017), SUN397 (Xiao et al., 2010), and performance metric.
SVHN (Netzer et al., 2021). We report top-1 accuracy
For NLP tasks, we utilize Flan-T5 (Chung et al., 2022) as our pre- trained language model. For fine-tuning, we employ the Flan-T5-base models on seven tasks derived from the GLUE bench- mark (Wang et al., 2018) with the same random seed 42 to initialize the parameter-efficien models. These tasks are CoLA, MNLI, MRPC, QQP. RTE, SST2, and STSB. We report Spear- man's p for STSB and accuracy for others.
Vision domain. All the fine-tuning experiments on image classification tasks using AdamW op- timizer. We fine-tune pre-trained C (Radford et al., 2021) downloaded from Hug- gingFace For image classification tasks we use the text prompt template 'a photo of [label]' to compute the text embeddings. We initialize all parameter-efficien modules (LoRA and L-LoRA) with the same random seed 42. We use batch size of 64 and learning rate of le-5 and a weight decay of 0 with a warm-up cosine scheduler for 6000 steps for downstream tasks.
Language domain. The fine-tuning experiments on NLP tasks in our study follow the same exper- imental setup as described in (Raffel et al., 2020) using Adam optimizer. We fine-tune pre-traine trained Man--T5bbaee model (Chung et al., 2022) download from HuggingFace ? In line with the previous setup, we convert all the downstream NLP tasks into a text-to-text format. For specific examples,
'https://huggigg co http:ppeaa/lllp--ttttbbbaaa Phttps: hhttps://huggingffaes co /oog/ooollleee t5
16

---

Published as a conference paper at ICLR 2024
Table 2: Hyperparameters for model fine-tuning.
Scenario
Learning rate
Steps
Batch size
Weight decay
LoRA
CLIP-ViT-B- on vision tasks
full fine-tuning
le-5 (warm-up cos)
6000
64
0.1
LoRA
le-5 (warm-up cos)
6000
64
0.1
16
L-LoRA
le-5 (warm-up cos)
6000
64
0.1
16
Flan-75-Base
NLP tasks
full fine-tuning
le-5,2e-5
2000
16
LoRA
(e-5,2e-5,3e-5,44
2000
16
16
L-LoRA
(e-5,2e-5,3e-5,4e-5
2000
16
16
Table 3: Summary of fine-tuning statistics.
Fine-tune diagram
Trainable params.
Total params.
Checkpoint
Trainable% %
VRAM
size
CLIP- V-T- B-16 on vision tasks (batch size=64)
full fine-tuning
85M
85M
100%
12.4GB
328MB
LoRA (r=16)
294K
86M
0.34%
9.2GB
1.2MB
L-LoRA (1=16)
294K
93M
0.32%
9.3GB
1.2MB
Flan-T3 Tan--T--Bsse on NLP tasks (batch size=16)
full fine-tuning
247M
247M
100%
2 16.3GB x 4
945MB
LoRA (r=8)
885K
248.5M
0.36%
11.3GB x
3.5MB
LoRA (1=16)
1,769K
249.3M
0.71%
2 11.4GB x 4
6,9MB
L-LoRA (r=8)
885K
291.8M
0.30%
11.3GB X 3.5MB
L-LoRA (1=16)
1,769K
293.5M
0.60%
11.4GB X 4
6.9MB
please refer to F.1. This standardized format facilitates the training process and ensures consistency across different tasks, enabling effective fine-tuning of the models. We initialize all the parameter- efficient modules (LoRA and L-LoRA) with the same random seed 42. For standard full fine-tuning, constant learning rate of le-5 or 2e-5 was used for 2000 steps with batch size of 16 and weight decay. For LoRA and L-LoRA fine-tuning, a sweep of learning rates from le-5 to 4e-5 was con- ducted, while keeping the other hyperparameters the same. The hyperparameter LoRA for both LoRA and L-LoRA but is not applicable for full parameter fine-tuning. The learning rate sweep aims find optimal values for different methods. Each fine-tuning run is conducted four GPUs, utilizing the power of distributed data parallelism (DDP). This approach allows for efficient training of the model by dividing the batch across multiple GPUs and processing it simultaneously.
to 16
Table 2 summarizes the key hyperparameters used during the fine-tuning of CLIP-ViT-B-LE and Flan-T5 ---b-e-- model under three different scenarios: standard full fine-tuning (FFT), fine-tuning with LoRA, and fine-tuning with L-LoRA.
Table presents a summary of the fine-tuning statistics for several different fine-tune diagrams. The table provides information on the trainable parameters, total parameters, trainable parameter percentage, VRAM utilization (with batch size of 16) and checkpoint sizes.
Taking the Tran-T5BBan model an example, full parameter fine-tuning updates all parameters, while leading to very high VRAM usage around 16.3GB per GPU and a large 945MB checkpoint size. On the contrary, the Low-Rank adaptation (LoRA) models have significantly fewer trainable parameters, ranging from 885K to 1.7M. This corresponds to only 0.3-0.7% of the total parameters being trainable. Consequently, the VRAM usage is much lower at 11.3-11.4G6 per GPU. The checkpoint sizes are also smaller, from 3.5-6.9MB. The linearized LoRA fine-tuning has VRAM usage similar to LoRA. They have more total parameters 291.8-299 209.8-233.5M because need to cache
17

---

Published as a conference paper at ICLR 2024
Table 4: Summary of training and inference cost.
Training (Adam)
Inference
Scenerio
Time
VRAM
Time
VRAM
CLIP-WIT-B-32 (batch size=64)
full fine-tuning (Figure ((a))
8.25 it/s
4.2GB
13.28 it/s
0.9GB
full linearization (Figure 3(b))
4.94 it/s
6.3GB
8.52 it/s
2 2.0GB
LoRA (r=16) (Figure 3(c))
8,43 it/s
2.6GB
13.01 it/s
2 0.9GB
L-LoRA (r= (6) (Figure 3(d))
7.55 it/s
2.7GB
10.65 it/s
2 L.0B
CLIP-ViT-B-16 (batch size=64)
full fine-tuning full linearization
2.97 it/s
2 12.4GB
8.92 it/s
R 1.6GB
1.49 it/s
20.1GB
2.50 it/s
R 3.7GB
LoRA (r=16)
3,83 it/s
= 9.1GB
8.60 it/s
~ 1.6GB
L-LoRA (r=16)
3.24 it/s
9.2GB
7.23 it/s
1.8GB
CLIP-ViT-L-1 (batch size=16)
full fine-tuning full linearization LoRA (r=16)
2.55 it/s
13.9GB
8.23 it/s
1.8BB
1.32 it/s
= 21.8GB
2.26 it/s
> 4.8GB
3.49 it/s
9.1GB
7.99 it/s
1.8GB
L-LoRA (r=16)
2.99 it/s
R 9.3GB
6.55 it/s
R 1.9GB
the initiation values of LoRA modules to compute the Jacobian vector products. The checkpoint sizes match LoRA as well.
In summary of Table 3, full parameter fine-tuning has the most parameters but is costly to fine-tune. LoRA reduces the trainable subset of weights, enabling more efficient adaptation. L-LoRA increases the total parameters while maintaining small trainable subset. Our partial linearization approach incurs minimal additional cost compared to standard LoRA fine-tuning. The training and inference costs increase only marginally and are negligible over LoRA.
Table summarizes the training and inference cost for different fine-tuning methods applied to CLIP vision models of varying sizes. The costs are broken down in terms of training/inference time (iterations/sec) and video RAM (VRAM) usage.
Several key trends can be observed from Table 4. Full linearization is the slowest for both training and inference due to the increased computational cost. It also requires the most VRAM. LoRA is faster than full tuning for training and on par for inference, while using less VRAM for training due to the saaameter-effcciee fine-tuning. L-LoRA slightly slower than LoRA. Its resource usage is comparable to LoRA.
In Figure 7, we report the single task performance of different fine-tuning methods: full parameter fine-tuning, LoRA fine-tuning and L--LRRA fine-tuning. These results demonstrate that nonlinear fine-tuning consistently achieves higher single-task performance compared to linearized fine-tuning. Among the non-linear methods, full fine-tuning gives the highest task-specific accuracy, while LoRA tuning performs slightly worse, but is more parameter efficient. L-LoRA fine-tuning exhibits the lowest individual task performance. However, provides improved multitask fusion capabilities. Although L-LoRA underperforms on single tasks, when evaluated on joint tasks, it can surpass LoRA fine-tuning due to enhanced representation disentanglement. The trade-off between special- ized single-task accuracy and generalizable multi-task performance is evident when comparing full fine-tuning LoRA and L--LORA in Section 5.1 and Appendix E.
Future work could further explore partial linearization, where some parameters are fine-tuned in a linear space while others remain nonlinear. This hybrid approach could trade-off between spe- cialized single-task performance and weight disentanglement. Specifically, partitioning the param- eters into fixed and linearly tuned subsets may combine the representation benefits of linearization with the expressiveness of nonlinear tuning. The non-linear parameters could capture task -secciic adaptations, maximizing per-task accuracy. Meanwhile, the linearized parameters could encourage independence in different directions of weight space between tasks.
18

---

Published as a conference paper at ICLR 2024
Tull: fleeetedddd
LoRA
L-LoRA
0.990
9:96
0.99
0.85
0.85
0.89
0.78
0.06
0.64
0.68
0.55
Score
0.6
0.50
0.54
0.0
DTD
BaneAA
GINRI Task
RUSISCAS
SUNT9
SVIIN
(a) Performance of task-specific CLIP-ViT-B-16 models downstream image classifi- cation tasks.
Tull Reconen
LoRA
L-LoRA
0.92
0.82
0.85
0.82 0.80
9:85
0.8
0.75
0.75
0..6
Score
0.6
6.45
0.0
MNLI
QQP
SI5B
Task
(b) Performance of task-specific Plan-T5-Base models downstream NLP task.
Figure 7: Performance of full fine-tuning LoRA, and L-LoRA on downstream tasks. Full fine- tuning achieves the best individual task performance, followed by LoRA and then L-LoRA which trades off some task-specific accuracy for improved parameter efficiency.
D LOss LANDSCAPE AND Cos SIMILARITY BETWEEN TASK VECTORS
MoVE MNLE EatA
1:00
0.12
6.12
0.165
0.16
0.10
0.06
MAVV MNU EXL
L:MM
0:29
0.30
0.21
0.23
0.43
0.27
STSR 5ST2 XTI: D00P MAPC MNNZ CMA
0.10
3,05
6.11
0.11
3.18
0.50
0.35
0:28
1,00
0.24
0.25
0.50
0.35
0.25
6.8
1.00
1.00
0.21
6:08
1.00
0.25
0.29
5.11
0.13
ogj
ITT: DgP
0.30
6:09
0.23
1.00
0.25
0.65
ITT:
0.19
0.27
0.25
0.04
6:25
6:90
0.79
644
S1S0 SNT2
10
8.10
6.66
0.6
0M
1:00
866
STND 5872
0:4h
6.26
0.24
0.19
025
1:00
0.5
0.29
6:11
0.27
62
0.06
6.15
6.11
0.08
0.12
0.05
1.50
0:27
6:25
0.35
0.22
0.99
0.25
1.0
0.09
6:04
0.13
0:51
0.00
10
1:00
Culo MNLLI MSFC
GOP
RTE
5572
ST5B
CoLA MNLLI MBFC
COP
RTE
8572
S158
CuL. MILL MRPPC
QCP
RIE
5512
ST58
(a) full fine-tuning
(b) LoRA fine-tuning
(c) L-LoRA
Figure 8: Similarity heatmap task vectors. These figures show heatmaps of the cosine similarity between task vectors from task-specific Flan-T5 model (Chung et al., 2022) fine-tuned on different NLP tasks from GLUE benchmark (Wang et al., 2018). The task vector represents the parameter differences between the fine-tuned models and the original pre-trained model.
(a) Cos similarity
matrix of task vectors when using full fine-tuning of the entire model. (b) Task vector similarities when using LoRA, parameter-efffiieet fine-tuning method. (c) Cos similarity of task vectors from L-LoRA, our proposed lincarized tuning approach that fine-tunes parameter-efficcent modules in tangent space.
19

---

Published as a conference paper at ICLR 2024
Figures 2 and 8 visualize the similarity between task-specific models fine-tuned different tasks using full fine-tuning, LoRA, and linearized LoRA (L-LoRA). Full fine-tuning results in a low sim- ilarity between task vectors, indicating greater independence between task-specific representations. In contrast, LoRA tuning induces higher similarity and entanglement between tasks. I.-LoRA im- proves over LoRA by further reducing the cosine similarity, demonstrating that linearized tuning of parameter-officient modules enhances disentanglement compared to standard LoRA. The results are consistent across both a vision model (CLIP) fine-tuned on image datasets, and a language model (Flan-T5) fine-tuned on NLP datasets. This supports the hypothesis that constrained optimization techniques like L-LoRA can improve representational disentanglement compared to unstructured tuning approaches.
E MULTI-TASK MODEL FUSION
E.1 FURTHER DETAILS ABOUT BASELINES
This section provides further implementation and configuration details for the baseline methods been compared in our experiments. An overview of the four baseline methods is provided as following. In Appendixes 1.1.1, 1.1. E.1.3 and E.1.4, the parameter settings used for each method in the experiments are explained.
Simple Weight Average: We average the trainable weights of the models fine-tuned on differ- ent tasks with different fine-tune methods, this method is also referred to as ModelSoup (Worts- man et al., 2022a) and AdapterSoup (Chronopoulou et al., 2023) in the literature. The averaged model is then evaluated on the validation set of each task. For full fine-tuned models, the weights the models are averaged directly (i.e., let = 1/n Ei= 0,). For parameter-efficient fine-tuning, average the added weights and then apply to the parameter-efficient modules, i.e., let Φ = D @ and f(z) = (2; Φ).
Task Arithmetic (Iharcoo et al., 2023; Zhang et al., 2023): As defined in Section 3.1, the task vector is computed on the set of trainable parameters. We compute the task vector for each task and then add them to construct a multi-task vector. The multi-task vector is then multiplied by scaling factor À element- wisely and added to the initial parameters of the pre- trained model to obtain a multi-task model, i.e.
Oo + X E, (0, - 0g) for full fine-tuning and Φo + X E, (Ø: Φb), (xx = So, (I; o) for parameter-efficient fine-tuning, where À is hyperparameter that the best-performing model is chosen on validation set.
Ties- (Yadav al., 2023): Ties-merging is method for merging multiple task-specific models into single multi-task model. This algorithm follows three steps (trim, elect sign of parameters, and disjoint merge) obtain merged task vector v. Given the final merged task vector the final model is chosen in similar way as task arithmetic, i.c. 0 = 0g + A for full fine-tuning and & = do + Av, f(x) foo (z; Φ) for parameter-efficciett fine-tuning, where is a hyperparameter that the best-performing model is chosen on validation set.
LoraHub (Huang et al., 2023): Lorahub is a method to apply arithmetic operations to parameter-efficien modules. With the help of few examples from novel task, LoraHub es- timates the optimal weights {we} for the parameter-efficien modules by utilizing a black-box optimization technique (Liu et al., 2020: Sun et al., 2022) to minimize L + a E, |Wy]. where is a hyperparameter. The final model is fxr = fa. (x: Φ), where & = >. W; s.t. >, a; = 1. In practical implementation, we construct the final model with task vector arithmetic because we have v = Φ do = (E, w;(s)) Φn Σg Wil;, so á = Φn + Σ, way. We evaluate LoraHub with both LoRA fine-tuning and L-LoRA fine-tuning.
E. SIMPLE WEIGHT AVERAGE
In our comparison, we employ the method known ModelSoup (Wortsman et al., 2022a) or AdapterSoup (Chronopoulou et al., 2023) in the literature, which involves averaging the trainable weights of models. We adapt simple averaging to models that have been fine-tuned on different tasks using different fine-tuning mie-ttning methods. By averaging the weights in this manner, we create a combined model that incorporates the knowledge and expertise gained from multiple tasks and training meth- ods. This averaged model is subsequently evaluated on the validation set of each respective task to assess its performance.
20

---

Published as a conference paper at ICLR 2024
For full fine-tuned models, we directly average the weights by computing the mean, represented as 0 Ei=1 i,, where n is the total number of models being averaged.
When it comes to parameter-efficiet fine-tuning, compute the average of the added weights and then apply them to the parameter-effiient modules. This is accomplished by calculating the mean of the added weights the parameter-efficien modules. Consequently, the function f(z) is defined as f(x) = So (z; Φ), 120 multi-task
n -i= Φi. and utilizing this average weight configuration in
taking into account the updated averaged weights. We create a total of 2 models for each fine-tuning method.
E.1.2 TASK ARITHMETIC
As defined in Section I. the task vector is computed on the set of trainable parameters. We compute the task vector for each task and then add them to construct multi-task vector. The multi-task vector then multiplied by scaling factor À element- wisely and added to the initial parameters of the pre-trained model to obtain a multi-task model, i.e. 0 = 0o - A > (0; - 0o) for full fine-tuning (Ilaarco et al., 2023) and & = Φ + A (o; - co). f(x) fe, (x; o) for parameter-effiiinn fine- tuning (Zhang et al., 2023), where À is hyperparameter that the best-performing model is chosen on validation set.
Following the practice from (Ilharco et al., 2023), add task vectors together and use single scaling coefficient for the sum of vectors, E {0, 0.05, 0.1..... 1.0). That is, for each set of tasks from the power set of (Tr)T= we construct 21 candidate models with different À values. We then evaluate the performance of each model on the validation set of each combination of tasks, and choose the best-performing model for each combination of tasks. Finally, evaluate the performance of the best-performing model on the test set of each task. Here we have created a total of 21 X (2 8) 2520 multi-task models for each fine-tuning method.
E. 1.3
TEES-MERGING
The Ties fies-nggging method, proposed in the paper by Yadav et al. (Yadav et al., 2023). is employed for merging multiple task-specific models into unified multi-task model. This merging process involves three key steps: trim, elect sign of parameters, and disjoint merge, ultimately resulting in a merged task vector denoted as v. Once the final merged task vector v is obtained, the selection of the final model parallels the approach used in task arithmetic. For full fine-tuning, the updated parameters are computed as 0 = 0o + Av, while for parameter-efficient mneettnnngg the updated parameters are computed é do Av, and the function f(x) is defined as foo (2; Φ). As previously mentioned, the hyperparameter the validation set.
chosen based on the performance of the model on
We sweep the hyperparameters & from the parameter trim step and scaling factor over a range of values (0.25, 0.50, 0.75, 1.00). This results in a total of 16 combinations, whereby each com- bination is evaluated to determine the best-performing model. Here we have created a total of 42 (2 -) 1920 multi-task models for each fine-tune method.
E.1.4 LORAHUB
LoraHub, as introduced by Huang et al. (Huang et al., 2023), is method specifically designed for applying arithmetic operations to pframeter-efficient modules. This technique leverages a set of example inputs from a novel task to estimate the optimal weights {w;}, } for the marameter-efficien modules. To achieve this, LoraHub utilizes black-box optimization technique, such as the one de- scribed in the works of Liu et al. (Liu et al., 2020) and Sun et al. (Sun et al., 2022). The optimization process aims to minimize the objective function C + a E, (wy). where a is a hyperparameter.
The final model obtained through LoraHub is defined f(z) fa, (z; Φ), where @ E, w. subject to the constraint E; a, = In practical implementations, the final model is constructed using task vector arithmetic. This is possible due to the relationship o do (E; w;o;:) do Σ, woy, which allows us to express ó as Øu + Σ, WiPt
We adapt the original implementation form (Huang al., 2023) and fix the hyperparameter 0.05. The maximum inference step for the gradient- free optimization algorithm Shiwa is 40, which results in up to 40 (27 -8) 4800 multi-task models for each fine-tune method.
21

---

Published as a conference paper at ICLR 2024
Table Normalized scores of seven-task model fusion (CLIP- across individual datasets.
Method
Cars
DTD
EuroSAT
GTSRB
RESISC45
SUN397
SVHN
Mean
Simple Average
FFT
0.77
0,62
0.76
0.49
0.71
0.87
0.56
0.68
LoRA
0.81
0.69
0.39
0.40
0.62
0.93
0.16
0.57
L-LoRA
0.87
0.86
0.41
0.58
0.74
0.97
0.19
0.66
Task Arithmetic
FFT
0.80
0.63
0.87
0.73
0.78
0.86
0.85
0.79
LoRA
0.86
0.72
0,51
0.50
0.73
0.91
0.48
0.67
L-LoRA
0.95
0.91
0.65
0.74
0.86
0,96
0.78
0.84
Tiex-Merging
FFT
0.78
0.65
0.92
0.69
0.83
0.88
0.93
0.81
LoRA
0.81
0.74
0.54
0.50
0.71
0.92
0.40
0.66
L-LoRA
0.89
0.91
0.60
0.69
0.82
0.97
0,51
0.77
Lorallub
FFT
0.93
0.90
0.32
0.30
0.47
0.77
0.13
0.55
LoRA
0.83
0.68
0.38
0.76
0.70
0.86
0,64
0.69
L-LoRA
0.94
0.88
0.73
0.85
0.85
0.91
0.80
0.85
E.2
EXPERIMENTAL RESULTS VISUALIZATION AND DISCUSSIONS
We fine-tune all models with hyperparameter settings as shown in Table 2. We construct multi- task models by utilizing task vectors specific to individual tasks, employing various model fusion algorithms. Due to the extensive volume data and multitude models in the validation set of downstream NLP tasks, feasibility constraints led run only the first 50 batches for each NLP task. This approach allowed us to manage computational resources while still obtaining a representative evaluation within practical limits.
In Figures and 11, we examine four different model fusion techniques simple averaging, task arithmetic, ties merging, and LoraHub for constructing multi-task models using increasing number task vectors on both vision and language domains.
Simple averaging doesn't account for the relationships interactions between different tasks. As more tasks are combined, destructive interference may occur between incompatible task vectors. This indicates that simply averaging task vectors is insufficient for effectively combining knowledge across many NLP tasks. In contrast, the other three fusion methods demonstrate steady gains or maintain average multi-task model performance as the number of task vectors increases.
For image classification tasks, ties merrimeggng exhibits the strongest overall performance, with the highest average normalized scores across different numbers of task vectors. This may be because ties- mer-ing introduces several innovations to address the challenges of parameter interference dur- ing model merging. The higher degree of interference between vision tasks, evident in the cosine similarity matrix in Figures 2 and 8, increases the importance of these techniques.
contrast, for NLP tasks, LoraHub demonstrates the strongest performance with the highest av- erage normalized scores. The learned routing between shared fixed parameters and task-specific adapters enables more effective fusion for language tasks.
In Table 5, present the normalized scores of seven-task model fusion (CLI--iT-B/IU across individual datasets. We observe that L-LoRA outperforms LoRA and FFT in most cases. This because L-LoRA fine-tuning is able to better disentangle the task-specific knowledge from the shared knowledge. Besides, different fusion techniques can be more aligned with certain parameter- efficient methods, leading substantial performance divergence depending the chosen approach.
Another key observation from Figures 11 is that LoRA performs better than L-LoRA when the number of task vectors being fused is small. As the number of task vectors increases, the average
22

---

Published as a conference paper at ICLR 2024
fall-finetune
LoRA
L-LoRA
Simple Average
Task Arithmetic
Ties Merging
Lural
0.7
Average Score
0.6
0.5
0.4
4
Number Tasks
Number Tasks
Number of Tasks
Number of Tasks
full-finetune
LoRA
L-LoRA
Simple Average
Task Arithmetic
Ties Merging
LoraHub
Average Nornalized Score
0.8
0.7
0.6
4
b
2
4
4
Number Tasks
Number of Tasks
Number of Tasks
Number of Tasks
(a) CLIP-ViT-B-16 model image classification tasks.
full-finetune
LoRA
L-LoRA
Simple Average
Task Arithmetic
Ties Merging
LoraHub
Average Score
0.2
0.1
0.0
2
4
6
4
4
4
Number of Tasks
Number of Tasks
Number of Tasks
Number of Tasks
full-finetume
LoRA
L-LoRA
Simple Average
Task Arithmetic
Ties Merging
Loral
Average Normalized Score
0.4
0.2
0.1
0.0
Number of Tasks
Number of Tasks
Number of Tasks
Number of Tasks
(b) Flan-T5-Base model on NLP tasks.
Figure 9: Multi-task model fusion: we construct multi-task models by utilizing task vectors spe- cific to individual tasks, employing various model fusion algorithms. In the evaluation, the x-axis represents the number of task vectors used in building the multi-task model, while the y-axis repre- sents the average (normalized) scores of the multi-task models across all seven downstream tasks. The line on the plot represents the average (normalized) scores of all multi-task models when con- sidering fixed number of tasks, while the shaded area corresponds to the standard deviation. The y-axis ticks are shared across these plots. more detailed comparison between LoRA and L-LoRA is shown in Figures 10 and 11.
23

---

Published as a conference paper at ICLR 2024
Average Normailized Score
LoRA
L-LoRA
LoRA vs L-LoRA
LoRA
0.7-
L-LoRA
0.6
2
6
Number of Tasks
Number of Tasks
Number of Tasks
(a) Simple Average
LoRA
L-LoRA
LoRA vs L-LoRA
Average Normalized Score
LoRA
L-LoRA
0.7-
0.6
6
2
Number of Tasks
Number of Tasks
Number of Tasks
(b) Task Arithmetic
LoRA
L-LoRA
LoRA vs L-LoRA
Average Normalized Score
0.8
0.7
LoRA L-LoRA
0.6
6
Number of Tasks
Number of Tasks
Number of Tasks
(c) Tics-Merging
LoRA
L-LoRA
LoRA vs L-LoRA
Average Normalized Score
0.7
LoRA
0.6
L-LoRA
2
Number of Tasks
Number of Tasks
Number of Tasks
(d) LoraHub
Figure 10: LoRA vs L-LoRA (CLIP-ViT-B-16 model vision tasks). We construct multi-task models by utilizing task vectors specific to individual tasks, employing various model fusion al- gorithms. In the evaluation, the x-axis represents the number of task vectors used in building the multi-task model, while the y-axis represents the average score of the multi-task model across the seven GLUE tasks. Each point on the plot corresponds the average score of a specific multi-task model, and the line represents the average score of all multi-task models considering a fixed number of tasks.
24

---

Published as a conference paper at ICLR 2024
Average Normallzzed Score
LoRA
L-LoRA
LoRA vs L-LoRA
00 00000
0600
LoRA L-LoRA
0.2
0000
0.1
0.0
2
Number Tasks
Number Tasks
Number Tasks
(a) Simple Average
LoRA
L-LoRA
LoRA vs L-LoRA
Average Normalized Score
080 000:000
0.3
00000
0.2
LoRA
0.1
L-LoRA
2
4
Number Tasks
Number Tasks
Number of Tasks
(b) Task Arithmetic
Average Normalized Score
LoRA
L-LoRA
LoRA vs L-LoRA
0.4
000
LoRA
0.3
L-LoRA
00:00
0.0
6
2
6
Number Tasks
Number Tasks
Number of Tasks
(c) Ties-Merging
LoRA
L-LoRA
LoRA vs L-LoRA
Averuge Normalized Score
0.4
LoRA
L-LoRA
0000000
6
2
4
6
Number of Tasks
Number of Tasks
Number of Tasks
(d) LoraHub
Figure 11: LoRA vs L-LoRA (Flan-TS-Base model on NLP tasks). We construct multi multi-task mod- els by utilizing task vectors specific to individual tasks, employing various model fusion algorithms. In the evaluation, the x-axis represents the number of task vectors used in building the multi-task model. while the y-axis represents the average score of the multi-task model across the seven GLUE tasks. Each point on the plot corresponds to the average score of a specific multi-task model, and the line represents the average score of all multi-task models considering a fixed number of tasks.
25

---

Published as a conference paper at ICLR 2024
performance of L-LoRA becomes better than that of LoRA. It becomes evident that LoRA excels in fusing a smaller number of tasks or leveraging its effectiveness through simple averaging. On the other hand, L-LoRA shows enhanced ability to exploit interactions within larger set of tasks. This is because the linearization of PEFT modules in I--LoRA is fine-tuned in tangent space, which linear space, while the PEFT modules in LoRA are fine-tuned in the original non-linear ambient space. The non-linear space fine-tuning results in a superior performance of the single-task model, which is also known as the non-linear advantage (Guillermo Ortiz-Jimenez al., 2023). However, as the number of tasks increases, L-LoRA becomes more effective than LoRA. This suggests that the learned linear combination of task vectors used L-LoRA able to leverage interactions between a larger number of tasks more effectively, and L-LoRA is better able to scale to handle multi-task model fusion across large number tasks.
F IMPLEMENTATION DETAILS
Here, we provide more details about the implementation of our method, including some basic code.
F.1 PREPROCESSED EXAMPLES
In this subsection, we provide some examples of the preprocessed data for the downstream tasks. We use the preprocessing steps described in (Raffel al., 2020).
F.1.1 CoLA
We assign label 0 as "unacceptable" and label "acceptable".
Original inputs:
sentence: Our friends won't 't buy this analysis, let alone the next one we propose.
label:
Preprocessed:
input:
cola sentence: Our friends won't buy this analysis, let alone the next pro- pose.
target: acceptable
F.1.2 MNLI
We assign label 0 as "entailment", label as "neutral" and label as "contradiction".
Original inputs:
hypothesis: Product and geography are what make cream skimming work.
premise: Conceptually cream skimming has two basic dimensions product and geogra- phy.
label:
Preprocessed:
input: mnli hypothesis: Product and geography what make cream skimming work.
premise: Conceptually cream skimming has two basic dimensions product and geography.
target: neutral
F.1.3 MRPC
We assign label 0 as "not_quivalent" and label 1 as "equivalent".
Original inputs:
26

---

Published as a conference paper at ICLR 2024
sentencel:
Amrozi accused his brother, whom he called "the witness" of deliberately distorting his evidence.
sentence2: Referring him as only "the witness", Amrozi accused his brother of deliber- ately distorting his evidence.
label:
Preprocessed:
input: mrpc sentencel: Amrozi accused his brother whom he called "the witness", of deliberately distorting his evidence sentence2: Referring to him as only "the witness". Amrozi accused his brother of deliberately distorting his evidence.
output: equivalent
F.1.4 QQP
We assign label O as "not_ "nott_upplictee and label 1 as "duplicate".
Original inputs:
question1: How is the life of math student? Could you describe your own experiences?
question2: Which level of prepration is enough for the exam jlpt5?
label:
Preprocessed:
input:
qqp questionl: How is the life of math student? Could you describe your own experiences? question2: Which level of prepration is enough for the exam jlpt5?
output: not nop_dupli
F.1.5 RTE
We assign label 0 as "entailment" and label as entt_eenailmmmt".
Original inputs:
sentencel: No Weapons of Mass Destruction Found in Iraq Yet.
sentence2: Weapons of Mass Destruction Found in Iraq.
label:
Preprocessed:
input: rie sentencel: No Weapons of Mass Destruction Found in Iraq Yet. sentence2: Weapons of Mass Destruction Found in Iraq.
output: not not_entailment
F.1.6 SST2
We assign label 0 as "negative" and label 1 as "positive".
Original inputs:
sentence: hide new secretions from the parental units
label:
Preprocessed:
input:
ss12 sentence: hide new secretions from the parental units
output: negative
27

---

Published as a conference paper at ICLR 2024
F.1.7 STSB
We format the 'label' argument as string with one decimal place, the corresponding Python code is x"" (::1f]" ....ffffrraaatt (label)'.
Original inputs:
sentencel: plane is taking off.
sentence2: An air plane is taking off.
label:
Preprocessed:
input: stsb sentencel: A plane is taking off. sentence2: An air plane is taking off.
output: 5.0
F.2
DISCUSSION ON THE CHOICE PROMPT TEMPLATES
Table Comparison of the templates used our experiments and better version of prompt tem- plates.
Task
Prompt Templates
input text
target text
CoLA Templates in Appendix F.1
"cola sentence: (sentence)"
"unacceptable" if la- bel=0 else "acceptable" "unacceptable" if la- bel=0 else "acceptable"
Improved Prompt Templates
"Indicate if the following sen- tence is grammatically correct not: "[sentence)". An- swere 'acceptable" or 'unac- ceptable'."
MNLI Templates in Appendix F.I
"mnli hypothesis: [hypothesis) premise: [premise]"
"entailment", "neutral", "contradiction" label 0, 1,2, respectively "entailment", "neutral",
Improved Prompt Templates
'(premise)' logically imply, contradict, or neutral to the hypothesis: "(hy- pothesis] Answere with "en- tailment', 'contradiction', "neutral'""
"Does the premise:
"contradiction" if label is
0, 1,2, respectively
RTE
Templates in Appendix F.1
"rte sentence1: [sentencel] sentence2: [sentence2]" "Does the text: '[sentence1 J entail that "seentenc2]] true? Provide 'yes" or 'no"."
"entailment" if label=0 else "not_ "ntt_eentalment" "yes" if label=0, else "no"
Improved Prompt Templates
To optimize the model's output, providing more explicit cues within the input is essential. An illustrative prompt, such as "sentence. Is this sentence 'positive' or "negative""" as suggested Ap- pendix F.1.6, signals clearly to the language model that the desired response should be categorized as either "positive' or 'negative', potentially leading to improved accuracy.
To validate this approach, conducted set of focused multi-task model fusion experiments target- ing three downstream tasks: CoLA, MNLI, and RTE. Across all fine-tuning processes, we employed the Adam optimizer, standardized the batch size to 16, and set the number of fine-tuning steps to 2000 for all models. While the learning rate was configured to le-5 for full fine-tuning, for both LoRA and L-LoRA, increased the learning rate to 4e-5.
Table provides comparison between the original prompt templates and an enhanced version. Table 7 outlines the individual performance of fine-tuned models using different prompt templates.
28

---

Published as a conference paper at ICLR 2024
Table 7: individual performance of fine-tuned models with different prompt templates
Fine-tuning Method
CoLA
MNLI
RTE
Mean
Prompt Templates in Appendix F.l
full fine-tuning
0.75
0.82
0.85
0.81
LoRA fine-tuning
0.69
0.76
0.83
0.76
L--LoRA fine-tuning
0.69
0.46
0.75
0.63
Better Prompt Templates
full fine-tuning
0.75
0.83(+1%)
0.86(+1 %6)
0.81
LoRA fine-tuning
0.69
0.83(+9%)
0.84(+11%)
0.79(+4%)
L-LoRA fine-tuning
0.69
0.82(+78%)
0.81(429%)
0.77(+22%)
Table Three-model fusion using simple model averaging, evaluate 'exact-match" accuracy validation datasets. These results show that a better and more predictable prompt yields superior results, with L-LoRA outperforming LoRA in terms of both absolute scores and normalized scores.
Prompt Templates in Appendix F.1 CoLA
Improved Prompt Templates
Method
MNLI RTE Mean
CoLA MNLI RTE MEAN
Absolute Score
FFT
0.67
0.37
0.54
0.53
0.70
0.78
0.82
0.76
LoRA
0.35
0.00
0.42
0.25
0.69
0.63
0,82
0.71
L-LoRA
0.35
0.00
0.23
0.20
0.69
0.73
0.81
0.74
Normalized Score
FFT 0.90
0.45
0.63
0.66
0.93
0.94
0.95
0.94
LoRA 0.50
0.00
0.50
0.34
1.00
0.76
0.98
0.91
L-LoRA
0.51
0.00
0.31
0.28
1.00
0.89
1.00
0.96
In Table 8, we evaluate 'exact-match' accuracy on validation datasets through a simple model av- eraging technique for three-model fusion. Our results indicate that more effective and predictable prompt yields superior performance, with L-LoRA outperforming LoRA in terms of both absolute scores and normalized scores. the "Absolute Score" section, the full fine-tuning (FFT) method appears show the highest scores with both sets of prompt templates, indicating superior perfor- mance in both original and improved settings. However, L-LoRA shows an increase in performance with the improved prompts compared to the original prompts, particularly in the MNLI and RTE columns and subsequently in the MEAN column. The "Normalized Score" section provides a rela- tive comparison of each method's performance. In this section, it suggests that despite FFT showing high performance in absolute terms, L-LoRA demonstrates increased normalized scores with bet- ter prompt templates, indicating greater effectiveness on weight disentanglement when adjusted to more appropriate prompting.
This experiment demonstrats that the choice prompt templates can impact the performance of these model fusion methods. It illustrates how more effective prompting strategy, especially one that better aligned with the model and task hand, can lead to significant improvements in performance. Besides, the highest normalized scores with these better prompt templates are achieved by L-LoRA, suggesting that the most responsive to prompt optimization among the methods tested.
29

---

Published as a conference paper at ICLR 2024
F.3 MODEL LINEARIZATION
class 1ineari zedMode Kragerinn ingeerinnnMMullee Module): init Isoit, model; En.Modulo, init_nodel: no.Rofuls
Initielines a Hnearised model.
Adgs: (nd.Modeelen
underlylne lying PyTenoh inearized.
mode_ (on, Hon.Hod The initial PyIorch modal used to conpute the linearization paraneters 1
detsuit: Nonel.
...
cupezt) -iriii..niis init_() selt.model model
12
12 init_ncdel is None:
13
init_model assect hasatcriseit, "parane0*)
paramse dI v.detsch for k, v in htt.mmooe..nmmme_orreeeeer
16
selt. celit.pacemo__eyee [x for & parenst]
self, pall.paaanao_vala Faraanaettttti for pasans0]] wel..paramao_ p.reguires_grro__ffal
20
supis_porome_to_dis (sole, cupis_parans):
23
Converts tuplo of paraneters to dictionazy with kays cocrespending to the paranstes nanes.
24
25
tuplo_parana [Tense,,, ...)): tupla of parametors.
27
28
Raturna:
Tenser]: dictionary keys corraspsading cccsesponsing
peranstez values.
31
32
(ouplespporaa Lon(eelf..promms_eeer stata_dict
34
for k, 99 in 2ip(eslr.parans_koy tuple_parmma): [Latm_diitt[A]
35
state_diet
38
del (self, sarga, skkaccy)): one
39
Computes linerrized model using first-order Taylor decenpesition.
42
Azds: Toxitionel argunentx to be passed to the nodel.
43
sekanggss Keyword azgunenta be passed the nodel.
46
Returns:
47
Lorch.Tenaoes Tensors The outpuz of the linearized model, computed using flest-order Tayloz decosposition
parans0 tupla valuee)
50
pazana par__parars____oto (OzdezedDiet tself
51
dperans tuple(p po for
sip(gerans, parane01)
xparan: ounctiooal__all;
se seldenddel, sel f.tull paranns dict (paran), args, kwargs
55
),
dparems,
59
out
Listing 1: Pytorch code to linearize model.
The LinearizedMel ledaaeiiddddeeeaaperrr class linearizes a given PyTorch model using a first-order Taylor expansion. The linearized model is computed using the forward method, which takes positional and keyword arguments and returns the output of the linearized model. The tuple_paramen ttsdsitt___________toe__eeeeeeet et method converts tuple of parameters dictionary with keys responding to the parameter names.
With the help of Lineari I class, we can linearize the LoRA modules in pre-trained language model, as the following code snippet shows:
def _iget_.aae zubboruu wa (nodels on Module,
keyl:
perent le, targes
targes
given key in PyTorch
Argas
model Modull ell The Pytorch model retrieve subnodules From,
key The key represencing sulmodule to retriwve.
Returns:
10
Tuple Maclul,, be, nn. Mniull,,,, strls tuple containing the pacent nodule, Larget nodulw, and Larget module
30

---

Published as a conference paper at ICLR 2024
pazent aet.blloatttt fow.aa..........ooo..o jopiikky...piic (1 target_nane (". [-1]
14
target medel.get_ o parent, target, carget_nana
18
def Linearize_erri (model: na. Nodule)))
Lineacises Goratayes nodulwa PyTozch oudel.
21
22
Args:
model (mn.Rodale)) The PyTorch model linearized.
25
Returns:
26
an.Modele: The linwarixed PyTorch model.
***
key, nodule model, menei_nnddllwedl
29
L2 isinstance (nodule, horalayez) and isinstance mnddle,, an, Lineae): log.debug convert (kay] to linearized zod 1ora layezf) petent, Larget, barget_naee _get_ abmoddles tmadel, keyl sutacte spazent, target_name, Lineari wodod (targat))
30
33
return nodel
Listing 2: Linearize all the LoRA modules in a LLM
Where the gget_ssbmmouules function takes PyTorcc model and key representing submod- ule in the model and returns tuple containing the parent module, the target module, and the target module name. The parent module is the module that contains the target module.
The linearize seteriie__loe cice__orr_mmocccc 1 function takes PyTorch model and linearizes all LoraLayer modules in the model. If LoraLayer module is also an instance of nn Linear, it is converted to Linear wewarriiieeoddeeeaeeer Wraper object. The function returns the linearized Py PyTorch model.
31


Reviews:
[{"review_id": "aBi4FrnLpB", "reviewer": "Reviewer_tr5D", "summary": "Efficient finetuning on the pretrained large model has been an important topic. In this work, a partial linearization method (L-Lora) is proposed under the context of PERF(parameter-efficient finetuning).  The key idea is applying linearization to adapter modules and applies task arithmetic over the linearized adapters. In practice, first-order Tayler expansion is used to linearize the model dynamics at time $t$. Based on the derivation from a neural tangent kernel theory, the hypothesis is that partial linearization of a subset of model parameters during fine-tuning can also improve weight disentanglement compared to full non-linear fine-tuning. CLIP and Flan-T5 are used to verify the hypothesis in vision-language and language domains.", "questions": "- From table 1, seems the L-LoRA method are outperforming full-finetuning under some model fusion settings, do we have some possible illustrations?\n- Is it possible to evaluate L-LoRA on image segmentation foundational models like SAM?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Evaluations are conducted on both vision-language and language tasks. \n- The proposed method achieved significant performance improvement, compared with the standard LoRA strategy.", "weaknesses": "- In vision domain, only the high-level vision task like image classification tasks evaluated, the mid-level and low level task are missing, for example, semantic segmentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3i61Guje6Y", "reviewer": "Reviewer_iJMj", "summary": "This paper proposes a new method for Parameter-Efficient Fine Tuning (PEFT) of large pre-trained foundational models for multi-task models. The authors build on prior work in weight disentanglement (Ortiz-Jimenez et al. 2023) and extend it to LoRA for better fusion of models. The authors hypothesise that partial linearisation of a model through the LoRA modules during fine-tuning can improve weight disentanglement, which is conducive to better task arithmetic. Results are shown on a variety of experiments whereupon the proposed models outperforms others on vision classification tasks.", "questions": "1. Section 4.2 introduces parameter scaling laws for weight disentanglement. However, what does this have to do with the method or key results? The scaling laws suggest over-parameterisation is necessary for weight disentanglement. I struggle to see the connection between this and the need to partially linearize a model for fine-tuning.\n\n2. A remark in the manuscript is made bottom of page 8 that \"...higher cosine similarity...implies greater redundancy and overlap...This results in more destructive task interference with naïve merging\". I don't follow this as it seems to be opposite to most work in multi-task training. In that setting, naïve training would favour similar tasks as it would not require methods to mitigate task interference (see GradNorm method). Why is this then opposite in the context of this paper?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. This is a very well written paper, motivations are clear, results are (mostly) well presented and it is clear to understand the results.\n\n2. The idea to perform partial linearisation on LoRA modules is interesting and nicely presented. The results, especially on CLIP-ViT-B-16 (Figure 5a) are compelling argument to the original hypothesis of the work being correct.", "weaknesses": "1. A key result of the paper is the result from Appendix A which allows the authors to hypothesize that partial linearization of a subset of module parameters (here LoRA) can improve weight disentanglement.\n\n    1a. This result needs to be in the main body of the text and properly explained. Without, it is difficult to \n    understand exactly why the authors claim this.\n\n    1b. Having checked the derivation in Appendix A, the authors show that the model output of a linearised \n    model is only determined by the gradient of the loss in the non-linearised model on task $t_i$. How \n    exactly then does this allow the authors to make the central hypothesis, which guides the presented \n    method?\n\n2. The results on the NLP task (Flan T-5-Base) need to be better explain. The method (L-LoRA) not only performs worse than full fine-tuning (so does LoRA fine-tuning) but also worse on average than LoRA. Why is this? Is the presented model only applicable to vision tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eomIzzF9zb", "reviewer": "Reviewer_HkFi", "summary": "This paper proposes a novel method to improve multi-task fusion for parameter-efficient fine-tuning techniques like LoRA fine-tuning. Specifically, their approach partially linearizes only the adapter modules and applies task arithmetic over the linearized adapters. This allows us to leverage the advantages of model fusion over linearized fine-tuning, while still performing fine-tuning and inference efficiently. Extensive experiments are conducted.", "questions": "### Major Concerns:\n1. The novelty of this paper is limited. [1] studies the linearized full fine-tuning while this paper simply replaces full fine-tuning with LoRA. Although the authors emphasize the proposed method is more efficient than [1], it is obvious since LoRA is more efficient than fully fine-tuning.\n2. What's the meaning of $\\phi_0$? For the full fine-tuning, $\\theta_0$ is the pre-trained model weight and is shared for every task $\\tau_i$. However, for LoRA, $\\phi_i$ is newly added parameters for each task $\\tau_i$. Thus, what is $\\psi_0$? Is $\\psi_0$ a shared initialization for LoRA matrixes of every task $\\tau_i$?\n3. In [2], the task vector of full fine-tuning is defined as $\\nu_i=\\theta_i-\\theta_0$, which means the parameter change of $\\theta_0$. Thus, why the task vector of LoRA in this paper is defined as the change of LoRA parameters $\\phi_i-\\phi_0$ rather than the change of $\\theta_0$ as in [3], i.e., $A_iB_i$, where $\\phi_i=[A_i, B_i]$ is the LoRA parameters.\n4. Table 1 only shows the average normalized scores over multiple datasets, so how about the performance of each dataset? Could you provide it in the Appendix?\n5. It seems the proposed L-LoRA method does not perform well on the NLP domain, according to the results in Table 1 and the similarity heatmap in Figure 8.\n6. Why use the proposed L-LoRA rather than the existing LoRA? The single-task fine-tuning results in Figure 7 show that L-LoRA has a large performance drop compared to LoRA in many datasets.  \n\n### Minor Concerns:\n1. The caption of Figure 3(c): not linearized?\n\n\n**References**\n\n[1] Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models. arXiv:2305.12827.\n\n[2] Editing Models with Task Arithmetic. ICLR, 2023.\n\n[3] Effective and Parameter-Efficient Reusing Fine-Tuned Models. arXiv:2310.01886.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The code is provided.\n2. This paper proposes a new linearized LoRA method.\n3. Extensive experiments are conducted.", "weaknesses": "1. The novelty of this paper is limited. This paper simply adapts the proposed method in [1] by replacing full fine-tuning with LoRA. Thus, the method in this paper is obviously more efficient than [1] since LoRA is more efficient than full fine-tuning.\n2. The definition of the task vector of LoRA in this paper seems to be unreasonable.\n3. The L-LoRA has a large performance drop on single-task fine-tuning (Figure 7) and a slight increase on the merged case (Table 1, especially in the NLP domain) compared to LoRA. Thus, it is unclear what are the advantages of L-LoRA compared to LoRA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q9sbKYO1po", "reviewer": "Reviewer_vkjS", "summary": "This paper proposes a novel approach to enhance multi-task fusion in large pre-trained models. The authors introduce partial linearization of adapter modules combined with task arithmetic, improving the fusion of multiple tasks while maintaining efficient fine-tuning and inference. Experimental results demonstrate that this method outperforms standard techniques, especially as the number of tasks increases. The contribution lies in its ability to construct unified multi-task models effectively and efficiently fuse fine-tuned task vectors, highlighting the benefits of partial linearization for scalable multi-task model fusion.", "questions": "- Can the proposed methods demonstrate effectiveness on SOTA models?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper exhibits a clear and logical structure, making it easy to comprehend.\n\n- The proposal's effectiveness is demonstrated through comprehensive experiments conducted on both NLP and image classification tasks. The visualization provided in Figure 6 offers intriguing insights into disentanglement error, further substantiating the proposal's efficacy.", "weaknesses": "- The absence of experiments conducted on larger-scale models diminishes the significance of the proposal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)