LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging
Authors: Ke Wang
Nikolaos Dimitriadis
Alessandro Favero
Guillermo Ortiz-Jimenez
François Fleuret
Pascal Frossard
Venue: ICLR 2025 Poster (2025)
Keywords: Model merging; Model editing; Catastrophic forgetting; Multi-task learning; OOD generalization
Primary area: foundation or frontier models, including LLMs
TL;DR: We apply a post-training layer-wise scaling to preserve zero-shot abilities and enhances model merging

Abstract: Fine-tuning pre-trained models has become the standard approach to endow them with specialized knowledge, but it poses fundamental challenges. In particular, (i) fine-tuning often leads to catastrophic forgetting, where improvements on a target domain degrade generalization on other tasks, and (ii) merging fine-tuned checkpoints from disparate tasks can lead to significant performance loss. To address these challenges, we introduce LiNeS, Layer-increasing Network Scaling, a post-training editing technique designed to preserve pre-trained generalization while enhancing fine-tuned task performance. LiNeS scales parameter updates linearly based on their layer depth within the network, maintaining shallow layers close to their pre-trained values to preserve general features while allowing deeper layers to retain task-specific representations. In multi-task model merging scenarios, layer-wise scaling of merged parameters reduces negative task interference. LiNeS demonstrates significant improvements in both single-task and multi-task settings across various benchmarks in vision and natural language processing. It mitigates forgetting, enhances out-of-distribution generalization, integrates seamlessly with existing multi-task model merging baselines improving their performance across benchmarks and model sizes, and can boost generalization when merging LLM policies aligned with different rewards via RLHF. Our method is simple to implement, computationally efficient and complementary to many existing techniques. Our source code is available at github.com/wang-kee/LiNeS.

Published as a conference paper at ICLR 2025
LINES: POST-TRAINING LAYER SCALING PREVENTS FORGETTING AND ENHANCES MODEL MERGING
Ke Wang"
Nikolaos Dimitriadis"
EPFL k. wanggeefl..co
EPFL nikolaos www.triadiis@@p.........
Alessandro Favero
Guillermo Ortiz-Jimenez
EPFL alessandro. favero@epfl.ch
Google DeepMind
gortizj@google.co com
François Fleuret
Paseal Frossard
University of Geneva, Meta FAIR francois. fleureetunigee.....
EPFL pascal. wwossardeeel....c..
ABSTRACT
Fine-tuning pre-trained models has become the standard approach to endow them with specialized knowledge, but it poses fundamental challenges. In particular, (i) fine-tuning often leads catastrophic forgetting, where improvements target domain degrade generalization on other tasks, and (ii) merging fine-tuned checkpoints from disparate tasks can lead significant performance loss. To address these challenges, we introduce LiNeS, Layer-increasing Network Scaling, a post-training editing technique designed to preserve pre-trained generalization while enhancing fine-tuned task performance. LiNeS scales parameter updates linearly based on their layer depth within the network, maintaining shallow layers close to their pre-trained values to preserve general features while allowing deeper layers to retain task-specific representations. multi-task model merging scenar- ios, layer-wise scaling of merged parameters reduces negative task interference. LiNeS demonstrates significant improvements in both single-task task and multi-task settings across various benchmarks in vision and natural language processing. It mitigates forgetting, enhances distriiiittiittonn generalization, integrates seamlessly with existing multi-task model merging baselines improving their per- formance across benchmarks and model sizes, and can boost generalization when merging LLM policies aligned with different rewards via RLHF. Our method is simple to implement, computationally efficient and complementary to many existing techniques. Our source code available at hetttu...omwwann-kk...
INTRODUCTION
Pre-trained models have become the backbone of modern machine learning pipelines (Bommasani et al., 2021; Touvron et al., 2023). Their introduction has shifted the paradigm from end-to-end training to fine-tuning (Zhuang et al., 2020), leading the proliferation of thousands of fine-tuned checkpoints derived from a few foundation models (Rombach et al., 2022; Team et al., 2023). To improve downstream performance across multiple tasks align with multiple preferences (Singh & Jaggi, 2020; Matena & Raffel, 2022; Ilharco et al., 2023; Yadav et al., 2023; Ramé et al., 2024a), model merging techniques combine available checkpoints, avoiding the costly process of joint fine-tuning (Ilharco et al., 2023; Yadav et al., 2023). However, specializing models introduces trade-offs, such as the forgetting of previously acquired knowledge (Aghajanyan al., 2021) phenomenon known as catastrophic forgetting (McCloskey & Cohen, 1989). Furthermore, merging checkpoints fine-tuned on different tasks can lead to significant performance degradation due to task interference (Yadav et al., 2023; Wang et al., 2024).
"Equal Contribution

---

Published as a conference paper at ICLR 2025
To mitigate catastrophic forgetting, many works propose regularizing the fine-tuning process (Aghajanyan et al., 2021; Kumar et al., 2022; Gouk et al., 2021; Razdaibiedina et al., 2023). Leveraging the insight that shallow layers capture generalizable representations (Yosinski et al., 2014; Neyshabur et al., 2020), Howard & Ruder (2018); Dong et al. (2022) apply lower learning rates to the shallow layers to retain general features. However, modifying the fine-tuning process can be complex and computationally expensive, This motivates the development of post-training model editing and model merging methods that directly edit the checkpoints in the weight space. For instance, Wortsman et al. (2022b); Ramé et al. (2022) mitigate catastrophic forgetting by interpolating weights between pre-trained and fine-tuned models. In multi-task settings, Yadav et al. (2023); Wang et al. (2024) propose methods to reduce interference among tasks when merging multiple checkpoints. Yet, significant performance degradation persists when merging multiple models, leaving this as an open challenge.
Most model merging methods, however, treat all layers equally, overlooking the earlier insight that shallow layers should remain close to their pre-trained weights to avoid losing the general represen- tations they encode. In this paper, we explore whether this insight can be leveraged post-training. We find that reducing the magnitude of shallow-layer updates after fine-tuning can retain single-task task performance gains while significantly mitigating forgetting.
We propose LiNeS, Layer-innreasee directly edits the residual, i.e., the difference between the fine-tuned and pre-trained checkpoint, by applying scaling coefficient that linearly increases with layer depth. This scaling effectively preserves the general features captured in the shallow layers of the pre-trained model while retaining task-specific features in the deep layers of the fine-tuned model. Moreover, we extend LiNeS to the multi-task model merging setting, where contributions from one task distort the general features also required by other tasks. By preserving the general features in the shallow layers, LiNeS mitigates task interference and improves multi-task performance.
Network Scaling, post-training, plug and-play method that
LiNeS demonstrates remarkable performance on diverse test scenarios and is orthogonal to existing post-training merging algorithms. It modifies the fine-tuned checkpoint to consistently retrieve nearly full performance on the fine-tuned task while significantly restoring generalization on other tasks. Furthermore, it can be seamlessly integrated with existing weight interpolation methods for improving out-of-distribution generalization (Wortsman et al., 2022b). When merging multiple models, LiNeS improves baseline methods for merging checkpoints fine-tuned multiple tasks in both computer vision and NLP benchmarks (Ilharco et al., 2023; Yadav et al., 2023; Wang al., 2024) and also enhances performance when merging checkpoints fine-tuned on the same task (Wortsman et al., 2022a) and merging LLM policies aligned with different rewards (Ramé et al., 2024a) via Reinforcement Learning with Human Feedback (RLHF) (Christiano et al., 2017).
Our contributions are as follows:
We propose LiNeS, a post-training editing technique that preserves the zero-shot generalization of pre-trained models while retaining fine-tuned knowledge by applying layer-wise scaling on parameter updates. For example, in image-classfficatin tasks with CLIP ViT-B/32 checkpoints, LiNes maintains on average 99.8% of performance on the fine-tuned task while preserving 97.9% performance of the pre-trained model on other control tasks, effectively mitigating catastrophic forgetting.
We demonstrate that LiNeS significantly enhances multi-task model merging baselines, consistently improving performance across benchmarks and architectures in both vision and NLP domains. For instance, we observe a 3.1 % and 4.0% improvement over Task Arithmetic (llarco et al., 2023) and Ties-merging (Yadav et al., 2023) respectively, for 20-task computer vision benchmark with ViT-L/14.
We show that LiNeS can be applied to enhance existing weight interpolation methods across various scenarios, improving disto--distribution generalization, merging multiple checkpoints fine-tuned on the same task with different syper-parament configurations, and merging LLM policies aligned with different rewards.
Our proposed method is simple to implement', orthogonal to many existing approaches, and im- proves performance in a wide variety of settings.
Py Torch pseudo-code in Appendix A.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
Representation collapse and regularized fine-tuning Pre-trained models such as CLIP exhibit strong zero-shot performance across diverse data distributions due to the robust and transferable feature representations learned during pre-training (Radford et al., 2021; Jia et al., 2021). However, fine-tuning on specific tasks often harms the zero-shot generalization performance on distributions different from the fine-tuning domain (Wortsman et al.. 2022b; Goyal et al., 2023; Aghajanyan al., 2021). This degradation arises from the distortion of pre-trained features during fine-tuning (Kumar et al., 2022), phenomenon referred to as representation collapse by Aghajanyan et al. (2021). To mitigate representation collapse, many works have proposed to regularize the fine-tuning process to preserve the general pre-trained features (Kumar et al., 2022: Goyal et al., 2023; Gouk al,, 2021; Zhang et al., 2022; Razdaibiedina et al., 2023; Shen et al., 2021; Lee et al., 2022). Some of these approaches take into account that different layers of model learn distinct features, with the shallower layers capturing more general features and deeper layers specializing in task-specific representations (Neyshabur et al., 2020; Yosinski et al., 2014; Adilova et al., 2024). Specifically, they apply layer-wise learning rate decay, preserving more of the pre- pre-trained features in the shallow layers while allowing deeper layers to specialize for the target domain (Clark et al., 2020; Bao et al., 2022; Dong et al., 2022; Howard & Ruder, 2018; Zhang et al., 2021). However, modifying the fine-tuning process is orders of magnitude more computationally expensive compared to post-training merging methods.
Weight interpolation and model merging Garipov et al. (2018); Draxler et al. (2018) showed that two solutions derived from separate training runs can be connected by nonlinear paths of low loss, while linear mode connectivity (Frankle et al., 2020) extended the paths to the linear case. These insights enabled the transfer of the benefits regarding robustness of (traditional) output ensembles (Hansen Salamon, 1990; Lakshminarayanan et al., 2017) to weight ensembles, reconciling the bias variance trade-off (Belkin et al., 2019) while eliminating the computational cost of multiple in- ferences (Fort et al., 2020). These findings can be leveraged to improve performance on single-task (Izmailov et al., 2018; Wortsman et al., 2021; Ramé et al., 2022; Wortsman et al., 2022a; Jang al., 2024),
(Wortsman et al., 2022b; Ramé et al., 2023), multi-task (Ilharco et al., 2022: Dimitriadis et al., 2023; 2025) and multi-objeccive alignment (Zhong al., 2024; Ramé al., 2024b) settings. Furthermore, model merging can also applied as a scalable approach to unify multi- ple task-specific models into single model with multi-task capabilities (Ilharco al., 2023; Yadav et al., 2023), despite performance loss compared to individual models. Several methods have tried to improve multi-task model merging by preserving the important parameters defined via the Fisher In- formation Matrix (Matena & Raffel, 2022; Tam et al., 2024), using heuristics (Davari & Belilovsky, 2023; Luo et al., 2023; Jin et al., 2023), randomly dropping and rescaling the task vector parameters (Yu et al., 2024) or by focusing on resolving weight interference caused by sign disagreements and redundant parameters (Yadav et al., 2023; Wang et al., 2024). Recent works use gradient descent to learn the layer-specific merging coefficients per task, e.g., Ada-merging (Yang et al., 2024) mini- mizes entropy in unlabeled test data while aTLAS (Zhang et al., 2024) optimizes using cross-entee cross-entropy loss on validation data. Compared to LiNeS, these methods do not incorporate any prior knowledge on early vs. deep layers and require training, resulting in significant computational overheads.
3 POST-TRAINING LAYER-WISE SCALING MITIGATES FORGETTING
In this section, we present the key insight of our work: Scaling down the updates of shallow lay- ers after fine-tuning can mitigate catastrophic forgetting and restore zero-shot generalization while preserving performance on the target task.
Notation We consider pre-trained model Og E N with N parameters. Fine-tuning on a specific task results in the fine-tuned weights O, The difference between these two sets of weights, т, = O, Oo. is referred to the task vector residual for task (Ilharco et al., 2023) and represents the updates made during fine-tuning.
Fine-tuning leads to catastrophic forgetting We quantitatively demonstrate the phenomenon of catastrophic forgetting with the following experiments. Consider the 8-task image classification benchmark studied in Ilharco al. (2023). We fine-tune CLIP ViT-B/32 model each task,
3

---

Published as a conference paper at ICLR 2025
Target task
Control tasks
100
50-
Pre trinee
Accuracy
80
00
Pretteined
40-
Flne-tuned
1.0
0.8
0.8
0.4
0.2
0.0
1.0
0.8
0.6
0.1
0.2
0.0
Downerale shallow layers
Downsaale deep layers
Figure Downscaling the shallow layers maintains the fine-tuned performance on target tasks (orange line, left), while restoring zero-shot performance from pre-trained model on control tasks (orange line, right). The performance for downscaling deep layers instead is presented in blue lines, which underperforms downscaling shallow layers in both cases. 7 represents the minimum scaling factor applied the layers, where smaller y leads to stronger downscaling strength, with ? restoring the original fine-tuned model.
measuring performance on the fine-tuned task referred to as the target task and the remaining 7 tasks the control tasks. The averaged results over all target and control task combinations, shown in Table 1, demonstrate that while fine-tuning significantly improves accuracy on the target task, drastically reduces accuracy on the control tasks, underscoring the loss of the model's zero-shot generalization abilities.
Shallow-layer updates impact minimally on target task accuracy Most parameter updates during the fine-tuning process redundant, as similar performance is achievable without updating most pre-trained weights (Yadav al., 2023; Wang et al., 2024; He et al., 2025). Moreover, prior work shows that task-specific features are often concentrated in deeper layers of the network (Neyshabur et al., 2020; Yosinski et al., 2014; Raghu et al., 2019). Based on these observations, we hypothesize that updates to the shallow layers contribute minimally to target tasks. To test this, progressively downscale the updates to shallow layers after fine-tuning. Specifically, we apply a scaling factor to the updates to f-th layer (8), defined as: X = 7 (1 (11-7) f-1 f- AŁ E [4], E [o, 1]. This linearly scales the updates from a factor of y for the first layer to for the last one. As result, fine-tuning updates to the shallow layers are scaled down more aggressively, with later layers experiencing progressively smaller reductions. We then reintroduce the scaled task vector into the pre-trained model and measure its performance the CLIP ViT-B/32 checkpoint fine-tuned across the tasks, where 7 is progressively decreased to strengthen the downscaling effect. We observe that, even with strong downscaling of shallow layers, the target task accuracy remains nearly unaffected. In contrast, when we downscale the deeper layers, target task accuracy drops significantly. These results support hypothesis that shallow-layer updates are largely unnecessary for maintaining accuracy on the target task.
Table Fine-tuning harms generalization control tasks. Our proposed post-training edition leads to a superior trade-off between performance target and control tasks.
Model / Accuracy
Target
Control
Pre-trained
48.3
48.3
Fine-tuned
90.5
38.0
Fine-tuned+LiNeS (ours)
90.3
48.0
the fine-tuned task. Figure I (left) shows the results of this experiment
Shallow-layer updates undermine zero-shot generalization While shallow-layer updates have minimal impact on target-task accuracy, they distort the general features learned during pre-training. which reside primarily in the shallow layers (Neyshabur et al., 2020; Yosinski et al., 2014; 4; Raghu control tasks is largely due
al., 2019). We hypothesize that the degradation of performance to these distortions in the shallow layers. Using the same experimental setup, we now evaluate the zero-shot performance on the control tasks, i.c., the other unseen tasks. As shown in Figure (right), as the strength of the shallow-laye downscaling increases, the accuracy on control tasks approaches the original pre-trained model's performance. This shows that by reducing the shallow- layer updates, we can restore most of the zero-shot performance that is lost during fine-tuning.
4

---

Published as a conference paper at ICLR 2025
Improved trade-off between target and con- trol performance To optimize the trade-off between target and control task performance, we select a scaling coefficient for each model that maximizes weighted balance be- tween these two objectives, as detailed in Ap- pendix C.I. After selecting the optimal scaling coefficient, the test results are shown in the fi- nal row of Table 1. Our post-training method preserves target task accuracy with minimal 0.2% difference while improving control task performance by 10%, compared to the fine- tuned model.
100
Control tasles normaliood nrcurary
80
70-
Pre-trained
Fine-tuned
60-
Fine-tuned LiNeS (ours)
50 0
We further apply the same method to a 20-task computer vision benchmark (Wang et al., 2024). For evaluation, report both the target task normalized accuracy and the control task normalized accuracy on the 19 tasks, where accuracy is normalized by the performance of the fine-tuned model for the target task and the zero-shot accuracy of the pre-trained model for the control tasks. We compare to fine-tuned models on each task and the pre-trained model as baselines. Figure 2 shows that fine-tuning degrades zero-shot generalization, as indicated by the performance drop on control tasks. In contrast, our post-training scaling method significantly improves generalization while maintaining near-full target task accuracy. On average, our method achieves a target task normalized accuracy of 99.8% and a control task normalized accuracy 97.9%. This demonstrates its effectiveness preserving both task-specific knowledge from fine-tuned checkpoints and the generalization capabilities of the pre-trained model. The full breakdown of results by task is available Figure in Appendix.
20
40
60
80
100
Turget task normalised anturacy (%)
Our linear scaling (LiNeS) retains performance on both control and fine-tuned target tasks. Each dot represents different model.
Figure 2:
In Appendix, we show that catastrophic forgetting happens with models fine-tuned with LoRA (Hu et al., 2022) as well. As shown in Table 11, higher expressivity in the form of higher ranks increases target accuracy for LoRA but the cost of lower performance significantly improves control performance while minimally affecting target accuracy. Furthermore, in Figure 10 of the appendíx, we show that similar benefits can be observed for convolutional architectures such as ConvNeXt (Liu et al., 2022). Finally, we provide performance comparison between editing models with LiNeS and regularized regularized-ine--uning based methods in Appendix C.8, including applying different learning rates per layer. Also in these cases, LiNeS demonstrates superior performance on control tasks, while being much more computationally efficient.
control tasks. Still, LiNeS
4 METHOD
Motivated by the results of the previous section for mitigating forgetting, we propose LiNeS for Layer-inereasing Network Scaling. a simple post-training technique that linearly rescales the updates of different layers in the task vector based on their depth in the network. LiNes is designed to retain general features in the shallow layers while preserving the task-specific adaptations in the deeper layers.
Given task vector with L layer blocks we apply the layer-wise linear scaling to adjust the con- tributions of shallow and deep layers using the following formulation:
TaNNss = concat (1 ......,,,))) (())(()) where ( = a
T. V [4].
(I)
As a result, the layers in T are progressively scaled with a factor between a for the first layer and a+33 3 for the last layer, with intermediate layers scaled with linearly increasing schedule depending on their depth. The final model 0 is then obtained by summing the pre- trainddidd model weights and the edited task vector, i.e., 0 Ou + TLNNs- Notice that, in Equation 1, a single-task residual or, in the context of model merging, a multi- task vector obtained by merging the residuals of multiple checkpoints fine-tuned starting from common initialization. Additional
correspond to either
5

---

Published as a conference paper at ICLR 2025
ImageNer/A
ImageNetSkotch
ObjectNet
41
41-
42-
LiNeo
00D
24
02
76.0
70.0
78.5
27.0
27.5
75.0
78.5
75.5
76.0
10.5
11.0
77.5
76.0
78.6
76.6
76.0
76.5
77.0
77.5
78.0
78.5
ImangoNes Accurary
ImageNect Accursey (56)
ImaageNet Accuracy (%)
ImageNeiR
VesageNett
010 Aequaay (80
LINGO
78.5
76.0
20.5
72.0
22.5
76.0
15.5
76.5
10.0
75.5
71.0
12.5
78.0
78.5
Imageieet Aomrary (56)
ImangeNee Arcumay (50
WISE-FT
WISE-FT LINeS
Fine-tuned
Finestued LINS
initiallzatton
Figure 3: Application of LiNeS to WiSE-FT (Wortsman et al., 2022b) improves performance on ImageNet and five different distribution shifts, resulting in dominating Pareto Front over WiSE-FT. LP initialization refers to the model initialized from linear probe (Kumar et al., 2022).
details on this process are provided in the next section. Setting = B = 0 corresponds to the pre- trained model, while a = 1, is the fine-tuned model the case that a single-task vector.
In practice, we find that tuning just one poper-parameter (either a or () is often sufficient to achieve good balance between target task performance and generalization. Specific details on hyper-parameter tuning for different applications are provided in the experimental sections. The linear scaling method introduced in Section 3 corresponds to LiNeS by setting a = y and = y. This formulation generalizes our previous approach, offering flexible way adjust the contributions of different layers based on the task requirements.
5 MODEL MERGING EXPERIMENTS
We empirically verify the effectiveness of applying LiNeS across diverse application domains. Section 5.1 presents results for improving robust fine-tuning (Wortsman al., 2022b) for OOD generalization; Section 5.2 focuses on improving existing multi-task merging methods (Ilharco et al., 2023; Yadav et al., 2023; Wang al., 2024) in both vision and NLP benchmarks. In Section 5.3, we apply LiNeS and improve the merging of single-task fine-tuned models within the setting of Model Soups (Wortsman et al., 2022a). and finally, we enhance merging foundation models fine-tuned on different rewards (Ramé et al., 2024a) in Section 5.4.
5.1 IMPROVING ROBUST FINE-TUNING FOR OOD GENERALIZATION
We first consider the setting of robust fine-tuning or WiSE-FT (Wortsman et al., 2022b), where lin- carly interpolating between the pre-trained and the fine-tume ine-ttned weights improves model performance on OOD datasets. The interpolation is equivalent scaling the residual т: (1-7)00+70 for y E (0. 1). We apply Li NiS to the residual т. Following Wortsman et al. (2022b), we evaluate CLIP models fine-tuned ageNetSketch (Wang et al., 2019), ImageNet-A (Hendrycks et al., 2021), ImageNet-R (Hendrycks al., 2020), ObjectNet (Barbu al., 2019), Ima (Recht et al., 2019).
Ou+YT,
ImageNet (Deng et al., 2009), considering 5 OOD datasets, namely Im-
We apply this L,i NeS to each of the 70 fine-tuned checkpoints provided by Wortsman et al. (2022a) setting a B 0.5. We present the average results in Figure 3, comparing the performance of WiSE-FT with and without applying LiNeS on the OOD datasets. Without applying WiSE-FT,
The checkpoints CLIP VIT-B/32 models fine-tuned ImageNet with different harerepa.

---

Published as a conference paper at ICLR 2025
Table 2: Results for multi-task model merging in vision classification benchmarks of tasks (Ilharco et al., 2023), 14 tasks, and 20 tasks (Wang et al., 2024) for different vision transformer architectures. Applying LiNeS improves baseline performance for all benchmark/archittectue combinations.
Method
with LiNeS
VIT-B/32
VIT-L/14
8 tasks
14 tasks
20 tasks
8 tasks
14 tasks
20 tasks
Zero-shot Fine-tuned
48.3
$7.3
56.1
64.8
68.3
65.3
90.5
89.5
90.4
94.0
93.3
94.0
Task Arithmetic
69.7
65.0
60.3
84,0
79.2
74.0
74.2 (++.55)
69.1 (+-4.1)
63.4 (+3.1)
86.5 (+2.5)
82.2 (+3.00
77.1 (+3.1)
73.6
67.6
63.1
85.6
79.3
75.6
Merging
77.2 (+336)
72.1 (+-15)
67.2 (+-.1)
88.0 (4:2.4)
82.5 (+-3.2)
79.6 (++1.0)
Consensus Merging
X
74.5
70.1
65.3
85.2
81.9
78.7
77.6 (+3.1)
73.6 (+3.5)
68.6 (+3.3)
(+2.1)
84.0 (+2.1)
81.0 (+2.3)
LiNeS already enhances both the ID and OOD performance of the fine-tuned models by a notable margin. Starting from this edited model and applying the WiSE-FT interpolation with the pre-trained weights leads to a Pareto Front (Caruana, 1997) that consistently dominates the one by WiSE-FT across all distribution shifts, illustrating the applicability of the proposed method across various distribution shifts. A granular result for applying LiNeS to each of the 70 checkpoints is provided in Appendix 10.2, further highlighting
universal effectiveness across models. We also report similar findings in Figure 11 in Appendix for CLIP ViT-B/16 checkpoint fine-tuned on ImageNet, using the same
as Wortsman et al. (2022b).
5.2 IMPROVING MULTI-TASK MODEL. MERGING
In this section, we extend LiNeS to improve multi-task merging algorithms, aiming to combine multiple models fine-tuned independently on different tasks into single model (Matena & Raffel, 2022; Ilharco et al., 2023; Ortiz-Jimenez et al., 2023; Yadav et al., 2023; Hazimeh et al., 2024). Task arithmetic (Ilharco et al., 2023) proposed to decouple the contributions of the pre-trained model and individual task vectors, first generating a multi-task vector TMTI. = _(T)..... Tr) with merging function g R x... R.V - RR,, and then adding back the pre-trained checkpoint with a scaling factor to construct a multi-task model 0 == 0g + X TMTL- The scalar coefficient is tuned using a held-out validation set. Recent works (Yadav et al., 2023; Wang al., 2024) follow the same protocol while improving the merging function g for retaining more task information. We refer Appendix B. for a more detailed explanation of these methods.
However, significant performance loss occurs between the merged multi task model and the original fine-tuned checkpoints. This performance decrease partially stems from interference (Yadav al., 2023; Wang et al., 2024) among task vectors, where the contribution of one task negatively impacts performance on others, leading to overall degradation. Task interference is linked catastrophic forgetting, as the individual task vectors lose a significant amount of generalization ability to other tasks after fine-tuning and merging them leads to interference among each other. Therefore, we can edit each task vector with LiNeS before merging to restore the generalization to other tasks, for simplicity, edit directly the merged multi-task vector to preserve the shallow and general features that are beneficial across tasks.
We enhance the merging methods by applying LiNeS on the merged multi-task vector TMIL- For the linear scaling schedule, we tune only |9 and set using heuristic that adjusts based on both the number of merged models and the merging method. Specifically, for task arithmetic which N
aggregates the individual task vectors through simple summation operation: Tsum Σ Ei=1 тo we set = 1/Nmodeet. For other merging strategies which result in TMT.. with different magniudees of norms, e.g., aggregation with summation leads to norm x Nmodees larger compared to averaging, we further multiply by scaling term [Tumm] /I [MTT_]] || to normalize their norm to simple summation. Overall, we set the intercept to:
Nusstes
1 Nmodels
[smm]]
where Tsum
Σ
T:
(2)
[T
i=1
7

---

Published as a conference paper at ICLR 2025
Table 3: Results for multi-task model merging methods in three NLP benchmarks with T5-large model. LiNeS improves baseline performance across merging methods and benchmarks.
TS-large el al., 2021)
Method
T.1 Ne
7
NLP tasks (Yadav al., 2023) 8 QA Lasks (Zhou al., 2022)
11 NLP tasks (Wang al., 2024)
Zero-shot
44.9
33.1
36.9
tuned
Arithmetic
x
71.9
63.8
63.6
76.4 (+-4.5)
67.6 (+3.8)
66.2 (+2.6)
Ties Merging
72.0 (+0.44
66.0 (+-3.)
66.4 (+2.44
73.5
68.6
67.5
Consensus Merging
75.4 (+1.9)
69.3 (+-07)
67,5 (++000
Therefore, we only tune 3 and search over the same range as the constant scaling used by the aforementioned merging techniques. As result, LiNeS shares the same computational requirements as the baseline merging methods; we provide more details for the hyper-parameetrs in Appendix B.2.1, as well as sensitivity analysis to the hyper-parameters in Appendix C.6. Specifically, we consider various multi-task model merging baselines, namely Task Arithmetic (Ilharco et al., 2023), Ties-merging (Yadav et al., 2023), Consensus Merging (Wang et al., 2024), enhancing them with LiNeS and evaluate on both computer vision and NLP benchmarks.
5.2.1 COMPUTER VISION
We experiment with the 8-task image classification benchmark proposed by Ilharco al. (2023), as well as the more challenging 14-task and 20-task benchmarks from Wang et al. (2024). Detailed descriptions of task composition appear in Appendix B.2.2. We also examine the efficacy of LiNeS across the model scale axis, studying three vision transformer (Dosovitskiy et al., 2021), namely VIT-B/32, ViT-B/16 and ViT-L/14, CLIP visual encoders (Radford al., 2021)
Table 2 presents the results for ViT-B/32 and ViT-L/14, while Appendix C.4 contains the ViT-B/16 experiments. We observe that LiNeS provides significant improvement to all baseline merg- ing methods across all tested scenarios, regardless of model sizes and total number of tasks. For example, for the 8-task benchmark with VIT-B/32, LiNeS improves task arithmetic by 4.5%, Ties- merging by 3.6% and consensus merging by 3.1%. For the challenging 20-task benchmark with ViT-L/14, LiNeS leads to consistent and significant improvements, improving task arithmetic by 3.1%, Ties-merging by 4.0% and consensus merging by 2.3%. The detailed performance on indi- vidual tasks for each tested scenario is presented in Appendix C.I1.
5.2.2 NATURAL LANGUAGE PROCESSING
We also evaluate the effectiveness of LiNes in NLP domain, including 7-task NLP benchmark (Yadav et al., 2023), an 8-task Question-Answeriin combined 11-task benchmark (Wang et al., 2024). Appendix B details the experimental settings. Following Tam et al. (2024), we adopt a variant of T5-large model (Raffel et al., 2020), namely 15-large-LM-Adapp (Lester al., 2021), and use their provided checkpoints. While TS-large tains both encoder and decoder networks, we apply LiNeS only to the decoder, as our findings in Appendix C.5 indicate that applying the edition the decoder leads similar observations to vision.
benchmark (Zhou al., 2022), and their
The performance of applying LiNeS to bascline methods with T5-lerge across various NLP tasks summarized in Table 3. LiNeS consistently improves multi-task performance across baseline merging methods and benchmarks with a notable margin. For example, on the 7 NLP tasks benchmark, LiNeS improves task arithmetic by 4.5 points, and consensus merging by 1.9 points. Meanwhile, LiNeS outperforms Ties-merging by 3.0% and 2.4% for the 8-QA benchmark and 11-NLP benchmark, respectively.
5.3 IMPROVING MODEL SOUPS FOR MERGING SINGLE- -ASK MODELS
Averaging in weight space multiple models fine-tuned on the same task derived from the same pre-trained model has been shown to increase target performance (Wortsman et al., 2022a; Ramé
8

---

Published as a conference paper at ICLR 2025
Table 4: LiNeS improves performance over Model Soups (Wortsman et al., 2022a), for both uniform and greedy soup in merging multiple checkpoints fine-tuned on ImageNet with different hyper-parameter configurations.
Resourtee bed Scaps Revendel Soupe
-0.8
SPT
-091
-061-
Method
Enhancements
ImageNet Acc.
-0.0
Averaged accuracy Best individual model
77.98
80.36
79.99
0.2
0.0
0.
Uniform soup
Task Arithmetic LiNeS
80.17
Reward
80.47 (+0.488
Figure 4: Applying LiNeS Rewarded Soups (Ramé et al., 2023) improves merging of LLM policies RL fine-tuned different rewards with a dominating Pareto Front.
81.01
Greedy soup
Task Arithmetic LiNeS
81.01
81.16 (+0.11))
al., 2022). In this section, we investigate whether L, NeS can enhance the test performance when merging single-task models.
We follow the setting in Model Soups (Wortsman et al., 2022a) and merge 70 CLIP ViT-B/32 checkpoints fine-tuned on ImageNet (Deng et al., 2009) using different hyper-parameters, plus the pre-trained checkpoint. We consider both variants introduced in Wortsman al. (2022a), namely uniform and greedy soup. We refer to Appendix B.3 for details regarding these methods and exper- imental settings. For both cases, the weight-averaging process can be decomposed as follows:
Narato
soup 0o + Tsoup: where Tsoup
(0:-0o)
Modells
We apply LiNeS to Tsoup fixing 1 and searching over B. As baseline, we also consider task arithmetic, where we search for constant scaling factor on Twuup- Note that both settings introduce one hyper-parametter deseription of the modifications. Table 4 summarizes the results and shows that LiNeS improves over vanilla soups and task arithmetic for both uniform and greedy soup by 0.48% and 0.15% on ImageNet, respectively. We report the best-performing model and the average performance as baselines. Finally, our proposed method compounds the gains from the greedy soup and leads to the best pesstrerforing model.
vanilla model soups, and refer Appendix B.3 for detailed
5.4 IMPROVING REWARDED SOUPS
In this section, we explore the effectiveness Li NeS for merging foundation models fine-tuned on different rewards. We consider the Rewarded Soups setting (Ramé et al., 2023), which interpolates the weights 0, and 02 two LLM policies, each optimized for a distinct reward R1 and Rg, respectively.
Starting with an LLM parameterized by weights Og, we first fine-tune using supervised fine-tuning (SFT) labeled demonstrations. From the resulting weights Oser, then apply Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017; Ouyang et al., 2022), training two independent policies via Proximal Policy Optimization (PPO) (Schulman et al., 2017) maximize the rewards Ri and Rt respectively. To merge these policies, we linearly interpolate the residuals
Ti 01 OSET and T2 =0,- ÓSFT, defining continuous set of rewarded policies:
Ons = osrT + Xm + (1 A)T2. A E [0, 1},
(4)
where the coefficient À models the user's preferences. We apply LiNeS to the weighted-sum residual: An + +(1 (1- A)T2, fixing a = B = 1 for computational reasons.
LLaMA-2 7B (Touvron al., 2023) and the Reddit Summary task (Stien- non et al., 2020), which consists of 14.9k post-summary pairs. We fine-tune the model using LoRA (Hu al., 2022) with FiaRA 64, Osann 128, and 0.05 dropout, We employ reward models:
In our experiment,

---

Published as a conference paper at ICLR 2025
which scores summaries based on human preferences and BART- faithful-sune somtmfryl-mmmarr---aae -ummary-detector (Chen et al., 2021) which evaluates the faithfulness the generated summary to the source post. To evaluate the models, we use subset of Ik samples from the test set, generate the responses, and compute the average score for each reward dimension.
In Table Soups+LiNeS. Lí NeS consistently outperforms the vanilla Rewarded Soups across the full prefer- ence space, Pareto dominating the baseline. This result highlights the generality of LiNeS.
present the empirical Pareto Fronts for both Rewarded Soups and Rewarded
DISCUSSION
We compare LiNeS with prior work that optimizes the scaling coefficients via backpropagation. Specifi- cally, Ada-merging (Yang et al., 2024) minimizes the entropy loss of the predictions on the test set, while aTLAS (Zhang et al., 2024) minimizes a cross en- tropy loss on validation samples. Both methods op- erate on a more fine- aeeeggaaidd level and introduce co-
0.4-
Sealing
0.3
0.2
Ada-merging
efficients per layer and per task, requiring all T + 1 checkpoints, for task vectors and pre-trained model respectively, to be stored in memory during their fine- tuning process.
&TLAS
LiNeS (ours)
0
10
Layer depth
We consider the 8-task computer vision benchmark with the ViT-B/32 visual encoder, and present the per-layer scalings in Figure 5. For aTLAS and Ada- merging, we report the average optimized scaling co- efficients for attention and linear layers in each block across tasks. Without requiring training, LiNeS leverages the inductive bias of neural networks achieve scaling very close Ada-merging or aTLAS, but with much less computational cost. Apart from the excessive memory overhead, both aTLAS and Ada-merging require multiple training epochs, making challenging to scale for large models. As we demonstrate Section 5.4, L.i NeN efficiently scales to large models like L.LMAA (Touvron et al., 2023).
Figure 5: Comparison of the scalings
obtained by different methods on 8-task merging benchmark with CLIP ViT-B/32.
7 CONCLUSION
In this work, we presented LiNeS, a novel method designed to mitigate catastrophic forgetting after fine-tuning process. By reducing the magnitude for parameter updates in the shallower layers, LiNeS improves the generalization performance of the edited model on control tasks while almost fully preserving performance on the fine-tuned tasks. Furthermore, we demonstrated the versatility of LiNeS in addressing task interference in multi-task model merging, where it consistently improves the baseline model merging methods across vision and NLP benchmarks. Our experiments confirm the broad applicability of Li NeS across various scenarios, from improving OOD generalization to enhancing multi-task and single-task model merging strategies, as well as improving merging LLM policies aligned with different rewards. Given its simplicity and ease of integration with existing methods, LiNeS offers practical and inexpensive solution for boosting the generalization and robustness fine-tuned models diverse application domains.
ACKNOWLEDGMENTS
The authors thank Adam Hazimeh and the anonymous reviewers for their constructive discussions and comments.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Linara Adilova, Maksym Andriushchenko, Michael Kamp, Asja Fischer, and Martin Jaggi. Layer- wise linear mode connectivity. In International Conference on Learning Representations (ICLR),
2024. URL https: wttp:////epppeeeeeee/mooee net/ forcmmid--Lm2hhh11 otgern/ewww....ttto/ooooeetteedd
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. Better Fine-Tuning by Reducing Representational Collapse. International Conference on Learning Representations (ICLR), 2021. URL https: wpwpreveeeeeeeeeeeee/eeeee net forum? 1d=0008sN700114
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image trans- formers. In International Conference on Learning Representations (ICLR), 2022. URL https: //tppnrevieww../////////////e//e net forumiid Bh2S25904.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. ObjectNet: large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems (NeurIPS), 2019. URL https: https::/prroceedings neurips.......... wttpppppe//c//////eeepee 97af07al 4ca.bacccca6668 aal.aaaccaaae.1ee1111111122233 Papeeee---appppppe .df..
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical birs-variance trade-off. Proceedings of the National Academy 2019.
of Sciences, 16663))
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the Op- portunities and Risks of Foundation Models, 2021. URL http:htp/////ae/ wwwpoogg//....ooooo..e/..oo....e 07258v3.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. 1ood-100-mining discriminative compo- nents with random forests. In IEEE European Conference on Computer Vision (ECCV), 2014. atttps://iine sttrss///inkk..ringgerr com/ chwpter/lC 1007/978 3-319-10599 10599-19-00011 29..
Rich Caruana. Multitask Learning. Machine Learning, 28(1):41-75, 1997.
Sihao Chen, Fan Zhang, Kazoo Sone, and Dan Roth. Improving Faithfulness in Abstractive Sum- marization with Contrast Candidate Generation and Selection. In North American Chapter of the Association for Computational Linguistics (NAACL), 2021. URL http: pttttp///axiv.... /ttp:/aax.vvrr///e.e......... a 2104. 09061v1.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Bench- mark and state of the art. Proceedings of the IEEE, 2017. URL http://arxdv..rgg/ 1703. 00121v1.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems (NeurIPS), 2017. URL
org/ /bbs/105 037 41v4.
Mircea Cimpoi, Subhransu Maji. Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014. URL http:://aateeeeee org/ abs
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv, 2018. URL http::t/axxi..oo//// abs/1811 111//111110 01 (1718v1.
Kevin Clark, Minh- Fanh-Thang Luong, Quoe V. Le, and Christopher D. Manning. Electra: Pre- training text encoders discriminators rather than generators. In International Conference Learning Representations (ICLR), 2020. URL https:/ http///pppeeeviieeeeee/ee// net forum?ide
/
T1 x1MH1 BtvB.
11

---

Published as a conference paper at ICLR 2025
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2011.
mlr. www.:/w...rrss.semmmmmmmmmmsee v15 coateslla.hmm.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. EMNIST: Extending MNIST to handwritten letters. In International Joint Conference on Neural Networks (IJCNN), 2017.
Mohammad- Davari and Eugene Belilovsky. Model breadcrumbs: Scaling multi-task model merging with sparse masks. In European Conference on Computer Vision (ECCV), 2023. URL https:/ /api. semanteatiiioe wesmmnnccschhlarr org/ Awrg/CorpsIDD:::. 2261772:266670000
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009. https:/ /ttps:///eeeeeee ore. leee. org/ abstract centt//ocuum /ww./mntt////////////oo/ooo.oc 5206845 488.
Nikolaos Dimitriadis, Pascal Frossard, and François Fleuret. Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task models. In International Conference on Machine Learning (ICML), 2023.
Nikolaos Dimitriadis, Pascal Frossard, and François Fleuret. Pareto low-rank adapters: Efficient multi-task learning with preferences. In International Conference on Learning Representations (ICLR), 2025. URL https: /ttp:s/////pe////ie/ee httpe/iwwweeeeeeeeeeeeeeee.eeeee
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Shuyang Gu, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Clip itself strong fine-tuner: Achieving 85.7% and 88.0% top-1 accuracy with vit-b and vit-l on imagenet. arXiv, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recogni- tion at Scale. In International Conference on Learning Representations (ICLR), 2021. URL http:// /tap:i///axii...//// org/ abs abs/ 2010 11929v2.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred A. Hamprecht. Essentially No Barriers in Neural Network Energy Landscape. In International Conference Machine Learning URL http: / /ttp///aiiiv.v..///i///ee/ea.aaa org/ 1803 00885v5.
(ICML). 2018.
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy. and Surya Ganguli. Deep learning versus kemnel learning: empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. In Advances in Neural Information Processing Systems (NeurIPS), 2020. URL patt:://arxie org/ abs abs/2010 15110v1.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear Mode Connectivity and the Lottery Ticket Hypothesis. In International Conference on Machine Learn- ing (ICML), 2020. URL httpt:tt////ttti www.oorgg/a..... abs/ 912. 05671v4.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P. Vetrov, and Andrew Gordon Wil- son. Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs. In Advances in Neural Information Processing Systems (NeurIPS), 2018. URL http: /awpiv.//////e///a/eeeeee www.vivooorooooooooooooooo hbss/11 8802. 10026v4.
Ian J Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, al. Challenges in representation learning: A report on three machine learning contests. In International Conference on Neural Information Processing (ICONIP), 2013. URL http: /ttp:///axxivi org/ attp//bs//1000 0414v1.
Henry Gouk, Timothy Hospedales, and Massimiliano Pontil. Distance-based regularisation of deep networks for fine-tuning. In International Conference on Learning Representations (ICLR), 2021. URL https: /twp//ppevvvieeeeeeeee ew n forum? Fd
12

---

Published as a conference paper at ICLR 2025
Sachin Goyal, Ananya Kumar, Sankalp Garg. Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), 2023. URL http:// http://aaxivv.// org/ abs//212 00638v1.
Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 12(10))993-100)
1990.
Adam Hazimeh, Alessandro Favero, and Pascal Frossard. Task addition and weight disentanglement in closed-vocabulary models. In Workshop on Efficient Systems for Foundation Models II @ ICML2024, 2024.
Yifei He, Yuzheng Hu, Yong Lin, Tong Zhang, and Han Zhao. Localize-and-stich:: Efficient model merging via sparse task arithmetic. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL httphhtp///// wwwpopppwwwweweeeeererreeeeeeeee
wee.ddiccWUBi1 6cl.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. Journal of Selected Topics in Applied Earth Observations and Remole Sensing, 2019. URL http: httt::://axiii / 1709. 00029v2.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Lixuan Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-o--distribution generalization. 2021 IEEE. International Conference on Computer Vision (ICCV), 2020.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 20211 URL http:/ /ttp:////xxvvv wtwgobs/////ooooooooo/.../e..eet 1t////99700000
Jeremy Howard and Sebastian Ruder. Universal Language Model Fine-tuning for Text Classifica- tion, In Association for Computational Linguistics (ACL), 2018. URL httt::t/////iii/ org/ abs/ /1801. 0614 6v5.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In Interna- tional Conference on Learning Representations (ICLR), httpss//cppereeveee
forum? diammm2/ddnZeeeeeeee99
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. arXiv, 2019. URL http:://// //attp://xxxvv.
axg/abs/1903 00277v2.
Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre. Shuran Song, Hannaneh Hajishirzi, Si- mon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpo- lating weights. In Advances in Neural Information Processing Systems (NeurIPS), 2022. URL http:/ /ttp///axvvv///////////ee//.aa.a org/ abs / hasg//////////
2208 05592v2.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing Models with Task Arithmetic. In International Conference on Learning Representations (ICLR), 2023. URL http: ataai.iiiiive/ii www.:vivvrrroooooooooooooo abs 2212. 04089v3.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wil- son. Averaging Weights Leads to Wider Optima and Better Generalization. In Conference Uncertainty in Artificial Intelligence (UAI), 2018. URL http::/axxiv...aii////// wttp:/wwww..g/////1//.3.... 05407v3.
Dong-Hwan Jang, Sangdoo Yun, and Dongyoon Han. Model Stock: All need is just few fine-tuned models. In European Conference on Computer Vision (ECCV), 2024.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- language representation learning with noisy text supervision. In International Conference on Machine Learning (ICML), 2021. URL http:htp//// www.v/ www.o/rr//////........o.. 00122.0511880 Bv2.
13

---

Published as a conference paper at ICLR 2025
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietres and Pengxiang Cheng. Dataless Knowledge Fusion by Merging Weights of Language Models. In International Conference on Learning Representations (ICLR), 2023. URL hitps: http:s:://pppeaeee
net/ ottp://ooromddaCCCCcattete
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. QASC: A Dataset for Question Answering via Sentence Composition, 2020. URL http: org/ abs/ 1910. 11473v2.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In International Conference on Computer Vision (ICCV) workshops, 2013.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images, 2009. https: / /www.www. , cs toronto. wew.uddriii wot.ott..eeoooorrrtt........ wtwpe/tteedddeeeeeeeeeeeeeeeeeee ffrr:///e/rieeeamnnaaaaaaarreeee
200
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine- Tuning can Distort Pretrained Features and Underperform Outtof--Dstribution. In International Conference on Learning Representations (ICLR), 2022. URL https: http///pppeeeeeeeeeeeeeeeeee net/
forum? d=
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles. In Advances in Neural Information Processing Systems (NeurIPS), 2017. URL http: //appi////xvvv..///a org/ abs / 612. 01474v3.
Yann LeCun. The MNIST database of handwritten digits, 1998. http: /stp://yaan... lecun. com/ exdb/mnte modb/mmist/.
Yoonho Lee, Annie Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. arXiv, 2022. URL http: // //aaxiiv.ca org/ abs /2210. .1.00..1145 11466v3.
Brian Lester, Rami Al-Rfou, and Noah Constant. The Power of Scale for Parfmeter-Efficient Prompt Tuning. In Empirical Methods in Natural Language Processing (EMNLP), 2021. URL http: / //arxiv.. org abs 24 08691v2.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thir- teenth international conference on the principles of knowledge representation and reasoning, 2012.
Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. Reasoning over paragraph effects in situations. In Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen (eds.), Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pp. 58-62, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/ v1/D19-5808. URL https::t////.. /acl anthology www.yyorggDDe.. 9-08./
Zhuang Liu, Hanzi Mao, Chao-Yuse Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11976-11986, 2022.
Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longho Huang, Jian Li, and Hang Zhao. Lem-lora: A universal stable-diffusion acceleration module. arXiv, 2023.
Michael Matena and Colin Raffel. Merging models with fesher- averaging. In Advances in Neural Information Processing Systems (NeurIPS), 2022. URL htRphttp//aa/i..oo abs/ 2111. 09832
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation. 1989. https://www
sciencedirect. cttcccccccceeeccecccccooeeccccce wcweecceeeeaeeeeeeeeeeeee wwt.ceeerrttiiieeee http:///a..aaaaaaa////////////// /ttp://11//00177711188833333113
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Read- ing digits in natural images with unsupervised feature learning. In Advances in Neural In- formation Processing Systems (NeurIPS), 2011. https: http:///tsttaag..oolapaa.paaa www.gelpis...ooooooooo com/ data /3tpa////naaaaaa////6//.... pdf.
pub-tools-pp wtttttoosspplilppuuululpeepoeeee
14

---

Published as a conference paper at ICLR 2025
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? Advances in Neural Information Processing Systems (NeurIPS), 2020. URL http:/ /ttt:://axiv...t/ orgoorgggao.....
httgg//b////e abs 2200//2008.00 11687v2..
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, 2008.
Guillermo OrtizJimenez, Alessandro Favero, and Pascal Frossard. Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models. Advances in Neural Information Processing Systems (NeurIPS), 2023. URL http:://ae http:://aaxxv..rrrr abs /bwp/305.... 12827....873....
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2022. URL https://prroceedingg.. neuri cc/ httpa//g.....ccc..o.poppoaaaaa/p file/ble
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2012.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Lan- com/blog/
guage Models are Unsupervised Multitask Learners, 2019.
modelsss.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Inter- national Conference on Machine Learning (ICML), 2021. URL http::/arxiv....tt org/ abs/ 2103. 00020v1.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learinng with Unified Text- to te-Text Transformer. Journal of Machine Learning Research (JMLR), 2020. URL http://
arxiv. arxiv.org/abbs abrrg//bs/// 1910 10683v4.
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. In Advances in Neural Information Processing Systems (NeurIPS), 2019. URL http: //axivv arxiv. org/ ottp//aa/1922..02055522
Alexandre Ramé, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord. Diverse weight averaging for dfi-s--distoibttion generalization. In Advances in Neural Information Processing Systems (NeurIPS), 2022. URL 2205. 097 39v2.
Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez-Paz. Model ratatouille: Recycling diverse models for distr--iistiiuttinn generalization. In Interna- fional Conference on Machine Learning (ICMI., 2023.
Alexandre Ramé, Guillaume Couairon, Corentin Dancette, Jean- Caan-Baptsste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by inter- polating weights fine-tuned diverse rewards. In Advances in Neural Information Processing Systems (NeurIPS), 2024a. URL http: aaxxiii.... .org/ htgg/ab//// abs/2306. 6.06. 04488v2.
Alexandre Ramé, Nino Vieillard, Leonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. WARM: On the benefits of weight averaged reward models. In In- ternational Conference on Machine Learning (ICML), 2024b. URL ht /pppnrevie.. net/forus forum? i d=s7 7RDNNJJYG.
15

---

Published as a conference paper at ICLR 2025
Anastasia Razdaibiedina, Ashish Khetan, Zohar Karnin, Daniel Khashabi. and Vivek Madan. Rep- resentation projection invariance mitigates representation collapse. In Empirical Methods in Nat- ural Language Processing (EMNLP). Association for Computational Linguistics, 2023. URL
https:/ a/c/ anchol org/ 2023. findi modinng-emmt 1 975/.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning (ICML), 2019. URL
http:://axiia org/ abs 1902. 10811v2.
Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky, Getting closer to Al com- plete question answering: set of prerequisite real tasks. In AAAI Conference on Artificial Intelligence (AAAI), 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An Ad- versarial Winograd Schema Challenge at Scale. Commun. ACM, 64(9):99-106, 2021. URL http://
hatp//bs/1// 1907 10041.142
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Common- sense reasoning about social interactions. arXiv, 2019. URL 1904. 09728v3.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv, 2017. URL http: /ttw/aaxviv/v.////////ee////aavv org/ abs/ 1707. 000700634722
Rishi Sharma, James Allen, Omid Bakhshandeh, and Nasrin Mostafazadeh. Tackling the Story Ending Biases in The Story Cloze Test. In Association for Computational Linguistics (ACL), 2018. URL https: / hat/acclantthhc http//aaaacooooo.oogogooo
Zhiqiang Shen, Zechun Liu, Jie Qin, Marios Savvides, and Kwang- Ting Cheng. Partial is better than all: Revisiting fine-tuning strategy for few-shot learning. In AAAI Conference on Artificial Intelligence (AAAI), number 11, 2021. URL http: / /arviv.. org/ abs / 2102 03 039834 vvv...
Sidak Pal Singh and Martin Jaggi. Model Fusion via Optimal Transport, In Advances in Neural Information Processing Systems (NeurIPS), 2020. https: http:/tttttss///e/eeeeeeeeeeoeee http://poooeennn....ee// wwwpiiinnnnnnnuurreeeeeee.ee.e.e .nurips...... cc// paper/208s hash/
Abstract.........aee....aaaac. httll.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Empirical Methods in Natural Language Processing (EMNLP), 2013. https:
//aclanthology org/D13-1170/ //
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The German traffic sign recognition benchmark: multi-class classification competition. In International Joint Confer- ence on Neural Networks (IJCNN), 2011. htap::://eeeexple.ec wwwloooeeeeeeeeeeeeeeeeeeeeeeeee ieee. org/ / document/
6033395.
Nisan Stienmon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions. In Empirical Methods in Natural Language Processing (EMNLP), 2019. URL htwps: peatttss:///ccaaatttet /aaaaaaaaaaollooooo/ooaa www.lllyyyyooooooooooooooo wwtp//rg//1i11 16088.
Derek Tam, Mohit Bansal, and Colin Raffel. Merging by Matching Models in Task Parameter Sub- spaces. Transactions on Machine Learning Research, 2024. URL https: openreviee. net/forunt 6ggNGG
16

---

Published as a conference paper at ICLR 2025
Gemini Team, Rohan Anil, Sebastian Borgeaud. Yonghui Wu, Jean- lean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev. Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models, 2023. URL http: / arwiv. org/ 2307 C9288v2.
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equiv- ariant CNNs for digital pathology. In International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), 2018. URL http: wttpvvaavvvv//aa///i//.../...ava org/ abs/ 03962v1.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. In Advances in Neural Information Processing Systems (NeurIPS), 2019. URL.
org/ abs 905 13549v2.
Ke Wang, Nikolaos Dimitriadis, Guillermo Localizing Task Information for Improved Model Merging and Compression. In International Conference on Machine Learning (ICML), 2024.
François Fleuret, and Pascal Frossard.
Mitchell Wortsman, Maxwell C Horton, Carlos Guestrin, Ali Farhadi, and Mohammad Rastegari. Learning Neural Network Subspaces. In International Conference on Machine Learning (ICML), 2021. URL http: //t///aa.t //axxivv.. /tggaas//////eaaaoo/e/./..s..a.r 21 02. 10472v3.
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. averaging weights of multiple fine-tuned models improves accuracy without in- creasing inference time. In International Conference on Machine Learning (ICMI.), 2022a. URL http:/ / /ttp/aaavvvv././//////.e//a.aava org.orggg//..oo..oooooo.o abs// 2203 05488233.
Model soups:
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022b. URL http: arxiiv. org/ abs 01903v3.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a Novel Image Dataset for Bench- marking Machine Learning Algorithms, 2017. URL http: arxiv. org/ abs/1708. 07747v2.
Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring a large collection of scene categories. International Journal of Computer Vision (IJCV), 2016. https:/..... /link. springer. wttp:ee....cartttteeeeeeoe ott.a//r..aaaa/.....111111111110
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES-Merging: Resolving Interference When Merging Models. In Advances in Neural Information Processing
Systems (NeurIPS), 2023. URL http: /aw.iv.ivvvv.////i///a.///aavavv org/ abs/ 01708v2.
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. AdaMerging: Adaptive Model Merging for Multi-Task Learning. In International Conference Learning Representations (ICLR), 2024. URL https:/ //wp///veeeeeeeeee/ee/ee/e. net forum?ide nzp ENg
17

---

Published as a conference paper at ICLR 2025
Yi Yang, Wen-tau Yih, and Christopher Meek. WikiQA: A Challenge Dataset for Open- Domain Question Answering. Empirical Methods in Natural Language Processing (EMNLP), 2015. URL httpt::/taacllae http:p/s///clanttoooog.....e.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems (NeurIPS), 2014. URL
fttp://arxiie org/ abs/ 20bs/1411 1411. 1792v1.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as free lunch. In International Conference Machine Learning (ICML), 2024. URL http: / //txxva..vvv//////////e///aaaa org/ abs /2b11 03099v3
Frederic Z. Zhang, Paul Albert, Cristian Rodriguez-Opazo. Anton van den Hengel, and Ehsan Abbasnejad. Knowledge Composition using Task Vectors with Learned Anisotropic Scal- ing. In Advances in Neural Information Processing Systems (NeurIPS), 2024. URL https: / //openzeview.. n -o
Haojie Zhang, Ge Li, Jia Li, Zhongjin Zhang, Yuqi Zhu, and Zhi Jin. Fine-tuning pre-trained language models effectively by optimizing subnetworks adaptively. In Advances in Neural In- formation Processing Systems (NeurIPS), 2022. URL http: wwaexiii.orrgaaoo abs/2211. 01 01642v1.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. Revisiting Few- sample BERT Fine-tuning. In International Conference on Learning Representations (ICLR), 2021. URL
net http://rwe.e...eeeeeeeeeeoeeaaaa formmm..--c1111443yy...
Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase Adversaries from Word Scram- bling. In North American Chapter of the Association for Computational Linguistics (NAACL), 2019. URL https: http::s:///claathologgooo org/ 9-1131
Yifan Zhong, Chengdong Ma, Xiaoyuan Zhang. Ziran Yang, Haojun Chen, Qingfu Zhang, Siyuan Qi, and Yaodong Yang. Panacea: Pareto alignment via preference adaptation for LLMs. In Advances in Neural Information Processing Systems (NeurIPS), 2024. URL https://
openreview.nee net/
Jing Zhou, Zongyu Lin, Yanan Zheng, Jian Li, and Zhilin Yang. Not All Tasks Are Born Equal: Un- derstanding Zero-Shot Generalization. In International Conference on Learning Representations (ICLR), 2022.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongho Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey transfer learning. Proceedings of the IEEE, 2020, URL http:// /ttp://arxiii org/ abs/1911 02685v3.
18

---

Published as a conference paper at ICLR 2025
Appendix
Table of Contents
A LiNeS Pseudocode
20
B Experimental Details
21
B.1 Descriptions of baseline model merging methods
21
B.2 Experimental details for multi-task model merging
21
Experimental details single-task model merging
22
C Additional Results
23
C.I. Different trade-offs for retention of task and control task performance
23
C.2 Detailed labels for Figure 2
23
C,3 Ablations of different choices of scaling function
23
C.4 Results for VTT6 for multi-task merging
24
Results for editing T5 with LiNeS
25
C.6 Sensitivity analysis for hyper-parameters
25
C.7 Experiments with CNN architectures
26
C.8 Experiments with Regularized Fine- Fine-nuning
26
C.9 Experiments with LoRA Fine-tuning
27
C.10 Additional results for improving WiSE-FT with 1,1 NeS
27
C.11 Detailed performance on individual tasks for multi-task model merging
28
19

---

Published as a conference paper at ICLR 2025
A LINES PSEUDOCODE
We provide here a python pseudocode for the scaling the task vectors.
def inn__ac ling tttak_vvctotrrr alpho=0.0, beta=t.0, 0, mom bum_hlooks=12): Progressively scales the task vector based layer depth.
Porametera:
vector :
A atetionarry tonaryves control parformantween the fine-tuned checkpoiat and the pre tzained checkpoint. alpha fleet
The minimum scaling factor for the blocks.
beta float
The maximus scaling costficient difference between the last and first block.
jum_biocka int
The totel number of layez blocks in the model.
Returns:
soafed_taak vector A copy of "Laek_vector" where each key is scaled based on the layez depth.
**.
import copy
copy task scaled_ copy. deeppooy (task_vector)
avoid modilying original
Generate the blocks corresponding the leyers of the model key_biocks [E" iaye for in range (nun_blocks)]
Create a scaling dietionary to store the scaling factor for each key scaling_die ()
for k in task_vecter ():
key assign scaling factor based for layer, block (nunerate
layer
i layer depth
if bleak in kt
scaling_dic(k] = alpha beta flayer (nun_blocks 1))
Scale the task vector acaled_task__eeee vector
on sealing dictionary
& Use alpha if layer is outside residual blocks
fask_veetor.vvettov
ing_dio.get (k, alpha)
for & in key_
return sea weeeedss___keeeeeeeeeeeeeeeeeese _ad_tss_veettrr
example: single-task fine-tured residua!
vector (k: theta_=[k] the:e_0[k] for k in (i caa.edd_sss___eeeeeeee sine_scaling task_vector, alpha=gamma, batas1.0 gamma, num nim_bloakss12
exampla: Seale the molt!-task vactors
k: stm [t thkta_0[k] for ft in st_aadels) keeya_0.keys(
acaled_ntv line_ dine_scaLing! alphas1 / (ff beto=bate, num_blocka=12
20

---

Published as a conference paper at ICLR 2025
B EXPERIMENTAL DETAILS
B.1 DESCRIPTIONS OF BASELINE MODEL MERGING METHODS
Task Arithmetic (Ilharco et al., 2023) generates multi-task vector by summing the in- dividual task vectors for each task. This multi-task vector is then added to the pre-trained checkpoint, with a scaling factor chosen based on validation set performance.
Ties-Merging (Yadav et al., 2023) resolves parameter conflicts during model merging by first pruning parameters with lower magnitudes from the individual task vectors, followed by addressing sign mismatches, and finally merging parameters with consistent signs with averaging operation. The resulting multi-task vector then added to the pre-trained check- point using a scaling factor determined from the validation set.
Consensus Merging (Wang et al., 2024) enhances existing model merging techniques by eliminating redundant weights in the multi-task vector. It first identifies the relevant subset of parameters for each task, then filters out weights that are relevant to either none or only one task. After removing these redundant weights, the refined multi-task vector is added to the pre-trained checkpoint with a scaling factor selected from the validation set.
While consensus merging be applied various merging methods, in all our experiments, evaluate only its application to task arithmetic.
B.2 EXPERIMENTAL DETAILS FOR MULTI-TASK MODEL MERGING
B.2.1 HYPER-PARAMETER TUNING
We list here the hyper-paramett search space for each model merging method in Table B.2.1, while we suggest the authors to the original papers for detailed description of these byper-parameters We highlight that applying LiNeS does not introduce extra computational cost in hyper- search for the baseline merging methods.
With MeB
Method
Hyper parameter search space
constant scaling term for multi-task vector: (8.0020000400550000000000030
Arithmetic
scaling term Eq. for the multi task vector: [0..0220000444550000000000000000
constant scaling term for multi-task vector. |1t.0/1000........0000000.01....
Ties-Merging
scaling temn 3 in Eq. for the multi task vectoo: http://wwww.....................
constant scaling term for multi-task vector: oet.cc|||.......0000.00000000000
Consensus Merging
x
weight-pruning threshold: [1.2]
scaling term Eq. for the multi task vector: vec.oe/00...2220300000.00...0000 weight prannig threshold: [1,
B.2.2 BENCHMARKS
Image classification For the benchmarks used in image classification, we utilized the 8-task benchmark initially proposed by Ilharco et al. (2023), well as the 14-task and 20-task bench- marks expanded by Wang et al. (2024).
The 8-task benchmark comprises the following tasks: Cars (Krause et al., 2013), DTD (Cimpoi et al., 2014). 4), EuroSAT (Helber et al., 2019), GTSRB (Stallkamp et al., 201 I), MNIST (LeCun, 1998), RESISC45 (Cheng et al., 2017), SUN397 (Xiao al., 2016), and SVHN (Netzer et al., 2011).
The 14-task benchmark includes the original eight tasks plus additional ones: CIFAR100 (Krizhevsky & Hinton, 2009), STL10 (Coates et al., 2011). Flowers102 (Nilsback & Zis- serman, 2008), OxfordlIIPPet (Parkhi et al., 2012), PCAM (Veeling et al., 2018), and FER2013 (Goodfellow et al., 2013).
The 20-task benchmark builds on the 14-task benchmark with the addition of: EMNIST (Cohen et al., 2017), CIFAR10 (Krizhevsky & Hinton, 2009), Food101 (Bossard et al., 2014), FashionMNIST (Xiao et al., 2017). RenderedSST2 (Socher et al., 2013; Radford et al., 2019), and KMNIST (Clanuwat et al., 2018).
21

---

Published as a conference paper at ICLR 2025
Natural Language Processing For NLP experiments, utilized benchmarks established by Yadav et al. (2023), Tam et al. (2024), and Wang et al. (2024).
The 7 NLP Tasks benchmark, explored in Yadav al. (2023), includes the following datasets: QASC (Khot et al., 2020), QuaRTz (Tafjord et al., 2019), PAWS (Zhang et al., 2019), Story Cloze (Sharma al., 2018), WikiQA (Yang al., 2015), Winogrande (Sak- aguchi et al., 2021), and WSC (Levesque et al., 2012).
The 8 QA Tasks (Tam et al., 2024) comprises the following datasets: CosmosQA Huang et al. (2019), QASC (Khot et al., 2020), QuAIL Rogers et al. (2020), QuaRTz (Tafjord et al., 2019), PAWS (Zhang et al., 2019), ROPES Lin et al. (2019), SociallQA Sap et al. (2019), and WikiQA (Yang et al., 2015).
The 11 NLP Tasks benchmark is union of these two benchmarks, as studied in Wang et al. (2024). contains the following tasks: QASC (Khot al., 2020), QuaRTz (Tafford al., 2019), PAWS (Zhang et al., 2019), Story Cloze (Sharma et al., 2018), WikiQA (Yang et al., 2015), Winogrande (Sakaguchi et al., 2021), WSC (Levesque et al., 2012), CosmosQA Huang al. (2019), QuAIL Rogers et al. (2020), ROPES Lin et al. (2019), and SocialIQA Sap et al. (2019).
B.3 EXPERIMENTAL DETAILS FOR SINGLE-TASK MODEL MERGING
B.3.1 DESCRIPTION OF MODEL SOUPS
Model soups (Wortsman et al., 2022a) is a model merging method which averages the weights of multiple fine-tuned models with different typer-paramedd configurations, improving accuracy of the merged model without increasing inference or memory costs. The authors of model soups proposed two methods:
Uniform soup: Averages the weights of all fine-tuned checkpoints, providing a simple and efficient way improve performance.
Greedy soup: Starting with the best- rest-performing checkpoint, greedily and iteratively adds the next best-performing checkpoint the soup, keeping those that improve accuracy of current collection of model checkpoints.
B.3.2 EXPERIMENTAL DETAILS FOR MODIFICATIONS TO MODEL SOUPS
We describe in detail the modifications to model soups, namely task arithmetic and our proposed LiNeS. For reference, model soups merges the checkpoints by averaging the weights of the indi- vidual checkpoints:
vanillaa
0p + Tsoup
(5)
Enhancing Model Soups with Task Arithmetic We enhance model soups with task arithmetic, by introducing a scaling factor Au to Troup in Equation 5. We search for this
within the range of [1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2.0). Note that Am 1.0 yields the vanilla model soups.
0hh = Oy + Au= -soup
(6)
Enhancing Model Soups with LiNeS We enhance model soups with LiNeS, by applying LiNeS to Tsoup in Equation 5. For the scaling, we apply directly the scaling introduced in Equation I.NNS fixing to while searching the value for 3 within the range of
to create scaled task vector soup [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]. Note that 0.0 yields the vanilla model soups.
LiNss 0g + LiNeS soup
Toup
We further note that, both Task Arithmetic and our proposed method introduce only one hyper- parameter model soups, while the computational cost for
search is the same.
22

---

Published as a conference paper at ICLR 2025
When applying the modifications to greedy soup, only apply them directly the selected subset of checkpoints after the greedy selection process.
We search for the hyper-panne parameter within the validation set and report the performance test set with the best hyper- parameter based on validation performance.
C ADDITIONAL RESULTS
C.I DIFFERENT TRADE-OFFS FOR RETENTION OF TASK AND CONTROL TASK PERFORMANCE
In Section 3, need to balance two competing objectives: maximizing accuracy the target task while preserving performance on the control task. To account for different user preferences, we scalarize these objectives by assigning varying weights to the target task accuracy. This weighting scheme can be adjusted depending on the scenario to reflect different priorities. Let Warget represent the weight assigned to the target task accuracy, and Marget and Moontml denote the normalized accu- racies for the target and control tasks, respectively. The optimal value of 7 is selected to maximize the following weighted trade-off on the validation set:
Marget Moontrol
To account for the high variance in control task performance and to emphasize the target task, we assign it weight of signifying that its accuracy prioritized twice much as the control task's accuracy.
Table 5: Validation results on the target vs control performance benchmark, presented in Section 3, averaged over the 8 tasks. We balance two competing objectives with various scalarization weights 2.
Warrett- In the main text, we use Wurget
Averaged normalized accuracy (%)
W Warget
Target task
Control tasks
1
99.8
101.9
2
100.0
101.5
5
100.2
100.8
C.2 DETAILED LABELS FOR FIGURE 2
We provide Figure 6 the detailed labels corresponding to each scatter dot. Each scatter dot cor- responds to applying specific model (FT for fine-tuned model; PT for pre-trained model; LS for fine-tuned model edited with LiNeS) on different tasks.
C.3 ABLATIONS OF DIFFERENT CHOICES OF SCALING FUNCTION
We provide this section an ablation study on applying different scaling functions for LiNeS. In LiNeS we used directly ( a + 3 T-I to scale different layers. Here test the performance multi-task model merging in vision benchmarks with the following choices for scaling functions f()): linear scaling, quadratic scaling and square root scaling:
linear scaling: ( = + 3 1-1
1-I
square root scaling: 100 = a + 3
L-1
2
quadratic scaling: ( = a + B.
We provide in Table C.3 the performance of different choices of scaling on vision benchmarks with ViT-B/32. While using quadratic scaling sometimes outperforms using identify function, especially with a larger number tasks during merging, the improvement not substantial. Therefore, we choose the linear scaling to keep the method simple and general.
23

---

Published as a conference paper at ICLR 2025
100
Dr
FE-IT
PEDDD
Contral nomtallzed accurocy
PT.BusSAT
LI-GTTSRL
PI-ENIST
PI-SNIST
L8-HNDSC
80
Fasooettt2
PTLCIPARIN
PT: PI-CIAAROO
1.5 PCAM
70
PI-PCAM
PT-CAM
LS-FER2245
PI-FERCOLL
PT-SERCOUL
1S. Pooduos
PI-CIARD
PT CT-L33AA
155 Pult
PT F.Bddo0
1.S. Rendked
60-
FT PaskiwMETTT
PT PuclieeMNnT
1.5 FAINIST
L8. 18-KMNET
PEIRINST
PEASUNIST
LSS-TE T0
PERSINST
PERSINES
Target: task normalized securacy (96)
Figure Figure 2 with detailed label information. Each scatter dot corresponds to applying specific model (FT for fine-tuned model; PT for pre-trained model; LS for fine-tuned model edited with LiNeS) on different task.
Table 6: Ablation study for applying different scalings for Li LiNeS on vision benchmarks with ViT- B/32.
VIT-B/32
Method
Scaling function
tasks
14 tasks
tasks
linear
74.2
69.1
63.4
Task Arithmetic
square
73.9
67.5
62.4
quadratic
73.8
69.2
64.6
linear square quadratic
77.2
72.1
67.2
Ties- Merging
76.9
70.4
65.6
76.1
71.6
67.4
lincar
77.6
73.6
68.6
Consensus Merging
square root
77.1
72.5
67.3
quadratic
77.1
73.9
69.0
C.4 RESULTS FOR VIT-B/16 FOR MULTI-TASK MERGING
We provide in Table C.4 the results complementary Table 2 for using VIT-B/16 the image coder, where we observe similar performance gains and observations by using I. Nee as in Table 2.
Table 7: Complementary to Table 2, for results obtained with ViT-B/16 as image encoder.
with LiNeS
VIT-B/16
Method
8 tasks
14 tasks
tasks
Zero-shot Fine-tuned
55.5
61.4
59.8
92.6
91.6
92.3
X
74.6
70.4
65.7
Task Arithmetic
/
77.6 (+3.0)
72.7 (+2.3)
67.7 (+2.0)
X
79.1
73
68.1
Ties-Merging
V
79.9 (+-0.8)
75.2 (+2.2)
71.2 (+3.1)
X
78.9
73.9
70.2
Consensus Merging
/
79.5 (+-066)
75.8 (+1.9)
72.0 (+1.8)
24

---

Published as a conference paper at ICLR 2025
C.5 RESULTS FOR EDITING T5 WITH LINES
We repeat here similar experiments we performed on ViT-B/32 model in Section 3 with T5-large (Raffel al., 2020). T5-large contains both encoder and decoder structure, with sequential residual blocks in both structures. We investigate separately how the shallow-layer updates in the encoder and decoder infect the target and control task accuracy.
We consider the averaged target and control task accuracy after applying LiNeS to
benchmark (Zhou et al., 2022), and plot in Figure 7 the
1. only the decoder part (left),
2. only the encoder part (middle),
3. both the encoder and decoder part (right).
Downscale only decoder
Dosmooal only meoder
Dommecai both encoder and decoder
Herg feal)
60-
50
10
0.5
0.8
94
0.0
1.0
0.8
1.0
08
06 02
00
Figure 7: The impact of downscaling the shallower-layer parameter updates T5-large model within the 8-question-answering benchmark (Raffel et al., 2020). Downscaling only on the decoder (left) architecture preserves full target performance, while slightly improving the control tasks per- formance. Downscaling on the encoder leads to performance degradation on target tasks.
We observe that, only downscaling the decoder architecture fully preserves full target perfor- mance, while downscaling on the encoder, or on both encoder and decoder, leads to performance drop on the target tasks. On the other hand, downscaling on the decoder part slight improves control generalization of the fine-tuned model, which we do not observe from downscaling on the encoder, or simultaneously on the encoder and decoder. We also note that, unlike the case in vision, the fine- tuned checkpoints this NLP benchmark actually improve over the zero-shot performance of the pre-trained model on control tasks.
These results motivate us to apply Li NeN to only the decoder part of T5-large when merging mul- tiple checkpoints, which preserves full target task accuracy while slightly improving control task performance, leading to similar observation in applying LiNeS to the ViT-B/32 architecture in vi- sion,
C.6 SENSITIVITY ANALYSIS FOR HYPER- HLYPER-PARAAE PARAMETERS
We provide in this section the sensitivity analysis for the hyper-parameters of L Nes. Specifically, we consider the setting in multi-task merging in the 8-task vision classification benchmark with ViT-B/32 CLIP model.
As explained in Section 5.2, LiNeS fixes a with a heuristic value by Equation 2 and only tunes 8 for multi-task merging. The slope pyper-parameter 3 is tuned within the same range as the uniform scaling coefficient À for the baseline merging methods. We compare in Figure 8 the sensitivity of averaged multi-task validation accuracy to the respective ppprmmpar,eters, i.e., À for baseline merging methods and 3 for the I.i NeS-enhanced merging methods. The results show that, for all three merging methods, including Task arithmetic, Ties-merging and Consensus, enhancing with LiNeS is less sensitive to parameer choices compared to the corresponding baseline method.
Furthermore, we perform an ablation treating a as a hyper-parameter and analyze the sensitivity to both a and B for LiNeS in two-dimensional grid for the same benchmark. The results are pre- scal-
sented in Figure 9. The results clearly demonstrate the necessity for applying ing, the optimal performance obtained with both > 0 and 3 > 0 for all three merging method. Note that the optimal configurations found by the ablation study are very close to the configurations found in method, shown in Table 8, by setting a via the heuristic and searching only for B.
25

---

Published as a conference paper at ICLR 2025
Task Arthmetti
Teseurping
mangins
Stpeegincane 20
Figure 8: Sensitivity to hyper-parametee in multi-task merging for the 8-task benchmark with CLIP ViT-B/32 model. The y-axis represents the averaged multi-task validation accuracy and x-axis represents the hyper-parameter value, i.e., for the baseline method and / for the method enhanced with LiNeS.
Teeneegihy
Conerrane leet ping
6
1
Figure 9: Sensitivity to both and B in multi-task merging for the 8-task benchmark with CLIP ViT-B/32 model. The heatmap represents the averaged multi-task validation accuracy, while x and y axis represent the | and respectively. The optimal configuration is annotated with red box.
C.7 EXPERIMENTS WITH CNN ARCHITECTURES
In this section, we apply LiNeS to CNN architectures. Specifically, we consider the ConvNeXt (Liu et al., 2022) architecture. First, we repeat the experiments presented in Section 3 regarding mitigating catastrophic forgetting. The final results are presented in Figure 10, where we observe similar findings with CLIP ViTs, i.c., LiNeS greatly improves the performance when applicd to the fine-tuned checkpoints while preserving most of the accuracy on target tasks. Furthermore, present in Table the results on multi-task model merging, following the experi- mental protocol established in Section 5.2. Again, we see that Li NeN improves the performance of baseline merging methods.
control tasks
C.8 EXPERIMENTS WITH REGULARIZED FINE-TUNING
In this section, we evaluate LiNeS against several regularized fine-tuning methods, focusing on their ability to preserve general features and mitigate catastrophic forgetting. The regularization strategies applied during fine-tuning described below:
Fine-tuning with Linear Layer-Wit Learning Rate Decay (LinLR): Applies a linear learning rate schedule where the learning rate linearly increases from 0.0 to the maximum value for all the layers.
2. Fine-tuning with Exponential Layer- Layer-Wise Wise Learning Rate Decay (ExpLR): Applies an exponential learning rate schedule where the learning rate is set to maximum for the deepest layers and decays by factor of 0.5 by each layer for the shallower layers.
3. Fine-tuning with First Half of Blocks Frozen (HalfFT): Freezes the parameters of the first half of the model's blocks during training.
4. Fine-tuning only the Final Block (LastFT); Freezes all blocks except the final block of the feature encoder.
26

---

Published as a conference paper at ICLR 2025
Table 8: and values for a set by our proposed heuristic in Equation 2.
Method
Task Arithmetic
0.125
0.3
Ties-mergin merging
0.21
1,4
Consensus merging
0.21
0.9
ConvNext
110
105
100
85-
Control
80
Fino-Tiee Tho-TTnndd-ae Tino (ours)
Fine-tuned
50
90
Targe normslized accuracy
Figure 10: Our linear scaling (LiNeS) retains performance on both control and fine-tuned target tasks for ConvNext architecture.
We present the results in Table 10. We observe that LiNeS, as a post-training editing method, outperforms the regularized fine-tuning methods in of restoring the zero-shot performance on the control tasks. We further emphasize that compared with the regularized fine-tuning methods, LiNeS benefits from many advantages such as efficiency, flexibility and computational cost.
C.9 EXPERIMENTS WITH LORA FINE-TUNING
We explore the applicability of the method models fine-tuned with LoRA (Hu al., 2022). For a layer with pre-trained weights W, E Rmxn, LoRA adds trainable matrices E Rmmxr and B R" for rank & min(n, m). The weights of the layer become:
W W. + BA
for a E E. Following common practice, set full fine-tuning We consider only the case of ViT-B/32 fine-tuned on 8 tasks and replicate the exper- iment presented in Table 1. Specifically, for each of the LoRA-fine-tuned models, we compute the accuracy the same (target) task as well the average performance for each of the remaining control tasks. Table reports the average over the cases for ranks r E {16, 32, 64, 128). We observe that LoRA fine-tuning has lower target performance compared to full fine-tuning and that increasing target performance comes at the cost more forgetting. In all the cases, LiNeS restores control performance while minimally affecting target performance.
and fine-tune with the same protocol used for
C.10 ADDITIONAL. RESULTS FOR IMPROVING WISE-FT WITH LINES
1.10.1 RESULTS FOR USING VrT-B/16 AS VISUAL ENCODER
We provide in Figure 11 the results for applying LiNeS for improving WiSE-FT, using ViT-B/16 as visual encoder. The ViT-B/16 checkpoint obtained through fine-tuning the CLIP checkpoint on ImageNet with the same hyper-parameter configurations in Wortsman et al. (2022b). From Figure 11
27

---

Published as a conference paper at ICLR 2025
Table 9: Multi-task model merging results using ConvNeXt architecture. LiNeS improves the results compared to uniform scaling for both Task Arithmetic and mers-ng
Method
LiNeS
Acc (%)
Norm. Acc (%)
77.9
83.8
Task Arithmetic
79.0 [+1.1]
84.8 [+1.0]
Ties-merging
79.7
85.8
V
80.3 [+0.6]
86.3 [+0.5]
Performance of different methods on target and control tasks, averaged over all target and Table 10:
control task combinations in the 8-task vision benchmark (Ilharco et al., 2023).
pre-trained
fine-tuned
FT+LiNeS
FT+LinLR
FT+ExpI.R
FT+HalfFT
FT+LasIFT
Target (%) 48.3
90.5
90.3
90.7
89.6
90.4
85.6
Control (%)
48.3
38.0
48.0
46.9
46.0
46.8
46.6
we observe that L Nes improves over WiSE-FT for both ID and OOD accuracies, leading to similar observations the results obtained with ViT-B/32.
ImapoNote
Nertkiih
Object
000 Aectines
50+
55
Long
72.5
80,0
70.0
72.5
5000
700
77.5
$5.0
ImagriNct
Accuneyy
ImageNeell
ImageNet
23
Litee
0.00
77.5
60.0
75.0
77.3
80.0
TeetgrNet
Figure 11: Results for improving WiSE-FT with LiNeS on with ViT-B/16 model fine-tuned on ImageNet.
C. INDIVIDUAL RESULTS FOR 70 CHECKPOINTS
We provide in Figure 12 individual results separately for the experiments on the individual model checkpoints. Note that here y-axis represents the averaged accuracy over 5 OOD datasets. From the figure, we observe that LiNeS consistently improves WiSE-FT terms both ID and OOD accuracies for most of the individual checkpoints.
C.11 DETAILED PERFORMANCE ON INDIVIDUAL TASKS FOR MULTI-TASK MODEL MERGING
Image Classification We provide the detailed performance each individual task for multi-task model merging in image classification benchmarks, complementary to the results in Table and Table C.4 where the accuracies averaged on the individual tasks.
28

---

Published as a conference paper at ICLR 2025
Table 11: Similar Table LoRA Fine-tuning harms generalization control tasks. Increased target performance rsults in higher levels of forgetting. Still, our proposed method LiNeS restores control performance for all ranks considered while minimally affecting target performance.
Target
Control
Pre-trained
48.3
48.3
Fine-tuned
90.5
38.0
Fine-tuned +LiNeS
90.3
48.0
16
84.4
44.2
r = 16 + LiNeS
84.3
46.7
32
85.8
42.8
32 + LiNeS
85.5
46.7
r
86.6
41.6
64 + LiNeS
86.4
46.2
r 128
87.5
41.6
128 LiNeS
87.2
46.3
The single-task performance is presented Figure 13 for ViT-B/32, Figure 14 for ViT-B/16, and Figure 15 for ViT-L/14. From the results observe that method demonstrates noticeable improvement over baseline merging techniques across individual tasks in all test scenarios.
Natural Language Processing We provide in Figure the detailed single-task performance for the three NLP benchmarks using T5-large, complementary to the results in Table 3. Similar to the observation in vision, our method provides consistent improvement over baselines across individ- ual tasks.
29

---

Published as a conference paper at ICLR 2025
3o
M00
75.4
261
940
70.
Madad
Maki
00.91
150
1A0
750
21.3
71.
140
714
150
20:
110
150
Modid
Make27
Made
50.3
50
70.
Mokk 27
Moke 26
Kodd
230
05.3
770
150
71
17.
800
500
100
Mak13
Mak4
Mi.ld
DAO
050
17.5
DA0
300
77-1
75.6
71.8
900
710
71.8
11.1
111
710
Maai
Makt 13
Make
Madd
800-
700
773
25.3
120
701
300
130
120
Mack4
Mall
500
2.3
MabEE3
62
400
75.8
75.
15.0
75a
77.
150
7..0
17.0
700
300
770
560
50
Maaki
Mal4
Ma.ld CA
4.3
2301
04.94
Maki
M7
Mak4
125
1200
15.
700 Via
100
15.8
WEEFT
WHE-FT Live
Flastrzed
Plastzzed USd
Figure 12: Performance of applying LiNeS to WiSE-FT to each VIT-B/32 checkpoint fine-tuned on ImageNet.
30

---

Published as a conference paper at ICLR 2025
Fino-tuned
TA
TIES
Consensus
Zero-soot
TA- TA-LINeS LINS
TIES LINeS
Consensus-aa LiNeS
DTD
STSRD
Euns&AT
Cats
RESISCIS
UTA
SVIN
BarsAA
Placouting
SUX2I
Clas
80
20
GISRE
ANNSST SVWY
UNSST
PCAM
Bpskmd8812
Eloveestee
Oetes
RESESCIS
CIEARICO
FRIeBI
SUNBIT
POAM
STE..
EXBOST CIPARIO
(e) Tel. Aulluske
ITD
CTSRR
RInSAT
SUNar
SWIIN
Band-
Fune&A
Cars
Pleveo
D713
SU Ni/ar
60
ETPARIN
20
CTSRR
INIST SMIN
MNITT
PCAM
MNR87
RESISCAK
CIEARACO
SUNEI
PCAE
PUNGST
EIFAROO Pealio
DTD
CTSRB EINEAT
SUNar CF7SS (7TSR
SUIIX
EuroSAT
Cars
Flovestrr
SUSS0
Clcs
80
CIEARIO
Okn
20
CTSRE
MNIST SVIEN
MINIST
POAM
RESISCYS
CIPARICO
SUN387
PCAM
1200007 CIFARTO
Poottt
Figure 13: Single-task accuracies for multi-task merging on image classification benchmarks for ViT-B/32.
31

---

Published as a conference paper at ICLR 2025
Fino-tuned
TA
TIES
Consensus
Zero-soot
TA- TA-LiNeS LINS
TIES LINeS
Consensus-aa LiNeS
DTD
STSRD
Euns&AT
Caus
RESISCIS
UTA
SVIN
BarsAA
Placoutng
SUNSO
20
20
GTSRE
ANIST SVWY
UNSST PCAZ
Bpokmo88822
Floessll
FERIOLL
RESESCYS
SVHN
CIEARICO
FRIOBI
SUNG07
POAM
STE..
EXBOST CIPAROO
(c) Auilauee
ITD
CTSRR
RInSAT
SWIIN
Band-""
Fune&A
Cara
Pleveo
D710
SU NN/a
Don
ETPARIN
20
ANIST SMIN
MNSTT
PCAM
MN887
RESISCAK
SVHN
CIEARACO
SUNEI
PCAM
PUNGST
EIFAROO Peatio
DTD
CTSRB
EINEAT
SUNar CF7SS (7TSRR
SVIIX
EuroSAT
Caas
SUSS
Clcs
CIFARIO
Ckn
20
GTSRE
ANIST SVIES
MINIST
POAM
RESISCYS.
CIPARICO
SUN387
PCAM
1200007 CIFARTO
Poottt
Figure 14: Single-task accuracies for multi-task merging on image classification benchmarks for ViT-B/16.
32

---

Published as a conference paper at ICLR 2025
Fino-tuned
TA
TIES TIES LINeS
Consensus
Zero-soot
TA- TA-LiNeS LINS
Consensus-aa LiNeS
DTD
STSRD
Euns&AT
Caus
RESISCIS
UTA
SVIN
BarsAA
Placouing
0770
SUNSW
20
GTSRA
ANIST SVWY
MNSST POAM
Bpskko888122
Floezsle
FERIOL3
RESESCIS
SVHN
CIEARICO
FRIOBI
SUNG07
PCAM
STE..
EXBOST CIPARIO
(c) Auilaueee
ITD
CTSRR
RInSAT
SWIIN
Band-"
FunS&A
Cara
Pleveo
D710
SU N//a
Don
EIPARIN
20
ANIST SMINN
MNST
PCAM
MN887
RESISCAS
SVHN
CIEARLCO
SUNEI
PCAM
PUNGST
EIFAROO Peatio
DTD
CTSRB
EINEAT
SUIIX
EuroSAT
Caas
SUSS
Clck
CIFARIO
Okn
20
GTSRE
ANIST SVIES
MINIST
POAM
RESISCYS
CIPARICO
SUN387
PCAM
BUS0007 CIFARTO
Poottt
Figure 15: Single-task accuracies for multi-task merging on image classification benchmarks for ViT-L/14.
33

---

Published as a conference paper at ICLR 2025
Fine-tunod
TA TA+LINGS
TIES TIES+LINeS
Consenstss
Zero shot
Cousenstas +iNNS
QualTTy
StorCClae
QualEE
QuAII.
WikGA
StoryClep
60
Winngi snal
40
20
PAWS
Wiki
CusnooQA
PAWB
WikE
QuaRt
RDPES
Cano
Winegrande
SwinIIA
QuAIL
(a) Tiak Antimncti 7 NLP vador
(M) Teak Asilincic 6 GA takk
(6) Taok Aralaustco 11 NLP Inlk
Qualtik
SteryClar
QUAREES
QASC
QuAII.
WlegA
SkiryCleie
Winograree
40
PAWS
Wikau
CaumonoQA
PAWS
Wikg
QuaRT
ROPES
WSC
Winogrande
SwialIQA
QUAIL
66) TIES 7 NLP kolo
(b) TIES 5 QA takka
(e) TEES
XLP real
QualITy
StoryCluee
Quald'
QASC
QuAII.
SodilIQA
Wikgg
SkoryClple
Witoog angle
40
PAWS
WilQu
CammonQA
MAWS
WSS
WikQ
QuaRT
ROPPS
Cocoola
Winogrande (A) Conernara 7 NLF tulo:
QASC
Swvialia
QuAIL
(b) Corsanens QA tanks
(6) Chconroon 11 NLP exaio
Figure 16: Single-task accuracies for multi-task merging on NLP benchmarks for Ta-lerge
34


Reviews:
[{"review_id": "pxTZwsbP4I", "reviewer": "Reviewer_oncK", "summary": "This paper introduces a post-training editing technique, LiNeS, designed to address catastrophic forgetting and facilitate model merging after fine-tuning. LiNeS scales parameter updates linearly with the depth of layers within the network. The technique has shown significant improvements across various domains, including OOD generalization, single-task and multi-task model merging, demonstrating its effectiveness.", "questions": "1. The basic assumption of this work is that the degradation of performance on control tasks is largely due to these distortions in the shallow layers. However, it lacks of formal theory.\n\n2. From Fig. 2 and Fig. 7, fine-tuned models have obtained high performance for both target task and control tasks in some cases. Can catastrophic forgettinh happen when the result is high?  Please discuss or analyze these specific cases where fine-tuning seems to improve performance on both target and control tasks.\n\n3. The result of \"Model 2\" in Fig. 10 is different from the other 69 curves, what is the reason?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper is easy-to-follow.\n\n2. The topic is essential yet the idea is moderate. Both regularized fine-tuning and model merging are important techniques for the community and the paper addresses two formulations simultaneously.\n\n3. A new method that seems well-motivated and performs well on a lot of benchmarks.", "weaknesses": "1. I like the fact that the authors simply scale the task vectors. But I was not clear why directly edits the difference between the fine-tuned and pre-trained checkpoint via a linearly scaling coefficient ($\\alpha+\\beta\\frac{l-1}{L-1}$). Also, compare to existing methods, like `Ties-Merging` and `Consensus Merging`, what are the core strengths of this work? More efficient or simpler?\n\n2. The point above also points to the limitation of the current framework, which is a lack of formal theory. It works very well experimentally, but some aspects are not very clear. Providing concrete theoretical analyses or proofs for linearly layer-wise scaling would be better.\n\n3. There are numerous typos and bad grammar throughout the paper. The authors should do a very careful proofreading and fix all the errors.\n\n4. Minor comments:\n- Some of the content in the appendix can be put into the body to ensure that the body is a full 10 pages.\n- It must be noted that the consistency in verb tense is not maintained throughout the document. For example, the second paragraph in the Introduction section.\n- line 112 \"task=specific\"\n-  Two identical sentences appear from line 223 to line 226.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TXUEsF3iy5", "reviewer": "Reviewer_KCXu", "summary": "This paper introduces LiNeS, a post-training editing technique designed to enhance the generalization ability of fine-tuned models. LiNeS applies a layer-wise linear scaling function to modulate parameter updates, reducing them more in shallow layers than in deep layers. The method is evaluated across several settings to demonstrate its effectiveness.", "questions": "- What could be an explanation for why the benefit of LiNeS appear to be smaller for deeper models (e.g. ViT-L/14 in Table 2) and models with smaller patch sizes (e.g. ViT-B/16 in Table 5)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-motivated and easy to follow.\n- The experimental results seem impressive.", "weaknesses": "- The validation set has a large influence on the OOD performance of the selected models [1]. I suggest the authors discuss the construction of the validation set and whether the same empirical trend can be observed with different validation sets, as suggested by [1].\n- The evaluation is limited to the transformer architecture. Including other model architectures, such as ResNet-based CLIP models, would validate whether the benefits of layer-wise scaling extend beyond transformer architecture.\n- There is a lack of sensitivity analysis for the hyper-parameter $\\alpha$ and $\\beta$ across different evaluation settings.\n- Typos should be checked. For example,\n    - enchancing → enhancing (Line 23)\n    - bcenchmarks → benchmarks (Line 24)\n    - task=specific → task-specific (Line 112)\n    - Figure 4 → Table 4\n\n[1] In search of lost domain generalization. In ICLR, 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7iZuo0L9Ro", "reviewer": "Reviewer_McJs", "summary": "This paper presents a novel post-training editing technique called LiNeS, designed to address the problem of catastrophic forgetting in large pre-trained models during fine-tuning. Catastrophic forgetting refers to the phenomenon where a model loses its generalization ability on other tasks when fine-tuned for a specific task. LiNeS balances the trade-off between retaining the generalization capability of the pre-trained model and enhancing performance on specific tasks through hierarchical scaling of parameter updates.", "questions": "- Are there additional experimental results for LLMs, including different models, datasets, and tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The LiNeS method introduces a novel post-training editing technique that prevents catastrophic forgetting through hierarchical scaling of parameter updates.\n- Extensive experiments validate the effectiveness of LiNeS across various scenarios, including single-task fine-tuning, multi-task model merging, and improved out-of-distribution (OOD) generalization.\n- The LiNeS method is applicable not only to visual tasks but also to natural language processing tasks, demonstrating its cross-domain versatility.\n- LiNeS can be integrated with existing multi-task model merging baselines.", "weaknesses": "- The article does not discuss the performance of LiNeS in long-term maintenance and updating of models, particularly when faced with evolving data distributions.\n- There is a lack of experimental results for LLM tasks.\n- Comparisons with the performance of other fine-tuning methods (such as LoRA) are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bbwKex0kBj", "reviewer": "Reviewer_tFK1", "summary": "The paper introduces a post-training technique designed to mitigate catastrophic forgetting; more specifically preserving pre-trained generalization while enhancing fine-tuned task performance. This is done by reducing the magnitude of parameter updates for the shallow-layers compared to the deeper-layers. The method boils down to a layer-wise linear interpolation between the fine-tuned and the pretrained model.", "questions": "1) Why a shallow layer must be updated less compared to a deeper layer? Theoretical analysis should be provided to why shallow layers need to be updated less than deeper layers and how does this method affects the model's representation space?\n2) Clarify exactly which tasks were used as target and control in your results, and to explain their task selection criteria. Add more baselines that aims at protecting both generalization and the fine-tuning to a downstream task. This would clarify the performance improvements?\n3) Conduct an ablation study comparing the linear scaling approach to a learned scaling approach, and analyze the resulting patterns in the learned scaling factors. \n4) How does the proposed method differ from applying different learning rates per layer (low LR for shallow layers and higher LR for deep layers) during fine-tuning? Include a comparison between the proposed post-training method and the layer-wise learning rate scheduling during fine-tuning. This would help clarify the unique benefits of the proposed approach.\n5) Provide an experiment to compare your proposed approach to freezing different shallow layers and updating the deeper layers using fine-tuning. This includes, but not limited to, freezing the feature extractor and finetuning the classifier.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) Overall the idea of the paper is pretty simple and should be easily reproducible. Further, the paper is clearly structured and mostly well-written.\n2) The paper thoroughly evaluates the proposed method across multiple tasks. \n3) The results clearly demonstrates strong trade-off performance on both target and control tasks.", "weaknesses": "1) Theoretical analysis have not been provided to support the proposed idea. \n2) Some results lack details. For example, Table 1 what is the selected target and control tasks? Were all other combinations tried out? Not enough baselines have been added. Comparing to pretraining and fine-tuning performances is not enough.\n3) How does learning the scaling factors perform compared to linearly scaling the parameter updates? Moreover, does the learned parameters have a consistent increasing trend from shallow-to-deep layers?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)