Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging
Authors: Zhenyi Lu
Chenghao Fan
Wei Wei
Xiaoye Qu
Dangyang Chen
Yu Cheng
Venue: NeurIPS 2024 poster (2024)
Keywords: Model Merging
Fusion of Experts
Efficiency
Large Language Models
Primary area: natural_language_processing

Abstract: In the era of large language models, model merging is a promising way to combine multiple task-specific models into a single multitask model without extra training. 
However, two challenges remain: (a) interference between different models and (b) heterogeneous data during testing. Traditional model merging methods often show significant performance gaps compared to fine-tuned models due to these issues. 
Additionally, a one-size-fits-all model lacks flexibility for diverse test data, leading to performance degradation. 
We show that both shared and exclusive task-specific knowledge are crucial for merging performance, but directly merging exclusive knowledge hinders overall performance. 
In view of this, we propose Twin-Merging, a method that encompasses two principal stages: 
(1) modularizing knowledge into shared and exclusive components, with compression to reduce redundancy and enhance efficiency; 
(2) dynamically merging shared and task-specific knowledge based on the input. 
This approach narrows the performance gap between merged and fine-tuned models and improves adaptability to heterogeneous data. 
Extensive experiments on $20$ datasets for both language and vision tasks demonstrate the effectiveness of our method, showing an average improvement of $28.34\%$ in absolute normalized score for discriminative tasks and even surpassing the fine-tuned upper bound on the generative tasks.

Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging
Zhenyi Lu Chenghao Fan 1.2 Wei Weil.2t Xiaoye Qu'
Dangyang Chen Yu Cheng School of Computer Science & Technology. Huazhong University of Science and Technology.
2 Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL), Ping An Property & Casualty Insurance Company of China, Ltd.,
The Chinese University of Hong Kong-
{luzhenyi529 facicofan 1egmai1 com, {weiw, xiaoye]@hust. edu. cn, chendangyang2278pingg.. com. en, chengyuêcse www.icsssecccck...e. cuhk.edu.
Abstract
In the era of large language models, model merging is promising way to combine multiple task-specific models into single multitask model without extra training. However, two challenges remain: (a) interference between different models and (b) heterogeneous data during testing. Traditional model merging methods often show significant performance gaps compared to fine-tuned models due to these issues. Additionally. one-size-fits, all model lacks flexibilityy for diverse test data, leading to performance degradation. We show that both shared and exclusive task-specific knowledge crucial for merging performance, but directly merging exclusive knowledge hinders overall performance. In view of this, we propose Twin-Merging, method that encompasses two principal stages: (1) modularizing knowledge into shared and exclusive components, with compression to reduce redundancy and enhance efficiency: (2) dynamically merging shared and task- specific knowledge based on the input. This approach narrows the performance gap between merged and fine-tuned models and improves adaptability heterogeneous data. Extensive experiments on 20 datasets for both language and vision tasks demonstrate the effectiveness of our method, showing an average improvement of 28.34% in absolute normalized score for discriminative tasks and even surpassing the fine-tuned upper bound on the generative tasks."
Introduction
In recent years, Large Language Models (LLMs) have demonstrated notable success across various Natural Language Processing (NLP) tasks [12, 16, 43, 61-63, 65, 68), including code generation [22, 56], solving math problems [2, 44], multilingualism [47], etc. These models, with billions of parameters, excel in various downstream tasks [25, 34, 72] but require extensive training on large datasets using thousands of GPUs. The considerable computational and energy costs [53] limit their specialization and deployment in
environments [38].
To tackle this challenge, model fusion has emerged promising solution [37]. One notable paradigm is model merging [29, 33, 76, 78), where multiple task-specific models, or "experts", are combined into a single unified model. This unified model quickly adapt new tasks without the need to retrain a large model. Various techniques, such as parameter averaging [6, 74], weight
Equal contribution.
t OOu implementation is available in httppp///ggttuuu..oo////ttttttoo
Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Meraed
Esclaase Krawledgy
Mexcl
SVD corpeasie
(A 0.)
8))
Eastseve
Wude
Kapwlelge
0g
9,
Knowlodge
Reare
Posssined
Tisk Specifie Expert
Skared Knowdadgy
Taak Specifie Expert
Hiniion
di Conveational Mergins
(1)) Icesmtedge Hountangkmmet
(II)) Dymomica Merginz
Figure Subfigure (I) shows that in conventional merging methods, parameters from different task-specific models and pre-trained model are weighted-summen into single multitask model for inference. Subfigure (II) illustrates that our Twin-Merging method first isolates shared knowledge, then extracts exclusive knowledge by identifying differences between task experts and the shared model. This exclusive knowledge then compressed into sparse vectors. Subfigure (II) shows that during testing, Twin-Merging dynamically merges shared and compressed specialized knowledge based on test inputs to form the final inference model.
interpolation [33, 46], and advanced strategies like task arithmetic [29, 51, 67, 78], have been developed for model merging. These techniques have been proven effective, enabling the integration of fine-tuned knowledge from diverse tasks into multi-task model without additional training.
However, merging models from different domains often sacrifices specific task performance, leading to large performance gap compared to the individual expert [31, 76]. Two major causes prevent the existing merging methods from reaching the theoretical upper-bound performance of individual experts: (1) Interference between models. Previous research shows that parameter redundancy and sign discrepancies [76], as well as the distribution gap between tasks [31], hinder effective model merging. We demonstrate that task-specific models often contain mixed knowledge, where the expertise in one model may be exclusive or detrimental to others. This redundancy interference can obstruct the integration of expertise across models [9]. (2) heterogeneity of data al test time. Previous methods pursue a single, statie optimal solution for various tasks. While a one-size-fits- all model avoids introducing new parameters, it might be inadequate suboptimal due to the unpredictable nature of test inputs [78]. It limits the utilization of complementary knowledge and leads to deteriorated performance [71].
To address the above issues, in this paper, we introduce Twin Merging, involving two principal stages: (1) Knowledge Modularization: Unlike previous research that migrates merging interference in parameter-we wise manner or searches merging coefficients, we decompose the knowledge possessed by experts into shared knowledge and exclusive task-specific knowledge, as shown in Figure I (II). First, we compress common knowledge into shared expert, serving to capture and consolidate common knowledge across varying tasks. Then we isolate exclusive knowledge based on the difference between the task experts and the shared expert, allowing diverse knowledge to be decomposed more finely. (2) Dynamic Merging: Inspired by Mixture of Experts (MoE) [80, 84, 85], we simplify the parameter merging problem into a conditional composition problem. Instead of pre-determining the best parameter combination for heterogencous data at test time, as illustrated in Figure 1 (III), introduce a router to dynamically merge shared and exclusive knowledge based on the test inputs. The shared model serves the foundation, and task-specific knowledge is conditionally injected according to the router.
We demonstrate the effectiveness of our proposed Twin-Merging method through extensive experi- ments on 12 datasets, covering both discriminative and generative tasks, various model architectures, and in-domain and out-of-domain setups. As shown in Figure 2b, Twin-Merging consistently outper- forms other merging methods across all datasets, surpassing the strongest baseline by an average of 28.34% in normalized scores for discriminative tasks and 3.86% for generative tasks on the scaled model (Qwen-14B) 14B). We validate the scalability, extensibility, generalization, and storage efficiency of Tein-Merging (Figure 2a). Remarkably, even with 99. 99.9% reduction in parameters, our method only experiences a slight 14 14% performance degradation. Our results establish Twin-Merging as a powerful and effective method for combining multiple fine-tuned models into single multi-task model.
To summarize, our contributions are as follows: (1) We introduce Twin-Merging, a novel model fusion method that reduces the performance gap between traditional model merging and fine-tuned models while enhancing adaptability to diverse data. (2) We investigate the impact of shared and exclusive task-specific knowledge merging performance, presenting innovative techniques for

---

COLA
S512
Sunguricttion
69
Twin-Merging (Ours)
MRPP
Individual
DARE
STSB
TrandiQA
66-
Arithmetic
Average
65
00
64
Ties Ties-Merging
QNLI
RTE
63-
Pre-trained
MNLI
Weight Averaging
.... Pramiieed
Tin--Mnnjngg
Fise
Arilamsic
Wright Awininn
Mao-inggg (W: (AAR)))))
#Parameters (x10%)
Task Anthootii
Twin giriing (Ous)
(a) The average performance on generative tasks vs, the number of parameters of Twin-Merging compared to various merging baselines, with different storage sizes indicated circle size.
(b) Comparison of absolute accuracy (%) of individual tasks for the NI.P benchmarks on RoBERTa and Qwen, covering 4 discrimina- tive and generative tasks.
Figure 2: The effectiveness of Twin-Merging in terms of performance and parameter-efffcieney
knowledge disentanglement and dynamic merging. (3) Twin-Merging is simple to implement with minimal hyperparameters, improves multi-task performance without retraining expert models, and can be combined with other merging methods for further gains. Our approach scales well with model size and task numbers and is storage-efficient.
2 Related Work
this section, we focus on model merging research, for additional related work on multi-task learning and Mixture of Experts, please see Appendix B. Model merging aims to fuse multiple fine-tuned task- specific models into one comprehensive multi-task model without additional training. FisherMerging [46] and RegMean [33], use straightforward weight averaging but require extra data and computation. Some works [1, 21, 58, 60, 70] bring models into a single low-loss basin and interpolate between them based on the linear mode connectivity (LMC) theory [15, 18, 20J. The weight permutations [1] and optimal transport [58] are utilized to better interpolate neural networks. However, recent studies [83] suggest that LMC might not always hold for fine-tuned models. Task-Arithmetic [28, 51] extends averaging to arithmetic operations in the parameter space for finer control over model behaviors, but the interference between the multiple models can be an issue. To tackle this challenge, advanced merging methods like Ties-Merging [76], AdaMerging [78] and DARE [79] have been proposed. These methods aim reduce task conflicts addressing parameter redundancy disagreements in signs, finding optimal merging coefficients, and reducing weight density, respectively. Jiang et al. [32] assume that tasks known and use task-specific knowledge improve performance. However, this assumption is often unrealistic since real-world data distributions are unpredictable. In contrast, our method addresses merging interference by modularizing shared and specifcc knowledge. We handle heterogeneous test data scenarios by introducing dynamic merging techniques.
3 Methodology
3.1 Analysis of the Performance Gap in Model Merging
In this paper, following the settings of model merging [29, 76, 79], we consider the case of tasks, where training for each task starts from pre-trained model weight O, and fine-tunes on Derain to obtain task-specific model 0,- Let (æ; 0) be language model accepting inputs x E X' and paramterized by weights 0 E O. Considering the real data distributions are diverse and challenging to represent with single task, to model such distributions, previous methods typically consider the
mixture of I' task test data: D > arD,, where ΣT ar 1, a 0 Vt. The model merging

---

considers the problem where we have T fine-tuned expert models {f( 0 and pre-trained weight 0,, composing multitask model 0 to approximate the optimal solution.
appt R 0 F(On, 0,, ,r)
(1)
Here F represents an arbitrary merging function. For example, in Task Arithmetic [28]. 0" 0o + - ((0;- 00).
Table 1: Merging without parameter interfer- ence and merging between similar tasks both cause performance degradation (Notice: these two experiments use different datasets).
Avg. Seore
60
Fine-tumed
Merged
Task
Normalized Score (Equation (4))
Fine-tuned Epoch
Fine-tuned upperbound.
With poraneter interference
Avg. Soore
80
Fine-tuned
100,00
1 I-modol merging
Merging
85,43
Without parameter interference
60
8-modols merging
Non-overlap Fine-tuned
100.00
Pre-trained lowerbound
Non-overlap Merging
82.21 3.21
20
100
Similar kasks
Sparsity Rate (%)
Finc-tuned
100.00
Similar-Tasse Merging
91.58 [] 8.42]
Figure 3: The impact of different ratios of shared knowledge and exclusive knowledge.
Although existing merging methods, like Task Arithmetic, can combine multiple task-specific models efficiently, they often exhibit significant performance gaps compared to single-task models. Prior research, such Ties Merging [76], attributes this phenomenon parameter interference. This term refers to the redundancy or sign discrepancies found in parameters located at the same position (e.g.. self-attention weights) across different task models, which in turn result in information conflicting and performance loss. Additionally, task interference, as noted in multi-task learning literature [13, 31], arises from the inherent differences between tasks. For instance, tasks such as summarization, mathematical reasoning, and code generation require the model to process information in distinct ways. These differences worsen interference when models trained on different tasks are merged.
To understand these performance drops, we conducted two experiments using Task Arithmetic. First, we fine-tuned Qwen- 14B with LoRA, assigning non-overlapping modules avoid parameter interference. Despite this, a 3.21% drop in performance occurred, indicating persistent interference. Second, using two similar summarization tasks (XSUM and DailyMail), we observed an 8.42% drop compared to individually fine-tuned models, confirming that interference persists even between similar tasks. These results suggest that interference model merging is not limited to parameter-wise and task-wise issues.
3.2 Interpreting Interference From the Perspective of Knowledge
To tackle the challenge of interference, examine the merging process finer-graines knowledge perspective. We identify two types of critical knowledge: (1) Shared knowledge, which benefits multiple tasks, and (2) Exclusive knowledge, which is useful only for specific task. Single-task models often contain both types, complicating the merging process and leading to interference. To validate our hypotheses, we conduct experiments that vary the ratio of task-specific and shared knowledge.
To examine the impact of shared knowledge, conducted full fine-tuning on each model for its specific task. Excessive fine-tuning epochs can lead to catastrophic forgetting [19], a phenomenon where the model retains task-specific knowledge but loses general knowledge. As the fine-tuning epochs increase, the shared knowledge gradually decreases. The top section of Figure 3 illustrates that the epoch count increases, merging performance significantly deteriorates, even though the fine-tuned model performs well on its task. This underscores the crucial role of shared knowledge in merging performance.
To explore the impact of exclusive knowledge, we merge a single task-specific model into the base model. We apply sparsity method (e.g., SVD) to reduce the ratios of task-specific weights in the

---

merging model from 100% (standard merging) to 0% (base model). As shown in the lower part of Figure 3. performance remains stable up to 90% sparsity. Notably, even with 99% sparsity rate, single-merged model outperforms multi-model merging, confirming the existence of exclusive knowledge, which is more pronounced with more models. This also underscores the value of unmerged task-specific knowledge, since the fine-tuning performance can be effectively restored by preserving unmerged task-specific information.
To summarize, both shared knowledge and un-merged task-specific knowledge play a vital role in merging performance. The exclusive nature of task mask-specific knowledge hinders the effectiveness of merging methods. Different types of knowledge need to be separated and modularized to achieve optimal performance. Thus, the first step of our Twin-Merghh weights into an expert containing shared knowledge and weights holding task-exclusiv knowledge before merging. Formally, we denote the shared expert as O, and the exclusive task specifcc knowledge as (1, 1 the detail of our method is illustrated in the following section.
approach is to explicitly partition the
3.3 Twin Merging
Our proposed Twin-Merging employs two main stages: knowledge modularization and dynamic merging. These stages are designed to narrow the performance gap and enhance adaptive knowledge composi- tion. Building on the formulation in Equa- tion (2). Twin-Merging preprocesses experts into shared experts, isolates and compresses exclusive knowledge into vectors, and dy- namically composes them during inference.
Algorithm Twin-Merging
Require:
language model [a; 0), pre-trained weight Og and I task-specific fine-tuned weights 1 trained router R parameterized by a full-connect layer Φ. em- bedding E'b, compression rank and pre-specffied weight (7e)e=1
1: Pre-calculation:
b Only excute once
2: Compute the shared expert 0.:
3: O, t 6o + Ei= (0, - 0a)
The preprocess stage comprises three steps: (I) Shared Expert: To separate shared knowledge across different models, we con- sider the pre-merged model as natural place- holder to encapsulate common knowledge that is important to all tasks (denoted as 0*). By leveraging established merging tech-
4: Extract exclusive knowledge vectors for each task- specific weight:
5:
ex (V. 0.), fort 1,.... T
6: Inference:
b Main loop
7: initialize output Y
8: for each input in inputs X do
9:
Calculate router weights:
niques such Task Arithmetic, we can read-
10:
[10 wr] + softmax (R(Em((z)):
ily extract the shared experts from the ini- tial merged model.
11:
Merge into a single expert 0":
(2) Exclusive Knowl-
12:
<- 0, + ΣT Wrth
edge: To convey task-specific information while separating common knowledge, we cal-
13:
Perform model inference to produce the output:
14:
Y + Y fx;;0")
culate the difference vector: V. 0, 0". This subtraction vector preserves un-merged task-specific information while discarding Ensure: Output Y for input X.
15: end for
the shared knowledge. (3) Compressed ex-
clusive vectors: For practical use and distribution, we apply singular value decomposition (SVD) to further compress the above exclusive knowledge into vectors for each task. Assuming v, has a rank-m decomposition, t; U,E, VT. achieve low-rank task space by selecting the top-r singular values, resulting in (r(r) (r)) We store only U.(r), Er(r), Vrr)).
In inference stage, adapting to unforeseen challenges is difficult, especially with varied test data. For example, if most of the data consists of certain type (denoted as D.),, we should tailor the merged model for that specific task to get the best results. Instead of pre-defining the best parameters, we propose a new approach that combines shared expertise with exclusive knowledge. Our method involves using the input to dynamically adjust to the current data, enabling utilize shared knowledge and apply specialized expertise based on the inputs.
0 F(
0
U1 CT
(2)
shared knowledge exclusive knowledge
During inference, we fine-tune a small fuser R parameterized by dp through empirical risk minimiza- tion on a small validation dataset. This fuser, trained to dynamically select the specific task experts,
5

---

replacing the need for complex optimization algorithms to determine fusion coefficients. The merging model is obtained by:
0' = -, +
We (V: - 0*)
t=1
[wi,"' WT softmax
R(Emb(<x);
Here, Emb(x) represents the sequence of the last-layer token embeddings from the shared expert f(;;
Experiments
4.1 Merging Experiment
Baselines We first compare Twin-Merging with several train-free model-merging methods on both discriminative and generative NLP benchmarks, including weight averaging, Task Arithmetic [28], Ties-Merging [76], and DARE Merging [79]. To compare with Merging methods that need validation dataset, we also conduct experiments on CV tasks with AdaMerging [78] and Surgery [77]. Details on these baselines provided in Appendix D.
Benchmarks For language discriminative tasks, following [76, 79], we use RoBERTa [42] as the backbone and evaluate on the 8-task GLUE benchmark [69]. More details are in Appendix D.2. For language generative tasks, we use Qwen-14B [3] as the primary model to demonstrate the effectiveness of our approach on large-scale language models. To reduce deployment costs, we utilize task-specific checkpoints fine-tuned with the LoRA method [26] (See Appendix A for details adapting Twin- Swin-Merging to LoRA). We evaluate our model on four scenarios: general knowledge (MMLU benchmark [24]), factualness (TruthfulQA [40]), safety (BBQ [52]), and summarization (CNN-DailyMail [48]).
For vision tasks, following AdaMerging [78], we use ViT-B/32 in CLIP [55] as the backbone on eight image classification datasets: SUN397 [75], Stanford Cars [35], RESISC45 [5], EuroSAT [23], SVHN [50], GTSRB [59], MNIST [10], and DTD [7]. We employ the best version of AdaMerging (layer-wise AdaMerging++) and the Surgery (AdaMerging version), and apply 90% sparsity for our Twin-Merging. Detailed information is provided in Appendix D.2.
Metrics We include individually fine-tuned models and the pre-trained model as upper and lower bounds on performance, respectively. To mitigate the effects of different task-speeific score ranges, performance is assessed using the average normalized score of the fine-tuned models. The normalized score of merged model f* is calculated as:
Score [f( 0*)] z>D,
Normalized Score
1
Σ
T
Score [ft 0,)] 2nD:
Table Performance on 8 Discriminative Tasks (RoBERTa) and Generative Tasks (Qwen-
Method
8 Discriminative Tasks
4 Generative Tasks
Avg.
Pretrained
41.69
91.06
66.37
Fine-tuned
100.00
100.00
100.00
Weight Averaging
52.56
95.74
74.15
Task Arithmetic
67.80
96.61
82.20
Task Arithmetic (w/ DARE)
64.66
98.52
81.59
Merg-ng
63.68
92.67
78.17
Ties-Merging (w/ DARE)
65.58
91.92
78.75
Twin-Merging (Rank-1)
86.00
100.96
93.48
Twin-Merging (Ours)
96.14
102.38
99.26

---

Table 3: Performance and Cost on CV Tasks (ViT-B/32)
Method
Avg. Normalized Score
Additional Time Cost
VRAM
Pretrained
52.02
18m48s
3.6GB
Fine-tuned
100.00
18m48s
28.8GB
Weight Averaging
72.30
18m50s
3.6GB
Task Arithmetic
76.50
21m34s
3.6GB
Ties-Merging
75.10
19m24s
3,6GB
AdaMerging
88.50
185m35s
3.6GB
Surgery
94.04
215m01s
32.4GB
Twin-Merging(Ours)
95.33
47m22s
5.0GB
Main Results Table 2 presents the results for all discriminative and generative benchmarks for language tasks, while Table provides the results for vision tasks. A comparison of each task is illustrated in Figure 2b, with detailed statistics provided in Table 8 and Table 9 in the Appendix D.7. We also list the full-finetuned LLAMA results in Appendix D.7.
For discriminative tasks, it approachs the upper bound of finetune performance in the GLUE bench- mark. Specifically, our methods improve over Task Arithmetic by 28 28.34%, Ties-Merging by 32.46%, and DARE-Merging by 30.56% in absolute normalized score. In Figure 2b, we observe that especially on the COLA task, where conventional merging methods fail to improve the result, our approach can still approach the upper bound of the COLA expert.
On generative tasks, Twin-Merginn achieves strong performance, outperforming Task Arithmetic and DARE Merging by 5.77% and 3. 3.86%. Two interesting insights emerge: (1) The performance gains for Owen-14B in generative tasks smaller compared to RoBERTa in discriminative tasks. This indicates that smaller models like RoBERTa gain more from task-specific knowledge, while large models like Qwen-14B perform well because its strong general knowledge. (2)Twin-Merging surpasses the upper bound set by fine-tuned experts on the generative benchmark. This may be due to the extensive knowledge in Qwen- 14B, where modularization and dynamic merging unlock further potential without additional fine-tuning. These findings highlight a promising path for improving large language models without retraining.
For vision tasks, Twin- Merring outperforms the AdaMerging and Surgery baselines with higher accuracy (95.33% vs. 94.04%) while being more efficient in time and storage (47m22s vs. 215m01s, 5.0GB vs. 32.4GB). AdaMerging uses task-wise or layer-wise learnable parameters to improve merging, and Surgery adds task-specific modules after merging, requiring training the validation set for all eight tasks. Surgery also needs prior knowledge of the task type before inference and involves multiple forward passes, leading to high VRAM usage. In contrast, method efficiently handles diverse test inputs with minimal time and storage costs.
Table 5: Performance (un-normalized") on unseen tasks
Table 4: Our method scalability (72B)
Method
QNI.I-NI.I-RTE
MMLU
Method
TruthfulQA
BBQ
Multi-Task Learning
44.63
63.74
Pretrained-728
94.48
89.51
Task Arithmetic
53.92
62.02
Fine-tuned
100
100
Task Arithmetic (w/ DARE)
54.27
63.09
Task Arithmetic
98.70
95.40
Ties Merging
54.09
64.62
Twin Merging
99.30
97.14
Ties Merging (w/ DARE)
54,72
63.13
Twin-Merging
55.86
65.98
Scalability of Twin-Megging Our method remains effective with scaled models (e.g.. 72B parame- ters), as shown in Table 4. To manage high deployment costs, we limited our evaluation and merged experts two tasks: BBQ and TruthfulQA. Twin-Merging consistently surpasses scaled pre-trained models and Task Arithmetic, highlighting our approach's 's scalability. Additionally, our method can be
Notice that we cannot directly normalize them as we do not have the corresponding expert on unseen datasets to get upper-bound performances. This leads to relatively lower scores due to the narrower score ranges for tasks like RTE (max 43 max 91.71 71.71 for QNLI) and MMLU (max 68 68.03).

---

easily integrated with other merging methods, as detailed in Appendix D.9, making it both extensible and scalable.
4.2 Unseen Generalization
As shown in Table 5, Twin-Merging method benefits from complementary collaboration among different experts. Since the corresponding task-specific experts are unavailable, we directly use the average of the unnormalized scores as the metrics. In the GLUE benchmark, when QNLI, MNLI, and RTE experts absent, combination for QNLI can be found in Figure 5a. For complex tasks like MMLU, which involves multiple-choice QA tasks across 57 categories, Twin-Merging demonstrates superior performance using the combined knowledge from TruthfulQA, BBQ, and CNN-DailyMail domains.
approach still outperforms traditional baselines. Details on the expert
4.3 Ablation Studies
Table 6: Ablation study of Twin-Merging
Method
RoBERTa
Qwen
Pretrain
41.69
91.06
Shared
67.80
96.61
Dynamic Merging
81.47
87.77
Pretrain Dynamic Merging
85,90
95.03
Shared + Dynamic Merging (Twin Merging)
96.14
102.38
To demonstrate the effectiveness of our approach, conducted ablation studies for Twin- Merging. summarized in Table 6. Removing dynamic experts from the Shared model leads to a significant performance loss (96.14 vs. 67.80), highlighting the need for dynamic merging. Replacing the shared expert with a task-specific expert also results in a clear drop in performance (96.14 vs. 81.47), showing the value of the shared expert in capturing common knowledge.
Additionally, applying dynamic merging directly pretrained model performs worse than Twin Merging (85.90 vs. 96.14), likely due to two factors: (I) Pretrained models may lack rich task knowledge, while the shared expert in Twin Merging captures diverse, task-specific knowledge. (2) Subtracting the pretrained model fails to fully consider exclusive knowledge specific to each task, leading to interference, as analyzed in Section 3.2.
Discussion We find that removing dynamic experts severely impacts RoBERTa but has less effect on Qwen- suggesting that smaller models rely more on task- biases, while larger models benefit more from general shared knowledge. This indicates that our method adapts effectively the varying knowledge requirements of models of different sizes.
Discussion Compared to simpler routing methods like direct route to task-specific expert or com- bining multiple experts based on multiple LoRA [27, 81], Twin Merging delivers better performance, especially on unseen tasks, by reducing interference and leveraging complementary knowledge.
Direct route task-specific expert refers to the fine-tuned baseline in Table 2. This approach assumes perfect routing and the absence of out-of-domain data, where each task uses its own dedicated expert. It represents the ideal scenario and serves gap for merging methods. Despite this, Twin Merging still improves performance on generative tasks (102.38 vs. 100.0) and unseen tasks (Table 5) by leveraging different sources of exclusive knowledge. Moreover, this baseline demands storing task-specific experts, which significantly increases storage, as discussed in Section 4.6.
oracle baseline to highlight the performance
In combining multiple experts, the lack of separation between shared and exclusive knowledge leads to interference, as conflicts between exclusive knowledge are inevitable (Section 3.2). There are two ways to combine experts: (1) Static Combination: This is akin to "Task Arithmetic" in LoRA (Table 2). Twin Merging outperforms static combinations (102.38 vs. 96.61). (2) Dynamic Combination: This matches "Pretrain + Dynamic Merging" method in Table 6, and Twin Merging again shows superior performance (102.38 vs. 97.03).

---

4.4
Scale to More Tasks
-X--
Pretmined
Weight Averaaing
Ties Merging
Ties Teer-Merging (W) DAKE)
-X-
Fine-tuned
Task Arthmetiic
Task Artthmetic (w DARE)
Twin-Merging (Ouns)
100
Nom (%6)
80
Storage Stze (G)
60
40
2
2
Figure 4: Averaged normalized accuracy vs. the number of tasks for various benchmarks. Twin- Merging maintains performance regardless of task number and compresses the fine-tuned checkpoints.
In the left panel of Figure 4, we examine the impact of the number of tasks on model merging performance. Conventional model merging methods degrade notably, especially with many tasks, nearly reaching pre-trained levels. However, Twin-Merging consistently outperforms other methods, approaching fine-tuned performance, with greater gains as the task count rises.
The right panel of Figure shows the performance-storage trade-offs. While model merging methods have constant storage cost, their performance remains low. In contrast, maintaining individual task-specific models guarantees strong performance but requires excessive storage. Twin-Merging achieves nearly 100% normalized accuracy across various tasks, balancing performance and storage efficiency by maintaining task-specific parameters with shared experts. This makes Twin-Merging a viable solution for scenarios demanding balance between performance and storage efficiency.
4.5
Router Analysis
COLL
VMRPC vSTS
VQQP
V:
v
v.NNI
v.TruthfuQQA
v:CNN
Wingh
Weight
IND
IND
IND
IND7
MMLU
TruthfuQA
BBQ
CNN
(a) The routing result on the QNLI dataset using different num- bers of GLUE experts, ranging from 2 twin vectors (UCALA and USSD2) to twin vectors (UCaLA, USSI-2, UMRPC. USTS-B. toge. UMNLI and UONLI). The router weights are Softmax normalized.
(b) The routing weight of Qwen ex-
perts (UhaaL..YTunnuiya.u Utwrannniiaa.aaaa..... www.g.
on four generative tasks (MMLU, Truth-
fulQA, BBQ. CNN-DailyMaii),
Figure 5: Twin-Merging routing decisions of the experts for various tasks.
Figure 5 shows the results of routing decisions among experts for the QNLI dataset and four generative benchmarks. As shown in Figure 5a, the router maximizes the use limited expert knowledge to address QNLI, task where the goal is to determine if the context sentence contains the answer the input question. For example, with only UCalA and USST.2 available, the router primarily uses UCoL which provides knowledge of sentence and word relations, while USST.2 is focused irrelevant sentiment classification. With six experts ranging from UCoLA to UNNLI. the router mainly leverages UMNLI for textual emtailment and Doop for question-answering capabilities. When UONLLI is included, the router naturally relies on QNLI-specific knowledge. These results demonstrate the fiexibility and adaptability of our Twin-Merging method, providing good interpretability. For larger models like Qwen- 14B, as shown in Figure 5b, the router plays a crucial role in selecting and combining specific knowledge. When experts have overlapping spec-fc knowledge, such as UruniulQA and UMMLU- the router may assign them similar weights.
4.6 Compression and Speed Analysis
Compression Analysis In the left panel of Figure 6, we explore sparsity rates from 0% 100%. Appendix E attachs detail qualtivie analysis of various Merging methods. Remarkably, our Twin-

---

Magnitude
Bemouli
ZIZ
SVD
Avg. Score
90
100
Arg. Score
(99,87,86.4)
80
70
Shared Export
(100.00.67.53.
20
60
100
10
30
50
70
90
98
Sparsity Rate (%)
Rate Spursificution (%)
Figure Twin-Merging performance vs. different sparsity levels and techniques for GLUE
99.8% compression rate. This suggests that performance relies on small fraction of task-specific parameters, aligning with previous findings [76, 79J. Our results also validate our hypothesis that redundant parameters can obscure critical knowledge, leading to performance degradation. Consequently, we primarily use a 90% sparsity rate our experiments to preserve performance while reducing storage costs. We also conducted an ablation study on sparsity methods, shown on the right side of Figure 6. SVD better retains task- susc-spe information compared to Magnitude [76] and Bernoulli Dropout [79]. As SVD is applied only once during preprocessing, it does not become an inference bottleneck.
Merging method maintains 86.4% performance
Table 7: cemputte-erformancee tradeoff in the generative benchmark.
Method
Training Tokens
Training Cost
Inference Cost (/1000 items)
Performance
Multi- Task Learning
536.35M
10h32min
236s
94.31
Model Merging
236s
96
Twin-Merging
0.57M
183s
275s
102.38
Speed Analysis Table presents the time cost for Twin-Merging in generative benchmarks. Al- though the training stage utilizes only 0.1% % of the total training budget, TTiin-erggng significantly improves general capabilities compared to multi-task learning. Compared to conventional model merging methods, Twin-Merging sacrifices minimal router training budget and incurs slight re- duction in inference speed for dynamically composing the twin vectors, thereby achieving superior performance. More detailed analysis and results are provided in Appendix E. In summary, our approach strikes a better balance between computational cost and performance.
Conclusions
In this paper, we introduce the Twin-Merging to merge language models, aiming to close the performance gap between conventional model merging techniques and fine-tuned models, while improving adaptability to data heterogeneity. By modularizing and dynamically merging shared and task-specific knowledge, Twin-Merging significantly outperforms existing model-merging methods and approaches the performance of fine-tuned models across various settings and domains. Our study highlights the impact of shared and exclusive task-specific knowledge on merging performance. benefits even strong scaled models like Qwen-72B, which already
We show that Twin-Merging perform well across domains. It extends to more tasks and merging methods, demonstrating better generalization on unseen data. By utilizing SVD, our solution retains 86% of the performance with only 0.1 1% of the parameters, approaching upper-bound performance with minimal storage increase as tasks grow, achieving a better tradeoff between computation and performance.
6 Acknowledgments
We thank the Shanghai AI Laboratory for supporting GPU resources. We also thank the anonymous reviewers for their comments on improving the quality this paper and Ne for their resource/technical support.
Heee, we assume that merging method does not retrain all task experts; instead, it reuses experts (e.g., down- loaded from model hubs like Huggingface [73]).
10

---

References
[1] Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference Learning Representations, 2023.
[2] Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: open language model for mathematics, 2024.
[3] Jinze Bai, Shuai Bai, Yunfci Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang. Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang. Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang. Jian Yang. Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023.
[4] Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gra- dient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pages 794-803. PMLR, 2018.
[5] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the Proceedings of the IEEE, 105(10):1855-883, 2017.
[6] Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. Fusing finetuned models for better pretraining, 2022.
[7] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed. and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and paftern recognition, pages 3606-3613, 2014.
[8] Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International conference on machine learning. pages 4057-4086, PMLR, 2022.
[9] Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng. Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models, 2024.
[10] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-14) 2012.
[11] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
[12] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 1171-4186, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10. 11111//11/1191... 1423. URI. http:///////caaaooooggooooo org/N19-1423,
[13] Chuntao Ding, Zhichao Lu, Shangguang Wang, Ran Cheng, and Vishnu Naresh Boddeti. Mitigating task interference in multi-task learning via explicit task routing with non-learnable primitives. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 7756-7765, 2023.
[14] Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. Multi-task learning for multiple language translation. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language
11

---

Processing (Volume 1: Long Papers), pages 1723-1732, Beijing, China, July 2015. Association for Computational Linguisties. doi: 10.3115/v1/115-111 1166. URL org/P15-1166.
[15] Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning, pages 1309-1318. PMLR, 2018.
[16] Chenghao Fan, Zhenyi Lu, Wei Wei, Jie Tian, Xiaoye Qu, Dangyang Chen, and Yu Cheng. On giant's shoulders: Effortless weak to strong by dynamic logits fusion. arXiv preprint arXiv:2406. 15480, 2024.
[17] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23 (120):1-39, 2022.
[18] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pages 3259-3269. PMLR, 2020.
[19] Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 128-1,2, 1999.
[20] Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31, 2018.
[21] Almog Gueta, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. Knowledge a region in weight space for fine-tuned language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguisties: EMNLP 2023, pages 3350-3370 Singapore, December 2023. Association for Computational Linguistics. doi: 10. 00.18633/v1/2022 2023. Findings-ennlp 95.
11t6://///00.....innn--mmm////// findings-emnlp.95. URL https: hatsss///caaaahhoggoeoe org/
[22] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming - the rise of code intelligence, 2024.
[23] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. 12(7):2217-2226, 2019.
[24] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. In International Conference Learning Representations, 2020.
[25] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan. Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022.
[26] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
[27] Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition, 2024. URL https://arxiv.. org/ hts//bb//23/// 13269.
[28] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference Learning Representations, 2022.
12

---

[29] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannanch Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023. URL ptttt:://openrweeee net/forum?id= 610Kwf8-jri,
[30] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024.
[31] Junguang Jiang, Baixu Chen, Junwei Pan, Ximei Wang, Dapeng Liu, Mingsheng Long, et al. Forkmerge: Mitigating negative transfer in auxiliary-task learning. In Phirty-sevente Conference on Neural Information Processing Systems, 2023.
[32] Weisen Jiang, Baijiong Lin, Han Shi, Yu Zhang, Zhenguo Li, and James T. Kwok. Byom: Building your own multi-task model for free, 2024.
[33] Xisen Jin, Xiang Ren, Daniel Preotiuc-Piete and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representationss 2022.
[34] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
[35] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[36] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding. In International Conference on Learning Represen- lations, 2021.
[37] Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey, 2023.
[38] Lucas Liebenwein, Cenk Baykal, Brandon Carter, David Gifford, and Daniela Rus. Lost in pruning: The effects of pruning neural networks beyond test accuracy. Proceedings of Machine Learning and Systems, 3:93-138, 2021.
[39] Chin-Yes Yew Lin and Eduard Hovy. Automatic evaluation of summaries using n- n-gram co- occurrence statistics. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 150-157, 2003. URL https: //ap::///clllo an anthology- wwoggooorggoo 2
[40] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), 2022.
[41] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 2021.
[42] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. 2019.
[43] Zhenyi Lu, Jie Tian, Wei Wei, Xiaoye Qu, Yu Cheng, Dangyang Chen, et al. Mitigating boundary ambiguity and inherent bias for text classification in the era of large language models. arXiv preprint arXiv:2406.0 07001, 2024.
13

---

[44] Haipeng Luo, Qingieng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct, 2023.
[45]
Maninis, lljaa Radosavovic, and lasonas Kokkinos. Attentive single-tasker -aalieg of multiple tasks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1851-1860, 2019.
[46] Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 2022.
[47] Taishi Nakamura, Mayank Mishra, Simone Tedeschi, Yekun Chai, Jason Stillerman, Felix Friedrich, Prateek Yadav, Tanmay Laud, Vu Minh Chien, Terry Yue Zhuo, Diganta Misra, Ben Bogin, Xuan-Son Vu, Marzena Karpinska, Arnav Varma Dantuluri, Wojeiech Kusa, Tommaso Furlanello, Rio Yokota, Niklas Muennighoff. Suhas Pai, Tosin Adewumi, Veronika Laippala, Xiaozhe Yao, Adalberto Junior, Alpay Ariyak, Aleksandr Drozd, Jordan Clive, Kshitij Gupta, Liangyu Chen, Qi Sun, Ken Tsui, Noah Persaud, Nour Fahmy, Tianlong Chen, Mohit Bansal, Nicolo Monti, Tai Dang, Ziyang Luo, Tien-Tung Bui, Roberto Navigli, Virendra Mehta, Matthew Blumberg, Victor May, Huu Nguyen, and Sampo Pyysalo. Aurora-m: The first open source multilingual language model red-teamed according to the u.s. executive order, 2024.
[48] Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. Abstractive text sum- marization using sequence-to-sqquenee rnns and beyond. arXiv preprint arXiv: /602.06023, 2016.
[49] Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning as bargaining game. In International Conference on Machine Learning, pages 16428-16446. PMLR, 2022.
[50] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng. et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 4. Granada, 2011.
[51] Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. In Thirty-sevntth Conference on Neural Information Processing Systems, 2023. URL http::///ponneeeeeee///eoooo 0A9f2jZDGW.
[52] Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thomp- son, Phu Mon Htut, and Samuel Bowman. Bbq: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguisties: ACL 2022, 2022.
[53] David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. 2021.
[54] Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: A kvcache-centric disaggregated architecture for Ilm serving, 2024. URL
220700
[55] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[56] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre. Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori. Wenhan Xiong, Alexandre Défosse7, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llaaa Open foundation models for code, 2024.
14

---

[57] Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja. al. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022.
[58] Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33:22045-22055, 2020.
[59] Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: a multi-class classification competition. In The 2011 international joint conference neural networks, pages 1453-1460. IEEE, 2011.
[60] George Stoica, Daniel Bolya, Jakob Bjorner, Taylor Hearn, and Judy Hoffman. Zipit! merging models from different tasks without training, 2023.
[61] Zhaochen Su, Juntao Li, Jun Zhang, Tong Zhu, Xiaoye Qu, Pan Zhou, Yan Bowen, Yu Cheng, et al. Living in the moment: Can large language models grasp co-temporal reasoning? arXiv preprint arXiv:2406.09077 09072, 2024.
[62] Zhaochen Su, Jun Zhang, Xiaoye Qu, Tong Zhu, Yanshu Li, Jiashuo Sun, Juntao Li, Min Zhang, and Yu Cheng. Conflictbank: A benchmark for evaluating the influence of knowledge conflicts in Ilm. arXiv preprint arXiv:2408.1200 2024.
[63] Zhaochen Su, Jun Zhang, Tong Zhu, Xiaoye Qu, Juntao Li, Min Zhang, and Yu Cheng. Timo: Towards better temporal reasoning for language models. arXiv preprint 2411:2246.14119, 2024.
[64] Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Rozière, Jacob Kahn, Daniel Li, Wen-tau Yih, Jason Weston, al. Branch-train-min Mixing expert llms into a mixture-of-experts Ilm. arXiv preprint arXiv:2403.078 2024.
[65] Xiaofei Sun, Linfeng Dong, Xiaoya Li, Zhen Wan, Shuhe Wang, Tianwei Zhang, Jiwei Li, Fei Cheng, Lingjuan Lyu, Fei Wu, and Guoyin Wang. Pushing the limits of chatgpt on nlp tasks, 2023.
[66] Anke Tang, Li Shen, Yong Luo, Nan Yin, Lefei Zhang, and Dacheng Tao. Merging multi-task models via weight-ensembling mixture of experts, 2024.
[67] Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. Parameter-efficient multi-task model fusion with partial linearization. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview net/forum?ice http:/orrmmiiiiiynnWWaWaaaaa
[68] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
[69] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2018.
[70] Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khaz- aeni. Federated learning with matched averaging. In International Conference on Learning Representations, 2019.
[71] Hongyi Wang, Felipe Maia Polo, Yuekai Sun, Souvik Kundu, Eric P Xing, and Mikhail Yurochkin. Fusing models with complementary expertise. NeurIPS 2023 Workshop on Distribution Shifts: New Frontiers with Foundation Models, 2023.
[72] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022.
15

---

[73] Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State-of-the-ar art natural language processing, 2020.
[74] Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roclofs, Raphael Gontijo Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, 2022.
[75] Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring large collection of scene categories. International Journal of Computer Vision, 119: 3-22, 2016.
[76] Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. In Thirty-seventh Conference on Neural Informa- tion Processing Systems, 2023.
[77] Enneng Yang. Li Shen, Zhenyi Wang, Guibing Guo, Xiaojun Chen, Xingwei Wang, and Dacheng Tao. Representation surgery for multi-task model merging. arXiv preprint 202.v:2402.02705, 2024.
[78] Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In The Twelfth International Conference on Learning Reprexentations, 2024. URL https: /tppss/////////i////eeee eepperrreieww net/forummid= nZP6NgD3QY.
[79] Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as free lunch, 2024.
[80] Jihai Zhang, Xiaoye Qu, Tong Zhu, and Yu Cheng. Clip-moe: Towards building mixture of experts for clip with diversified multiplet upcycling. arXiv preprint arXiv:2409. 2024.
[81] Jinghan Zhang, Junteng Liu, Junxian He, et al. Composing purameter-efficient modules with arithmetic operation. Advances in Neural Information Processing Systems, 36:12589-12610, 2023.
[82] Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoe V Le, James Laudon, et al. Mfxture-of-experts with expert choice routing, Advances in Neural Information Processing Systems, 2022.
[83] Zhanpeng Zhou, Zijun Chen, Yilan Chen, Bo Zhang, and Junchi Yan. Cross-task linearity emerges in the pvetraining-finetunin paradigm, 2024.
[84] Tong Zhu, Daize Dong, Xiaoye Qu, Jiacheng Ruan, Wenliang Chen, and Yu Cheng. Dy- namic data mixing maximizes instruction tuning for oftture-of-experts arXiv preprint arXiv:2406.11 1256,, 2024.
[85] Tong Zhu, Xiaoye Qu, Daize Dong, Jiacheng Ruan, Jingqi Tong, Conghui He, and Yu Cheng. Llama-moe: Building mixture-ofeexperts from Ilaaa with continual pre-training. arXiv preprint 240i 16554, 2024.
[86] Barret Zoph. Designing effective sparse expert models. In 2022 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pages 1044-1044, 2022. doi: 10. 10.11001P0PP555544.300.00111
16

---

A Twin Merge on LoRA
Here, we will demonstrate that our Twin-Merging method can seamlessly applied to LoRA module
[26], where the base model is fixed and additional task-specific information is injected through matrix, i.e. O, - On + LoRA,, where LoRA, represents the fine-tuned LoRA module for the /-th task. let 0, 0 + LoRAg, we can prove that Twin-Merging the 0 is equivalent to Twin-Merging on the
LoRA module.
0" 0,
Σ We SVD, 0.)
Twin Swwin-Mrgging.om on
00 + LORA,
Wc * SVD,
(0n (On+LORA)) - (0o + LORAA))
(5)
0o + LoRA, +
Σ Wt SVD, LORA.)
Twin-Merging on LoRA
0. LORA
where we denote LORA* LORRA, +IT SLOR LORA.).
B More relative research
Multi-Task Learning. The multi-task training typically learns multi-task features by simultaneously optimizing task-specific objectives, facilitating integration diverse knowledge into the model. Existing works mainly focus mitigating task conflicts [41] and catastrophic forgetting [19] by parameter sharing [45], adjusting suitable objectives [14, 57], find suitable task weighting [4, 49], and minimizing negative transfer [31]. In an where models
growing larger, and the number of task scenarios is increasing, what we need to explore is more cost-effective approach to multi-task learning. Therefore focus is multi-task scenarios that do not require acquiring integrating multi-task data and do not involve additional updates to existing experts.
Mixture of Experts. To enhance model scalability without increasing computational costs, the mixture of experts (MoE) paradigm introduces conditional routing of inputs to a subset of learnable parameters. Several efforts have extended feedforward networks (FFNs) within Transformers to incorporate MoE layers, such GShard [36] and Switch Transformer [17]. These models typically employ learnable top-2 or top-1 routing strategies to scale MoE language models an extremely large size [30]. Recent studies have focused on challenges such as load balancing of experts [8, 82), training instability [86), expert specialization [9, 66), and synchronization reduction [64]. However, these methods often require substantial multi-task data and costly joint training. In contrast, our approach directly reuses task-specific experts, leading to the natural specialization of experts in different domains. We only require minimal fine-tuning for small router calculate fusion weights, making our method highly efficient.
C The Merging Interference and Limited Generalization
To illustrate the challenge in determining the optimal merging coefficient and the limitations of pre- specified coefficients with unpredictable data, we consider COLA and SST-2 in-domain experts. We merge them using Task Arithmetic and evaluate on the eight discriminative tasks from the GLUE benchmark. Only COLA and SST-2 are tasks, while the others are unseen. Since the merging coefficient is crucial for performance [51, 78), we conduct an extensive grid search for coefficients ranging from 2 to
large dark-blue region indicates consistent optimal performance, which is why Task Arithmetic can work with various weights. Conventional methods search this region for optimal performance across
17

---

COLA
SST-2
MRPC
STS-B
Pana
1.00
78574
0
0-
BOXK
0.75
-1-
-1-
-2
-2
-2
-2
0.25
-2
0
-1
2
-2
1
QQP
MNLI
QNLI
RTE
-0.00
2.
-0.25
6aT
78872
90o4
OOOK
ÓCOLA
00.50
0-
0-
-075
-1.00
-2
-2-
2
-2
2
0
2
-1
0
2
-2
7OOLA
TOOLA
7OOLA
TOOLA
Figure 7: The visualizations show normalized performance across eight GLUE tasks, highlighting the impact of combining expertise from the COLA and SST-2 domains (expert indicated by red vectors) through Task Arithmetic. Performance scores are normalized, with the unmerged pretrained model set to zero and other results scaled to the [-1, 1] range. The x-axis (*COLA) and y-axis (TSST2) represent the merging weights for COLA and SST-2 expertise. Blue regions indicate improved performance over the pretrained model, while red regions indicate deterioration.
all in-domain tasks, avoiding the red region. However, this is computationally expensive and does not scale well with increasing number of tasks. Additionally, it cannot handle unseen tasks, as the same coefficients can produce different patterns across tasks. For example, setting coefficients "COLA and " to 1 leads to performance drops in MRPC and QNLI, but gains in MNLI, QQP, and RTE.
Furthermore, merging performance is not always single cluster. For example, within the range of [-2, 2]. STS-B and QNLI already show complex patterns, making it difficult to find an optimal weight for tasks when task-specific experts are limited. Although Yang et al. [78] propose unsupervised entropy minimization to find optimal coefficients, this method is limited to classification tasks and has limited adaptability.
To address this, we propose reformulating the problem of fusing models as supervised learning task. Specifically, we train a router dynamically merge task-specific experts, as detailed in Section 3.3.
D Experiment Details
Here we detaily illustrate the setting of our experiments.
D.I Compute Resources Used and Runtimes
We executed all our experiments on Nvidia A100 GPUs equipped with 80GB RAM. Single-task LoRA models for Owen-14B on four generative tasks required 1-2 hours per task, Single-task LoRA for Qwen-72B need 10 hours single GPUs to train. while the multitask vector took around 10 hours on single GPUs of 500M tokens. The RoBERTa model needs 15 minutes per task on GLUE datasets.
Our router is implemented as a three-layer linear network with Leaky ReLU activations and batch normalization. We train the router the validation dataset with learning rate of 5e-4 for 10 epochs. The validation set consists the integration of in-domain downstream tasks, not the general text corpus. The validation set is taken from split of the training set, and at most 1,000 items for router training for each task.
'In fact. the MNLI and QNLI are very similar tasks about Natural Language Inference (NLI) [69]. This demonstrates that task similarity does not guarantee similar merging performance patterns.
18

---

Merge experiments were efficient, with evaluations consuming less than minutes. The inference generally fast within 4 minutes per 1000 items for generative tasks and less than 30 seconds per 1000 items for discriminative tasks. The detail comparison of the training cost and inference cost of different methods are detailed in Table 7.
D.2 Employed Datasets and Associated Licences
Discriminative Tasks. we conduct experiments on the GLUE benchmark [69] with eight discrimi- native tasks, which is designed for classification tasks except for STS-B for the regression task. The detail of eight dataset can be found in the paper of Wang et al. [69]. Consistent with prior research [79], We split 10% of the training test set.
validation and employ the original validation data the
The licenses of QNLI, COLA, and STS-B are licensed under CC-BY-SA. QQP is licensed under MIT. SST-2 and MRPC are licensed under Apache 2.0. MNLI is licensed under OANC. RTE is licensed under CC BY 4.0. Thus, these datasets in GLUE are available for non-commercial research purposes.
Generative Tasks. We conducted experiments on four benchmarks:
1. MMLU [24]: This benchmark tests general and STEM knowledge across 57 subjects, from elementary professional levels. We used Exact-Match the metric.
2. TruthfulQA [40]: This benchmark assesses the truthfulness of language models with 817 questions spanning categories like health, law, finance, and politics. Exact-Match was used as the metric.
3. BBQ [52]: This dataset highlights social biases against protected classes nine social dimensions relevant to U.S. English-speaking contexts. Exact-Match was the metric.
4. MaNl [48]; This dataset is used for text summarization, requiring models to generate summaries of news stories. ROUGE-2 scores [39] were used for evaluation.
We evaluated these tasks using the HELM benchmark in a few-shot setting.
For MMLU and TruthfulQA, which lack official training sets, used the Dolly-15k dataser for MMLU and the BigBench-samples dataset for TruthfulQA.
The GSM8K and MMLU datasets are under the MIT License. TruthfulQA and CNN-Daily Mail are under the Apache-2.0 License. BBQ is under the CC-BY 4.0 License. These datasets are available for commerccalal research purposes.
Vision Tasks.
1. SUN397 [75]:A scene classification dataset containing 108,754 images across 397 classes, with each class having at least 100 images.
2. Stanford Cars [35]: A car classification dataset comprising 196 classes and a total of 16, 185185 images, divided equally between the training and test sets.
3. RESISC45 [5]: A remote sensing image scene classification dataset with 45 classes and 31,500 images, approximately 700 per class.
4. EuroSAT [23]: A satellite image classification dataset consisting of 27,000 labeled and geo-referenced images across 10 classes.
5. SVHN [50]: A real-world digit classification dataset extracted from Google Street View images. It includes 10 classes, with 73,257 training samples, 26,032 test samples, and 531,131 additional simple samples.
6. GTSRB [59]: A traffic sign classification dataset containing over 50,000 images across 43 classes of traffic signs.
7. MNIST [10]: benchmark dataset for image classification, containing grayscale images of handwritten digits across 10 classes. The training set has 60,000 images, and the test has 10,000 images, with balanced distribution among classes.
'https: hhttps://huggingfaac http://hhhw...oooooooooooootttta
19

---

8. DTD [7]: A texture classification dataset with 47 classes and a total of 5,640 images, with approximately 120 images per class.
D.3 Language Model Backhone
For discriminative tasks, we used RoBERTa-basd basse" [42] as our pre-trained backbone and fine-tuned for each dataset to create supervised models. We conducted separate fine-tuning for the RoBERTa- base model on each dataset for 10 epochs. Our selected hyperparameters included a batch size of 64 and learning rate set at le
For generative tasks, employed Qwen-14BB* the backbone and applied LoRA [26] for task- specific fine-tuning. In the case of generative tasks, the fine-tuning process for Qwen-14B involved the utilization of LoRA with rank to 32, batch size of 128, and learning rate of 2e-4 for epochs. For Qwen-72B we employ the same setting with QLoRA technique [11].
D.4 Non-Overlapping Merging
To serperate the impact of parameter wise interference, we design the non-overlapping experiment based on Qwen LoRA modules follows: (1) Firstly, we obtain standard merging experts by injecting the LoRA A module into both the "wl" and "c_proj"" weights of the Qwen-based model, and fine-tune them on two different tasks, resulting in two distinct models. Then combine it into single model to obtrain standard merging results. (2) Next, we performe a con-overlapping fine-tuning by injecting LoRA only to "wl'" 1" on one task and "c_ "__projj" on another. producing two models with task-specific knowledge in different modules. (3) Finally, we combined the non-overlapping checkpoints to get the merged results. Since task specific knowledge was injected into separate modules, parameter- wise interference was minimized. The results are shown in the upper section of Table 1.
D.5 Sparsification Methods Details
In Figure 6, we conduct comparative analysis employing various sparsification methods. The specifics of each method are outlined below:
Magnitude. Following the setting in Ties fies-Megging [76], we retain solely the 4% largest- magnitude values while resetting the remaining values to zero.
Bernoulli-Dropout, Adhering to the methodology introduced in DARE [79], we employ a parameterized Bernouli distribution to sample sparse mask m'. This mask is then applied to the parameters 8 and subsequently rescaled with respect to the mask rate k.
m' Bernoulli(k),,
8 m' ④ 5*,
8' 8' (1-
Singular value decomposition (SVD). Assuming that matrix M has a rank-m decompo- sition, expressed as M U,E, V² where U, Bdou xm Σ; E Rm V, E RdoxXm. We compress the matrix M by selecting only the top-r singular values from E, denoted Here, Ue(r) E R xr Er(r) E R'xr V E R xr represent sub- matrimatss of U. Σ., V? This transformation significantly reduces the task- specific parameter dimensionality from x (dout + din + 1) to x (dout + din + 1). the maximum m typically equals to the hidden size of the language model (e.g., m 768 for RoBERTa-base and m 4096 for Qwen-14B) and can be reduced to 1, resulting in significant reduction parameters and storage effectiveness.
as M, = Ex (r))
During merging, we decompress these matrices by extending (r) (r) Vr(r) to size- m by filling with zeros, allowing us to recover M via their product. This operation is only at the matrix level; once we obtain the merged matrix, we discard the decompressed matrices. ensuring efficient storage.
hnttps://hugginggfac. co/Faceeookkt/rrberrk. base >nttps://hugiingfFan ace. http:///nnnnaeeee//////////e///e 1488
20

---

D.6
Baselines Details
Here we will elaborate on the baselines utilized in our main comparison experiment, as outlined in Table 2 and Figure 2b.
Individual means that each task uses the corresponding fine-tuned model, which has no interference between tasks but cannot perform multiple tasks simultaneously. It serves as the upper-bound performance for each specific task.
Weight Averaging [6, 74] is the simplest form of model merging, which straightforwardly averages the parameters of multiple models. It serves as lower bound for model merging.
Task Arithmetic [28] first introduces the concept of "task vectors" and merges them into the pre-trained model to execute multi-task learning.
Ties-Merging [76] addresses task conflicts by eliminating redundant parameters. The process involves three steps: Trim, Elect Sign, and Disjoint Merge.
Task Arithmetic (w/ DARE) [79] This variant incorporates the Bernoulli-Drogee technique
for 70% sparsification before employing Task Arithmetic [28] for merging.
Ties-Merging (w/ DARE [79] Similar to the previous approach, this variant integrates
Bernoulli-Dropout for 70% sparsification, followed by Ties-Merging [76] for the merging process.
AdaMerging [78] assumes access to an offline test set and dynamically adapts to it by introducing additional coefficients at every layer, conducting unsupervised training across multiple iterations on the test set (without labels) to refine the model.
Surgery [77] assumes that test data IDs are accessible during inference, allowing it to insert corresponding task-specifis adapters to leverage task-specific knowledge.
The coefficient for Task Arithmetic and Ties-Merging are decided by a small scale grid search on validation datasets. The coefficient of 0.7 is consistently applied for DARE Merging, following the previous papers [79].
D.7
Detail Results
In Table 2, we present only the average normalized scores across various tasks. In this section, detail the statistical performance of all tasks, with discriminative results displayed in Table 8 and generative results shown in Table 9.
Table 8: The detail statistics of different merging performance on 8 discriminative tasks. Bold
numbers indicate the best-averaging performance across different model merging methods.
Model
COLA
STS-2
MRPC
STS-B
QQP
QNLI
MNLI
RTE
Avg.
Pre-trained Fine-tuned
0.00
53.76
85.01
4.01
37.48
$3.05
37.09
71.19
41.69
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
Weight Averaging
0.00
59.21
85.79
46.99
45.37
63.94
48.00
71.19
52.56
Task Arithmetic
8.35
88.26
89.57
32.84
82.05
85.40
75.54
80.43
67.80
Ties-Merging
31.76
88.86
86.18
10.94
61.05
85,94
83.01
69.56
64.66
Task Arithmetic (w/ DARE)
0.00
88.14
86.61
30.19
84.33
79.09
63,95
77.16
63.68
Ties-Merging (w/ DARE)
11.82
95.52
85.75
9.43
86.77
88.67
83,13
63.59
65.58
Twin-Merging (Rank-1)
51.24
98.67
89,20
76.31
92, 16
93.24
96.45
90.76
86.00
Twin-Merging (90% compressed)
101.01
99.88
99.41
79.89
99.14
99.67
96.68
93.47
96.14
We primarily merge using LoRA for the Qwen-14B models because fully finetuning them to obtain task-specific experts would require huge amount of resources and computation (finetuning the full 14B model requires at least 8 A100 GPUs). However, to further demonstrate, we provide experiments on fully fine-tuned LLaMA 7B models for generative tasks (GSM8k and TruthfulQA). where our approach still exhibits superior performance.
D.8 More Advantages of Our Method
Beyond strong performance and efficiency, our method offers key benefits in handling distribution shifts and LLM deployment, surpassing approaches like AdaMerging and Surgery.
21

---

Table The detail statistics of different merging performance on 4 generative tasks. Bold numbers indicate the best- aestrainggg performance across different model merging methods. Underlines indicate second best performance each task across different model merging methods.
Model
MMLU
TruthfulQA
BBQ
Mail
Avg.
Pretrained
101.37
94.35
86.27
82.24
91.06
Fine-tuned
100.00
100.00
100.00
100.00
100.00
Weight Averging
99.63
92.04
88.01
103.28
95.74
Task Arithmetic
98.93
98.23
83.65
105.62
96.61
Task Arithmetic (w/ DARE)
99.22
96.90
88,56
109.40
98.52
Vies-Merging
99.88
92.04
89,92
88.83
92.67
Ties-Merging DARE)
101.41
97.66
86.81
81,80
91.92
Twin-Merging (rank-1)
99.40
95.58
93.46
115.39
100.96
Twin-Merging (rank-16)
99.87
98.23
97.00
114.43
102.38
Table 10: Performance of LLaMA-7B
Method
Avg.
Inference Time (/1000 items)
Task-Arite Crithei
69.89
186s
Twin Merging
88.18 18
198s
AdaMerging tackles distribution shifts in image domains by requiring access to test sets, even in unseen domains. This is problematic for online settings where test data is unpredictable. In offline scenarios, scaling test sets to large sizes is inefficient. Additionally, AdaMerging's entropy-based optimization limits it to classification tasks, which restricts its applicability as generative models like LLaMA and GPT-4 become more prominent.
Our method overcomes these limitations by dynamically adapting to test inputs, as shown in Table 5. It does not require specific validation datasets for each in-domain expert, making it more flexible. For example, the open- open-source Dolly dataset can represent the MMLU expert, and we do not need validation sets for QNLI, MNLI, RTF, MMLU. Instead, we assume that combining in-domain knowledge helps address out-of-domain inputs, as supported by [71]. This dynamic merging ensures better generalization to diverse inputs, unlike AdaMerging, which relies entropy approximations without supervision.
Our approach is also optimized for gradient updates, aligning with current trends [54]. Key advantages include:
LLM deployment with continuous data streams and no
Handling Unpredictable, Heterogeneous Data: Our dynamic merging technique efficiently addresses diverse streaming inputs.
Reducing Latency and Storage:
We shift router training and knowledge modularization to preprocessing, and apply SVD techniques to minimize storage needs.
Broad Applicability: Our method works across NLP and CV tasks, including generative tasks with Qwen-14B and up 72B models, supporting large-scale Al deployment.
D.9 Compatibility of Twin-Merging with Other Merging Methods
Table 11: Our method extensibility other model merging methods
Method
RoBERTa
Qwen
Weight Average
52.56
95.74
Twin-Merging Weight Average
96.23
100.08
Task-Arithmetic
67.80
98.52
Twin-Merging Task-Arithmetic
96.14
102.38
Ties Ters-Mggging
63.68
92.67
Twin-Merging Ties fies-Merging
96.34
102.35
22

---

To evaluate the compatibility of Twin-Merghh with other merging methods, we conducted exper- iments using different techniques to create a shared expert, followed by dynamically merging the twin vectors. The results in Table 11 demonstrate that our method integrates seamlessly with primary merging techniques, leading to significant improvements. For example, when combined with our approach, the baseline Weight Average method improves from 52.26 to 96.23 on GLUE, approaching the performance fine-tuned experts. Notably, our method complements Ties-Megging particularly well, suggesting that better isolation of shared knowledge enhances the overall performance of Twin-Merging.
E Inference Efficiency
Assume we have T tasks, the fine-tuned model have P= Py + Pa parameters, where Pj are frozen and Pa are activated.
Parameter Count and Storage Cost Assuming each float parameter uses 16 bits (either fp16 or 6) Fine-tuned models require Pa + Pς) bytes of storage. Pretrained models, including those using Weight Average, Task Arithmetic, Ties-Merging and DARE Merging techniques, each need 2P bytes of storage per model. For Twin-Merging, with the router having Pr parameters (P, < P) and a compression rate of k%, it need to store 27kPa - 2P + P, bytes including a shared expert, compressed exclusive task-specific vectors, and the router. We can select to compress the model matrix to rank 1 for best storage. These strategies enhance the accessibility and sustainability of task-specific models, fostering wider advancements and applications. Visual representations can be found in Figure 2a and Figure 4.
Computation FLOPs Analysis Our method mainly introduce the extra time cost due to routing and dynamical merging. However, as the inference process typically involves hundreds of forward passes (e.g., 300 tokens for summarization tasks), the additional computing is usually neglectable. Assuming context length 8, task number T, layer number m, the introduced FLOPs Multiply-accumulee operation can be computed mm24sh² hbs for routing,
+ 9h) for merging (excluding norm parameter), while generating I. tokens typically requires Σt= 24m(thh2 + 4b/2 FL.OPs. Given that n < 7. and are typically truncated, the additional consumption is neglectable in generation tasks. We demonstrate the actual time cost in Table 7, which adds only 0.039 seconds per sample while bringing significant performance improvements.
Comparison with other baselises Moreover, approach offers significant performance improve- ments with these additional computing resources. As shown in Table 2 and the "More Baseline" section, we achieve absolute normalized improvement of 28. 34.3%% for RoBERTa, 18.83 18.83% on ViT- B-32 compared to Task Arithmetic, 9.71% compared to Twin-Merging on Qwen- 14B. Traditional model merging methods often overlook the heterogeneous nature of test inputs, leading to substantial performance gaps. Advanced merging techniques like AdaMerging and Surgery typically require costly training and searching processes, demonstrated in Table 3. In contrast, our method achieves superior performance to fine-tuned models with minimal cost and storage requirements (47m22s vs. 215m01s, 5.0GB vs. 32.4GB) due to dynamic merging and SVD techniques.
Speedup variants Currently, our method supports batch inference for the routing process, while the dynamic merging process handles inputs sequentially. However, it is straightforward to extend our approach support merging in batches or groups. We can achieve this by first obtaining router weights in batch, then grouping similar data items using the following strategy: (1) Divide into Bins Based on Argmax Indices: First, we divide the data into several bins according the arg-max indices of the router logits. (2) Cluster Within Each Bin: Then, we cluster (by KMeans) within each bin to group the logits (we set the group number to 20). (3) Average Weights Within Each Group: Within each group, the router weights are averaged to obtain merged model. Fach group corresponds to one merged process, and the group size is typically larger than the batch size, making it very efficient. We have added group-wise experiment on RoBERTa to illustrate this;
23

---

Table 12: Performance of group-wir wise variant.
Method
Avg. Normalized Score
Time
Tast-Arit
67.80
4m52s
Twin- Twin-Merging
96.14
9m31s
Twin-Merging (group-wise, group number=20)
90.02
5m14s
Limitations and Future Work
Our approach shares common limitations with existing merging methods: (1) The underlying theory behind why and when weight interpolation works not fully understood, though recent works [51, 83] have made interesting observations about weight disentanglement and cross-task linearity. (2) Currently, merging is limited to models with the same architecture and it may be difficult to find suitable fine-tuned model with specific capacities.
Additionally, while our method focuses on shared and exclusive task-specific knowledge, providing a way approach fine-tuner mne-tuned model performance and potentially surpass it without additional training, we observe there may be other types of knowledge that remain unexplored: (1) Evil knowledge: Useless for any task and distracts the model, obscuring critical knowledge during merging. (2) Irrelevant knowledge: Has no impact on merging performance. Our experiments validate the existence of the irrelevant knowledge since we demonstrate that dropping 90% of parameters retains most of the fine-tuned performance, but we have not investigated evil knowledge. Future work may include further investigation and decomposing these different types of knowledge better ignite the model's s full potential without retraining.
G Broader Impacts
This paper presents work whose goal is advance the field of machine learning and model merg- ing research. In terms of positive social impact, twin-merging techniques can achieve multi-task performance foundation models without retraining expert models, significantly reducing com- putational and energy costs. Our proposed knowledge modularization and compression techniques make the task-specific enhanced model more accessible and sustainable, paving the way for broader applications and advancements in the field. These techniques effectively align unaligned models by leveraging experts, thus mitigating the harmfulness and biases present the original models. Addi- tionally, model merging allows the unified model to benefit from the strengths of each task-specific model, even for tasks with private or inaccessible data, enhancing commercial and safety benefits. However, improper merging of biased models may contaminate the merged model. This issue be addressed by merging a de-biat expert or using sparsity techniques to minimize the impact.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's 's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Appendix F provides a detailed report on our limitations and future work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
25

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section 3.3 provides the pseudo-code our method and Appendix provides the training and test details of each method we experiment with.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification: Our paper uses publicly available datasets and provides the complete code and execution scripts in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co oublic/guides/Codoe www.essiommosooll....
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: ctppubllclllccc httpc//i/ee//oeeeeeeeeeeeeeeeeee
wwwsssoloiciiooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Appendix D provides the training and test details of each method we experi- ment with.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The paper does not include error bars for the experiments. This omission is due to the high computational cost associated with calculating error bars for large language models. Additionally, the training-free algorithms used in the experiments are deterministic, ensuring reproducibility and consistency of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
27

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides computer resources and information needed to reproduce experiments in Appendix D.I.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Yes, the research conducted in the paper conforms in every respect with the
NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We analyze both the positive and negative societal impacts in the Appendix G.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
28

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the creators original owners of assets used in the paper properly cited and Appendix D.2 provides licensing information for datasets used in our work.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
29

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
We submit the code in the supplementary material with detail scripts and guidance.
Justification:
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with humans subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing or research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
30

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "M9UxMlSOv3", "reviewer": "Reviewer_vkK5", "summary": "In this paper, the authors introduce the Twin-Merging to merge language models, aiming to close the performance gap between conventional model merging techniques and fine-tuned models, while improving adaptability to data heterogeneity. By modularizing and dynamically merging shared and task-specific knowledge, the authors show that Twin-Merging outperforms existing model-merging methods and approaches the performance of fine-tuned models across various settings and domains.", "questions": "Please see the Weaknesses", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper is overall well written and easy to follow. The idea of dynamic merging to dynamically merge shared and exclusive knowledge based on the test inputs is interesting.\n\n+ The experiments show clear superiority over prior methods.", "weaknesses": "+ The proposed method dynamically merges the models with the varying inputs, which can be extremely time-consuming in practice. It would be better if the authors propose some mechanisms to address this issue.\n\n+ In figure 3, the authors compare 1-model merging with 8-model merging. Why here 8-model merging is used for comparisons? Why not 2-models merging is compared? More explanations should be provided here.\n\n+ Some highly related works are missing in the related work section, such as MuDSC[1]. The differences between these works should be clarified.\n\n[1] Training-Free Pretrained Model Merging, CVPR 2024\n\n[2] REPAIR: REnormalizing Permuted Activations for Interpolation Repair, ICLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer `vkK5`\n\n> Q1. The proposed method dynamically merges the models with the varying inputs, which can be extremely time-consuming in practice. \n\nPlease refer to the \"Inference Efficiency\" section of our global rebuttal. Our method achieves superior performance (95.33 vs 94.04) with less time and storage cost compared to AdaMerging and Surgery baselines  (47m22s vs 215m01s, 5.0GB vs 32.4GB). As shown in Table 7, our approach adds only 0.039 seconds per sample while improving performance by 28.34% for discriminative tasks. Additionally, our method supports common inference speedup techniques and offers efficient group-wise merging variants.\n\n> Q2. In Figure 3, Why 8-model merging is used for comparisons? Why not 2-models merging is compared?\n\nWe chose to highlight the 8-model merging scenario to illustrate the degradation gap that can occur in practical scenarios, as typically merging multiple models is more common. \nWe actually have demonstrated the performance for 2 to 8 model merging in the left figure of Figure 4. \n\n> Q3. More works (MuDSC and REPAIR) in relative works\n\nThank you for your suggestion. MuDSC and REPAIR can be categorized as Linear Mode Connectivity (LMC) based methods, which we have discussed in the relative work section (L79-L82). \nMuDSC addresses the issue of inconsistent similarity calculations in activation and weight spaces by designing a merging framework with dual-space constraints to ensure high similarity in both spaces between models. \nREPAIR addresses the problem of collapsed variance in activations during interpolation by rescaling pre-activations, thereby mitigating performance degradation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x1m72dMFPb", "reviewer": "Reviewer_Q3uE", "summary": "`Twin-Merging` proposes a method for task merging which tackles two issues:\n  * Task interference: The proposed `Twin-Merging` explicitly model shared vs task-specific knowledge to potentially reduces redundancies across the task vectors, which may lead to subpar task merging results\n  * Dynamic merging: Usually, task merging weights are only task dependent and determined only once. In contrast, here, the weights are determined at the input level using a router akin to Mixture of Experts design.", "questions": "* I do not fully understand the conclusion of **Section 3.1**:\n\t  * the assumption of `Ties Merging` is that redundant parameters can be hard to merge. But they do not really make assumptions on whether these parameters were trained jointly/with overlap > therefore I'm not sure how the LoRA experiments contradicts this insight: Even if the LoRA modules are trained separately, they may still have redundant statistics at the end of training ?\n\t  * Similarly, the notion of *similar tasks* or *task interference* is hard to define in general.  So it is not clear to me how significant the results of the XSUM/DailyMail experiment are.\n  * Based on the result of table 4.5 it looks like Twin Merging does not often behave as a model merging method ? It seems that most of the time the samples are routed to their respective task-specific experts. If that insight is true, it is not too surprising that `TwinMerging` performs on-par with the finetuned baseline.\n  * In Table 7: Shouldn't `Model Merging` and `Twin Merging` also integrates the cost of training the initial task vectors in **training cost** ? Otherwise it seems unfair to the MTL baseline.", "limitations": "The paper discusses limitations inherent to task merging in appendix F. However, as discussed in the Weaknesses section.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* Compressing the task vectors using SVD is beneficial for memory usage\n* Performance improves over standard  task merging approaches\n* Experiments are also conducted on large scale models (72B)", "weaknesses": "* **The per-input routing design seems highly impractical**.  If I understood correctly, the task merging weights are compute on-the-fly for each individual input. This means that:\n\t  * The design does not support batching as we need a different merged model for each input in the batch\n\t  * Every time a new input comes, we need to first get the merging weights by executing the fuser, then uncompress the task vectors, build the merged model, and finally run the inputs through it. This does not seem very hardware friendly, even if the task vectors are low-rank compressed using SVD. In contrast, in standard task merging, we only ship one merged model.\n\t  * Because the fuser $\\mathcal{R}$ takes as inputs the  *last-layer token embeddings from the shared expert* (line 186), does it mean that every input require two forward passes (one through the fuser, then one through the merged model) ?\n- The paper does not really convey the importance/novelty of **knowledge modularization** strongly enough. In essence, the basic task arithmetic (**A**) already performs some form of modularization where the pretrained model is the *shared knowledge* and task finetuned models (task vectors) are *task specific*. In contrast, knowledge modularization(**B**) **redefined the task vectors** relatively to the merged model (rather than the pretrained one) and compress them via SVD.  In my opinion, comparing **A** and **B** would be a better ablation experiment than the one in Table 6 where the merged expert is replaced by a random specific one.\n\n- **Baseline**: Since the proposed design uses additional validation data (to finetune the router), it would be fair to also compare to the more recent/stronger baseline `AdaMerging: # Adaptive Model Merging for Multi-Task Learning` (Yang et al), which also assumes extra data available.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer `Q3uE`\n\n> Q1. The design does not support batching\n\nWhile the router process supports batching, the dynamic merging process currently handles inputs sequentially. However, **it is straightforward to extend the merging process to support batching**. As detailed in the \"Inference Efficiency\" section of our global rebuttal, we can achieve this by clustering and rearranging data items based on the router logits into groups. This approach significantly reduces the number of merging operations required, with minimal impact on performance. By doing so, we maintain the benefits of our approach while efficiently processing inputs in batches.\n\n> Q2. Does every input requires two forward passes (through the fuser and through the merged model) ?\n\nFor the fuser, yes, it requires one forward pass to induce the merging weights. However, the merging model typically requires hundreds of forward passes for generation (e.g., 300 tokens for summarization). Therefore, the additional cost is typically negligible, referring to analysis in the \"Inference Efficiency\" section of the global rebuttal.\n\n> Q3. The basic task arithmetic (A) already performs some form of modularization where the pretrained model is the shared knowledge and task finetuned models are task-specific. Knowledge modularization(B) redefined the task vectors relative to the merged model and compressed them via SVD. Comparing A and B would be a better ablation experiment than the one in Table 6.\n\nThank you for your suggestion. We have revised the ablation study table as shown below: \n\n|Method|RoBERTa|Qwen|\n|-|-|-|\n|Pretrain+Dynamic Merging(A)|85.90|97.03|\n|**Shared+Dynamic Merging(TwinMerging,B)**|**96.14**|**102.38**|\n\nWe observe that A performs worse than B (85.90 vs 96.14), which can be attributed to two main reasons:\n- The pretrained model may contain relatively sparse shared knowledge that benefits the input tasks. In contrast, the shared expert, constructed by merging task-specific experts, contains more abundant and diverse shared knowledge.\n- Modulizing knowledge by subtracting the pretrained model does not effectively mitigate interference, as it does not consider exclusive knowledge specific to each task. This explains the performance gap between the task vectors and the fine-tuned experts, as analyzed in Section 3.2.\n\n> Q4. Compare to AdaMerging which assumes extra data is available.\n\nThank you for your suggestion. \nWe add AdaMerging which need extra validation data in the \"More Baseline\" Section of the global rebuttal.\nOur approach still outperforms them with less time costs (95.33>88.50). \n\n> Q5. They do not really make assumptions on whether these parameters were trained jointly/with overlap. therefore I'm not sure how the LoRA experiments contradicts this insight: Even if the LoRA modules are trained separately, they may still have redundant statistics at the end of training ?  \n\nTo clarify, our study focuses on the \"parameter interference\" phenomenon as defined by the Ties-Merging paper[1]. This refers to the conflict of parameters **at the same position across task experts**, e.g., the sign disagreements of the up-projection layer in different task models, which is the main focus of the Ties-Merging method.\nOur Section 3.1 experiment demonstrates that even when task-specific modules are trained without overlap thus merging without overlap, interference still occurs.\nThis indicates that Ties-Merging approach does not fully resolve parameter interference.\n\n> Q6. The notion of similar tasks/task interference is hard to define in general. So it is not clear to me how significant the results of the XSUM/DailyMail experiment are. \n\nWe use \"task interference\" from MTL literature [2] to describe the distinct nature of different task types. For instance, summarization, math reasoning, and code generation each require different forms of responses. Conversely, XSUM and DailyMail are both summarization tasks, handled similarly by the model. Our experiments showed that even similar task types experience interference, prompting us to explore finer-grained relationships between tasks, such as the knowledge types (Sec 3.2).\n\n> Q7. It seems that most of the time the samples are routed to respective task-specific experts. If that insight is true, it is not too surprising that TwinMerging performs on-par with the finetuned baseline. \n\nTo clarify, we **do not directly route samples to task-specific experts** (equivalent to the fine-tuned baseline in Table 2), which is highly impractical when facing unknown test distributions. \nInstead, we combine shared and exclusive knowledge, which leads to **even better performance sometimes**, e.g., averaging 102% over the fine-tuned baseline on generative tasks (Table 2) and 101% on COLA (Table 8), and better unseen generalization (Table 5). \nBy isolating different types of knowledge and composing them dynamically, we avoid redundancy and leverage complementary information from both shared and exclusive sources, resulting in improved performance.\n\n> Q8. Shouldn't Model Merging and Twin Merging also integrates the cost of training the initial task vectors in training cost ? Otherwise it seems unfair to the MTL baseline.\n\nWe did not include training time in Table 7 because merging methods can directly download task experts from Hugging Face/PyTorch Hub without post-training. However, to demonstrate, the training time and cost for custom fine-tuning from scratch are as follows:\n\n|Method|TrainingTokens|TrainingCost|Performance|\n|-|-|-|-|\n|MTL|536.35M|10h32min|94.31|\n|Task-Arithmetic|536.35M|10h32min|96.61|\n|Twin-Merging|536.92M|10h35min|102.38|\n\nOur approach shows an 8.5% performance improvement over MTL with only a 0.1% increase in training tokens and a 0.4% increase in training time.\n\n[1] TIES-Merging: Resolving Interference When Merging Models [NIPS23]\n\n[2] Mitigating Task Interference in Multi-Task Learning via Explicit Task Routing with Non-Learnable Primitives [CVPR23]", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "quQxIpOHgc", "reviewer": "Reviewer_2gMR", "summary": "This paper attempts to resolve an issue of destructive interference with model merging techniques. It proposes to maintain a shared base model and separate task-specific knowledge structures that can dynamically be combined at test time. \nThe paper presents some a nice to buttress the drive home their methodology", "questions": "* How are the $v_t$s represented / stored (Algorithm 1) ?\n    * Are the v_ts stored as the low rank matrices that are later multiplied to obtain the appropriate matrix size ?  As written, it seems like each $v_t$ is of size equal to the size of the full parameter space and so ends up as a memory burden.\n* For Figure 4 (right), is the fine-tuned storage size much larger because you are doing full fine-tuning instead of LoRA fine-tuning ?\n    * If you are doing LoRA fine-tuning is the gap because the rank of the lora matrices are larger >> than the rank of the final $v_t$s above ? \n* Maybe I missed this but where is the “(Best Storage)” option for Twin Merging in Table 2  described ?\n* For Table 4, would it be possible to provide the results for Twin-Merging only before showing the additive results ? It’s hard to make out whether the other methods are contributing anything at all", "limitations": "Broader impact statements and limitations are discussed in the paper", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Code provided\n- Initial analyses of the existence of interference even when separate parameter sets are fine-tuned with LoRA is interesting though a bit obvious in hindsight be useful to empirically validate\n- Initial experiments validate the hypotheses of interference and shared / exclusive knowledge\n- Reasonably thorough experimentation", "weaknesses": "1. My primary issue with the paper is why this is a viable alternative to just keeping the base model back-bone and LoRA low-rank vectors of similar rank to the $v_t$s (Algo 1) (and instantiating the task specific model on the fly).  Twin-Merge has memory overhead (v_t for T tasks) which brings it more into the realm of “saving task-wise adapters/LoRA” whose size grows as the number of tasks. It is unclear from the existing set of experiments why a practitioner would use this method over keeping separate task low-rank (small memory footprint) adapters and just using these at test time (either statically per task or dynamically — https://arxiv.org/pdf/2306.14870 ) ?\n    1. I understand that the paper performs a primary comparison to other model merging methods like DARE and Task Arithmetic but these methods have zero extra memory overhead (after the model is merged) — they don’t enjoy the expressivity that test time adaptation + extra memory gives them. \n2.  Claims of generalization to unseen tasks might not be valid. Specifically, it is interesting to see that the boost in performance for Twin Merging in Table 2 is much more significant than for Table 5 with unseen generalization. This makes me wonder if the deltas in table 5 are actually significant. \n\n\nI am open to raising my score if the authors can convince me of the utility of the method -- in light of [1] above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer `2gMR`\n\n> Q1. Why this is a viable alternative to just keeping the base model and LoRA low-rank vectors? Why we would use this method over keeping separate task low-rank adapters and just using these at test time ?\n\nTable 2 has proven that our knowledge modularization technique outperforms directly using task-specific LoRA at test time in generative tasks, exhibiting an average score of 102%.\nCompared to several common strategies to utilize task-specific adapters, Twin Merging is still a promising method to improve performance:\n-  **Directly Route to Top-1 Expert**, which is equivalent to the fine-tuned baseline in Table 2: This is the simplest but highly impractical approach, as it relies heavily on the router. It has the worst expected performance for out-of-domain data when the router cannot properly predict. **Twin Merging outperforms this method (102.38 > 100.0) and has superior performance on unseen tasks (Table 5)**, as it can benefit from complementary knowledge from different exclusive sources. Additionally, routing to the top-1 expert requires storing all experts, which can be a large storage requirement when used with fully fine-tuned models.\n-  **Combining Multiple Experts**: Without the isolation of shared and exclusive knowledge, combining multiple experts suffers from interference problems, as the redundancy of shared knowledge may obscure the key knowledge required for tasks, and conflicts between the exclusive knowledge are also unavoidable (analyzed in Section 3.2).\n    - *Statically Combining*: This is equivalent to \"Task Arithmetic\" when using LoRA. **Twin Merging outperforms static combination (102.38 > 96.61)**.\n    - *Dynamically Combining*: This is equivalent to the method (A) in the following table. **Twin Merging(B) outperforms dynamic combination (102.38 > 97.03)**.\n\n|Method|RoBERTa|Qwen|\n|-|-|-|\n|Pretrain+Dynamic Merging(A)|85.90|97.03|\n|Shared+Dynamic Merging(TwinMerging,B)|96.14|102.38|\n    \n> Q2. Baselines like DARE and Task Arithmetic don’t enjoy the expressivity that test time adaptation + extra memory gives them.\n\nWe add baselines like AdaMerging and Representation-Surgery that have extra time cost and memory consumption in the \"More Baseline\" section of the global rebuttal. \n**While consuming less time cost and memory consumption, our approach superior in performance (95.33 > 94.04 > 88.50)**. \nAs analyzed in the \"Inference Efficiency\" section of the global rebuttal, our method actually introduces neglectable cost compared to the total generation (0.039s per sample in Table 7), and can be further optimized by the group-wise merging.\n\n> Q3. Twin Merging Performance in Table 2 is much more significant than in Table 5 with unseen generalization.\n\nThis is because we present **unnormalized scores** in Table 5. We cannot directly normalize them as we do not have the corresponding expert on unseen datasets to get upper-bound performances (as noted in line L242). \nThis leads to relatively lower scores due to the narrower score ranges for tasks like RTE (max 66.43 vs max 91.71 for QNLI) and MMLU (max 68.03).\n\nIf we use the maximum score from Table 2 as the upper bound for normalization, \nwe observe more significant improvement for Table 5, e.g. 91.16 -> 96.98 for MMLU (We present the \"unstrictly-normalized\" scores in parentheses):\n\n|Method|QNLI+MNLI+RTE|MMLU|\n|-|-|-|\n|MTL|44.63(55.87)|63.74(93.69)|\n|Task-Arithmetic|53.92(67.42)|62.02(91.16)|\n|Twin-Merging|55.86(71.92)|65.98(96.98)|\n\nThe scores are lower than in Table 2 because the in-domain knowledge may be unrelated or even harmful to the test data, leading to a performance downgrade. \nHowever, our approach helps mitigate this effect by dynamically adjusting the merging weight.\n\n> Q4. How are the $v_t$ represented/stored? Are the $v_t$ stored as the low rank matrices? It seems size of $v_t$ is equal to the size of the full parameter space.\n\nWe do not directly store $v_t$ since it is the same size as the original parameter. As detailed in Appendix D5, we further compress $v_t$:\nGiven the size-$m$ decomposition $v_t = \\mathbf{U}_t \\mathbf{\\Sigma}_t \\mathbf{V}_t^T$, we select the top-$r$ singular values to form $\\mathbf{U}_t(r) \\mathbf{\\Sigma}_t(r) \\mathbf{V}_t(r)$. **We store only $\\mathbf{U}_t(r)$, $\\mathbf{\\Sigma}_t(r)$, and $\\mathbf{V}_t(r)$**.\n\nDuring merging, we decompress these matrices by extending $\\mathbf{U}_t(r)$, $\\mathbf{\\Sigma}_t(r)$, and $\\mathbf{V}_t(r)$ to size-$m$ by filling with zeros, allowing us to recover $v_t$ via their product. This operation is only at the matrix level; once we obtain the merged matrix, we discard the decompressed matrices, ensuring efficient storage.\n\n> Q5. For Figure 4, is the fine-tuned storage size much larger because you are doing full fine-tuning ? If you are doing LoRA fine-tuning is the gap because the rank of the lora matrices are larger than the rank of the final $v_t$ ?\n\nYes, we mainly show the full fine-tuning results for RoBERTa in Figure 4. \nAs analyzed in Appendix E, for LoRA fine-tuning, the typical rank is 32, but in our experiments, we can use a rank-1 for the best storage efficiency and still obtain good results (refer to Table 9), which is much smaller in storage.\n\n> Q6. Where is the \"Best Storage\" option  \n\nThe \"Best Storage\" option refers to the rank-1 compression via SVD, as illustrated in Table 8 and Table 9. The detailed storage analysis is provided in Appendix E.\n\n> Q7. Showing the additive results for the Ablation Study\n\nThanks for your suggestion. We show the additive style ablation as follows:\n\n|Method|RoBERTa|Qwen|\n|-|-|-|\n|Pretrain|41.69|91.06|\n|Shared|67.80|96.61|\n|Dynamic Merging|81.47|87.77|\n|**Shared+Dynamic Merging(Twin Merging)**|**96.14**|**102.38**|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gqGv6AJi6D", "reviewer": "Reviewer_LqLU", "summary": "This paper investigates a dynamic and compressive merging method for adapting large-scale models to multiple tasks. The authors claim that adjusting the ratio between shared knowledge and exclusive knowledge is crucial for high-performing model merging, and they devise an algorithm that learns proper coefficients to merge those two kinds of knowledge in a data-dependent manner. They further compress the task-specific exclusive knowledge via SVD to reduce memory storage during merging. They validate their method with language models on discriminative and generative tasks.", "questions": "- details on router network architecture and its training (refer to weakness section)\n- details on the validation set construction (refer to weakness section)\n---\nIf there is a misunderstanding from me, please don't hesitate to refute it. I would be happy to discuss this further.", "limitations": "Lack of scalability to larger models due to increasing inference cost of instance-dependent dynamic merging.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Remarkable performance improvement compared with baselines\n- The proposed dynamic merging method can be memory-intensive during inference time because it requires all task-specific modules to be stored, but the author mitigates this somewhat via SVD compression. The combination of dynamic merging and compressive merging is very impressive, which induces better performance while considering practical usefulness", "weaknesses": "- Lack of technical details on the most important part of the methodology\n    - The router module plays a crucial rule that produces input-dependent merging coefficients during inference time. However, the authors do not provide any technical details on the training of the router module (including appendix).\n        - Model architecture and training configurations should be provided.\n    - Moreover, a description of the validation set the authors to use for the router training should be provided. The authors only describe that dataset as '**_a small validation set_**'. Does the validation set consist of the integration of downstream tasks? or general text corpus? and how the construction of the validation set affects the final merged model performance.\n        - I believe details on the construction and amount of samples for the validation set should be presented.\n- Limited scope of validation\n    - While they evaluate their method with fully fine-tuned models on discriminative tasks, they only validate it on the parameter-efficient fine-tuning regime for the generative tasks. As one of the proposed method's main contributions is parameter compression, it would be more effective when they show the applicability of their method on fully fine-tuned model merging with decoder-only LM or encoder-decoder LM.\n- Concerns about scalability\n    - While the compressive merging reduces the requirements of memory storage during the inference phase, I speculate that the amount of computation for the merging operation is still a huge burden. The input-dependent dynamic merging requires computations for merging operation per every sample, and I wonder about the **runtime and computation cost during inference phase when the fully fine-tuned models are the target of validation** rather than the parameter-efficient fine-tuning regime in Table 7.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer `LqLU`\n\n> Q1. Lack of technical details on the router (Model architecture and training configurations ), validation set for router training. Does the validation set consist of the integration of downstream tasks? or general text corpus? \n\nThanks for your suggestion, we will revise our paper in the next version.\nOur router is implemented as a three-layer linear network with Leaky ReLU activations and batch normalization. \nWe train the router on the validation dataset with a learning rate of 5e-4 for 10 epochs.\n\nThe validation set consists of the integration of in-domain downstream tasks, not the general text corpus.  The validation set is taken from a split of the training set, and we use at most 1,000 items for router training for each task. \n\n> Q2. how the construction of the validation set affects the final merged model performance.\n\nAs for how the construction of the validation set affects the final merged model performance, if the dataset is too small or imbalanced, it can affect the router's precision and degrade the merging performance. Therefore, we typically ensure the sample numbers from different tasks are the same. Practically, using 1,000 items  (which is about 2% of the discriminative task test sets) is sufficient to achieve comparable performances.\n\n> Q3. They only validate it on the parameter-efficient fine-tuning regime for the generative tasks. They should show the applicability of their method on fully fine-tuned model merging.\n\nWe primarily merge using LoRA for the Qwen-14B models because fully finetuning them to obtain task-specific experts would require a huge amount of resources and computation (finetuning the full 14B model requires at least 8 A100 GPUs). \nHowever, to further demonstrate, we provide experiments on fully fine-tuned LLaMA 7B models for generative tasks (gsm8k and truthfulqa), where our approach still exhibits superior performance. \n\n| Method |  avg. normalized score | Inference Time (/1000 items)|\n|-|-|-|\n| Task-Arithmetic  | 69.89  |  186s  | \n| Twin-Merging  | 88.18  |  198s  |\n\nWe have also verified the effectiveness of our method on fully fine-tuned RoBERTa (as shown in Table 2) and ViT-B/32 models (in the global rebuttal section).\n\n> Q4. I speculate that the amount of computation for the merging operation is still a huge burden. The input-dependent dynamic merging requires computations for merging operation per every sample. I wonder about the runtime and computation cost during the inference phase when the fully fine-tuned models are the target of validation rather than the parameter-efficient fine-tuning regime in Table 7.\n\nPlease refer to the analysis in the \"Inference Efficiency\" section of the global rebuttal. Our approach introduces negligible cost in typical generation scenarios and can be easily optimized by group-wise merging. \n\nAs for runtime and computation costs for fully fine-tuned models, please refer to the LLaMA 7B time cost in the response to Q3 (186s->198s) and the ViT-B/32 results in the global rebuttal (47m22s vs 215m01s).  \nFrom these tables, we can see that our method introduces only an extra 0.01 seconds per sample for LLaMA7B and is faster than methods like AdaMerging and Surgery, which also improve performance through additional training and modules.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Rebuttal

Thank all four reviewers for their constructive feedback which has helped us to improve the clarity and contribution of our work.
The below contains a rebuttal for remarks that are common to most reviewers.

## 1. More Baseline (To `Q3uE`, `2gMR`)

To compare with baselines that require additional datasets and test-time adaptation, we add experiments on ViT-B-32 for 8 CV tasks merging (SUN397 Cars RESISC45 EuroSAT SVHN GTSRB MNIST DTD) following the AdaMerging paper [1] and Surgery [2].  

| Method |  Avg. Normalized Score | Additional Time Cost. | VRAM |
|-|-|-|-|
| **Pretrained**| 52.02 | 18m48s | 3.6GB |
| **Fine-tuned**| 100.00| 18m48s | 28.8GB | 
| **Weight Averaging**| 72.30 | 18m50s | 3.6GB |
| **Task Arithmetic** | 76.50 | 21m34s | 3.6GB |
| **Ties-Merging** | 75.10 | 19m24s | 3.6GB |
| **AdaMerging**| 88.50 | 185m35s | 3.6GB |
| **Surgery**| 94.04 | 215m01s | 32.4GB |
| **Twin-Merging (Ours)**| **95.33** | 47m22s | 5.0GB |

We use the best version from AdaMerging and Surgery, and the 90% sparsity for our twin Merging.
AdaMerging introduces task-wise or layer-wise learnable parameters to improve the merging performance, 
while Surgery adds post-merging task-specific modules to shift representation towards the input tasks. 
They both need to be trained on the eight task val set for a long time. 
Moreover, Surgery needs to know the task type before inference and requires eight finetuned models and the merged model forward during the merging process, thus exhibiting large VRAM.
**In contrast, our method can robustly handle heterogeneous test inputs and has very efficient storage, exhibiting minimal time cost.**

## 2. Inference Efficiency (To `vkK5`, `Q3uE`, `2gMR`, `LqLU`)

- Currently, our method supports batch inference for the routing process, while the dynamic merging process handles inputs sequentially. However, **it is straightforward to extend our approach to support merging in batches or groups**. We can achieve this by first obtaining router weights in batch, then grouping similar data items using the following strategy:
    1. Divide into Bins Based on Argmax Indices: First, we divide the data into several bins according to the arg-max indices of the router logits.
    2. Cluster Within Each Bin: Then, we cluster (by Kmeans) within each bin to group the logits (we set the group number to 20).
    3. Average Weights Within Each Group: Within each group, the router weights are averaged to obtain a merged model. Each group corresponds to one merged process, and the group size is typically larger than the batch size, making it very efficient.
        We have added a group-wise experiment on RoBERTa to illustrate this:
 
        | Method|  Avg. Normalized Score  | Time  |
        |-|-|-|
        | Task-Arithmetic | 67.80 |  4m52s  | 
        | Twin-Merging | 96.14 |  9m31s  |
        | **Twin-Merging (group-wise)** | 92.02 |  5m14s  | 

- We acknowledge the extra time cost due to routing and dynamical merging. However, as the inference process typically involves hundreds of forward passes (e.g., 300 tokens for summarization tasks), the additional computing is usually neglectable. Assuming context length $s$, task number $T$, layer number $m$, the introduced FLOPs ( Multiply–accumulate operation ) can be computed as $m(24sh^2 + 4bs^2h)$ for routing, $Tm(12h^2 + 9h)$ for merging (excluding norm parameter), while generating $L$ tokens typically requires $ \sum_{l=s}^{L} 24m(lh^2 + 4 bl^2h)$ FLOPs. **Given that $n \ll L$ and $s$ are typically truncated, the additional consumption is neglectable**. 
We demonstrate the actual time cost in Table 7, which adds only 0.039 seconds per sample while bringing significant performance improvements.

- Moreover, our approach offers significant performance improvements with these additional computing resources. 
As shown in Table 2 and the "More Baseline" section, we achieve an absolute normalized improvement of 28.34% for RoBERTa, 18.83% on ViT-B-32 compared to Task Arithmetic, 9.71% compared to Twin-Merging on Qwen-14B.
Traditional model merging methods often overlook the heterogeneous nature of test inputs, leading to substantial performance gaps. 
Advanced merging techniques like AdaMerging and Surgery typically require costly training and searching processes, as demonstrated in the "More Baseline" section.
**In contrast, our method achieves superior performance to fine-tuned models with minimal cost and storage requirements (47m22s vs 215m01s, 5.0GB vs 32.4GB) due to dynamic merging and SVD techniques.**

- Furthermore, after merging, inference uses a single model per batch. This allows us to leverage optimizations like KV cache, Group Query Attention, efficient FFN/attention, and model compression techniques. Our method is also compatible with inference engines like FlashDecoding, DeepSpeed, and vLLM. 

[1] AdaMerging: Adaptive Model Merging for Multi-Task Learning [ICLR24]

[2] Representation Surgery for Multi-Task Model Merging [ICML24]

Meta-review: The work proposes Twin-Merging, a two-stage method for model merging: (1) modularizing knowledge into shared and exclusive components, with compression to reduce redundancy and enhance efficiency; (2) dynamically merging shared and task-specific knowledge based on the input. During testing, Twin-Merging dynamically merges shared and compressed specialized knowledge based on test inputs to form the final inference model. Compared to conventional merging methods, the new method better addresses adaptability to data heterogeneity and therefore further bridges the gap between merged models vs. fine tuned models. 

The final recommendations from the reviewers are unanimously acceptance (5, 6, 6, 6).

The paper is well written. 
The proposed method is technically sound. 
The method is verified by quite extensive experiments.

The two major common concerns raised by most reviewers are more baselines and discussion/evaluation over inference efficiency. Both have been addressed in the rebuttal. Please incorporate the new results and discussions in the final version.
Decision: Accept (poster)