Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging

Authors: zhenyi  lu
chenghao  fan
wei  wei
†  xiaoye qu
dangyang  chen
yu  cheng
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 09:03:32

Abstract: In the era of large language models, model merging is a promising way to combine multiple task-specific models into a single multitask model without extra training. 
However, two challenges remain: (a) interference between different models and (b) heterogeneous data during testing. Traditional model merging methods often show significant performance gaps compared to fine-tuned models due to these issues. 
Additionally, a one-size-fits-all model lacks flexibility for diverse test data, leading to performance degradation. 
We show that both shared and exclusive task-specific knowledge are crucial for merging performance, but directly merging exclusive knowledge hinders overall performance. 
In view of this, we propose Twin-Merging, a method that encompasses two principal stages: 
(1) modularizing knowledge into shared and exclusive components, with compression to reduce redundancy and enhance efficiency; 
(2) dynamically merging shared and task-specific knowledge based on the input. 
This approach narrows the performance gap between merged and fine-tuned models and improves adaptability to heterogeneous data. 
Extensive experiments on $20$ datasets for both language and vision tasks demonstrate the effectiveness of our method, showing an average improvement of $28.34\%$ in absolute normalized score for discriminative tasks and even surpassing the fine-tuned upper bound on the generative tasks.

Main paper: 
 introduction: 
 In recent years, Large Language Models (LLMs) have demonstrated notable success across various Natural Language Processing (NLP) tasks [12, 16, 43, 61-63, 65, 68] , including code generation [22, 56] , solving math problems [2, 44] , multilingualism [47] , etc. These models, with billions of parameters, excel in various downstream tasks [25, 34, 72] but require extensive training on large datasets using thousands of GPUs. The considerable computational and energy costs [53] limit their specialization and deployment in resource-constrained environments [38] .To tackle this challenge, model fusion has emerged as a promising solution [37] . One notable paradigm is model merging [29, 33, 76, 78] , where multiple task-specific models, or "experts", are combined into a single unified model. This unified model can quickly adapt to new tasks without the need to retrain a large model. Various techniques, such as parameter averaging [6, 74] , weight Figure 1 : Subfigure (I) shows that in conventional merging methods, parameters from different task-specific models and a pre-trained model are weighted-summed into a single multitask model for inference. Subfigure (II) illustrates that our Twin-Merging method first isolates shared knowledge, then extracts exclusive knowledge by identifying differences between task experts and the shared model. This exclusive knowledge is then compressed into sparse vectors. Subfigure (III) shows that during testing, Twin-Merging dynamically merges shared and compressed specialized knowledge based on test inputs to form the final inference model. interpolation [33, 46] , and advanced strategies like task arithmetic [29, 51, 67, 78] , have been developed for model merging. These techniques have been proven effective, enabling the integration of fine-tuned knowledge from diverse tasks into a multi-task model without additional training.However, merging models from different domains often sacrifices specific task performance, leading to a large performance gap compared to the individual expert [31, 76] . Two major causes prevent the existing merging methods from reaching the theoretical upper-bound performance of individual experts: (1) Interference between models. Previous research shows that parameter redundancy and sign discrepancies [76] , as well as the distribution gap between tasks [31] , hinder effective model merging. We demonstrate that task-specific models often contain mixed knowledge, where the expertise in one model may be exclusive or detrimental to others. This redundancy or interference can obstruct the integration of expertise across models [9] . (2) heterogeneity of data at test time. Previous methods pursue a single, static optimal solution for various tasks. While a one-size-fitsall model avoids introducing new parameters, it might be inadequate or suboptimal due to the unpredictable nature of test inputs [78] . It limits the utilization of complementary knowledge and leads to deteriorated performance [71] .To address the above issues, in this paper, we introduce Twin Merging, involving two principal stages:(1) Knowledge Modularization: Unlike previous research that migrates merging interference in a parameter-wise manner or searches merging coefficients, we decompose the knowledge possessed by experts into shared knowledge and exclusive task-specific knowledge, as shown in Figure 1 (II). First, we compress common knowledge into a shared expert, serving to capture and consolidate common knowledge across varying tasks. Then we isolate exclusive knowledge based on the difference between the task experts and the shared expert, allowing diverse knowledge to be decomposed more finely. (2) Dynamic Merging: Inspired by Mixture of Experts (MoE) [80, 84, 85] , we simplify the parameter merging problem into a conditional composition problem. Instead of pre-determining the best parameter combination for heterogeneous data at test time, as illustrated in Figure 1 (III), we introduce a router to dynamically merge shared and exclusive knowledge based on the test inputs. The shared model serves as the foundation, and task-specific knowledge is conditionally injected according to the router.We demonstrate the effectiveness of our proposed Twin-Merging method through extensive experiments on 12 datasets, covering both discriminative and generative tasks, various model architectures, and in-domain and out-of-domain setups. As shown in Figure 2b , Twin-Merging consistently outperforms other merging methods across all datasets, surpassing the strongest baseline by an average of 28.34% in normalized scores for discriminative tasks and 3.86% for generative tasks on the scaled model (Qwen-14B). We validate the scalability, extensibility, generalization, and storage efficiency of Twin-Merging (Figure 2a ). Remarkably, even with a 99.9% reduction in parameters, our method only experiences a slight 14% performance degradation. Our results establish Twin-Merging as a powerful and effective method for combining multiple fine-tuned models into a single multi-task model.To summarize, our contributions are as follows: (1) We introduce Twin-Merging, a novel model fusion method that reduces the performance gap between traditional model merging and fine-tuned models while enhancing adaptability to diverse data. (2) We investigate the impact of shared and exclusive task-specific knowledge on merging performance, presenting innovative techniques for Twin-Merging (Ours) (a) The average performance on generative tasks vs. the number of parameters of Twin-Merging compared to various merging baselines, with different storage sizes indicated by circle size. knowledge disentanglement and dynamic merging. (3) Twin-Merging is simple to implement with minimal hyperparameters, improves multi-task performance without retraining expert models, and can be combined with other merging methods for further gains. Our approach scales well with model size and task numbers and is storage-efficient. 
related work: 
 In this section, we focus on model merging research, for additional related work on multi-task learning and Mixture of Experts, please see Appendix B. Model merging aims to fuse multiple fine-tuned taskspecific models into one comprehensive multi-task model without additional training. FisherMerging [46] and RegMean [33] , use straightforward weight averaging but require extra data and computation. Some works [1, 21, 58, 60, 70] bring models into a single low-loss basin and interpolate between them based on the linear mode connectivity (LMC) theory [15, 18, 20] . The weight permutations [1] and optimal transport [58] are utilized to better interpolate neural networks. However, recent studies [83] suggest that LMC might not always hold for fine-tuned models. Task-Arithmetic [28, 51] extends averaging to arithmetic operations in the parameter space for finer control over model behaviors, but the interference between the multiple models can be an issue. To tackle this challenge, advanced merging methods like Ties-Merging [76] , AdaMerging [78] and DARE [79] have been proposed. These methods aim to reduce task conflicts by addressing parameter redundancy or disagreements in signs, finding optimal merging coefficients, and reducing weight density, respectively. Jiang et al. [32] assume that test tasks are known and use task-specific knowledge to improve performance. However, this assumption is often unrealistic since real-world data distributions are unpredictable. In contrast, our method addresses merging interference by modularizing shared and task-specific knowledge. We handle heterogeneous test data scenarios by introducing dynamic merging techniques. 
analysis of the performance gap in model merging: 
 In this paper, following the settings of model merging [29, 76, 79] , we consider the case of T tasks, where training for each task t starts from pre-trained model weight θ 0 and fine-tunes on D EQUATIONHere F represents an arbitrary merging function. For example, in Task Arithmetic [28] , Fine-tuned Epoch Although existing merging methods, like Task Arithmetic, can combine multiple task-specific models efficiently, they often exhibit significant performance gaps compared to single-task models. Prior research, such as Ties Merging [76] , attributes this phenomenon to parameter interference. This term refers to the redundancy or sign discrepancies found in parameters located at the same position (e.g., self-attention weights) across different task models, which in turn result in information conflicting and performance loss. Additionally, task interference, as noted in multi-task learning literature [13, 31] , arises from the inherent differences between tasks. For instance, tasks such as summarization, mathematical reasoning, and code generation require the model to process information in distinct ways. These differences worsen interference when models trained on different tasks are merged.θ * = θ 0 + T t=1 γ t (θ t -θ 0 ).To understand these performance drops, we conducted two experiments using Task Arithmetic. First, we fine-tuned Qwen-14B with LoRA, assigning non-overlapping modules to avoid parameter interference. Despite this, a 3.21% drop in performance occurred, indicating persistent interference. Second, using two similar summarization tasks (XSUM and DailyMail), we observed an 8.42% drop compared to individually fine-tuned models, confirming that interference persists even between similar tasks. These results suggest that interference in model merging is not limited to parameter-wise and task-wise issues.To tackle the challenge of interference, we examine the merging process at a finer-grained knowledge perspective. We identify two types of critical knowledge: (1) Shared knowledge, which benefits multiple tasks, and (2) Exclusive knowledge, which is useful only for a specific task. Single-task models often contain both types, complicating the merging process and leading to interference. To validate our hypotheses, we conduct experiments that vary the ratio of task-specific and shared knowledge.To examine the impact of shared knowledge, we conducted full fine-tuning on each model for its specific task. Excessive fine-tuning epochs can lead to catastrophic forgetting [19] , a phenomenon where the model retains task-specific knowledge but loses general knowledge. As the fine-tuning epochs increase, the shared knowledge gradually decreases. The top section of Figure 3 illustrates that as the epoch count increases, merging performance significantly deteriorates, even though the fine-tuned model performs well on its task. This underscores the crucial role of shared knowledge in merging performance.To explore the impact of exclusive knowledge, we merge a single task-specific model into the base model. We apply a sparsity method (e.g., SVD) to reduce the ratios of task-specific weights in the merging model from 100% (standard merging) to 0% (base model). As shown in the lower part of Figure 3 , performance remains stable up to 90% sparsity. Notably, even with a 99% sparsity rate, a single-merged model outperforms multi-model merging, confirming the existence of exclusive knowledge, which is more pronounced with more models. This also underscores the value of unmerged task-specific knowledge, since the fine-tuning performance can be effectively restored by preserving unmerged task-specific information.To summarize, both shared knowledge and un-merged task-specific knowledge play a vital role in merging performance. The exclusive nature of task-specific knowledge hinders the effectiveness of merging methods. Different types of knowledge need to be separated and modularized to achieve optimal performance. Thus, the first step of our Twin-Merging approach is to explicitly partition the weights into an expert containing shared knowledge and weights holding task-exclusive knowledge before merging. Formally, we denote the shared expert as θ s and the exclusive task-specific knowledge as {v t } T t=1 , the detail of our method is illustrated in the following section.Require: language model f (x; θ), pre-trained weight θ0 and T task-specific fine-tuned weights {θt} T t=1 , trained router R parameterized by a full-connect layer ϕ, embedding Emb, compression rank r and pre-specified weight {γt} T t=1 Our proposed Twin-Merging employs two main stages: knowledge modularization and dynamic merging. These stages are designed to narrow the performance gap and enhance adaptive knowledge composition. Building on the formulation in Equation (2), Twin-Merging preprocesses experts into shared experts, isolates and compresses exclusive knowledge into vectors, and dynamically composes them during inference.The preprocess stage comprises three steps:(1) Shared Expert: To separate shared knowledge across different models, we consider the pre-merged model as a natural placeholder to encapsulate common knowledge that is important to all tasks (denoted as θ * ). By leveraging established merging techniques such as Task Arithmetic, we can readily extract the shared experts from the initial merged model. (2) Exclusive Knowledge: To convey task-specific information while separating common knowledge, we calculate the difference vector: v t = θ t -θ * . This subtraction vector preserves un-merged task-specific information while discarding the shared knowledge. (3) Compressed exclusive vectors: For practical use and distribution, we apply singular value decomposition (SVD) to further compress the above exclusive knowledge into vectors for each task. Assuming v t has a rank-m decomposition, v t = U t Σ t V T t , we achieve a low-rank task space by selecting the top-r singular values, resulting in U t (r)Σ t (r)V t (r) T . We store only U t (r), Σ t (r), V t (r) T .In inference stage, adapting to unforeseen challenges is difficult, especially with varied test data. For example, if most of the data consists of a certain type (denoted as D u ), we should tailor the merged model for that specific task to get the best results. Instead of pre-defining the best parameters, we propose a new approach that combines shared expertise with exclusive knowledge. Our method involves using the input x to dynamically adjust to the current data, enabling us to utilize shared knowledge and apply specialized expertise based on the inputs.EQUATIONDuring inference, we fine-tune a small fuser R parameterized by ϕ through empirical risk minimization on a small validation dataset. This fuser, trained to dynamically select the specific task experts, replacing the need for complex optimization algorithms to determine fusion coefficients. The merging model is obtained by:EQUATIONHere, Emb(x) represents the sequence of the last-layer token embeddings from the shared expert f (x; θ s ). 
merging experiment: 
 Baselines We first compare Twin-Merging with several train-free model-merging methods on both discriminative and generative NLP benchmarks, including weight averaging, Task Arithmetic [28] , Ties-Merging [76] , and DARE Merging [79] . To compare with Merging methods that need validation dataset, we also conduct experiments on CV tasks with AdaMerging [78] and Surgery [77] . Details on these baselines are provided in Appendix D.Benchmarks For language discriminative tasks, following [76, 79] , we use RoBERTa [42] as the backbone and evaluate on the 8-task GLUE benchmark [69] . More details are in Appendix D.2. For language generative tasks, we use Qwen-14B [3] as the primary model to demonstrate the effectiveness of our approach on large-scale language models. To reduce deployment costs, we utilize task-specific checkpoints fine-tuned with the LoRA method [26] (See Appendix A for details on adapting Twin-Merging to LoRA). We evaluate our model on four scenarios: general knowledge (MMLU benchmark [24] ), factualness (TruthfulQA [40] ), safety (BBQ [52] ), and summarization (CNN-DailyMail [48] ).For vision tasks, following AdaMerging [78] , we use ViT-B/32 in CLIP [55] as the backbone on eight image classification datasets: SUN397 [75] , Stanford Cars [35] , RESISC45 [5] , EuroSAT [23] , SVHN [50] , GTSRB [59] , MNIST [10] , and DTD [7] . We employ the best version of AdaMerging (layer-wise AdaMerging++) and the Surgery (AdaMerging version), and apply 90% sparsity for our Twin-Merging. Detailed information is provided in Appendix D.2.Metrics We include individually fine-tuned models and the pre-trained model as upper and lower bounds on performance, respectively. To mitigate the effects of different task-specific score ranges, performance is assessed using the average normalized score of the fine-tuned models. The normalized score of merged model θ * is calculated as: Main Results Table 2 presents the results for all discriminative and generative benchmarks for language tasks, while Table 3 provides the results for vision tasks. A comparison of each task is illustrated in Figure 2b , with detailed statistics provided in Table 8 and Table 9 in the Appendix D.7.EQUATIONWe also list the full-finetuned LLaMA results in Appendix D.7.For discriminative tasks, it approachs the upper bound of finetune performance in the GLUE benchmark. Specifically, our methods improve over Task Arithmetic by 28.34%, Ties-Merging by 32.46%, and DARE-Merging by 30.56% in absolute normalized score. In Figure 2b , we observe that especially on the COLA task, where conventional merging methods fail to improve the result, our approach can still approach the upper bound of the COLA expert.On generative tasks, Twin-Merging achieves strong performance, outperforming Task Arithmetic and DARE Merging by 5.77% and 3.86%. Two interesting insights emerge: (1) The performance gains for Qwen-14B in generative tasks are smaller compared to RoBERTa in discriminative tasks. This indicates that smaller models like RoBERTa gain more from task-specific knowledge, while large models like Qwen-14B perform well because its strong general knowledge. ( 2)Twin-Merging surpasses the upper bound set by fine-tuned experts on the generative benchmark. This may be due to the extensive knowledge in Qwen-14B, where modularization and dynamic merging unlock further potential without additional fine-tuning. These findings highlight a promising path for improving large language models without retraining.For vision tasks, Twin-Merging outperforms the AdaMerging and Surgery baselines with a higher accuracy (95.33% vs. 94.04%) while being more efficient in time and storage (47m22s vs. 215m01s, 5.0GB vs. 32.4GB). AdaMerging uses task-wise or layer-wise learnable parameters to improve merging, and Surgery adds task-specific modules after merging, requiring training on the validation set for all eight tasks. Surgery also needs prior knowledge of the task type before inference and involves multiple forward passes, leading to high VRAM usage. In contrast, our method efficiently handles diverse test inputs with minimal time and storage costs. Scalability of Twin-Merging Our method remains effective with scaled models (e.g., 72B parameters), as shown in Table 4 . To manage high deployment costs, we limited our evaluation and merged experts to two tasks: BBQ and TruthfulQA. Twin-Merging consistently surpasses scaled pre-trained models and Task Arithmetic, highlighting our approach's scalability. Additionally, our method can be easily integrated with other merging methods, as detailed in Appendix D.9, making it both extensible and scalable.As shown in Table 5 , Twin-Merging method benefits from complementary collaboration among different experts. Since the corresponding task-specific experts are unavailable, we directly use the average of the unnormalized scores as the metrics. In the GLUE benchmark, when QNLI, MNLI, and RTE experts are absent, our approach still outperforms traditional baselines. Details on the expert combination for QNLI can be found in Figure 5a . For complex tasks like MMLU, which involves multiple-choice QA tasks across 57 categories, Twin-Merging demonstrates superior performance using the combined knowledge from TruthfulQA, BBQ, and CNN-DailyMail domains. Subtracting the pretrained model fails to fully consider exclusive knowledge specific to each task, leading to interference, as analyzed in Section 3.2.Discussion 1 We find that removing dynamic experts severely impacts RoBERTa but has less effect on Qwen-14B, suggesting that smaller models rely more on task-specific biases, while larger models benefit more from general shared knowledge. This indicates that our method adapts effectively to the varying knowledge requirements of models of different sizes.Discussion 2 Compared to simpler routing methods like direct route to task-specific expert or combining multiple experts based on multiple LoRA [27, 81] , Twin Merging delivers better performance, especially on unseen tasks, by reducing interference and leveraging complementary knowledge.Direct route to task-specific expert refers to the fine-tuned baseline in Table 2 . This approach assumes perfect routing and the absence of out-of-domain data, where each task uses its own dedicated expert. It represents the ideal scenario and serves as an oracle baseline to highlight the performance gap for merging methods. Despite this, Twin Merging still improves performance on generative tasks (102.38 vs. 100.0) and unseen tasks (Table 5 ) by leveraging different sources of exclusive knowledge. Moreover, this baseline demands storing all task-specific experts, which significantly increases storage, as discussed in Section 4.6.In combining multiple experts, the lack of separation between shared and exclusive knowledge leads to interference, as conflicts between exclusive knowledge are inevitable (Section 3.2). There are two ways to combine experts: (1) Static Combination: This is akin to "Task Arithmetic" in LoRA ( In the left panel of Figure 4 , we examine the impact of the number of tasks on model merging performance. Conventional model merging methods degrade notably, especially with many tasks, nearly reaching pre-trained levels. However, Twin-Merging consistently outperforms other methods, approaching fine-tuned performance, with greater gains as the task count rises.The right panel of Figure 4 shows the performance-storage trade-offs. While model merging methods have a constant storage cost, their performance remains low. In contrast, maintaining individual task-specific models guarantees strong performance but requires excessive storage. Twin-Merging achieves nearly 100% normalized accuracy across various tasks, balancing performance and storage efficiency by maintaining task-specific parameters with shared experts. This makes Twin-Merging a viable solution for scenarios demanding a balance between performance and storage efficiency. Figure 5 shows the results of routing decisions among experts for the QNLI dataset and four generative benchmarks. As shown in Figure 5a , the router maximizes the use of limited expert knowledge to address QNLI, a task where the goal is to determine if the context sentence contains the answer to the input question. For example, with only v CoLA and v SST-2 available, the router primarily uses v CoLA , which provides knowledge of sentence and word relations, while v SST-2 is focused on irrelevant sentiment classification. With six experts ranging from v CoLA to v MNLI , the router mainly leverages v MNLI for textual entailment and v QQP for question-answering capabilities. When v QNLI is included, the router naturally relies on QNLI-specific knowledge. These results demonstrate the flexibility and adaptability of our Twin-Merging method, providing good interpretability. For larger models like Qwen-14B, as shown in Figure 5b , the router plays a crucial role in selecting and combining specific knowledge. When experts have overlapping task-specific knowledge, such as v TruthfulQA and v MMLU , the router may assign them similar weights.Compression Analysis In the left panel of Figure 6 , we explore sparsity rates from 0% to 100%. Appendix E attachs detail qualtivie analysis of various Merging methods. Remarkably, our Twin- Merging method maintains 86.4% performance even at a 99.8% compression rate. This suggests that performance relies on a small fraction of task-specific parameters, aligning with previous findings [76, 79] . Our results also validate our hypothesis that redundant parameters can obscure critical knowledge, leading to performance degradation. Consequently, we primarily use a 90% sparsity rate in our experiments to preserve performance while reducing storage costs. We also conducted an ablation study on sparsity methods, shown on the right side of Figure 6 . SVD better retains task-specific information compared to Magnitude [76] and Bernoulli Dropout [79] . As SVD is applied only once during preprocessing, it does not become an inference bottleneck. More detailed analysis and results are provided in Appendix E. In summary, our approach strikes a better balance between computational cost and performance. 
conclusions: 
 In this paper, we introduce the Twin-Merging to merge language models, aiming to close the performance gap between conventional model merging techniques and fine-tuned models, while improving adaptability to data heterogeneity. By modularizing and dynamically merging shared and task-specific knowledge, Twin-Merging significantly outperforms existing model-merging methods and approaches the performance of fine-tuned models across various settings and domains. Our study highlights the impact of shared and exclusive task-specific knowledge on merging performance. We show that Twin-Merging benefits even strong scaled models like Qwen-72B, which already perform well across domains. It extends to more tasks and merging methods, demonstrating better generalization on unseen data. By utilizing SVD, our solution retains 86% of the performance with only 0.1% of the parameters, approaching upper-bound performance with minimal storage increase as tasks grow, achieving a better tradeoff between computation and performance. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We analyze both the positive and negative societal impacts in the Appendix G.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All the creators or original owners of assets used in the paper are properly cited and Appendix D.2 provides licensing information for datasets used in our work.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• We recognize that the procedures for this may vary significantly between institutions locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Here, we will demonstrate that our Twin-Merging method can be seamlessly applied to LoRA module [26] , where the base model is fixed and additional task-specific information is injected through matrix, i.e., θ t = θ 0 + LoRA t , where LoRA t represents the fine-tuned LoRA module for the t-th task. let θ s = θ 0 + LoRA s , we can prove that Twin-Merging on the θ is equivalent to Twin-Merging on the LoRA module.θ * = θ s + T t=1 w t * SVD r (θ t -θ s ) Twin-Merging on θ = θ 0 + LoRA s + T t=1 w t * SVD r (θ 0 + LoRA t ) -(θ 0 + LoRA s ) = θ 0 + LoRA s + T t=1 w t * SVD r (LoRA t -LoRA s )Twin-Merging on LoRAEQUATIONwhere we denote LORA * = LORA s + T t=1 w t * SVD r (LoRA t -LoRA s ).Multi-Task Learning. The multi-task training typically learns multi-task features by simultaneously optimizing task-specific objectives, facilitating the integration of diverse knowledge into the model. Existing works mainly focus on mitigating task conflicts [41] and catastrophic forgetting [19] by parameter sharing [45] , adjusting suitable objectives [14, 57] , find suitable task weighting [4, 49] , and minimizing negative transfer [31] . In an era where models are growing larger, and the number of task scenarios is increasing, what we need to explore is a more cost-effective approach to multi-task learning. Therefore our focus is on multi-task scenarios that do not require acquiring or integrating multi-task data and do not involve additional updates to existing experts.Mixture of Experts. To enhance model scalability without increasing computational costs, the mixture of experts (MoE) paradigm introduces conditional routing of inputs to a subset of learnable parameters. Several efforts have extended feedforward networks (FFNs) within Transformers to incorporate MoE layers, such as GShard [36] and Switch Transformer [17] . These models typically employ learnable top-2 or top-1 routing strategies to scale MoE language models to an extremely large size [30] . Recent studies have focused on challenges such as load balancing of experts [8, 82] , training instability [86] , expert specialization [9, 66] , and synchronization reduction [64] . However, these methods often require substantial multi-task data and costly joint training. In contrast, our approach directly reuses task-specific experts, leading to the natural specialization of experts in different domains. We only require minimal fine-tuning for a small router to calculate fusion weights, making our method highly efficient.To illustrate the challenge in determining the optimal merging coefficient and the limitations of prespecified coefficients with unpredictable data, we consider COLA and SST-2 as in-domain experts. We merge them using Task Arithmetic and evaluate on the eight discriminative tasks from the GLUE benchmark. Only COLA and SST-2 are seen tasks, while the others are unseen. Since the merging coefficient is crucial for performance [51, 78] , we conduct an extensive grid search for coefficients ranging from -2 to 2.A large dark-blue region indicates consistent optimal performance, which is why Task Arithmetic can work with various weights. Conventional methods search this region for optimal performance across all in-domain tasks, avoiding the red region. However, this is computationally expensive and does not scale well with an increasing number of tasks. Additionally, it cannot handle unseen tasks, as the same coefficients can produce different patterns across tasks. For example, setting coefficients γ COLA and γ SST-2 to 1 leads to performance drops in MRPC and QNLI, but gains in MNLI, QQP, and RTE. 4Furthermore, merging performance is not always a single cluster. For example, within the range of [-2, 2], STS-B and QNLI already show complex patterns, making it difficult to find an optimal weight for all tasks when task-specific experts are limited. Although Yang et al. [78] propose unsupervised entropy minimization to find optimal coefficients, this method is limited to classification tasks and has limited adaptability.To address this, we propose reformulating the problem of fusing models as a supervised learning task. Specifically, we train a router to dynamically merge task-specific experts, as detailed in Section 3.3.Here we detaily illustrate the setting of our experiments.We executed all our experiments on Nvidia A100 GPUs equipped with 80GB RAM. Single-task LoRA models for Qwen-14B on four generative tasks required 1-2 hours per task, Single-task LoRA for Qwen-72B need 10 hours on single GPUs to train. while the multitask vector took around 10 hours on single GPUs of 500M tokens. The RoBERTa model needs 15 minutes per task on GLUE datasets.Our router is implemented as a three-layer linear network with Leaky ReLU activations and batch normalization. We train the router on the validation dataset with a learning rate of 5e-4 for 10 epochs.The validation set consists of the integration of in-domain downstream tasks, not the general text corpus. The validation set is taken from a split of the training set, and we use at most 1,000 items for router training for each task.Merge experiments were efficient, with evaluations consuming less than 2 minutes. The inference is generally fast within 4 minutes per 1000 items for generative tasks and less than 30 seconds per 1000 items for discriminative tasks. The detail comparison of the training cost and inference cost of different methods are detailed in Table 7 .Discriminative Tasks. we conduct experiments on the GLUE benchmark [69] with eight discriminative tasks, which is designed for classification tasks except for STS-B for the regression task. The detail of eight dataset can be found in the paper of Wang et al. [69] . Consistent with prior research [79] , We split 10% of the training set as a validation set and employ the original validation data as the test set.The licenses of QNLI, COLA, and STS-B are licensed under CC-BY-SA. QQP is licensed under MIT. SST-2 and MRPC are licensed under Apache 2.0. MNLI is licensed under OANC. RTE is licensed under CC BY 4.0. Thus, these datasets in GLUE are available for non-commercial research purposes.Generative Tasks. We conducted experiments on four benchmarks:1. MMLU [24] : This benchmark tests general and STEM knowledge across 57 subjects, from elementary to professional levels. We used Exact-Match as the metric. 2. TruthfulQA [40] : This benchmark assesses the truthfulness of language models with 817 questions spanning 38 categories like health, law, finance, and politics. Exact-Match was used as the metric. 3. BBQ [52] : This dataset highlights social biases against protected classes in nine social dimensions relevant to U.S. English-speaking contexts. Exact-Match was the metric. 4. CNN-DailyMail [48] : This dataset is used for text summarization, requiring models to generate summaries of news stories. ROUGE-2 scores [39] were used for evaluation.We evaluated these tasks using the HELM benchmark5 in a few-shot setting.For MMLU and TruthfulQA, which lack official training sets, we used the Dolly-15k dataset 6 for MMLU and the BigBench-sampled dataset for TruthfulQA.The GSM8K and MMLU datasets are under the MIT License. For discriminative tasks, we used RoBERTa-base7 [42] as our pre-trained backbone and fine-tuned it for each dataset to create supervised models. We conducted separate fine-tuning for the RoBERTabase model on each dataset for 10 epochs. Our selected hyperparameters included a batch size of 64 and a learning rate set at 1e -5 .For generative tasks, we employed Qwen-14B8 as the backbone and applied LoRA [26] for taskspecific fine-tuning. In the case of generative tasks, the fine-tuning process for Qwen-14B involved the utilization of LoRA with a rank set to 32, a batch size of 128, and a learning rate of 2e -4 for 3 epochs. For Qwen-72B we employ the same setting with QLoRA technique [11] .To serperate the impact of parameter-wise interference, we design the non-overlapping experiment based on Qwen LoRA modules as follows: (1) Firstly, we obtain standard merging experts by injecting the LoRA module into both the "w1" and "c_proj" weights of the Qwen-based model, and fine-tune them on two different tasks, resulting in two distinct models. Then we combine it into a single model to obtrain standard merging results. ( 2) Next, we performe a non-overlapping fine-tuning by injecting LoRA only to "w1" on one task and "c_proj" on another, producing two models with task-specific knowledge in different modules. ( 3) Finally, we combined the non-overlapping checkpoints to get the merged results. Since task-specific knowledge was injected into separate modules, parameter-wise interference was minimized. The results are shown in the upper section of Table 1 .In Figure 6 , we conduct a comparative analysis employing various sparsification methods. The specifics of each method are outlined below:• Magnitude. Following the setting in Ties-Merging [76] , we retain solely the k% largestmagnitude values while resetting the remaining values to zero.• Bernoulli-Dropout. Adhering to the methodology introduced in DARE [79] , we employ a parameterized Bernoulli distribution to sample a sparse mask m t . This mask is then applied to the parameters δ and subsequently rescaled with respect to the mask rate k.EQUATION• Singular value decomposition (SVD). Assuming that matrix M has a rank-m decomposition, expressed asM = U t Σ t V T t where U t ∈ R dout×m , Σ t ∈ R m×m , V t ∈ R din×m .We compress the matrix M by selecting only the top-r singular values from Σ t , denoted asM r = U t (r)Σ t (r)V t (r) T . Here, U t (r) ∈ R dout×r , Σ t (r) ∈ R r×r , V r t ∈ R din×r represent sub-matrices of U t , Σ t , V T t .This transformation significantly reduces the taskspecific parameter dimensionality from m × (d out + d in + 1) to r × (d out + d in + 1), as the maximum m typically equals to the hidden size of the language model (e.g., m = 768 for RoBERTa-base and m = 4096 for Qwen-14B) and r can be reduced to 1, resulting in a significant reduction in parameters and storage effectiveness. During merging, we decompress these matrices by extending U t (r), Σ t (r), V t (r) T to sizem by filling with zeros, allowing us to recover M via their product. This operation is only at the matrix level; once we obtain the merged matrix, we discard the decompressed matrices, ensuring efficient storage.Here we will elaborate the baselines utilized in our main comparison experiment, as outlined in Table 2 and Figure 2b .• Individual means that each task uses the corresponding fine-tuned model, which has no interference between tasks but cannot perform multiple tasks simultaneously. It serves as the upper-bound performance for each specific task.• Weight Averaging [6, 74] is the simplest form of model merging, which straightforwardly averages the parameters of multiple models. It serves as a lower bound for model merging.• Task Arithmetic [28] first introduces the concept of "task vectors" and merges them into the pre-trained model to execute multi-task learning.• Ties-Merging [76] addresses task conflicts by eliminating redundant parameters. The process involves three steps: Trim, Elect Sign, and Disjoint Merge.• Task Arithmetic (w/ DARE) [79] This variant incorporates the Bernoulli-Dropout technique for 70% sparsification before employing Task Arithmetic [28] for merging.• Ties-Merging (w/ DARE) [79] Similar to the previous approach, this variant integrates Bernoulli-Dropout for 70% sparsification, followed by Ties-Merging [76] for the merging process.• AdaMerging [78] assumes access to an offline test set and dynamically adapts to it by introducing additional coefficients at every layer, conducting unsupervised training across multiple iterations on the test set (without labels) to refine the model.• Surgery [77] assumes that test data IDs are accessible during inference, allowing it to insert corresponding task-specific adapters to leverage task-specific knowledge.The coefficient for Task Arithmetic and Ties-Merging are decided by a small scale grid search on validation datasets. The coefficient of 0.7 is consistently applied for DARE Merging, following the previous papers [79] .In Table 2 , we present only the average normalized scores across various tasks. In this section, we detail the statistical performance of all tasks, with discriminative results displayed in Table 8 and generative results shown in Table 9 . We primarily merge using LoRA for the Qwen-14B models because fully finetuning them to obtain task-specific experts would require a huge amount of resources and computation (finetuning the full 14B model requires at least 8 A100 GPUs). However, to further demonstrate, we provide experiments on fully fine-tuned LLaMA 7B models for generative tasks (GSM8k and TruthfulQA), where our approach still exhibits superior performance.Beyond its strong performance and efficiency, our method offers key benefits in handling distribution shifts and LLM deployment, surpassing approaches like AdaMerging and Surgery. AdaMerging tackles distribution shifts in image domains by requiring access to test sets, even in unseen domains. This is problematic for online settings where test data is unpredictable. In offline scenarios, scaling test sets to large sizes is inefficient. Additionally, AdaMerging's entropy-based optimization limits it to classification tasks, which restricts its applicability as generative models like LLaMA and GPT-4 become more prominent.Our method overcomes these limitations by dynamically adapting to test inputs, as shown in Table 5 .It does not require specific validation datasets for each in-domain expert, making it more flexible. For example, the open-source Dolly dataset can represent the MMLU expert, and we do not need validation sets for QNLI, MNLI, RTE, or MMLU. Instead, we assume that combining in-domain knowledge helps address out-of-domain inputs, as supported by [71] . This dynamic merging ensures better generalization to diverse inputs, unlike AdaMerging, which relies on entropy approximations without supervision.Our approach is also optimized for real-world LLM deployment with continuous data streams and no gradient updates, aligning with current trends [54] . Key advantages include:• Handling Unpredictable, Heterogeneous Data: Our dynamic merging technique efficiently addresses diverse streaming inputs.• Reducing Latency and Storage: We shift router training and knowledge modularization to preprocessing, and apply SVD techniques to minimize storage needs.• Broad Applicability: Our method works across NLP and CV tasks, including generative tasks with Qwen-14B and up to 72B models, supporting large-scale AI deployment. To evaluate the compatibility of Twin-Merging with other merging methods, we conducted experusing different techniques to create a shared expert, followed by dynamically merging the twin vectors. The results in Table 11 demonstrate that our method integrates seamlessly with primary merging techniques, leading to significant improvements. For example, when combined with our approach, the baseline Weight Average method improves from 52.26 to 96.23 on GLUE, approaching the performance of fine-tuned experts. Notably, our method complements Ties-Merging particularly well, suggesting that better isolation of shared knowledge enhances the overall performance of Twin-Merging.Assume we have T tasks, the fine-tuned model have P = P f + P a parameters, where P f are frozen and P a are activated.Parameter Count and Storage Cost Assuming each float parameter uses 16 bits (either fp16 or bf16): Fine-tuned models require 2(T P a + P f ) bytes of storage. Pretrained models, including those using Weight Average, Task Arithmetic, Ties-Merging, and DARE Merging techniques, each need 2P bytes of storage per model. For Twin-Merging, with the router having P r parameters (P r ≪ P ) and a compression rate of k%, it need to store 2T kP a + 2P + P r bytes including a shared expert, compressed exclusive task-specific vectors, and the router. We can select k to compress the model matrix to rank 1 for best storage. These strategies enhance the accessibility and sustainability of task-specific models, fostering wider advancements and applications. Visual representations can be found in Figure 2a and Figure 4 .Our method mainly introduce the extra time cost due to routing and dynamical merging. However, as the inference process typically involves hundreds of forward passes (e.g., 300 tokens for summarization tasks), the additional computing is usually neglectable. Assuming context length s, task number T , layer number m, the introduced FLOPs ( Multiply-accumulate operation ) can be computed as m(24sh 2 + 4bs 2 h) for routing, T m(12h 2 + 9h) for merging (excluding norm parameter), while generating L tokens typically requires L l=s 24m(lh 2 + 4bl 2 h) FLOPs. Given that n ≪ L and s are typically truncated, the additional consumption is neglectable in generation tasks. We demonstrate the actual time cost in Table 7 , which adds only 0.039 seconds per sample while bringing significant performance improvements.Comparison with other baselises Moreover, our approach offers significant performance improvements with these additional computing resources. As shown in Table 2 and the "More Baseline" section, we achieve an absolute normalized improvement of 28.34% for RoBERTa, 18.83% on ViT-B-32 compared to Task Arithmetic, 9.71% compared to Twin-Merging on Qwen-14B. Traditional model merging methods often overlook the heterogeneous nature of test inputs, leading to substantial performance gaps. Advanced merging techniques like AdaMerging and Surgery typically require costly training and searching processes, as demonstrated in Table 3 . In contrast, our method achieves superior performance to fine-tuned models with minimal cost and storage requirements (47m22s vs. 215m01s, 5.0GB vs. 32.4GB) due to dynamic merging and SVD techniques.Speedup variants Currently, our method supports batch inference for the routing process, while the dynamic merging process handles inputs sequentially. However, it is straightforward to extend our approach to support merging in batches or groups. We can achieve this by first obtaining router weights in batch, then grouping similar data items using the following strategy: (1) Divide into Bins Based on Argmax Indices: First, we divide the data into several bins according to the arg-max indices of the router logits. (2) Cluster Within Each Bin: Then, we cluster (by KMeans) within each bin to group the logits (we set the group number to 20). (3) Average Weights Within Each Group: Within each group, the router weights are averaged to obtain a merged model. Each group corresponds to one merged process, and the group size is typically larger than the batch size, making it very efficient. We have added a group-wise experiment on RoBERTa to illustrate this: F Limitations and Future WorkOur approach shares common limitations with existing merging methods: (1) The underlying theory behind why and when weight interpolation works is not fully understood, though recent works [51, 83] have made interesting observations about weight disentanglement and cross-task linearity.(2) Currently, merging is limited to models with the same architecture and it may be difficult to find a suitable fine-tuned model with specific capacities.Additionally, while our method focuses on shared and exclusive task-specific knowledge, providing a way to approach fine-tuned model performance and potentially surpass it without additional training, we observe there may be other types of knowledge that remain unexplored: ( This paper presents work whose goal is to advance the field of machine learning and model merging research. In terms of positive social impact, twin-merging techniques can achieve multi-task performance of foundation models without retraining expert models, significantly reducing computational and energy costs. Our proposed knowledge modularization and compression techniques make the task-specific enhanced model more accessible and sustainable, paving the way for broader applications and advancements in the field. These techniques effectively align unaligned models by leveraging experts, thus mitigating the harmfulness and biases present in the original models. Additionally, model merging allows the unified model to benefit from the strengths of each task-specific model, even for tasks with private or inaccessible data, enhancing commercial and safety benefits. However, improper merging of biased models may contaminate the merged model. This issue can be addressed by merging a de-bias expert or using sparsity techniques to minimize the impact.Answer Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Justification: Yes, the research conducted in the paper conforms in every respect with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Notice that we cannot directly normalize them as we do not have the corresponding expert on unseen datasets to get upper-bound performances. This leads to relatively lower scores due to the narrower score ranges for tasks like RTE (max 66.43 vs max 91.71 for QNLI) and MMLU (max 68.03).Here, we assume that merging method does not retrain all task experts; instead, it reuses experts (e.g., downloaded from model hubs like Huggingface[73]).In fact, the MNLI and QNLI are very similar tasks about Natural Language Inference (NLI)[69]. This demonstrates that task similarity does not guarantee similar merging performance patterns.https://github.com/stanford-crfm/helmhttps://huggingface.co/datasets/databricks/databricks-dolly-15khttps://huggingface.co/FacebookAI/roberta-basehttps://huggingface.co/Qwen/Qwen-14B 


Decision: The work proposes Twin-Merging, a two-stage method for model merging: (1) modularizing knowledge into shared and exclusive components, with compression to reduce redundancy and enhance efficiency; (2) dynamically merging shared and task-specific knowledge based on the input. During testing, Twin-Merging dynamically merges shared and compressed specialized knowledge based on test inputs to form the final inference model. Compared to conventional merging methods, the new method better addresses adaptability to data heterogeneity and therefore further bridges the gap between merged models vs. fine tuned models. 

The final recommendations from the reviewers are unanimously acceptance (5, 6, 6, 6).

The paper is well written. 
The proposed method is technically sound. 
The method is verified by quite extensive experiments.

The two major common concerns raised by most reviewers are more baselines and discussion/evaluation over inference efficiency. Both have been addressed in the rebuttal. Please incorporate the new results and discussions in the final version.

Reviews: review: paper_summary: paper_summary: In this paper, the authors introduce the Twin-Merging to merge language models, aiming to close the performance gap between conventional model merging techniques and fine-tuned models, while improving adaptability to data heterogeneity. By modularizing and dynamically merging shared and task-specific knowledge, the authors show that Twin-Merging outperforms existing model-merging methods and approaches the performance of fine-tuned models across various settings and domains.
strength_weakness: strength_weakness: + The proposed method dynamically merges the models with the varying inputs, which can be extremely time-consuming in practice. It would be better if the authors propose some mechanisms to address this issue.

+ In figure 3, the authors compare 1-model merging with 8-model merging. Why here 8-model merging is used for comparisons? Why not 2-models merging is compared? More explanations should be provided here.

+ Some highly related works are missing in the related work section, such as MuDSC[1]. The differences between these works should be clarified.

[1] Training-Free Pretrained Model Merging, CVPR 2024

[2] REPAIR: REnormalizing Permuted Activations for Interpolation Repair, ICLR 2023.
questions: questions: Please see the Weaknesses
limitations: limitations: N/A
paper_summary: paper_summary: `Twin-Merging` proposes a method for task merging which tackles two issues:
  * Task interference: The proposed `Twin-Merging` explicitly model shared vs task-specific knowledge to potentially reduces redundancies across the task vectors, which may lead to subpar task merging results
  * Dynamic merging: Usually, task merging weights are only task dependent and determined only once. In contrast, here, the weights are determined at the input level using a router akin to Mixture of Experts design.
strength_weakness: strength_weakness: * **The per-input routing design seems highly impractical**.  If I understood correctly, the task merging weights are compute on-the-fly for each individual input. This means that:
	  * The design does not support batching as we need a different merged model for each input in the batch
	  * Every time a new input comes, we need to first get the merging weights by executing the fuser, then uncompress the task vectors, build the merged model, and finally run the inputs through it. This does not seem very hardware friendly, even if the task vectors are low-rank compressed using SVD. In contrast, in standard task merging, we only ship one merged model.
	  * Because the fuser $\mathcal{R}$ takes as inputs the  *last-layer token embeddings from the shared expert* (line 186), does it mean that every input require two forward passes (one through the fuser, then one through the merged model) ?
- The paper does not really convey the importance/novelty of **knowledge modularization** strongly enough. In essence, the basic task arithmetic (**A**) already performs some form of modularization where the pretrained model is the *shared knowledge* and task finetuned models (task vectors) are *task specific*. In contrast, knowledge modularization(**B**) **redefined the task vectors** relatively to the merged model (rather than the pretrained one) and compress them via SVD.  In my opinion, comparing **A** and **B** would be a better ablation experiment than the one in Table 6 where the merged expert is replaced by a random specific one.

- **Baseline**: Since the proposed design uses additional validation data (to finetune the router), it would be fair to also compare to the more recent/stronger baseline `AdaMerging: # Adaptive Model Merging for Multi-Task Learning` (Yang et al), which also assumes extra data available.
questions: questions: * I do not fully understand the conclusion of **Section 3.1**:
	  * the assumption of `Ties Merging` is that redundant parameters can be hard to merge. But they do not really make assumptions on whether these parameters were trained jointly/with overlap > therefore I'm not sure how the LoRA experiments contradicts this insight: Even if the LoRA modules are trained separately, they may still have redundant statistics at the end of training ?
	  * Similarly, the notion of *similar tasks* or *task interference* is hard to define in general.  So it is not clear to me how significant the results of the XSUM/DailyMail experiment are.
  * Based on the result of table 4.5 it looks like Twin Merging does not often behave as a model merging method ? It seems that most of the time the samples are routed to their respective task-specific experts. If that insight is true, it is not too surprising that `TwinMerging` performs on-par with the finetuned baseline.
  * In Table 7: Shouldn't `Model Merging` and `Twin Merging` also integrates the cost of training the initial task vectors in **training cost** ? Otherwise it seems unfair to the MTL baseline.
limitations: limitations: The paper discusses limitations inherent to task merging in appendix F. However, as discussed in the Weaknesses section.
paper_summary: paper_summary: This paper attempts to resolve an issue of destructive interference with model merging techniques. It proposes to maintain a shared base model and separate task-specific knowledge structures that can dynamically be combined at test time. 
The paper presents some a nice to buttress the drive home their methodology
strength_weakness: strength_weakness: 1. My primary issue with the paper is why this is a viable alternative to just keeping the base model back-bone and LoRA low-rank vectors of similar rank to the $v_t$s (Algo 1) (and instantiating the task specific model on the fly).  Twin-Merge has memory overhead (v_t for T tasks) which brings it more into the realm of “saving task-wise adapters/LoRA” whose size grows as the number of tasks. It is unclear from the existing set of experiments why a practitioner would use this method over keeping separate task low-rank (small memory footprint) adapters and just using these at test time (either statically per task or dynamically — https://arxiv.org/pdf/2306.14870 ) ?
    1. I understand that the paper performs a primary comparison to other model merging methods like DARE and Task Arithmetic but these methods have zero extra memory overhead (after the model is merged) — they don’t enjoy the expressivity that test time adaptation + extra memory gives them. 
2.  Claims of generalization to unseen tasks might not be valid. Specifically, it is interesting to see that the boost in performance for Twin Merging in Table 2 is much more significant than for Table 5 with unseen generalization. This makes me wonder if the deltas in table 5 are actually significant. 


I am open to raising my score if the authors can convince me of the utility of the method -- in light of [1] above.
questions: questions: * How are the $v_t$s represented / stored (Algorithm 1) ?
    * Are the v_ts stored as the low rank matrices that are later multiplied to obtain the appropriate matrix size ?  As written, it seems like each $v_t$ is of size equal to the size of the full parameter space and so ends up as a memory burden.
* For Figure 4 (right), is the fine-tuned storage size much larger because you are doing full fine-tuning instead of LoRA fine-tuning ?
    * If you are doing LoRA fine-tuning is the gap because the rank of the lora matrices are larger >> than the rank of the final $v_t$s above ? 
* Maybe I missed this but where is the “(Best Storage)” option for Twin Merging in Table 2  described ?
* For Table 4, would it be possible to provide the results for Twin-Merging only before showing the additive results ? It’s hard to make out whether the other methods are contributing anything at all
limitations: limitations: Broader impact statements and limitations are discussed in the paper
paper_summary: paper_summary: This paper investigates a dynamic and compressive merging method for adapting large-scale models to multiple tasks. The authors claim that adjusting the ratio between shared knowledge and exclusive knowledge is crucial for high-performing model merging, and they devise an algorithm that learns proper coefficients to merge those two kinds of knowledge in a data-dependent manner. They further compress the task-specific exclusive knowledge via SVD to reduce memory storage during merging. They validate their method with language models on discriminative and generative tasks.
strength_weakness: strength_weakness: - Lack of technical details on the most important part of the methodology
    - The router module plays a crucial rule that produces input-dependent merging coefficients during inference time. However, the authors do not provide any technical details on the training of the router module (including appendix).
        - Model architecture and training configurations should be provided.
    - Moreover, a description of the validation set the authors to use for the router training should be provided. The authors only describe that dataset as '**_a small validation set_**'. Does the validation set consist of the integration of downstream tasks? or general text corpus? and how the construction of the validation set affects the final merged model performance.
        - I believe details on the construction and amount of samples for the validation set should be presented.
- Limited scope of validation
    - While they evaluate their method with fully fine-tuned models on discriminative tasks, they only validate it on the parameter-efficient fine-tuning regime for the generative tasks. As one of the proposed method's main contributions is parameter compression, it would be more effective when they show the applicability of their method on fully fine-tuned model merging with decoder-only LM or encoder-decoder LM.
- Concerns about scalability
    - While the compressive merging reduces the requirements of memory storage during the inference phase, I speculate that the amount of computation for the merging operation is still a huge burden. The input-dependent dynamic merging requires computations for merging operation per every sample, and I wonder about the **runtime and computation cost during inference phase when the fully fine-tuned models are the target of validation** rather than the parameter-efficient fine-tuning regime in Table 7.
questions: questions: - details on router network architecture and its training (refer to weakness section)
- details on the validation set construction (refer to weakness section)
---
If there is a misunderstanding from me, please don't hesitate to refute it. I would be happy to discuss this further.
limitations: limitations: Lack of scalability to larger models due to increasing inference cost of instance-dependent dynamic merging.

Comments: title: Thank you
Sincerely looking forward to your feedbacks
Response
comment: We appreciate the reviewer's recognition of our work's effectiveness and the score increase. We will revise our work to include clearer illustrations. Thank you for your time and valuable suggestions!
I want to express my severe gratitude for the authors' kind response and my regret for my unprofessionalizm, such as the score-reversing behavior.

Thanks to the discussion with the authors, I could also extend my sight, not only gain a clearer understanding of the paper.

I will raise my rating accordingly.
We sincerely appreciate your valuable feedback and concerns regarding the clarity of our descriptions.

We hope our response has effectively addressed all your concerns.  Your insights are crucial for improving our work, and we are open to  further discussion if you have any questions about our response.

With the effectiveness in merging performance (even outperforming the oracle at times), efficient storage, minimal time cost, and a  reasonable assumption regarding the validation dataset—as recognized by  Reviewer `2gMR` , `Q3uE` and `LqLU`—we believe that our  approach will become increasingly practical and significant in the era  of large language models. We hope that these insights and outcomes can  contribute to the community. We appreciate your time and would be very  grateful if you could re-evaluate the paper’s rating.
We sincerely appreciate your valuable feedback and concerns regarding the clarity of our descriptions and the validation sets.

We hope our response has effectively addressed all your concerns. Your insights are crucial for improving our work, and we are open to further discussion if you have any questions about our response.

With the effectiveness in merging performance (even outperforming the oracle at times), efficient storage, minimal time cost, and a reasonable assumption regarding the validation dataset—as recognized by Reviewer `2gMR` and `Q3uE`—we believe that our approach will become increasingly practical and significant in the era of large language models. We hope that these insights and outcomes can contribute to the community. We appreciate your time and would be very grateful if you could re-evaluate the paper’s rating.
We thank the reviewer for their feedback and for raising the score! We will address the additional comments below.

> Q1. I wonder how group-wise merging would perform

The insight is that the router logits indicate the relevance of different exclusive knowledge modules to specific input samples. 
Inputs have similar router logits require similar knowledge, i.e., similar merging models. 
To leverage this similarity, inputs can be grouped based on their router logits. Within these groups, the merging weights are expected to be similar and can be approximated by an averaged representation.

To begin, you can divide the inputs into bins based on the arg-max indices of the router logits, which represents the most relevant domain or knowledge module for each input.
To further refine into groups, apply K-means clustering within each bin directly on the router weights. 
Once the clustering is done, average the router weights within each group. The model is then merged based on these averaged router weights, allowing the inference for the entire group to be performed using a single model.

> Q2. the paper introduces several new assumptions departing from traditional task merging (labeled data + extra parameters/router model + per-input dynamic behaviour requiring extra processing for batched inference)

To clarify, we want to highlight that these assumptions are not entirely new but build upon those in previous works such as FisherMerging (NeurIPS22), DARE (ICML24), and Surgery (ICML24). FisherMerging, for instance, utilizes a validation dataset to adjust merging weights. DARE introduces pre-merging techniques like Sparsify to enhance performance, while AdaMerging and Surgery focus more on post-merging techniques.
Specifically, AdaMerging assumes access to an offline test set and dynamically adapts to it by introducing additional coefficients at every layer, conducting unsupervised training across multiple iterations on the test set (without labels) to refine the model. Surgery goes even further by assuming that test data IDs are accessible during inference, allowing it to insert corresponding task-specific adapters to leverage task-specific knowledge.

In contrast, our key insight is that **in-domain knowledge, when combined appropriately, can effectively address out-of-domain inputs**, eliminating the need for offline test dataset access or test ID information.
To achieve this, we significantly reduce the additional parameters required, moving from several task-specific adapters across the entire model to highly sparsed exclusive knowledge representation and a single, simple MLP that infers optimal weights for combining modularized knowledge based on the test input. We train this MLP using a small validation dataset rather than an unlabeled test set, which is more suitable for LLM serving, a current trend in AI.

It is important to emphasize that, like previous methods, our approach only uses **a single merged model** to actually perform the task during the inference phase.

In summary, while our method introduces techniques across preprocessing, additional parameters, and post-merging stages—similar to previous methods—it is distinct in its approach and insight. This distinction is validated by our experimental results, which demonstrate superior performance with only 1/4 the effort of AdaMerging and 1/5 the effort of Surgery (47m22s vs. 185m35s/215m01s) and just 15.4% of the storage cost of Surgery (5.0GB vs. 32.4GB). We aim to further refine these assumptions in future work.

> Q3. the writing would further benefit from making these differences clearer for fair comparison

We thank the reviewer for the suggestion. We will further elaborate on the differences between previous merging methods and incorporate the AdaMerging comparison into our revised version of the paper.
Thanks for your feedback. We will address the concerns below.

> Q1: image domain distribution shift & varying amounts of validation set

We want to clarify that **our method addresses distribution shifts through dynamic merging, which adapts to test inputs**, as shown in our NLP results (Table 5) and Section 4.5, although the preprocessing stage (router training/knowledge modularization) is indeed agnostic to the test distribution.

To address your specific concern, we conduct additional experiments on image domain shifts using Gaussian noise corruption. The results demonstrate that Twin-Merging is robust against image domain distribution shift.

| Method | Avg. Before Corruption | Avg. After Corruption |
|-|-|-|
| Task-Arithmetic  | 84.3  |  65.6  | 
| AdaMering  |  91.7 |  73.4  | 
| **Twin-Merging**  | **96.9** |  **79.2**  |

In terms of varying amounts of validation set, we actually have a related experiment in Figure 4, where the router validation set is varied from 2,000 to 7,000, and we can observe consistent superior performance. 
To further demonstrate, We also add experiments with reducing the validation data number to 100 per task for generative tasks, which is 1/10 of the original size, and we observe that the performance does not change significantly:

| Method | val-1000 | val-100 |
|-|-|-|
| Twin-Merging  | 102.38  |  101.23 | 

> Q2: harder to gather validation sets for each merging ingredient fine-tuned model from different institutions (due to commercial / privacy issues)

Firstly, to clarify, **our approach does not strictly require the validation set to be taken from each in-domain dataset used by the experts**. For example, we can utilize the opensource Dolly dataset to represent the MMLU expert (L619-L620). Furthermore, **we do not need to gather validation datasets for all experts**. As illustrated in Table 5, our approach still works effectively without gathering specific validation datasets for QNLI, MNLI, RTE, and MMLU.

In practice, we can select a subset of accessible experts and gather representative validation data for them, which is typically not difficult.
This actually stems from the key assumption of our approach, that **in-domain knowledge contains complementary elements that can effectively address out-of-domain inputs when combined properly [1,2]**. By dynamically inferring optimal weights to combine this modularized knowledge based on the test input, our method offers better collective generalization against the unpredictable heterogeneous input.
In contrast, AdaMerging may limits in imprecise entropy approximation and the lack of supervised guidance.
To better demonstrate the advantage of our approach, we conducted additional generalization experiments using the same settings as in the AdaMerging paper:

| Method | EuroSAT | MNIST | Avg. Acc |
|-|-|-|-|
| AdaMering  | 45.9  | 90.1 | 68.0  | 
| **Twin-Mering**  | **53.2** | **92.9** | **73.5** |

Secondly, we want to emphasize that our approach is primarily designed for **real-world LLM online serving**, where models are deployed with a continuous stream of inputs without gradient updates, aligning with trends in LLM deployment [3,4].
- **Unpredictable, Heterogeneous Data Streams**: The nature of these data streams means that traditional batch techniques are inefficient, and any single expert is insufficient. This is why we introduce our dynamic merging technique.
- **Latency and Storage Considerations**: To address critical concerns around latency and storage, we shift time-consuming router training and knowledge modularization to the preprocessing stage. Additionally, we employ SVD techniques to reduce storage requirements. A detailed analysis of the FLOPs for these serving scenarios is provided in the global rebuttal.
- **Broad Applicability**: Our method applies to both NLP and CV domains, spanning discriminative and generative tasks, making it highly adaptable for deploying and scaling AI models.

In contrast, the AdaMerging post-training technique incurs significant latency due to its reliance on test-time training, making it more suitable for offline evaluations rather than real-time LLM deployment.

[1] Fusing Models with Complementary Expertise [ICLR24]

[2] Knowledge Fusion of Large Language Models [ICLR24]

[3] Mooncake: Kimi's KVCache-centric Architecture for LLM Serving 

[4] LLM Inference Serving: Survey of Recent Advances and Opportunities
Dear authors,

thanks for your response and clarifications:
  * The new ablation on A vs B is more convincing in showing the benefit of explicitly building shared/specific task vectors
  * The group-wise variant of Twin-Merging is interesting, and it's good to see that performance of the method does not drop significantly in that case. However I do wonder how it would perform 

Since my main concerns have been addressed, I will raise my score to weak accept: Overall I think the paper is technically solid, and I appreciate the authors' effort in clearly portraying the efficiency/memory cost of the method. However, I also think that the paper introduces several new assumptions departing from traditional task merging (labeled data + extra parameters/router model + per-input dynamic behaviour requiring extra processing for batched inference), and the writing would further benefit from making these differences clearer for fair comparison (e.g. it would be more fair to make Adamerging the standard baseline in the experiments section since it also requires extra but unlabelled data in contrast to Ties-Merging).

**Note:** ~[NIPS23]~ -> [NeurIPS23]
Sorry for my reverse (6 -> 5), but I would like to discuss my second worry with the authors further.

After reviewing the authors' responses, I still lean toward the negative side due to their reliance on the validation set.
While the method is agnostic to the test distribution, its effectiveness is unclear under severe distribution shifts in image domains (that AdaMerging focuses on) compared to distribution shifts in language domains. That is, I still doubt the performance sensitivity against distribution shifts and varying amounts of validation set.

Moreover, in the real-world deployment scenario, I think it is much harder to gather validation sets for each merging ingredient fine-tuned model from different institutions (due to commercial / privacy issues) compared with test-time unlabeled incoming samples.

Given that, my concerns about Twin-Merging's reliance on ID validation still seem to be weaknesses, even though it boosts performance significantly.
Thanks for your detailed rebuttal.

Authors' claims are convincing and I will raise my score 5 -> 6
We thank the reviewer for raising the score! We will address the additional comments below.

> Q1. why listed Top-1 expert as baseline in original paper, not the "soft merging" method 

We choose the "Route to Top-1 Expert" as an **oracle** baseline to highlight the performance gap between common merging techniques and **the ideal scenario**. 
Because this baseline typically performs the best [1], as shown in the table below (taken from the response to Q1):

| Method |  RoBERTa | Qwen|
|-|-|-|
| Top-1 (oracle)  | 100.00  |  100.00  | 
| Pretrain+Dynamic Merging(A)  | 85.90  |  97.03  | 
| Shared+Dynamic Merging(TwinMerging,B)  | 96.14 |  102.38  |

The "soft merging" (A) performs worse than the oracle, due to the interference between the different models, which is consistent with the findings in [1]. 
However,Twin-Merging mitigates it by modularizing shared and exclusive knowledge, leading to improved performance and, in some cases, even surpassing the oracle baseline (102.38 > 100.00).

We appreciate the reviewer's suggestion to include the "soft merging" as a baseline for better clarity. We will incorporate this into our revised version of the paper.

> Q2. unsufficient evidence to say your method is better than the simple LoRA adapter router baseline -- esp given the relatively small delta (102.38 > 100.0) (how does this breakdown to individual task scores btw ?)

We’ve provided a detailed breakdown of the scores in Table 9, Appendix D7. 
It’s important to note that the LoRA router baseline represents the **oracle performance**, which typically achieves the best results, as explained in the above. 
Surpassing this baseline is extremely challenging. 
However, our method achieves nearly identical performance ( 99.87 vs. 100 on MMLU) or even outperforms the oracle in some datasets (over a 14% improvement on CNN/DM). 

Additionally, a key limitation of the simple LoRA router baseline is its difficulty in adapting to unseen tasks, as the router often struggles to predict the correct top-1 result. 
In contrast, our approach demonstrates better performance on unseen tasks (as shown in Table 5), leveraging complementary knowledge from different exclusive sources to enhance collective intelligence.

[1] Exploring the Benefits of Training Expert Language Models over Instruction Tuning [ICML23]
Hi authors,
Thanks for your response. And including the additional baselines.
> Directly Route to Top-1 Expert

I'm wondering why you decided to do only Top-1 expert in this baselines instead of soft router weights like you do. 
I think a better comparison would have been to have Top-K or even just a soft routing like you guys have in the paper. This would then be a better way of demonstrating that Twin merge is superior. As it stands I don't think there is sufficient evidence to say your method is better than the simple LoRA adapter router baseline -- esp given the relatively small delta (102.38 > 100.0) (how does this breakdown to individual task scores btw ?)

Based on the explanations and updated experiments. I'm raising my score -- but it would be great if the more extensive version of the experiment above (as I mentioned, is included in the paper)
We thank the reviewer for the feedback. We will address the additional concerns below.

> Q1

Our original approach significantly outperforms both AdaMerging and Surgery in terms of speed and performance. 
As evidenced by 1st table from the global rebuttal, our method completes in just 47m22s, compared to 185m35s / 215m01s for AdaMerging/Surgery. 
Despite the substantial time savings, our approach also delivers superior performance, achieving a score of 95.33, versus 88.50 / 94.40  for AdaMerging / Surgery, 
**the latter methods invest over 10 times the effort to marginally improve performance**.

Furthermore, our group-wise variant that matches the efficiency of Task-Arithmetic (5m14s vs 4m52s) still holds superior performance (92.02 vs 67.80). We will include this variant in the revised version of our paper.

> Q2

To clarify, AdaMerging requires **the actual test sets** (as confirmed in Section 3.2.2 of the original paper, or the code from `src/datasets/common.py` line 142 in the original codebase). This means that, for the example in Table 5, for the unseen domain dataset, 
AdaMerging theoretically requires access to these unseen domain test sets to approximate the optimal model, though without true labels. 
This poses a significant challenge for online deployment scenarios, where test inputs are unpredictable and streaming.
Even if we can access to the test set in in offline scenarios, scaling the test set to a large number, such as 1,000, makes the process highly inefficient.

Additionally, AdaMerging relies on entropy optimization for unsupervised training, making it **applicable only to classification tasks**. 
Given the growing dominance of large **generative** foundation models like LLaMA, GPT-4, and diffusion models,  AdaMerging’s focus on classification limits its scalability and applicability. 

In contrast, our approach only requires in-domain validation data corresponding to each expert. For instance, in Table 5, we used 5 experts to handle 3 unseen datasets, necessitating only the validation sets corresponding 5 experts. 
This is because the validation dataset is meant to enable the router to learn and identify specific exclusive knowledge in domain. 
Please notice that this training process is **agnostic to the actual test distribution**.  Whether the test distribution scales to 10, 100, or even 1000 tasks, we still only need those initial 5 validation datasets.
Our method, however, is versatile and can handle any type of task, including generative tasks with Qwen-14B in Table 2, and scales up to **72B** on generative tasks (Table 3), offering broader applicability and alignment with current AI trends.
I appreciate the authors' kind response! Some of my concerns are addressed.

* I still think that TwinMerging's inference time overhead compared with the vanilla task arithmetic in Table 2 of global response is significant (about two times), given that group-wise extension is not included in the reviewed manuscript.
* Moreover, as from your answer, TwinMerging requires labeled samples from ALL test domains in advance (even though it is validation splits), which is an optimistic assumption compared with AdaMerging, which only requires unlabeled test domain data.