Knowledge Fusion of Large Language Models via Modular SkillPacks
Authors: Guodong DU
Zhuo Li
Xuanning Zhou
Junlin Li
Zesheng Shi
Wanyu Lin
Ho-Kin Tang
Xiucheng Li
Fangming Liu
Wenya Wang
Min Zhang
Jing Li
Venue: ICLR 2026 Poster (2026)
Keywords: Knowledge Fusion
Model Merging
Large Language Model
Task Vector
Primary area: foundation or frontier models, including LLMs
TL;DR: GraftLLM enables efficient cross-capability transfer in large LLMs via compact SkillPacks, preserving knowledge, preventing forgetting, and supporting heterogeneous model fusion.

Abstract: Cross-capability transfer represents a key challenge in large language model (LLM) research, particularly in multi-task integration, model compression, and knowledge fusion. Recent works such as FuseLLM and FuseChat have shown the potential of transferring multiple model capabilities to lightweight models, thereby enhancing adaptability and efficiency. This motivates our investigation into more efficient methods for cross-capability transfer. However, existing merging approaches primarily focus on small, homogeneous models, limiting their applicability.
For large, heterogeneous models, knowledge distillation with full-parameter fine-tuning often overlooks the student model’s inherent capability and risks catastrophic forgetting, while PEFT methods struggle to effectively absorb knowledge from source LLMs.
To address these issues, we introduce **GraftLLM**, a novel grafting-based method that stores source model capabilities in a target model + SkillPack format. This approach preserves general capabilities, reduces parameter conflicts, and supports forget-free continual learning and model fusion. We employ a module-aware adaptive compression strategy for parameter updates, ensuring efficient storage while **preserving task-specific knowledge**. The resulting SkillPack serves as a compact and transferable knowledge carrier, ideal for **heterogeneous LLM fusion**.
Experiments across various scenarios demonstrate that GraftLLM outperforms existing techniques in knowledge transfer, knowledge fusion, and forget-free learning, providing a scalable and efficient solution for cross-capability transfer.

Published as a conference paper at ICLR 2026
KNOWLEDGE FUSION OF LARGE LANGUAGE MODELS VIA MODULAR SKILLPACKS
Guodong Du Zhuo Li', Xuanning Zhou', Junlin Li', Zesheng Shi', Wanyu Lin', Ho-Kin Tang', Xiucheng Li,, Fangming Liu Wenya WangS, Min Zhang', Jing Li
Harbin Institute of Technology, Shenzhen, China
2The Hong Kong Polytechnic University *Nanyang Technological University
4Huazhong University of Science and Technology, China
duquodong 7 gmail. com
jingli, www.t.phaaaaaaao com
ABSTRACT
Cross-capability transfer represents a key challenge in large language model (LLM) research, particularly multi-task integration, model compression, and knowledge fusion. Recent works such as FuseLLM and FuseChat have shown the potential of transferring multiple model capabilities to lightweight models, thereby enhancing adaptability and efficiency. This motivates our investigation into more efficient methods for cross-capability transfer. However, existing model merging approaches primarily focus on homogeneous models, limiting their applicability. For large, heterogencous models, knowledge distillation with full-parameter Line-tuning often overlooks the student model's inherent capability and risks catastrophic forgetting, while PEFT methods struggle to effectively absorb knowledge from source LLMs. To address these issues, we introduce SLaftLLM, novel grafting-bart based method that stores source model capabilities in a target model + SkillPack format. This approach preserves general capabilities, reduces parameter conflicts, and supports forget-free continual learning and model fusion. We employ a module-aware adaptive compression strategy for parameter updates, ensuring efficient storage while preserving task-specific knowledge. The resulting SkillPack serves as a compact and transferable knowledge carrier, ideal for heterogeneous LLM fusion. Experiments across various scenarios demonstrate that Gra LILMM outperforms ex- isting techniques in knowledge transfer, knowledge fusion, and forget-free learning, providing scalable and efficient solution for cross-capability transfer. The code is publicly available at: https: //t:://iii
//gtttiii /github. com /owoom/duudooog////oo GGrafftLLl
7.7
GraftLLM (ours)
ESBI-Meerring
Teig-ierging
7.5
Qwen Seeni5-Chhi--2e
Average Score
TIRE
PCB- Meeging
Mixtral instruct
7.3-
Tres Marging
LLSMAD 08 insmect
Routed LoRA
PCS-Merging
7.1
IntorUM2 Chat 208
Tein-ennghg
GrafILLM
LM
FaseChat 3.0
Hecess
@pgieChatts....p
Source Models
GraFLLLN toursl
N812-50L44-30.0X
Parameter Morging Methods
6.7
7
15
25
Parameters (x10*)
Figure Comparison of explicit knowledge fusion methods for heterogeneous LLMs MT- Bench, including parameter size analysis.
Figure 2: A comprehensive comparison of im- plicit knowledge fusion methods for heteroge- neous LLMS across multiple benchmarks.
M Corresponding author.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
INTRODUCTION
Cross-capability transfer (Pan et al., 2023; Zhong et al., 2025; Yang et al., 2022; Fujii et al., 2024;
Zhao et al., 2024) aims to combine migrate different skills and task abilities across LLMs, enabling a single model to benefit from capabilities originally distributed among multiple specialized sources. This paradigm has has received increasing attention in LLM research, driving progress in key applications such as multi-task fusion (Yang et al., 2024d), model compression (Wang et al., 2024c; Huang al., 2024), and continual learning (Tang et al., 2025). KnowPara (Zhong al., 2024) employs sensitivity-basee techniques to extract and align knowledge-specific parameters (Panigrahi al., 2023) across different models. FuseLLM (Wan al., 2024a) and FuseChat (Yang et al., 2025b) showcase the potential of distilling multiple models into lightweight target, while EvoLLM (Akiba et al., 2025) introduces an evolutionary approach (Du et al., 2024c;a) to automatically combine diverse open-source models without extra training data. These methods collectively tackle the core challenge of efficient and reliable knowledge transfer across heterogeneous LLMs.
Existing knowledge grafting methods (Deng et al.. 2024) aim to enable cross-capability transfer but are mostly limited to small (Panigrahi et al., 2023; He et al., 2024) or structurally identical models (Yu et al., 2023), which constrains their applicability to heterogeneous LLMs. To address this challenge, we introduce GraftLLM, which encodes model capabilities as a combination of a target model and lightweight SkillPack. A SkillPack is modular, task-specific set of parameter deltas obtained via distillation from heterogeneous source models. This design preserves the strengths of both target and source models, enhances parameter and storage efficiency, mitigates forgetting, and facilitates multi-task transfer and model fusion by reducing parameter conflicts (Yadav et al., 2024). By contrast, knowledge distillation-a widely adopted grafting strategy typicallyy follows two paradigms: cull-parameter distillation and PEFT-based fine-tuning, The former often disregards the student model's intrinsic capabilities and risks catastrophic forgetting (Alexandrov et al.. 2024), while the latter, though more parameter-efficien generally underperforms full fine-tuning and struggles to absorb sufficient task knowledge from source models.
We consider a heterogeneous capability transfer scenario where source model capabilities are extracted via synthetic data (Yang et al., 2025b), integrated into the target model through full-model fine- tuning, and further refined with preference optimization (e.g.. DPO (Rafailov et al., 2023)). The resulting parameter deltas capture the specialized knowledge gained during this process. To enable efficient storage and transfer, compresses these deltas before and after specialization. By adapting pruning (Yu et al., 2023), low- rank decomposition (Lu al., 2024), and adaptive quantization (Ping al., 2024) to each module's structure, our method balances compression ratio with task knowledge retention. The compressed representation, termed a SkillPack, serves as a compact, transferable knowledge unit, supporting scalable integration and continual specialization without catastrophic forgetting.
introduce module-aware adaptive compression strategy that
Stronger Ability
New Ability
Contaminated, Toxic
Multi-Teachers
/ Privacy Leakage
Source Model
Capability Transfer
Original Task
J
Continual Learning
Unlearning
I Fusion
Knowledge Distillation
AW Stronger
len Forgetting
Ww
Exposed & Risky
Compete & Conflict
Fully Fine-Tuned Model
Knowledge Grafting
Pretrained
SkillPack
Stronger
Forget-Free
Private, Clean & Safe Routing Ensemble
Figure 3: Comparision of knowledge distillation and knowledge grafting in various scenarios.
We anticipate that the knowledge grafting method will provide advantages in various scenarios, as illustrated in Fig. 3. (1) First, it nearly matches full-parameter distillation in learning from a source model with superior original-task capability. (2) Second, since grafting does not alter the target model's parameters, it proves highly effective for forget-free learning, allowing the source model to acquire new abilities. (3) Additionally, the grafted modules can be easily unloaded, facilitating
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
unlearning, detoxification, decontamination, and other processes, thus helping mitigate issues like privacy leakage. (4) Finally, GraftLLM employs a routing mechanism to support model fusion and multi-task learning, avoiding parameter competition and conflict, further enhancing its applicabilityy.
To empirically validate the effectiveness of Graft we conducted extensive experiments in various cross-capabilityy transfer scenarios, demonstrating our approach's advantages from three perspectives: (1) Knowledge Transfer and Compression: using LLaMA3 as the target, we grafted capabilities from sources like Dwwen22.5-2BBInnsrrue (Yang et al., 2024a) under SFT and DPO settings, significantly outperforming PEFT and Twin-Merging (Lu et al., 2024) on general and task-specific tasks. (2) Knowledge Fusion: we tested on 10 benchmarks under both explicit and implicit heterogeneous model fusion scenarios, with LLaMA3. 1AAAA.1-8B 1-8B-18BBBae --s-Inst (Dubey et al., 2024a) and D.ven-2.5-7B-Instrun (Yang et al., 2024a) as target models, showing substantial improvements over existing methods, as shown in Fig. 1 and 2. (3) Forget-free Learning: our method better mitigates catastrophic forgetting, achieving stronger forget-free learning performance.
This paper makes three significant contributions: (I). We highlight the necessity of cross-capability transfer between heterogeneous large language models and identify limitations in existing methods regarding generalization and adaptability. (2). We propose traftLIa which structures cross-model capabilities as SkillPack, offering high performance, forgetfulness resistance, and easy integration for practical applications.
(3). Experiments show Graft LLM significantly improves performance in knowledge transfer and compression, heterogeneous model fusion, and forget-free learning tasks.
2 RELATED WORK
Knowledge Distillation Knowledge distillation (Hinton et al., 2015) plays a crucial role in enabling capability transfer (Wan et al., 2024a; Zhong et al., 2024; 2025) across heterogeneous large language models (LLMs). Despite the progress made by knowledge distillation methods in merging large language models (LLMs), two main approaches have emerged: one involves complex multi-task training (Yang al., 2025b) for model sharing, but often fails to achieve optimal performance for individual tasks (Shen et al., 2024; Yang et al., 2024c); the other uses pairwise distillation (Wan al., 2024b; Yan al., 2025) followed by parameter merging (Li et al., 2023a; Matena & Raffel, 2022). but conflicts between tasks during fusion can lead to performance degradation (Yadav et al., 2024). To address this, routing mechanisms (Muqeeth et al., 2024; Li et al., 2024a) have been introduced to preserve single-task performance while reducing task interference (Yang et al., 2024e). However, routing requires each branch to be highly parameter-efficien to minimize resource usage (Lu al., 2024; Kang et al., 2024). While PEFT methods such LoRA (Wu et al., 2024) introduce lightweight adapters, they often fall short of the performance achieved by full-parameter fine-tuning (Ding et al., 2023). To address this, we propose a strategy that first fine-tunes all parameters and then modularizes them, providing stronger support for routing and fusion.
Model Fusion Most existing model merging approaches primarily focus on homogeneous settings, where models share the same pre-trained backbone. Within this scope, Model Grafting (Panigrahi et al., 2023) was first proposed as a technique to transplant a small subset of fine-tuned parameters onto the pre-trained model, effectively recovering the performance of the original fine-tuned model. Meanwhile, Task Arithmetic (Ilharco et al., 2023; Zhang et al., 2023) introduced the concept of task vectors, and Ties Ties-Mggging (Yadav et al., 2024) demonstrated the importance of pruning these vectors. Building on this idea, subsequent works like DARE (Yu et al., 2023) and TSV-Merge (Gargiulo al., 2025) applied it merging large language models. Beyond task vector pruning, methods such as mask localization (Panigrahi et al., 2023; He et al., 2024), singular value decomposition (SVD) (Wang et al.. 2024f: Yuan al., 2023), and quantization (Frantar et al., 2022: Lin al., 2024) have also been widely adopted for model compression and merging, For example, Model Tailor (Zhu et al., 2024) generates sparse masks based on salience and sensitivity scores, while Talls Mask (Wang et al., 2024c) and EMR-Merging (Huang et al., 2024) introduce additional masks to localize task-specific information and reduce storage costs. SVD is applied in various contexts: Twin-Merging (Lu et al., 2024) uses it for modular routing, KnOTS (Stoica et al., 2024) for LoRA fusion, and D (Gu et al., 2025) for MoE-based LLMs. Methods like BitaDelta (Liu et al., 2024) and Calt--Come (Ping al., 2024) incorporate quantization for further compression. In our Graft LLM work, propose a moodule-adattiv delta compression strategy for merging heterogeneous LLM that balances performance and storage efficiency. More comparisons with related work provided in App. A.
3

---

Published as a conference paper at ICLR 2026
METHODOLOGY
GraftLLM
dule-based Verification
Module-Aware Adaptive Compression
DPO Data
Reward Model
Sample
Responses
SFT Data
Pruning
Quantization
Decomposition
Source Model
Difference
Embedding Head
MLP
Attention
Task Vector
SkillPack
Target Model
Fully Fine-Tuned
(a) Specific Knowledge as SkillPack
Gemma
Routed GraftLLM
Ensembled GraftLLM
Data
Data
Mistral
Data 1
Qwen
Router
LLaMa
(b) Knowledge Fusion
(c) Forget-Free Learning
Figure 4: Overview of GraftLLM. GraftLLM transfers capabilities across heterogeneous LLMs
and extracts them into compact modular SkillPacks, enabling efficient knowledge fusion.
METHODOLOGY
In Sec. 3.1, formalize the problem of efficient LLM fusion. Sec. 3.2 introduces our proposed method, GraftLLM which enables cross-capability transfer between heterogeneous models and encapsulates the acquired knowledge into a compact SkillPack. Finally, See. 3.3 illustrates how the modularity and composability of SkillPacks support downstream applications, including heteroge- neous model fusion and forget-free learning.
3.1 PROBLEM SETTING
We consider a heterogeneous adaptation scenario involving source model 6 and target model 0g.. transfer capabilities from the source to the target, we adopt a two-stage training pipeline: supervised fine-tuning (SFT) followed by direct preference optimization (DPO).
The parameters after this two-stage adaptation are denoted fig. and we define the difference from the original parameters the delta parameters:
A = 1gr 0gg
which captures the task-specific adaptation knowledge and serves the foundation for subsequent modular compression and transfer.
To enable efficient storage and transfer, we compress A using a module-specific adaptive strategy. Each submodule m E .M is compressed with dedicated operator Cm,.),, selected based on its functional role and sensitivity. The compression may involve pruning, low-rank decomposition, or quantization, with bitwidth adaptively assigned according to the importance of each component. The resulting compressed update is:
Ao = (Cmm (Am (A0m) IEE::
(2)
which forms SkillPack-a compact, transferable representation of the acquired task knowledge, suitable for heterogeneous model fusion, as shown Fig. 4.
3.2 KNOWLEDGE AS SKILLPACK
To achieve compact and transferable skill representations, propose module-aware adap- tive compression strategy,
previous uniform compression methods methods-applie dif- ferent operations based each module's role, sensitivity, and compression difficulty. As
4

---

Published as a conference paper at ICLR 2026
METHODOLOGY
shown in Fig. and Fig. 6, moderate pruning preserves performance for the Embedding and Output Head. For Attention modules, the fast-decaying singular value spectrum al- lows low-rank SVD compress projection matrices without significantly reducing represen- tational capacity. MLP modules, with strong nonlinear transformations, require conservative compression to retain critical singular vectors and avoid performance degradation. Accord- ingly, we assign modulc-specific compression operators to the delta parameters as follows:
Embedding and Output Head. We apply magni- tude pruning with a retention ratio a, preserving
the weights with the top proportion of absolute magnitudes:
Accuracy [%]
60
55
Agembed Prune, Agembed).
(3)
50
Embedding & Output Head
Attention Modules
Attention Modules. For attention blocks, we apply low-rank decomposition using SVD:
Modules
0.10 0.20 0.30 0.40 0.50 0.60 Sparsity Ratio Pruning
0.80
Agmlp UΣVT, s.t. rank(2))
(4)
Figure 5: Performance of delta pa- rameters across modules under differ- ent pruning ratios.
where Agmlp E R x his U, Rh E, E I and V, E Rhaxr correspond to the top r components.
75
MLP Modules. For MLP modules, we employ conservative SVD scheme that retains essential ranks, with the truncation rank determined by the cu- mulative explained variance under an energy thresh- old B:
70
Accuracy (%g
65
60
S
55
Embedding Output
Σ of Σ min(dendda)
Attention Modules
(5)
50
MLP Modules
To further reduce storage overhead, we apply mixed-precision quantization to the pruned matrix or SVD-derived compo- nents. Each SVD component is quantized with bit precision k. adaptively chosen based on its importance in the decom- position.
Ratio of Reduced Rank
Performance of delta pa- rameters across modules under differ- ent reduced rank ratios.
Figure 6:
Quanty (0,x) argmin 10x
0x1
(6)
where Quanty denotes k-bit quantization operator (k: 1). For each group of singular vectors indexed by [r] Thegin Tend, we apply group-wise quantization with GPTQ (Frantar et al., 2022) as follows:
Quant,
[r]
ÜH
Quant, (UM) Enr VTr x)
where En denotes the diagonal matrix of singular values corresponding to the selected rank range. The quantization precision & can be adaptively adjusted across different groups based on the relative importance of singular values.
3.3 SKILLPACK COMPOSITION AND ROUTER MECHANISM
GraftL.LM enables modular and composable integration of task-specific knowledge across hetero- geneous LLMs through SkillPacks A0. Each SkillPack first decoded through dequantization obtaion A0(d)), and then reconstructed via truncated SVD to recover the task-specific delta parameters:
Agidg) ~ UEV = A6,
(8)
where U, Σ, V are obtained from the truncated SVD decomposition. The reconstructed delta A/ is then added back to the base model parameters to produce the final fused model:
O Ogg A0.
5

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTAL SETUP
To support flexible and selective integration across tasks, router function R is introduced. The router determines which SkillPack is applied to which submodule or task-specific region the target model. For example, for set of n SkillPacks [A;))=-> the fused model is computed as:
Oused
Oggt + Σ R(AO))
(10)
where R can be instantiated in two ways:
lightweight feed-forward network trained to predict the most suitable source model or SkillPack based on input features.
Classifier-bssed router:
Manual task-type assignment: a deterministic mapping from known task types their corresponding SkillPacks.
During inference, we typically use top-1 routing for efficiency, ensuring that only the most relevant SkillPack is activated. This strategy minimizes inference overhead while preserving the benefits of modular, task-specific delta parameters. More details provided in App. B.3
Overall. this formulation provides unified interface for modular knowledge transfer, enabling both heterogeneous model fusion and task-adaptive capability integration in a principled, efficient, and scalable manner.
4 EXPERIMENTAL SETUP
4.1 BASELINE METHODS
For pairwise LLM grafting, we evaluate two categories of baselines: (1) PEFT methods, comparing LoRA under varying rank settings in both SFT and DPO stages; (2) Task Vector Compression,
which evaluates full-parameter tuning followed by magnitude pruning (Yu et al., 2023; Yadav et al., 2024), SVD (Lu et al., 2024; Stoica et al., 2024), or quantization (Ping et al., 2024; Yang et al., 2025a) across varying compression ratios.
For heterogeneous knowledge fusion, we benchmark against: (1) Multi-teacher distillation (e.g.. FuseLLM (Wan et al., 2024a)); (2) Parameter merging approaches such as Task Arithmetic (Ilharco et al. 2023), TIES-Merging (Yadav et al., 2024), SCE-Merging (Wan al., 2024b), PCB-Merging (Du et al., 2024b). DARE (Yu et al., 2023), and InfiFusion (Yan et al., 2025); (3) Routing-basse methods, including Routed LoRA (Hu et al., 2022) and Twin-Merging (Lu et al., 2024), and (4) Mask-based fusion strategies like TALL Mask (Wang et al., 2024c) and EMR-Merging (Huang et al., 2024), which leverage unified task vectors and localization.
For forget-free learning, we use LoRA, Model Grafting (Panigrahi et al., 2023), and Model Tailor (Zhu al., 2024) as baselines. Details of all baselines provided in App. D.
4.2 DATASETS AND ARCHITECTURES
To showcase the effectiveness of Gr Graft L.M.M,,, we conduct a comprehensive evaluation across multiple domains, including instruction following, question answering, reasoning, mathematics, and coding. We use 10 established benchmarks, grouped into four categories, with domain-specific response sampling strategies to ensure fair comparison. Full benchmark details are available in App. E.3.
For pairwise LLM grafting in Sec. 5.1 and Fig. 7, 8, we uses Llama-3. 1. --8--Instruct as the target model, grafting capabilities from strong source model Brenn225-72B-Innstrue
(Yang al., 2024a). For explicit knowledge fusion in Sec. 5.2 and Tab. 1, we follow the FuseChat 2.0 (Wan et al., 2024b) setup by fusing chat-centric L.LMs of varying architectures and scales, using OpenChat-3.55777 (Wang et al., 2024b) as the pivot model and six representative chat models as sources. For implicit fusion in Sec. 5.2 and Tab. 2, we adopt the FuseChat 3.0 (Yang et al., 2025b) setup with Llama-3.1 1.B--....-----tttttte and Dwwen22.5-BBllnstrue as target models and 4 different stronger LLMs. For forget-free learning in Sec. 5.3 and Tab. 3. we sequentially acquire math and coding abilities using both SFT and DPO datasets. Further architectural details are provided in App. E, while additional implementation details can be found in App. F, covering training procedures F. hyperparameter settings F.2, and computational resources and runtimes F.3.

---

Published as a conference paper at ICLR 2026
5 RESULTS
Grafting by SFT from Source Models
Grafting by DPO from Source Models
SFT
SFT+DPO
Average Score MMLU
73
Average Score on Math
72-
GM fours) SVD+ Quantization
69
GraftLLM (ours)
SVD Quantization
Pretrained
SVD
Pruning
Pruning
Pretrained
LoRA
LoRA
70
67
10
# Parameters (x10%)
Parameters (x109)
Figure 7: Comparison of parameter efficiency and MMLU performance across different meth- ods for LLM capability transfer with SFT.
Figure 8: Comparison of parameter efficiency and average performance on GSM8K and MATH across different methods under the DPO setting.
RESULTS
this section, we evaluate Gra ft LLM in various settings, comparing with other methods, including pairwise heterogeneous LLM grafting 5.1, knowledge fusion 5.2, and forget-free learning 5.3, while also highlighting its potential for unlearning tasks like model detoxification (see App. C.2).
5.1 PAIRWISE GRAFTLLM
We demonstrate the effectiveness of parameter--fficieet capability transfer between paired models. Fig. 7 and 8 show that while PEFT and other compression methods perform reasonably well in simple SFT scenarios, their effectiveness drops significantly-on or even fails under more complex DPO settings. In contrast, our method consistently achieves performance close to a fully fine-tuned target model, highlighting its robustness and efficiency.
5.2 GRAFTLLM FOR KNOWLEDGE FUSION
We explore two approaches LLM knowledge fusion: explicit fusion, which aligns tokens and probability distributions, and implicit fusion, which transfers knowledge through generated data.
Explicit Knowledge Fusion. shown in Tab. compared to the best results from Merging- LLM fusion, our approach achieves a significant performance boost without introducing a large number additional parameters. Compared to routing-based fusion methods, our approach achieves better performance with lower parameter cost. Unlike Twin Merging, which relies on higher ranks, our method delivers superior results more efficiently. Compared to TALL-Mask and MMM-MMgggig, avoid the overhead introduced by using unified task vector.
Compared to the source models, our approach improves the target model, OpenChat-33,5 with only 28% increase in parameter size, achieving performance comparable to Dixxtra-8x7B--Instruu and Qwen1.5 1.wenl.--hha--222 In fact, on MT-Bench, our model outperforms all source models, setting new benchmark. Additionally, on AlpacaEvval 2.0, it shows an 8.07% improvement over the best parameter fusion method. More result deails are in App. C.2.
Implicit Knowledge Fusion. We evaluate the effectiveness of implicit heterogeneous model fusion on 10 benchmark tasks, as shown in Tab. 2, comparing three representative methods. (1) PCB-Merging (pairwise distillation + parameter fusion) distills knowledge from multiple models and merges their parameters, but suffers from conflicts between source models, limiting its ability to balance multi-task performance. (2) Twin- Twin-Merging (pairwise distillation + routing) uses model decomposition for routing-
basedd based fusion, but experiences significant performance loss during decomposition. resulting in the weakest performance overall. (3) FuseChat-3 (multi-teacher distillation) integrates knowledge from multiple tasks, yet still falls short of task-specific upper bounds especially on the GPQA (Rein
7

---

Published as conference paper at ICLR 2026
5 RESULTS
Table 1: Overall results of explicit LLM knowledge fusion on AlpacaEval 2.0 and MT-Bench. The best-perforning results for both parameter merging and routing-based methods are shown in bold, while the performance difference between the two highlighted in green.
AlpacaEval
MT-Bench (GPT-4-01255 255-Preee--
Model
#Params
Win Rate
L.C Win Rate
Ist Turn
2nd Turn
Average Score
Source LLMs
OpenChat-3 5-78 Wang al. (2024a)
7B
10,20
14.90
7.14
6.55
6.84
Starling-I M--7B-alpha
(2023)
NI12 SOLAR 10.7B Kim et al. (2023)
10.71
12.22
18.13
7.11
6.36
6.74
M2.
(2024)
7.78
Mixtral 8x7BB Instruct Jang et (2024)
8x7B
18.30
23.70
7.76
7,00
7.38
Qwenl 5-Chht--222
(2023)
Multi-teacher Distillation
Rusel Wan et al. (2024a)
7B
10.56
14,50
7,36
6.40
6.88
Pairwise Distillation Parameter Merging
Task Arithmetioun.san Ilharco al. (2023)
7B
10.67
15.78
7.54
6.78
7.22
Tjes-Merging Yaday et (2024)
11.55
16.73
7,59
7.03
7.31
SCE- Mergingaatoo Wan (2024b)
11.63
16.89
7.61
7,05
7.33
PCB-Meggingssatsse Du (2024b)
11.82
17.22
7.71
7.36
PCB CCB-Mrring+DARR Eic Yu et (2023)
7B
11.96
17.35
7.79
6.99
7.39
oosio Yan (2025)
11.74
17.21
7.68
Pairwise Distillation Rouler
Rosted LoRA :512
14.18
10.16
15.48
7.14
6.75
6.95
Routed LoRA r1024
12.57
7.52
Maskieass Wang et (2024c)
16.7B
13.69
22.76
7,92
7.14
7.53
EMR-Merging seattss Huang (2024)
14.52
Twin-Merging 1512
14.1B
12.20
19,90
7.74
7.07
7,40
Twin-Merging r1024
15.93
Routed GraftLLM (Ours)
9.28
16.56(+4.6)
25.42(+88.07)
8.05(+0026)
7.35(+0.27)
7.70(+0.31)
Table 2: Overall results of implicit LLM knowledge fusion across 10 benchmark tasks.
Llama-3.1 3.10a-331-B--amm 3.ntucctttt
Qwen-2.5-778 Dwon-2.5-7B-Instruuc
Category
Benchmark
PCB
Twin- Merging
Fuse Chat-3
Routed GraftLLM
PCB. Merging
Twin- Merging
Fuse Chat-3
Routed
Base
Merging
Base
GraftLLM
MML.U-Pro
49.7
48.6
50.2
48.8
$1.3
54.0
53.7
54.5
52.8
55.4
General
MMLU-rodux
70.5
71.5
72.6
73.0
75.1
75.3
76.2
GPOA-Diamond
33.6
35.4
36.1
34.8
37.7
34.7
34.2
36.8
33,9
38.1
GSM8K
85.9
87.2
86.4
88.0
88.2
91.7
91.5
91.3
91.7
92.0
Mathematics
MATH abake C-T)
50.7
54.2
55.2
54.8
55.9
75.0
73.2
72.1
73.5
75.0
AMC 23
25.0
30,0
27.5
37.5
35.0
52.5
52.5
50.0
57.5
35.0
Coding
HumanEval sbxes
68.3
69.8
70.5
72.0
85.4
83.1
81.9
79.9
MBPP dwti
66.9
71.7
70.3
71.4
72.8
80,2
82.7
81.6
83.1
84.5
Instruction
AlpacaEval-2 (USS)
28.3
61.2
52.4
65.4
64.8
34.2
58.9
53.3
63.6
61.5
Following
MT- MI-Bench
8.4
8.6
8.4
8.6
Average
48.7
53.8
52.8
$5.2
56.0
59.0
61.4
60.4
62.0
63.2
et al.. 2023) benchmark. where other tasks offer little benefit. In contrast, method combines the performance strengths of pairwise distillation with the parameter efficiency of modular routing, effectively reducing task conflicts and fusion costs. When using LLaMA3. 1--.3..18-BB1 --s-ruutttutt and Qwen- 2.5-7B -.--7B-Inttt as target models, our approach achieves average performance gains of 0.8 and 1.2, respectively, demonstrating significant advantages. More result deails are in App. C.I.
5.3 GRAFTLLM FOR FORGET-FREE FREE LEARNING
Table 3: Forget-free learning results on code and math tasks using 1---MA...--BB-InSe
Additional Parameters
Code Benchmarks (Original task)
Math Benchmarks (New task)
Method
HumannEval
HumanEEval+
MBPP
AMC23
Average
LLaMA3, 1-8B-Instrucc
68.3
61.6
66.9
54.8
85.9
50.7
25.0
59.0
Multi LoRA r256
1,483
68.8
61.8
67.7
55.6
86.2
51.3
25.0
59.5
Model Graftingnesann
803M
70.4
63.9
69.1
57.5
87.2
53.4
27.5
61.3
Model Tailorecung
803M
71.4
64.2
71.1
59.4
87.6
54.5
27.5
62.2
GraftLIM (ours)
803M
72.0
65.2
72.2
61.8
88.2
55.9
35.0
64.3
8

---

Published as a conference paper at ICLR 2026
6 ANALYSIS
We evaluate Graft in a forget-free learning setting, where L 1--B-Innstruch is first trained on code (original task) and then on math (new task), using data generated from stronger source models. The final model is evaluated math. Under the same 10% parameter budget as prior methods like Model Grafting and Model Tailor, Gra fLLLM delivers consistently stronger performance while mitigating forgetting, outperforming existing approaches by an average of 2.1% (Tab. 3). More deails are in App. C.I.
seven benchmarks in total-four for code and three for
5.4 PERFORMANCE ON HIGHLY DISTINCT FUSION DOMAINS
Our method is explicitly designed to decouple conflicting task behaviors into separate SkillPacks, preventing cross-task interference and enabling near-lossless capability fusion- when the underlying tasks are highly distinct. To validate this design in scenarios with highly divergent domains, conducted experimental setup inspired by AdaptLLM (Cheng et al., 2023). We first report the performance of models fine-tuned individually on each domain. As shown in the table below, a model fine-tuned one domain suffers substantial degradation on other domains, sometimes performing worse than the base model, highlighting the limitations of traditional merging approaches in handling conflicting updates.
experiment involving finance, law, and biomedicine, following
In contrast, SkillPack-basel fusion effectively isolates domain-specifie delta parameters and recom- bines them with minimal interference, achieving hear-lossless multi momii-domain performance. Further- more, even when compressing the model to just 10% of its original parameters, our method still reaches nearly 99% of the original performance. This means that the performance originally requiring three separate 7B fine-tuned models can now be matched by GraftLLM with only an additional 30% of parameters, demonstrating significant parameter efficiency without sacrificing accuracy.
Table 4: Performance of GraftLLM compared with baselines across Biomedieine, Finance, and Law
domains. The average column also indicates relative performance to the reference.
Methods
Params
Biomedicine
Finance
Law
Average
LLaMA-7B
7B
44.2
58.6
34.2
45.7
LLaMA-7B-BiS
7B
47.3
57.9
34.5
46.6
LLaMA-7B-Fiing -inacee
7B
43.7
63.4
34.0
57.0
ILLMAA7B--La
7B
44.1
58.2
38.5
46.9
AdaptI.LM-P7
3x7B
47.3
63.4
38.5
49.7100%
FuseChatpcrre 25]
45.6
60.1
36.3
47.390%
Twin Swin-MergingiNaae
15.3B
45.9
61.3
36.7
47.9 47.9%6%
(rrr)LLM(Ouss)
9.1B
47.2
63.4
38.2
49.6g9%
ANALYSIS
6.1 ABLATION STUDY
We conduct an ablation study to assess the impact of each component within module-aware adaptive strategy. As shown in Tab. 5, evaluate the effect of replacing removing individual compression modules, using the ALaMA3.1-87 Instructt model on the GSM8K and MATH validation sets. To ensure fairness, all configurations maintain comparable overall compression ratio of approximately 5%. The results reveal that different model components benefit from tailored compression strate- gies. Quantization emerges as a critical factor for preserving performance, with maxed-precision quantization causing minimal degradation. In contrast, the MLP modules exhibit high sensitivity to compression: applying suboptimal methods to these layers leads to notable performance drops, underscoring their importance in the compressed model architecture. More details of the ablation study are provided in App. B.
Table 5: Ablation study on module-aware adaptive strategy on the MATH task.
Methods
Null
wlo Quantization
w/o Mixed Quant
Pruning
SVD
Low Rank SVD.
Embedding and Output Head
71.3
71.8
71.8
72.1
71.9
71.7
MLP Modules
68.7
69.2
71.5
70.2
72.1
71.2
Attention Modules
70.7
71.3
71.8
71.2
71.8
72.1

---

Published as a conference paper at ICLR 2026
7 CONCLUSIONS
6.2 EFFECT OF TASK DIFFICULTY AND DATA SETTINGS
We assess our method across diverse settings, including varying sample sizes, numbers of source models, and task difficulties. We also study the impact of different compression ratios under both SFT and DPO paradigms. As shown in Fig. 9a and 9b, with a compression ratio (CR) 10%, our method consistently retains nearly 100% of the original performance both SFT and DPO settings. In CR ratio 5%, performance decreases as task difficulty increases particularly under DPO- - highlighting the greater challenge of compression in preference-alignee scenarios. Fig. shows our method is robust cntructiionfollovwee tasks, highlighting both strengths
on simpler tasks but less stable on DPO- and limitations across alignment challenges. Further analysis of compression hyperparameters, including rank and exxedpprecisio ratios, is in App. B.2.
Performance on MT-Beech
Performance Code Benchmark
11.00
1.00
De0.5% ON
Average Accuno
0.99
0.922
0.98
0.30
0.98
0.96
DVO 1:78 ON
0.97
DMC 15%
5TT 15% Ch
DP3 15% Cto
2
0.96
Number of Samples (x30ff)
Number of Source Mbdils
Varlouz Tasks
(a) Training Samples
(b) Source Models
(c) Task Difficulty
Figure Performance trends across task difficulty using LLAMA3.I-BBInstree
6.3 ROUTER BEHAVIOR AND SKILLPACK INTERACTION
We further examine how the router behaves under different task relationships and evaluate whether combining multiple SkillPacks can provide additional benefits. When SkillPacks correspond to highly similar tasks(e.g.. different Chat LLMs). their latent features naturally overlap, making the distinction between experts less pronounced. In these cases, top-1 routing is inherently ambiguous, and ensembling the outputs of multiple lkillPacks-weigeeee performance gains, although with increased inference cost.
by the router can yield noticeable
In contrast, when SkillPacks correspond to clearly distinct domains such as finance and biomedicine, the router learns a confident mapping to the most relevant expert. Cross-expert ensembling brings little benefit because their knowledge does not reinforce each other. This matches our empirical findings. Overall, the router leverages complementary skills for aligned tasks, avoids interference for divergent ones, and preserves efficiency through selective activation.
Table 6: Analysis of router behavior and multi-SkillPack ensembling across three fusion settings.
Explicit LLM Fusion (AlpacaEval 2.0)
Implicit LLM Fusion (Avg. of 10 benchmarks)
Finance-Law-Bio LLM Fusion
Inference Overhead
Methods
FuseChat
11.63 / 16,89
55.2 56.0 (+0.8)
47.33
1x
Grafting Top 1 SkillPack
16.56 (+4.9) / 25.42 (+8.5)
49.62 (+0.23)
1.03x
Grafting Top 2 SkillPacks
17.33 (+0.8) / 26.28 (+0.9)
56.3 (+0.3)
49.64 (+0.02)
2.03x
Grafting Top. 3 SkillPacks
17.49 (+0.2) 26.72 (+0.4)
56.4 (+0.1)
49.62 (-0.01)
3.03x
6.4 LIMITATION AND FUTURE WORK
While our approach provides insights into knowledge transfer between LLMs, it relies on the quality of prior supervised fine-tuning (SFT) and Direct Preference Optimization (DPO), with suboptimal distillation limiting its ability to fully capture source model capabilities. Future work may explore alternative inference strategies and develop automated, robust methods for compression operations during deployment to improve efficiency, scalability, and robustness.
7 CONCLUSIONS
We present GraftLLM, a scalable framework for efficient cross-capability transfer in large language models. By compressing task-specific updates into modular SkillPacks, method preserves knowledge while avoiding interference and forgetting. Experiments show strong performance in knowledge fusion and continual learning, outperforming prior methods under various settings.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
ETHICS STATEMENT
This research was conducted in full accordance with established ethical standards in artificial in- telligence and machine learning. All experiments utilized publicly available datasets and models under their respective licenses, without involving any personally identifiable sensitive information. The proposed methods are intended strictly for academic and scientific purposes, aiming to advance understanding in machine learning rather than for deployment in high-stakes prrision-making without appropriate safeguards.
We acknowledge that advances in Al systems entail potential societal risks, including concerns related to fairness, misuse, privacy, and environmental impact arising from computational resource demands. To address these issues, prioritize responsible reporting of results, transparent disclosure of limitations, and clear distinction between research contributions and downstream applications.
Future research building upon this work should continue evaluate potential ethical implica- moons-particularlly regarding bias, safery, and dual-use risks-and implement appropriate measures promote beneficial, equitable, and responsible outcomes.
REPRODUCIBILITY STATEMENT
Our implementation, including all code, training scripts, and evaluation datasets, is available at: http://giiaabbbomuuuuuuooo/gggg2
ACKNOWLEDGEMENTS
This work supported in part by National Natural Science Foundation of China (62476070), Shen- zhen Science and Technology Program UCC1100242201123300000 GXWD20231 128 128103232001, KOTDD20240722
Department of Science and Tech- ZDSYS2023062609 0024A15150115400 and National Key R&D Program of China nology of Guangdong (SQ2004YFEE0200522) This work also supported part by the Major Key Project of PCL under Grant PCI.2025 A10 and PCI P024204, and in part by the Shenzhen Science and Technology Program under Grant RCJC20231211889 181000
REFERENCES
Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. Nature Machine Intelligence, pp. 1-10, 2025.
Anton Alexandrov, Veselin Raychev, Mark Mueller, Ce Zhang, Martin Vechev, and Kristina Toutanova. Mitigating catastrophic forgetting in language transfer via model merging. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 17167-17186, 2024.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.0 2tt.:22080.77322, 2021.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. InternIm2 technical report. arXiv preprint 2403:22403.12977 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint 1rXiv:2107.03374 2021.
Daixuan Cheng, Shaohan Huang, and Furu Wei. Adapting large language models to domains via reading comprehension. arXiv preprint arXiv:2309. 20.55530000 2023.

---

Published as a conference paper at ICLR 2026
REFERENCES
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXhv:2110.14168, 2021.
Wenlong Deng, Yize Zhao, Vala Vakilian, Minghui Chen, Xiaoxiao Li, and Christos Thrampoulidis. Dare the extreme: Revisiting delta- patam-arrmetter pruning for fine-tuned models. arXiv preprint arXiv:2410.09344, 2024.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang. Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficien fine-tuning of large-scale pre- tre-trained language models. Nature Machine Intelligence, 3(3):220-2333 2023.
Guodong Du, Runhua Jiang, Senqiao Yang. Haoyang Li, Wei Chen, Keren Li, Sim Kuan Goh, and Ho-Kin Tang. Impacts of darwinian evolution on pre-trained deep neural networks. In 2024 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2024a.
Guodong Du, Junlin Lee, Jing Li, Runhua Jiang, Yifei Guo, Shuyang Yu, Hanting Liu, Sim Kuan Goh, Ho-Kin Tang, Daojing He, and Min Zhang. Parameter competition balancing for model merging. Advances in Neural Information Processing Systems (NeurIPS), 37, 2024b.
Guodong Du, Jing Li, Hanting Liu, Runhua Jiang, Shuyang Yu, Yifei Guo, Sim Kuan Goh, and Ho-Kin Tang. Knowledge fusion by evolving weights of language models. In Findings of the Association for Computational Linguistics ACL 2024, pp. 11727-11742, 2024c.
Guodong Du Zitao Fang. Jing Li, Junlin Li, Runhua Jiang, Shuyang Yu, Yifci Guo, Yangneng Chen, Sim Kuan Goh, Ho-Kin Tang, Daojing He, Honghai Liu, and Min Zhang. Neural parameter search for slimmer fine-tuned models and better transfer. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), 2025a. doi: 10. 18.18V53/v/ 2025.acl- 1570. URL https: / www.nnchlllooooooooooooooooooola org/2025 acl longl.o............ 1570/.
Guodong Du, Xuanning Zhou, Junlin Li, Zhuo Li, Zesheng Shi, Wanyu Lin, Ho-Kin Tang, Xiucheng Li, Fangming Liu, Wenya Wang. Min Zhang, and Jing Li. Knowledge grafting of large language models. arXiv preprini 1xXiv:2505.18500 2025b. URL https: htapps://arxae /arxivivv.vv.////////a/////avvvr org abs/ 2505. 18502.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024a.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024b.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-comtrolled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024.
Elias Frantar, Saleh Ashkhoos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210. 77323, 2022.
Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki lida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. Continual pre-training for cross- mooss-lingual llm adaptation: Enhancing japanese language capabilities. In First Conference on Language Modeling, 2024.
Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli. Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodola. Task singular vectors: Reducing task interference in model merging. Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 18695- 18705, 2025.
Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al. Are done with mmlu? arXiv preprint
2024.
12

---

Published as a conference paper at ICLR 2026
REFERENCES
Hao Gu, Wei Li, Lujun Li, Qiyuan Zhu, Mark Lee, Shengjie Sun, Wei Xue, and Yike Guo. Delta decompression for moe- based ilms compression. arXiv preprint arXiv:2502. 2025.
Zhuoyuan Hao, Zhuo Li, Wu Li, Fangming Liu, Min Zhang, and Jing Li. Echoes as anchors: Probabilistic costs and attention refocusing in Ilm reasoning. The Fourteenth International Conference Learning Representations (ICLR), 2026.
Yifei He, Yuzheng Hu, Yong Lin, Tong Zhang, and Han Zhao. Localize-and-stichh: Efficient model merging via sparse task arithmetic. Transactions Machine Learning Research (TMLR), 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021b.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 1503.05500.0055 025331, 2015.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang. Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint XXXv:2307.13269, 2023.
Chenyu Huang, Peng Yc, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. Emr-merging: Tuning-fie free high-perfone high-perffrmance model merging. Advances in Neural Information Processing Systems, 37:122741-122766 2024.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401 240.0088,, 2024.
Junmo Kang, Leonid Karlinsky, Hongyin Luo, Zhen Wang, Jacob Hansen, James Glass, David Cox, Rameswar Panda, Rogerio Feris, and Alan Ritter. Self-moe: Towards compositional large language models with self-specialized experts. arXiv preprint arXiv:2406. 12034, 2024.
Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, et al. Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling. arXiv preprint 12. 2023.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen, Oliver Stanley, Richárd Nagyfi, et al. Openassistant conversations democratizing large language model alignment. Advances in Neural Information Processing Systems, 36, 2024.
Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystify efficient smoe with hints from its routing policy. In The Twelfih International Conference on Learning Representations (ICLR), 2024a.
Tianle Li. Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena harda-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.1 /939, 2024b.
13

---

Published as a conference paper at ICLR 2026
REFERENCES
Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey. arXiv preprint arXiv:2309. 15698, 2023a.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaBval: An automatic evaluator of instruction-followee following models, 2023b.
Zhuo Li, Guodong Du, Weiyang Guo, Yigeng Zhou, Xiucheng Li, Wenya Wang, Fangming Liu, Yequan Wang, Deheng Ye, Min Zhang, et al. Multi-objective large language model alignment with hierarchical experts. arXiv preprint 200222550..0025, 2025.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-awarc weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6: 87-100, 2024.
James Liu, Guangxuan Xiao, Kai Li, Jason D Lee, Song Han, Tri Dao, and Tianle Cai. Bitdelta: Your fine-tune may only be worth one bit. Advances in Neural Information Processing Systems (NeurIPS), 37: 37:13579-13600, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- on Learning Representations, 2019.
Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. Twin- Twin-merging Dynamic integration of modular expertise in model merging. Advances in Neural Information Processing Systems, 37:78905-78935. 2024.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations, 2024.
Michael Matena and Colin A Raffel. Merging models with fisher weightee averaging. Advances in
Neural Information Processing Systems (NeurIPS), 35:17703-17716, 2022.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 35: 35:17359- 2022.
Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In Proceedings of the Eleventh International Conference on Learning Representations (ICLR), 2023.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint 20Xx:22306.02707. 2023.
Mohammed Muqeeth, Haokun Liu, Yufan Liu, and Colin Raffel. Learning to route among specialized experts for zero-shot generalization. In Proceedings of the 41st International Conference on Machine Learning, pp. 36846, 2024.
Zizheng Pan, Jianfei Cai, and Bohan Zhuang. Stitchable neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16102-16112, 2023.
Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora. Task-specific skill localization in fine-tuned language models. arXiv preprint arXiv:2302.06600 2023.
Bowen Ping, Shuo Wang. Hanqing Wang, Xu Han, Yuzhuang Xu, Yukun Yan, Yun Chen, Baobao Chang, Zhiyuan Liu, and Maosong Sun. Delta-come: Training- free delta-compression with mixed- preccsior for large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
14

---

Published as a conference paper at ICLR 2026
REFERENCES
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 01155372-537444 2023.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google- google-proof q&a benchmark. arXiv preprint urXiv:2311. /2022, 2023.
Li Shen, Anke Tang, Enneng Yang. Guibing Guo, Yong Luo, Lefei Zhang, Xiaochun Cao, Bo Du, and Dacheng Tao. Efficient and effective weight-ensembling mixture of experts for multi-task model merging. arXiv preprini arXiv:2410.214 1804, 2024.
George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, and Judy Hoffman. Model merging with svd to tie the knots. arviv preprint arXiv:2410.19775, 2024.
Anke Tang, Enneng Yang, Li Shen, Yong Luo, Han Hu, Bo Du, and Dacheng Tao. Merging models on the fly without retraining: A sequential approach to scalable continual model merging. arXiv preprint 200002500 2025.
Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge fusion of large language models. arXiv preprint arXiv:240J. 10491, 2024a.
Fanqi Wan, Longguang Zhong, Ziyi Yang. Ruijun Chen, and Xiaojun Quan. Fusechat: Knowledge fusion of chat models. arXiv preprint arXiv:2408. 07990, 2024b.
Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. Openchat: Advancing open-source language models with mixed-quality data. In The Twelfth International Conference on Learning Representations, 2024a.
Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. Openchat: Advancing open-source language models with mixed-quality data. In The Twelfth International Conference on Learning Representations (ICLR), 2024b.
Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jimenez, François Fleuret, and Pascal Frossard. Localizing task information for improved model merging and compression. arXiv preprint arXiv:2405.07813, 2024c.
Mengru Wang. Ningyu Zhang, Ziwen Xu, Zekun Xi, Shumin Deng, Yunzhi Yao, Qishen Zhang, Linyi Yang, Jindong Wang, and Huajun Chen. Detoxifying large language models via knowledge editing. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL), pp. 3093-3118, 2024d.
Peng Wang, Ningyu Zhang, Xin Xie, Yunzhi Yao, Bozhong Tian, Mengru Wang, Zekun Xi, Siyuan Cheng, Kangwei Liu, Guozhou Zheng, et al. Easyedit: An easy-to-use knowledge editing frame- work for large language models. arXiv preprint arXiv:2308.07269 2023.
Peng Wang. Zexi Li, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. arXiv preprint arXiv:2405. 147688 2024e.
Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-llm: Truncation-aware singular value decomposition for large language model compression. arXiv preprini arXiv:2403.07378 2024f.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-pro: more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024g.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-as natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Dyiemonstrations, pp. 38-45, 2020.
15

---

Published as a conference paper at ICLR 2026
REFERENCES
Xun Wu, Shaohan Huang, and Furu Wei. Mixture of lora experts. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024.
Zhaoyi Yan, Yiming Zhang, Baoyi He, Yuhao Fu, Qi Zhou, Zhijie Sang, Chunlin Ji, Shengyu Zhang, Fei Wu, and Hongxia Yang. Infifusion: A unified framework for enhanced cross-model reasoning via Ilm fusion. arXiv preprint wrXiv22501.02795 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2.5 technical report. arXiv preprint 151 2024a.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint 0rXiv:240..1212,, 2024b.
Enneng Yang. Li Shen, Zhenyi Wang. Guibing Guo, Xiaojun Chen, Xingwei Wang, and Dacheng Tao. Representation surgery for multi-task model merging. In Forty-first International Conference Machine Learning (ICML), 2024c.
Enneng Yang, Zhenyi Wang. Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In The Twelfth International Conference on Learning Representations (ICLR), 2024d.
Jinluan Yang, Anke Tang, Didi Zhu, Zhengyu Chen, Li Shen, and Fei Wu. Mitigating the backdoor effect for multi-task model merging via safety-aware subspace. arXiv preprint arXiv:2410.13910, 2024e.
Xingyi Yang, Daquan Zhou, Songhua Liu, Jingwen Ye, and Xinchao Wang. Deep model reassembly. Advances in neural information processing systems, 14:25739-2575:3 2022.
Yan Yang, Yixia Li, Hongru Wang. Xuetao Wei, Jiangiao Yu, Yun Chen, and Guanhua Chen. Impart: Importance-aware delta-sparsification for improved model compression and merging in llms. arXiv preprint Nt.Xii:2504.133235. 2025a.
Ziyi Yang, Fanqi Wan, Longguang Zhong, Tianyuan Shi, and Xiaojun Quan. Weighted-reward preference optimization for implicit model fusion. arXiv preprint arXiv:2412.03187, 2024f.
Ziyi Yang, Fanqi Wan, Longguang Zhong, Canbin Huang, Guosheng Liang, and Xiaojun Quan. Fusechat-3.0s Preference optimization meets heterogeneous model fusion. arXiv preprint arXiv:2503.04222, 2025b.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. arXiv preprint arXiv:2311.03099, 2023.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, 2024.
Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation- aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821, 2023.
Jinghan Zhang, Junteng Liu, Junxian He, et al. Composing parameter-efficien modules with arithmetic operation. Advances in Neural Information Processing Systems (NeurIPS), 36:12589- 12610, 2023.
Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE transactions on knowledge and data engineering, 34(12):5586-566) 2021.
16

---

Published as a conference paper at ICLR 2026
REFERENCES
Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui, and Xuanjing Huang. Llama beyond english: An empirical study on language capability transfer. arXiv preprint arXiv:2401.01055, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In NeurlPS Datasets and Benchmarks Track, 2023.
Lianmin Zheng, Wei-Lin Chiang. Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Prucessing Systems, 36, 2024a.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. LlamaFactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 400-410, 2024b.
Ming Zhong, Chenxin An, Weizhu Chen, Jiawei Han, and Pengcheng He. Seeking neural nuggets: Knowledge transfer in large language models from a parametric perspective. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
Ming Zhong, Aston Zhang, Xuewei Wang, Rui Hou, Wenhan Xiong, Chenguang Zhu, Zhengxing Chen, Liang Tan, Chloe Bi, Mike Lewis, et al. Law of the weakest link: Cross capabilities of large language models. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving Ilm helpfulness & harmlessness with rlaif, November 2023.
Didi Zhu, Zhongyi Sun, 7exi Li, Tao Shen, Ke Yan, Shouhong Ding, Kun Kuang, and Chao Wu. Model tailor: Mitigating catastrophic forgetting in multi-modal large language models. arXiv preprint 4rXiv:2402.12048, 2024.
17

---

Published as a conference paper at ICLR 2026
REFERENCES
APPENDIX
Novelty and Contribution
19
Additional Analysis
20
B.1 Additional Ablation Studies
20
B.2 Additional Hyperparameters Analysis
20
B.3 Router for SkillPacks
21
Additional Results
C.1 Result details of Knowledge Fusion an Forget-free Learning
21 ................................
Detoxification with GraftLLM
C,3 Case Studies on SkillPack Knowledge
23
Baselines details
23
Datasets details
24
E.1
Traing Datasets for Explicit Knowledge Fusion
24
E.2 Traing Datasets for Implicit Knowledge Fusion
25
E.3 Evaluation Benchmarks
25
Implementation Details
27
F.1
Training Details
27
F.2
Hyperparameter Settings
27
F3
Computational Resources and Runtimes
28
The Use Large Language Models (LLMs)
29
18

---

Published as a conference paper at ICLR 2026
A NOVELTY AND CONTRIBUTION
Appendix for Knowledge Fusion of LLMs via Modular SkillPacks
OVERVIEW
This paper proposes knowledge grafting approach that efficiently transfers capabilities from hetero- geneous LLMs to a target LLM using modular SkillPacks. The appendix is structured according to our key contributions. We also make the project code available via anonymous link for reproducibility: https: / anonymous 4open. scwer.cee httpiccce//rraaatiiii 6DaGDCda326B
Appendix A (Novelty and Contribution) provides additional experimental results knowl- edge compression as well as task-level results from the knowledge fusion experiments.
Appendix B (Additional Analysis) includes ablation studies, hyperparameter analysis, and time cost evaluation for the search process.
Appendix C (Additional Results) outlines the computational resources and runtimes, along with the training details and evaluation metrics.
Appendix D (Baselines details) provides detailed baseline description.
Appendix (Datasets details) provides detailed dataset description.
Appendix (Implementation details) provides a detailed dataset description.
A NOVELTY AND CONTRIBUTION
We underscore the importance of cross-capability transfer across heterogeneous L.LMs and identify key limitations in current methods regarding generalization and adaptability. To this end, we propose GraftLLM, which encapsulates transferable skills as compact SkillPacks, offering high performance, robustness forgetting, and practical integrability. To clearly demonstrate the innovation of our method, we conduct a comparative analysis with existing oftte-of-thearr baseline methods.
Comparison with Multi-Teacher Distillation. (Wan et al., 2024a; Yang al., 2024f; Zhang & Yang, 2021) Our method offers several advantages:
1. It avoids the complex training procedures required by multi-task learning.
2. It achieves higher single-task performance ceilings.
3. It is naturally suited for distributed training and federated learning scenarios.
Comparison with Pairwise Distillation + Parameter Merging. Compared to approaches that directly merge parameters after distillation (Du et al., 2024b; 2025a; Yan et al., 2025), our method:
Employs a routing mechanism to avoid conflicts between capabilities from different source models.
2. Circumvents the challenge merging parameters with large differences.
3. Ensures balanced parameter allocation across tasks to mitigate interference.
Comparison with Pairwise Distillation Router.
1. Compared to methods like TALL-Mask (Wang et al., 2024c) and EMR-Merging (Huang et al., 2024) that rely on unified task vectors, our method achieves superior parameter efficiency.
2. Compared to Twin Mern-Merging (Lu et al., 2024), our approach supports large language models and preserves near full-task performance.
19

---

Published as a conference paper at ICLR 2026
B ADDITIONAL ANALYSIS
Comparison with PEFT-based distillation Router. Our method provides significantly stronger performance by first applying farammerre fine-tuning to fully extract the capabilities of the source model, and then using compression to reduce storage overhead. This enables approach to preserve critical task knowledge that PFFT-based methods (e.g., LoRa-MoF) (Ding et al., 2023; Hu et al., 2022; Du et al., 2025b) often fail capture, especially in complex scenarios such Direct Preference Optimization (DPO), where lightweight adapters struggle to inherit nuanced decision boundaries and preference reasoning from the teacher model.
Comparison with Delta Compression Methods. We introduces several key improvements:
1. We propose a module-adaptive delta compression strategy, which prioritizes and com- presses parameter updates based on the functional significance of each module aspect not considered in previous works (Liu et al., 2024; Ping al., 2024; Yang et al., 2025a).
2. Previous compression schemes typically neglect challenging tasks such DPO, limiting their robustness across diverse applications.
3. Our method is specifically designed for source-model capability transfer, whereas prior techniques target more general or different objectives.
4. We explicitly consider downstream usability in scenarios like knowledge fusion and forget- free learning, improving long-term flexibility.
5. Our method achieves better trade-off between performance and storage efficiency compared to earlier approaches.
B ADDITIONAL ANALYSIS
B.I ADDITIONAL ABLATION STUDIES
To verify the generality of conclusions, conduct additional ablation study the HumanEyal Plus benchmark, using the same LLaMA3. 1- IB-Innstuce model and keeping the overall com- pression ratio at approximately 5% across all settings, as discussed in Section Analysis 6.1. Similar to the results on math task, we examine the effect of replacing or removing individual compression modules. The results in Table 7 show consistent patterns: Mixed-pre seeeddprecision quantization continues to be the key to maintaining performance, while MLP modules remain highly sensitive to compression. Applying suboptimal strategies to MLP layers results in clear performance drops, again highlighting the importance of our module-aware adaptive strategy across different tasks.
Table 7: Ablation study module-aware adaptive strategy on the HumanEyal Plus benchmark.
Methods
Null
w/o Quantization
w/o Mixed Quant
Pruning
SVD
Low Rank SVD.
Embedding and Output Head
64.3
64.8
64.8
65.2
64.9
64.4
MLP Modules
62.7
63.8
64.8
65,2
64.4
Attention Modules
64.7
64.9
64.1
64.4
65.2
B.2 ADDITIONAL HYPERPARAMETERS ANALYSIS
We further analyze the impact of compression hyperprrameers--iicdddnggg the SVD decomposition rank and mixed-precision quantization ratios-on on model performance. Experiments are conducted on the LLaMA3.1 --BB- -B-Instruct model using the GSMSK and MATH validation sets, with results reported as normalized accuracy. As shown in Figure 10, increasing the SVD decomposition rank and quantization consistently improve performance. Notably, the Attention module is more amenable to compression, achieving 100% performance retention with rank of 1000 under double- double-precision settings. In contrast, the MLP module requires higher compression costs to reach comparable retention, highlighting the effectiveness of our proposed module-aware adaptive strategy. Quantization-relle related parameters, such as bit-width, are kept consistent with prior work, specifically Delta-CoMe (Ping et al., 2024), which employs training- maining-free delta compression with mixed precision. Pruning ratios and the mergy-preserve threshold are determined via simple grid search to balance compression efficiency and model performance.
employing
20

---

Published as a conference paper at ICLR 2026
ADDITIONAL RESULTS
1.01
1.01
Acsendion
1.0
1.0
Module
0.395
0.906
0.99
0.986
0.982
Normaiized
0.977
Normalized aizaalizee
0.972
0.972
0.97
0.97
0.954
0.95
600
0.35
Single
Double
Triple
Rank of SVD Decomposition
M Mixed-precision Quantization
(a) Rank of SVD
(b) Quantization Strategy
Figure 10: Performance trends across hyperparameters using LLaMA3.1-8D-Instee
B.3 ROUTER FOR SKILLPACKS
The routing function R is conditioned on ei- ther the source model or the task type and dynamically assigns each SkillPack to the tar- get model. In the case of explicit knowl- edge fusion, the router is implemented as lightweight feed-forward network consisting of four fully connected layers (input - 4096 -+ 1024 256 + 5 outputs), equipped with GELU activations and LayerNorm. The input features drawn from the Llama2-7B embedding head, whose 4096-dimensional hidden representation is directly fed into the router. The router is trained using the same dataset employed for source-model distillation, with a batch size of 256, learning rate of 0.001, and approximately 5,000 training steps. We train the router using the training datasets provided by FuseChat 2.0 (Wan et al., 2024b). Specifically, we collect the training loss on this dataset from five target models, each obtained through pairwise distillation from different source model. These loss values serve as supervision signals, while input features are extracted from the embedded representations of the input data. A five-way classifier is then trained to predict the most suitable source model for each input. To evaluate the effect of routing quality, we vary the amount of training data used for the classifier to obtain models of different capabilities, and analyze their impact on fusion performance, as shown Table 8. Since performance differences among source models pronounced on the MT-Bench dataset, the improvements from routing are more significant compared to AlpacaEval 2.0. For implicit knowledge fusion and forget-free learning experiments, we directly assign different Skil/Packs based task types, without training an additional router.
Table 8: Impact of the volume of training data on
router effectiveness in explicit LLM fusion.
Number of Samples
5000
10000
20000
90000
MT MrT-Bench AlpachuEval Win Rate)
7.28
7.36
7.56
7.70
20.37
23.46
25.31
25.42
C ADDITIONAL RESULTS
C.I RESULT DETAILS OF KNOWLEDGE FUSION AN FORGET-TREE LEARNING
We provide additional details to supplement our previous experiments, including the results of pairwise distillation and subsequent task vector compression the heterogeneous knowledge fusion setting, shown in Table 9 for explicit knowledge fusion and Table 10 for implicit knowledge fusion. These results demonstrate that our proposed Graft framework effectively preserves performance in most cases.
In addition, we present more details on the forget-free learning experiments in Table 11, further highlighting the extent of catastrophic forgetting in forget-free learning scenarios and showcasing the advantages our method. Finally, we include visualizations related to knowledge fusion Figure 11 and Figure 12, offering more intuitive understanding of the behavior of different methods.
C.2 DETONIHICATION WITH GRAFTLLM
We validate the effectiveness of our proposed GrMftLLM method the detoxification setting. Specifically, extract a detachable SkillPack from the detoxified model obtained through knowledge
21

---

Published as a conference paper at ICLR 2026
C ADDITIONAL RESULTS
Table 9: Result details of explicit knowledge fusion on AlpacaEval 2.0 and MT-Beech. Reported metrics include pairwise distillation, pairwise distillation with modular-aware adaptive compression 10% storage cost, and the performance retention ratio (%).
Distillation
Distillation Compression
AlpacaEvval 2.0
MT- TTT-Bench
AlpacaEval 2.0
MT-Bench
Model
#Params
Win
LC Win
Average Score
Win Rateys
LC Win Rate
Average Score
OpenChut- 3.5-BB Surting
7B
11.28
16.14
7.22
11.26
16.06
7.20
OpenChat-3.5-7B SOLAR
11.15
16.21 ow.s)
7.16
OpenChut-3.51 InteraLM
11.93
15.33
7.23
11.54
15.33
7.12 (2..8)))
OpenChat-3.5-78 Mixtral
7B
11.71
16.46
7.28
11.41.7744
16.23 (e..)
7.24 (00.)
OpenChat-3.40 Qwee
11:13
15.10
7.23
11.13
15.03
7.17 (117000.0
Table 10:
Result details of implicit knowledge fusion on various tasks. Reported metrics include pairwise distillation, pairwise distillation with modular-aware adaptive compression at 8% storage cost, and the performance retention ratio (%). For tasks where performance degrades after distillation, report results using the original target model instead.
Model
Pairwise Distillation
Pairwise Distillation Compression
General Tasks
MMLU-Pro
MMLU-redux
GPQA-Din
MMLU-Pro
MMLU-redux
GPQA-Dia
1LaMa-3.1-80-Imstie Qwen-225-77-Insttuut
$1.3
73.0
37.7
51.3
73.0 76.2
37.7 38.1
55.6
76.6
38.1
55.4
Mathematics Tasks
GSM8K 88.8
MATH
AMC23
GSM8K
MATH
AMC23
LLaMa--31-88B-mmse -8s-strucc
56.2
37.5
88 (m0.3
55.9 75.0
35.0 (3.3)
Qwen-225-7B-Instrucc
92.6
75.3
57.5
92.0 (80.4)
55.0
Code Tasks
HumanFyal
MBPP
HumanByal
MBPP 72.8
L.aMMa3.1-88- -----------n
72.0
73.0
(2.0 85.6 85.6
Qwen-2.5-7B-Imsste
85.7
84.8
84.5
Instruction Following
AlpacaEval 2.0
MT-Bench
AlpacaEval 2.0
MT-Bench
J.aMa-311-88-mmatt
65.4
9.0
64.8(99.1) (p0.1)
64.8 61.5 (w6.0)
8.8
Qwen-2.5-7B-Insss
63.6
(.7
38
Models
Parameter Merging Methods
Average Score
26
GraftLLM (ours)
Twin Marging
Owan-255-
2-w-225----------- Indrrcc
23
TALL Mask
80.0
GraftLLM
20-
SOLAK AimeeLM2.CCat..
17.
GraftLLN (ours)
Scutect LeRA
14
LM LM-76--------00 78-a
25 # Parameters (x109)
Figure 11:
Comparison of explicit knowledge fusion methods for heterogeneous LLMs on Al- pacaEval 2.0, including parameter size analysis.
Figure 12: A comparison of implicit knowledge fusion methods for heterogeneous LLMs with Qwen-22557BBIInstrun as target model.
editing. This modular SkillPack can be seamlessly integrated into the base model, enabling it to retain both detoxification capability and general-purpose performance. Our experiments are conducted on (Dubey et al., 2024b). We select three existing
the mainstream chat model knowledge editing methods as baselines: FT-L (Meng et al., 2022). WISE (Wang et al., 2024e), and DINM (Wang al., 2024d). Other common approaches, such as ROME (Mitchell et al., 2022) and MEMIT (Meng et al., 2023), require identifying specific model regions based on knowledge entities for parameter modification, making them less suitable for LLM detoxification tasks.
22

---

Published as a conference paper at ICLR 2026
D BASELINES DETAILS
Table 11: Result details of continual learning code and math tasks with modular-aware adaptive compression at 10% storage cost.
Additional Parameters
Original fask (Code)
New task (Math)
Method
HumanEval
HumannEval++
MBPP
MBPP+
GSM8K
MATH
AMC23
Average
LLaMA3,1-8B Instrstt
61.6
54.8
25.0
Sequential Distillation
69.1
63.2
67.4
35.9
87.5
35.7
30.0
61.3
Distillation
72.0
73.0
62.7
85.2
50.3
22.5
Alter Compression
803M
72.0
65.2
72.2
61.8
85.2
50.1
20
61.3
Distillation on Math
67.8
60.8
66.2
54,3
88.8
56.2
37.5
61.7
After Compression
60.2
Multi LoRA r256
1.48B
68.8
61.8
67.7
55.6
86.2
51.3
25.0
59.5
Model Graftinguess.ss
70.4
69.1
57.5
87.2
53.4
27.5
61.3
Model Taileries.
803M
71.4
64.2
71.1
59.4
87.6
54.5
27.5
62.2
GraftLLM (ours)
803M
72.0
65.2
72.2
61.8
88.2
55.9
35.0
64.3
We conduct our experiments on the Sa Sefeiit benchmark (Wang et al., 2024d) using the EasyEdit framework (Wang et al., 2023). For all methods involving training components, uti- lize the training and validation sets for model development and evaluate the final performance on the test As shown in Table 13, our method achieves detoxification performance comparable DINM, while better preserving the general capabilities of the base model. Overall, Graft LLM outperforms the previous best approach by 1.76 points in terms of the average score across detoxification and general tasks.
C.3 CASE STUDIES ON SKILLPACK KNOWLEDGE
We present qualitative examples illustrating how the router handles inputs that combine biomedical, financial, and legal terminology, as shown in Tab. 12. For each input, we report the selected SkillPack and the corresponding domain knowledge it encodes. As shown in the table below, the router consistently selects the SkillPack aligned with the dominant semantic cue, and each SkillPack captures coherent, dopain-specific knowledge- instance, dose-respan responee reasoning in Biomedicine and regulatory interpretation Law. These examples provide a clear qualitative understanding of the information encoded each SkillPack and how the router effectively manages mixed-domain queries.
Table 12: Qualitative examples illustrating router behavior and the knowledge captured by SkillPacks.
Input Example
Router (Top-1) Biomed
Knowledge Captured in SkillPack
Does increasing the dose im- prove overall return under the current treatment policy?
Captures clinical reasoning such as dose- response relationships, treatment efficacy, and pharmacological effects.
What regulatory constraints apply when reporting adverse outcomes that may affect finan- cial liability?
Law
Encodes legal concepts including regulatory compliance, reporting requirements, and liabil- ity interpretation.
Evaluate whether com- pound shows significant im- pact on long-term yield.
Biomed
Stores biochemical pharmacological knowl- edge regarding compound effects, biological im- pact, and experimental outcomes.
D BASELINES DETAILS
This section provides a detailed description of the baseline, as outlined below.
FuseLLM (Wan et al., 2024a) first to introduce multi-teacher distillation for fusing knowledge from heterogeneous large language models.
FuseChat 2.0 (Wan et al., 2024b) fuses chat LLMs of different scales and structures through lightweight pairwise fine-tuning into target models of the same size. uses a statistics -bsed token alignment for compatibility and merges the targets in parameter space.
23

---

Published as a conference paper at ICLR 2026
E DATASETS DETAILS
Table 13: Detoxification performance and general performance of vanilla LLMs and various detoxifi- cation methods on SafeEdit. The detoxification performance (detoxification success rate) is multiplied by 100. DG-Avg represents the average performance across the four DG metrics. The best and second-best results on each model are highlighted in bold and underlined, respectively.
Detoxification Performance
General Performance
Method
Average (t)
DS
DGanto
DGaterA
DGuhe
DG
DG-Avg
Fluency
KQA
CSum
1LLMMA3-B-Insstee
14.82
55.41
31.14
13.88
31.43
32.97
7.89
64.83
25.81
29.26
FT-Losottesn
82.18
91.75
90.90
79.83
93.81
90.57
6.42
63.03
25.51
53.54
81.43
81.24
81.99
80.30
78.10
5.64
62.99
25.90
50.81
DINMacize
82.89
99.24
98.87
99.70
99.78
99.40
1.20
62.98
28.18
54.33
Graft LIM (ours)
82.83
98.84
98.46
99,70
99.34
99.08
7.89
64.83
25.81
56.09
FuseChat 3.0 (Yang et al., 2025b) further introduces implicit model fusion and DPO-based strategy to enhance alignment and integration performance across heterogeneous LLMs.
Task Arithmetic (Ilharco et al., 2023) first defines the concept of "task vectors" and merges these vectors into pre-trained model execute multi multi-task learning. The model is produced by scaling and adding the task vectors to the initial model as pm. = 0ut + Ei=1 Ft-
Vieg-Mgrging (Yadav et al., 2024) further solves the task conflit problem in Task Arith- metic (Ilharco al., 2023). It eliminates redundant parameters and resolves symbol conflicts through three steps: Trim, Elect Sign, and Disjoint Merge.
DARE (Yu et al., 2023) sets the majority delta parameters zero and rescale the rest by 8' = 0 (1/(1 pj) where is the proportion of delta parameters dropped, therefore efficiently reduces parameter redundancy.
LoraHub (Huang et al., 2023) employs Low-rank Adaptations to dynamically combine task-specific modules for cross-task generalization, and adapts to new tasks by configuring 0'
- =Σ Wg Ox.-
PCB-Merging (Du et al., 2024b) effectively adjusts parameter coefficients through balancing parameter competition within model population.
InfiFusion (Yan al., 2025) enhances Universal Logit Distillation with Top-K selection and Logits Standardization to improve cross- model alignment. Top-K filters noisy outputs, while standardization ensures consistent logit distributions across diverse models.
TALL-Mask (Wang et al., 2024c) localize the tasc-specific information in multi-task vector, which deactivates irrelevant parts for each task in the merged multi-task vector with binary masks.
EMR-Merging (Huang et al., 2024) first selects a unified model from all weights, then generates lightweight task-specific modulators-masse and rescalers-to align direction and magnitude with each source model.
Delta-CoMe (Ping al., 2024) propose mixed-precision delta-compression method that employs varying bit-widths for different singular vectors based on their singular values
Model Grafting (Panigrahi et al., 2023) introduces the concept of skill localiza- tion innntifnni where task spec-ipe skills reside within the model and proposes method to efficiently acquire them.
Model Tailor (Zhu et al., 2024) derives sparse mask to identify the "model patch" through fusion of salience and sensitivity analysis, and then decorates the patch to enhance performance.
E DATASETS DETAILS
E. TRAING DATASETS FOR EXPLICIT KNOWLEDGE FUSION
We use a comprehensive training dataset, MOSECHAT--MITTURE (Wan et al., 2024b), from various sources. This dataset covers different styles and capabilities, featuring both human-writted and
24

---

Published as a conference paper at ICLR 2026
E DATASETS DETAILS
model-ggnerated, and spanning general Instruction-folloiing and specific skills. These sources include:
Orca-Best': :
We sampled 20,000 examples from Orca-Best, which is filtered from the GPT-4 (1M) partition Orca (Mukherjee et al., 2023) based on maximum length and clustering of instructions.
Capybara?: We incorporated all the 16,000 examples of Capybara, which is high-quality collection of multi-turn synthetic conversations.
No-Robots We included all the 9,500 examples of No No-Robots, which is a dataset created by skilled human annotators for supervised fine-tuning.
ShareGPT-GPT4*. We utilized all 6,200 examples from ShareGiTTGPPT, which exclusively uses dialogues generated by GPT-4 ShareGPT.
Oasst-Top1": We selected 5,000 examples from Oasst-Topl, which is refined version of Oasstl (Köpf et al., 2024), a numan-annotate assistant-style conversation dataset.
MetaMathQA :: We sampled 10,000 examples from MetaMathQA (Yu et al., 2024), which is augmented from the GSM8K (Li et al., 2025) and MATH (Hendrycks et al., 2021 datasets for mathematics problem-solving.
OSS-Instruct ?
We chose 10,000 examples from OSS-Instruct (Hao et al., 2026), which contains code instruction data synthesized from open-source code snippets.
Evol-Alpaca 8. We sampled 10,000 examples from Evol-Alpaca, which is a code instruction dataset generated by GPT-4 with evol-instruct proposed by WizardCoder (Luo al., 2024).
Python-Code 9: We selected 10,000 examples from Python-Code, which comprises instructions and responses generated by GPT-3.5 and GPT-4 for python code generation.
We followed the data processing code in FastChat (Zheng et al., 2024a) to clean instances containing non-English special characters. Then, split long conversations into blocks with maximum length of 2048 tokens, resulting in the final FUSECHAT-MIXTURE with 95,000 samples.
E.2 TRAING DATASETS FOR IMPLICIT KNOWLEDGE FUSION
The training datasets used the implicit knowledge experiments are listed in Table 14. Additionally, we provide the Hugging Face repository names and corresponding links for the target LLMs, source LLMs, and the reward model.
E.3 EVALUATION BENCHMARKS
AlpacaEval-2 (Li et al., 2023b) comprises 805 instructions from five different datasets and assesses models using two metrics: length-controlled (L.C) win rate and raw win rate (WR) (Dubois et al., 2024). GPT-4-Preview-1 106 serves as both the baseline model and the evaluator for the other models.
MT-Bench (Zheng et al., 2023) contains 80 multi-turn dialogues across eight categories, including writing, roleplay, reasoning, math, coding, extraction, STEM, and humanities. Each response is evaluated by GPT-4 on a scale from to 10, with the average score reported for each dialogue turn across the 80 dialogues. We use GPT-4-0613 the judge model following the official setting.
MMLU-Pro (Wang et al., 2024g) an enhanced version of the MMLU dataset, designed ad- dress issues such noisy data and reduced difficulty due to advances in model capabilities and
25

---

Published as a conference paper at ICLR 2026
E DATASETS DETAILS
Table 14: Details of open-source models and datasets used in Implicit Knowledge Fusion.
Name
Huggingface ID
Target LLMs
Llama-3 3.1--B-I1mmBB I88-ttrutt Qwen-2.5-7B- Instruct
IalllaaLaaanaa 3,11-11 3,1-8B-Inmm 8B-Instruct
Qwen/Qwwen..-77--mae Instruct
Source LLMs
Mistral- aranlll-agg-
TisttallLarge--nsstttte
Semma-2-27B-44
google/gemma-222751
Owen-2.5-72B-Innn Instutrttt -lama-3.1-700-Inssei
Qween//Oeen255-77---nnnnuu
mottellammlllaa-33--700----
Reward Model
AmoRRM-LLMMA--8B-00.
RLAFHow/AAmoRM-LLaas---
Datasets
UltraFeedback
princeton-nlpy
Magnie-rro-DPO
Migpie-Align/Magpeeee -lamp-----aaa-..... 1..mma---1-10
HelpSteer2
avidia/HolpSteer2
0penttahhnsttuutt22
widdal@ppenMaablnstnuua22
LectCode
greengerong/lettoode
Self-Oss- SC2
Alpaca-GPT4-Zh
llamafactory/alpaacuuu gpt4
Sigpie-Qwen2-Pro222
Ptwe:p///MMMiiieei-e--e-----oo-- -Chinese
increased data contamination. MMLU-Pro increases challenge levels by expanding multiple-choice options from to 10, requiring reasoning across more questions, and incorporating expert-reviewed annotations for improved quality and reduced noise.
MMLU-redux (Gema et al., 2024) is a re-annotated subset of the MMLU dataset created through manual assessment from 14 human experts. GPQA-Diamond (Rein al., 2023) is challenging knowledge benchmark crafted by PhD-level domain experts in biology. physics, and chemistry. The dataset contains questions that are straightforward for experts but difficult for laypersons. We evaluate the highest quality diamond set comprising 198 questions.
Arena-Hard (Li et al., 2024b) is challenging intrructionfoolloweee benchmark that closely aligns with the human preference ranking from Chatbot Arena, a crowd-sourced platform for evaluating LLMs. spans 250 high-quality topic clusters including 500 well-defined technical mooblem-solving queries. We report the win rate against GPT-4-0314 using GPT-4-Previees 106 as the judge model. GSM8K (Cobbe et al., 2021) is set of grade-school math word questions that evaluates mathematical reasoning capabilities.
MATH (Hendrycks et al., 2021a) a dataset of math problems ranging in difficulty from middle school to high school competition level. It tests a wide range of mathematical skills, including algebra, calculus, number theory, and probability.
AMC 23 (Yang al., 2024b) refers to the 2023 American Mathematies Competition, featuring 25 multiple-choice questions that test advanced high school mathematics, including trigonometry, advanced algebra, and elements of calculus.
HumanEval (Chen et al., 2021) evaluates code generation capabilities by presenting models with function signatures and docstrings and requiring them to implement the function body in Python.
MBPP (Austin et al., 2021) is a dataset of simple programming problems designed to assess the ability of models to generate short Python code snippets from natural language descriptions.
cottsss//hugiingffccccooo// -M0/at mo- idat ioation-aee amc
26

---

Published as a conference paper at ICLR 2026
F IMPLEMENTATION DETAILS
F IMPLEMENTATION DETAILS
F.1 TRAINING DETAILS
Explicit Knowledge Fusion In this experiment, we primarily focus on effectively integrating
chat LLMs with diverse architectures and varying model sizes. We select six representative (Zhu
source models: 1.penhhat33.5-11 (Wang et al., 2024a), -tta-ling-MM--B-a--- et al., 2023), -H2-SOLAR-10- 7B (Kim al., 2023), InternIM2-Chat-2200 (Cai et al., 2024), 1exxtral-8x7B-111
(Wang et al., 2024a), and Owen-1.5-Chan 5- (Bai et al., 2023). We use OpenChat- 5B as the pivot model and starting point for generating target LLMs, given its well-balanced size and performance. Initially, we apply pairwise knowledge fusion to produce five target models with uniform architecture. Subsequently, these target models' knowledge is combined through either parameter merging or routing mechanisms.
Implicit Knowledge Fusion The construction of data plays a vital role facilitating the Implicit Model Fusion approach showcased in FuseChat-3.0 (Yang et al., 2025b). We perform SFT+DPO on four task-specific datasets to obtain four individually fine-tuned models, each achieving strong performance on its corresponding task. We describe the procedures for selecting prompts, sampling responses, and assembling the dataset, explaining the reasoning behind each design choice.
To enhance the target LLMs' abilities across multiple iields-including instruction following, math, coding, and Chinese -e assemble a diverse dataset. This is done by carefully choosing samples from well-regarded open-source community datasets, followed by specific filtering and preprocessing steps ensure quality and relevance.
Prompt Selection:
Response Sampling: For each prompt in the curated dataset, we generate responses primarily from four leading source LLMs. Our response sampling strategy is tailored to each domain, leveraging vLLM" (Zhu et al., 2023) as the inference backend. We perform multiple sampling runs using different random seeds to ensure diversity. The sampling parameters for each source model are as follows: for Mi stral-laa -ge-all---rr-----eeeeeeeeeee Inee-Instruus ruct 24007, and I. ama 3. -3. 3.1 T we set the temperature to 0.8 and top-p to 0.95; for Qwen 2. 5- (Math) - Instruct. use a temperature of 0.7, top-p of 0.8, and a repetition penalty of 1.05.
Preference Pairs: To construct preference pairs from models with diverse output styles, we select the best and worst responses generated by the same source model for each pair. This antra-model pairing strategy reduces reward bias caused by heterogeneous response styles, prevents reward hacking, and provides more controlled and reliable preference signal. The data construction process varies by domain: for instruction-following and conversational data, we use external reward model to evaluate responses; for mathematics and coding domains, responses are verified through rule-based systems.
The final dataset D consists of 158,667 entries, with 94,539 allocated the SFT phase (Dsrr) and 64,128 preference pairs for the DPO phase (Dppo). The dataset composition is provided in Table 15.
F.2 HYPERPARAMETER SETTINGS
Explicit Knowledge Fusion we follow a same experiment setting as FuseChat 2.0 (Wan et al., 2024b), train the target LLMs using batch size of 128 and maximum length of 2048 single node with 8x80GB NVIDIA A800 GPUs for three epochs, which takes approximately hours. The models optimized using the AdamW (Loshchilov & Hutter, 2019) optimizer with 33 0.9 and B2- 0.999. We use weight decay of 0.0 and gradient clipping of 1.0. A cosine learning rate schedule is employed, with maximum learning rate of 5e-6 and a warmup ratio 0.03. Our training framework is implemented based on the HuggingFace Transformers (Wolf et al., 2020).
Implicit Knowledge Fusion In our SFT experiments, we use the Llama-Factory library (Zheng al., 2024b) to implement the fine-tum fine-tuning. For all target models, perform fine-tuning for 3 epochs, with a batch size of 128 and maximum sequence length of 2048 tokens. cosine learning rate
"https://gif httb//ttthhtooooo comb wttp:/hb..oom/mmmmooooo///////// Phttps:/ //g thub. com www.coom//hiee howga /.l.aa/a/aaaaa MMM- MaMAtaaa--orrrry
27

---

Published as a conference paper at ICLR 2026
F IMPLEMENTATION DETAILS
Table 15: The constitution of Implicit Knowledge Fusion dataset in SFT phase and DPO phase. As no suitable reward models were available for Chinese, we used all samples for SFT and omitted the DPO phase.
Category
Dataset
Count
#DSFT
#DDpo
Instruction Following
UltraFeedback
51,098
20,439
30,659
Magpie-Pro-DPO
20,374
8,149
12.225
HelpSteer2
9,435
3,774
5,661
Mathematics
OpenMathInstruutt22
51,803
40,188
11,615
Coding
LeetCode
3,113
1,877
1,236
Self-Oss-Instruct-SC7
12,892
10,160
2,732
Chinese Language
Alpaca-GPT4-Zh
2,471
2,471
0
Magpie-Qwenn2Pro--ZZ
7,481
7,481
0
Total
158,667
94,539
64,128
Table 16: Hyperparameters for different target models during the SFT and DPO stages.
Target Model
SFT Learning Rate
DPO Learning Rate
DPO A
DPO Loss Type
Llama-3.1 1.--3..----sttttee D0wen-255-7B-Inntteee
5 x 10 6 2 10
8 x 10 7
10
LIN-DPO
x 10 7
0.01
LDPO
schedule with warmup ratio of 0.1 is employed. DPO experiments, we utilize the alignment- handbook 13 as the training framework for DPO. All post-SFT target models undergo training for one epoch with batch size of 128 and a maximum sequence length of 2048. cosine learning rate schedule with warmup ratio of 0.1 is used. Checkpoints are saved every 100 steps, and the best checkpoint from the last two is selected. The hyperparameter configurations for different models are detailed in Table 16.
F.3 COMPUTATIONAL RESOURCES AND RUNTIMES
We report the resource consumption and runtime of our module-adaptive compression strategy in Table 10. The overall compression overhead calculated as the weighted sum of each module's storage cost, with weights corresponding to the proportion of parameters in each module. Additionally, we provide the compression time for each module, which mainly depends on the parameter count and the amount of data used for GPTQ (Frantar et al., 2022). In all our experiments, we use the C4 validation split as the calibration set for GPTQ, which widely adopted in previous GPTQ-based quantization research. Since pruning and SVD typically require only a few minutes, we omit their time costs and primarily report the runtime for guantization. practice, the compression strategy can be adjusted according to task complexity and model type.
Table 17: Storage and time costs of our proposed adaptive compression strategy. The Storage Cost is defined as the ratio of the compressed module size that of the corresponding original parameters.
Modules
Compression Strategy
Quantization Strategy
Storage Cost (%)
Time Cost
Embedding & Head
Pruning with a = 0.5
4bits
12.5
8 minutes
MLP Module
with 1400
[8. 3. 2] bits for rank [20, 180, 1200]
5.43
53 minutes
Attention Module
SVD with 1000
[8. bits for runk [20, 980]
5.59
22 minutes
In addition, provide a detailed comparison of training and inference resource usage across different
methods (see Tab. 18). In all experiments, we adopt top-1 router activation, so that during inference, the computational overhead primarily comes from the router forward pass and the SkillPack de- compression step. Both of these steps introduce negligible increase in latency, accounting for about
http://gtthhbb..omomggggggggg/// htw.g//faccc.olgggnnnnnnanannnnn gnmentthandbboo
"httts://hugging http:/aa....ooaaaattttttee al 1enai hataa///////////// ddtti on.
28

---

Published as a conference paper at ICER 20B USE OF LARGE LANGUAGE MODELS (LLMS)
3% of the total inference time, demonstrating that our method maintains efficiency comparable to baseline models.
Table 18: Resource analysis of GraftLLM compared to representative LLM fusion baselines.
Method
Model Size
Router Size
Training Cost (GPU h)
Compression Cost
Inference Overhead
FuscI.LM
7B
132
1.00x
FuseChat
7B
132 h
4 min
1.00x
Twin-Merging
21B
21M
132 + 8 min
9 min
1.03x
GraftLLM (Ours)
9.2B
21M
132 + min
163 min
1.03x
G THE USE OF LARGE LANGUAGE MODELS (LLMS)
During the preparation of this manuscript, large language models were used solely for minor stylistic enhancements and occasional grammatical corrections. All conceptual insights, analytical reasoning. and interpretive conclusions were generated entirely by the authors. No algorithmic assistance was sought in the formulation, design, or substantive content of the work, and full scientific responsibility lies solely with the human contributors.
29


Reviews:
[{"review_id": "Ontd2CDDoV", "reviewer": "Reviewer_Evyv", "summary": "This paper identifies a critical challenge in cross-capability transfer for LLMs, i.e., existing methods like full-parameter knowledge distillation often cause catastrophic forgetting, while standard PEFT methods are ineffective at absorbing comprehensive knowledge from powerful source models. To address this, the authors propose GraftLLM, a novel grafting-based framework. The core idea is to distill and store capabilities from source models into modular, compressed \"SkillPacks.\" These SkillPacks can then be attached to a target model to fuse new capabilities, preserving the target's original knowledge. The method employs a \"module-aware adaptive compression\" strategy to create these SkillPacks, aiming to reduce parameter conflicts and support \"forget-free\" continual learning and model fusion.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- I think the authors did a very good job of identifying a key gap: the lack of a method that both effectively absorbs deep knowledge from a source model (like distillation) and preserves the target model's inherent capabilities (which distillation often fails to do).\n- The method introduced is sound and the reframing of model fusion as a modular composition problem is quite novel. \n- A \"forget-free\" method for adding new, complex capabilities to a base model is highly sought after. This approach has clear significance for building specialized models on demand without the need for costly full-scale retraining.", "weaknesses": "- I think the baseline comparisons is a bit ambiguous. The abstract positions the work against distillation and PEFTs, and mentions FuseLLM/FuseChat for small models. However, it's unclear if GraftLLM is benchmarked against current SOTA model merging techniques for large models. These methods also aim to fuse capabilities and are a crucial point of comparison. \n- The framework's practicality hinges on the cost of creating the SkillPacks. Can the authors please quantify the overhead of their method? What is the computational cost (e.g., GPU hours) to create a typical SkillPack? What is the inference-time impact (e.g., added latency, extra VRAM) of using a target model + 1 SkillPack?\n- A key test for a fusion method is its ability to handle conflicting or highly distinct capabilities (e.g., fusing a \"math\" SkillPack and a \"poetry\" SkillPack). How does GraftLLM handle fusing multiple SkillPacks that may have conflicting parameter updates like this example?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZeXFhNeEUB", "reviewer": "Reviewer_vnrD", "summary": "This paper introduces GraftLLM, a framework for transferring capabilities across heterogeneous large language models by creating and composing modular \"SkillPacks.\" The method involves fine-tuning a target model, extracting the parameter delta, applying a module-aware compression strategy, and composing these SkillPacks using a routing mechanism. The approach is evaluated on tasks including pairwise knowledge transfer, heterogeneous model fusion, and forget-free continual learning.", "questions": "See weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper tackles the highly relevant and challenging problem of fusing knowledge across heterogeneous LLMs. The proposed \"module-aware adaptive compression\" strategy is an intuitive and empirically effective contribution for creating compact, transferable knowledge modules. The experimental results consistently demonstrate strong performance across multiple benchmarks, outperforming several baselines in knowledge fusion.", "weaknesses": "The technical contribution of this work appears to be incremental. The proposed pipeline—distilling knowledge, calculating a delta, compressing it, and then composing these modules—is conceptually very similar to existing frameworks like FuseChat (pairwise distillation followed by merging) and LoraHub (dynamic composition of LoRA modules). The main novelty seems to lie in the \"module-aware adaptive compression\" strategy, but the overall framework feels like a combination of established techniques rather than a new approach. The paper would be stronger if it more clearly articulated its core conceptual departure from these related works.\n\nFurthermore, the analysis provided is limited and somewhat misaligned with the paper's main theme of knowledge fusion. The primary analysis focuses on ablation studies of the compression strategy (Table 4) and its impact on performance loss. While this validates a component of the method, it does little to illuminate the dynamics of the fusion process itself. An analysis of inter-SkillPack interference, the behavior of the router on ambiguous inputs, or a qualitative look at what knowledge is captured within a SkillPack would have been more insightful for a paper centered on fusion.\n\nThe paper is difficult to follow in several key areas, with missing details and confusing notation that undermine the reader's confidence in the methodology and results.\n\n1.  **Inconsistent Notation**: The notation for the base model is inconsistent. Equation (10) uses `θ_tgt`, while the description on line 270 uses `θ_target`. This should be standardized for clarity.\n\n2.  **Missing Quantization Details**: The method utilizes GPTQ for quantization, which typically requires a calibration set to determine quantization parameters. The paper fails to mention what calibration data was used, how it was selected, or its size. This is a critical detail for reproducibility.\n\n3.  **Confusing Formulation of the Router**: Equations (10) and (11) are nearly identical, yet they are meant to describe different scenarios. This implies that the router `R` and the SkillPacks themselves might function differently in each context, which is highly confusing. The paper should explicitly state this difference. Moreover, the appendix reveals that two different types of routers are used—one trained as a classifier and another based on manual, task-type assignment. This crucial distinction is not clearly made in the main paper. Important details about the trained router's architecture, training data, and hyperparameters are also missing.\n\n4.  **Misleading Claims about Parameter Size**: The paper claims its method is efficient, yet in Table 1, the parameter count for \"Routed GraftLLM (Ours)\" is 9.2B, a 28% increase over the 7B base model. This increase can hardly be considered minor. Why does a method centered on *compression* result in a significantly larger final model?\n\n5.  **Flawed Comparison in Continual Learning**: Table 3 evaluates forget-free learning. The setup is problematic for two reasons. First, it is missing some critical results: the performance of the LLaMA3 model after being fine-tuned sequentially on different stages. Second, the paper states that the GraftLLM router for this experiment is a manual one based on \"task-type.\" If the model is explicitly told which task it is performing (e.g., \"this is a math problem, load the math SkillPack\"), then the comparison to other methods that do not receive this oracle information is unfair.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "74nqU3yjon", "reviewer": "Reviewer_HvCk", "summary": "The paper addresses the problem of LLMs merging. The paper proposed an approach which consists of storing source model capabilities in a target model + SkillPack format. According to the paper, this approach preserves general capabilities, reduces parameter conflicts, and supports forget-free continual learning and model fusion. According to the paper, the approach proposed outperforms existing techniques.", "questions": "- What is Modular Skill Pack?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "The paper addresses a well known problem when developing or adapting LLMs. Instead of developing a completely new model, the goal is to merge existing models. The paper proposes a new approach which  preserves general capabilities, reduces parameter conflicts, and supports forget-free continual learning.", "weaknesses": "The main issue with the paper is self contained which makes its review difficult. The paper used several key technical terms such as “cross-capability transfer”, “SkillPack”, MLP, etc. without providing the meaning, which can make the paper very difficult to understand by newcomers.\n\nThe problem addressed by the paper is not clear and there is a confusion in the paper contribution:\n- The paper is addressing the problem when models are structurally identical\n- What is the difference between heterogeneous models (line 58) and structural different model (line 60) that the paper wants to solve? These terms lead to confusion. The paper should clarify all the terms used.\n\nThe methodology is not understandable, which did not make the reproducibility easy. Several capital terms are not clearly defined. Figure 4 is not well explained. Section 3.2 is not clear\n\nSome description of key terms (line 267) are provided in additional materials. However, the paper should be self contained.\n\nExperimentation environment, hyper-parameter, etc. are not provided in the experimentation section.\n\nThe evaluation of the time complexity and difficulties to integrate the models are not provided.\n- Line 26: for heterogeneous heterogeneous LLM fusion –> there is a redundancy of the term “heterogeneous”\n- Missing experimentation and results summary in the abstract. The paper could present to which approach the proposed approach was compared to.\n- SkillPack, cross-capability transfer should be defined before first use in the introduction and abstract\n- In the abstract, the paper could introduce the experiments, present to which baseline their approach was compared to and summarize some results.\n\n- Line 106: “SFT and DPO” should be defined before first use\n- Line 112-113 is not clear. \n- The figures presented in the paper should be accompanied by a description text which makes its understanding easy.\n- The definition of SkillPack in line 215 is too short and not understandable for such a technical term. The paper should provide a clear explanation of what SkillPack is (with examples).\n- The ablation may provide the evaluation (costs in time of the different approaches) and  the integration difficulties.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DBV5hMx3f3", "reviewer": "Reviewer_zW7i", "summary": "The paper presents a graft method, GraftLLM, that extracts the source models' capabilities as SkillPack and then dynamically assigns capabilities to the target model based on the tasks. Moreover, this method employs a combination of compression techniques, including pruning, quantization, and decomposition, to make the SkillPack compact and efficient. The router function is trained on FuseChat 2.0 to explicitly select SkillPack based on the source model and task type. The experiments show that GraftLLM outperforms the other merging or fusion methods with comparable parameters on multiple tasks.", "questions": "1. Runtime and complexity concern:\nCould you provide a runtime comparison against the competing methods, including both training and validation phases? The proposed framework integrates several techniques, which may introduce substantial computational overhead and tuning difficulty despite the parameter reduction. A comparative runtime analysis would clarify the practical efficiency of the approach.\n\n2. Clarification of the router function:\nCould you elaborate on the formal definition and empirical behavior of the router function? It appears to be the key innovation that unifies the multiple techniques and enables dynamic capability transfer across heterogeneous models. A clearer explanation would help readers understand its role and significance.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Innovative router-based skill integration:\nThe paper introduces a novel router function that dynamically assigns SkillPacks to the target model, enabling flexible and adaptive composition of capabilities. This mechanism allows the target model to seamlessly adapt to diverse tasks without full retraining.\n\n2. Support for heterogeneous model fusion:\nUnlike most prior works that focus on homogeneous architectures, the proposed method is applicable to heterogeneous LLMs, demonstrating strong generality and robustness across models with different structures and parameterizations.\n\n3. Strong empirical performance with compact models:\nThe proposed GraftLLM achieves superior performance across multiple benchmarks while maintaining a comparable parameter count to competing methods, highlighting its efficiency and effectiveness.", "weaknesses": "1. Terminology and clarity issues:\nThe paper assumes substantial prior knowledge and introduces multiple technical terms without sufficient explanation. For instance, PEFT is used in the abstract and appears frequently throughout the paper, but it is never explicitly defined. Similarly, the term modules is used ambiguously; it would help if the authors clarified that they refer to specific neural network components, such as attention layers or MLP blocks. Additionally, there are several minor typographical errors, particularly in Section 3, that should be corrected for readability.\n\n2. Methodological complexity and unclear contribution emphasis:\nWhile the proposed approach is empirically strong, it combines more than five techniques, which makes it conceptually complex and somewhat difficult to parse. According to the ablation study, the optimal configuration involves using pruning for the embedding and output head, SVD for MLP layers, and low-rank SVD for attention layers. However, in such a composite pipeline, the reported parameter reduction may not translate into practical efficiency—training and validation runtimes could be substantially longer than for competing methods. Although a runtime analysis is included, it lacks a direct comparison with other baselines. Moreover, if similar compression strategies (e.g., pruning, SVD) were applied to the competitors, they might outperform the proposed method. The paper would be stronger if the authors highlighted the most impactful component rather than treating all sub-techniques as equally central, as the current presentation dilutes the core contribution.\n\n3. Insufficient explanation of the router function:\nThe router function appears to be the conceptual centerpiece of the paper, yet its formulation and role are not clearly articulated in the main text. A more detailed exposition—possibly with a schematic or mathematical description—would make the contribution clearer and could inspire future extensions of this idea.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sabBnj6fNh", "reviewer": "Reviewer_ZGH4", "summary": "This paper introduces GraftLLM for transferring capabilities from heterogeneous source LLMs into a single target LLM. The key contribution of GraftLLM is to compress task-specific knowledge distilled from various source models into compact, module-wise \"SkillPacks\", effectively preserving the target model's knowledge and preventing catastrophic forgetting. These lightweight SkillPacks can be flexibly routed and integrated into the target model, offering a plug-and-play method for practical deployment. Comprehensive experiments span pairwise grafting, knowledge fusion and forget-free learning demonstrate the effectiveness of GraftLLM.", "questions": "1. What is the inference overhead of the proposed method? Specifically, does loading multiple SkillPacks significantly increase inference latency? Additionally, what is the computational cost associated with training the router?\n  \n2. For training the router module in explicit model fusion, what is the rationale for using training loss values as supervision signals? Furthermore, how is the accuracy of this routing mechanism ensured, and have you analyzed whether a potential distribution shift between the router's training data and the test set could affect its performance?\n  \n3. In the implicit fusion experiments (Table 2), could you clarify how the models for GraftLLM's pairwise distillation step were trained? What were the individual performances after distilling from each source model, before the final fusion? The overall performance gain over the FuseChat-3 baseline is modest; could you elaborate on the trade-off between this performance gain and the complexity/overhead of the GraftLLM approach in this specific scenario?\n\n4. I am willing to raise my score if the authors can satisfactorily address the weaknesses and questions I have raised.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper introduces an innovative framework that creatively combines knowledge fusion, model compression, and model rounting. The introduced \"SkillPack\" is flexible and plug-and-play, offering a cost-effective solution for scalable, efficient, and forget-free knowledge transfer across heterogeneous LLMs.\n2. The work is technically strong. The proposed module-aware adaptive compression strategy is well-motivated and well-designed to preserve performance while minimizing parameter overhead. The experimental methodology is rigorous, featuring comprehensive comparisons against various baselines across three distinct scenarios.", "weaknesses": "1. The presentation of the paper has several issues that hinder readability. These include, but are not limited to:\n\n- **Grammatical errors**: For example, the word \"heterogeneous\" is repeated on line 26. \n\n- **Mismatched citations**: For instance, the citations for FuseChat  (Yang et al., 2025b) on line 54 and synthetic data  (Wan et al., 2024b) on line 72 appear to be swapped. \n\n- **Mismatched table captions**: The caption for Table 6 does not correspond to its content. The caption of Table 10 should appear before the table.\n\n- **Logical contradiction**: Appendix B.3 states, \"A five-way classifier is then trained to predict the most suitable source model for each input,\" which suggests the routing function $\\mathcal{R}$ selects the best SkillPack on a per-prompt basis. However, this appears to conflict with the description in Section 3.3 (lines 270-272), which defines $\\mathcal{R}$ as a submodule-specific router that \"dynamically assigns each SkillPack to the appropriate submodules.\" The authors should rewrite Section 3.3 to clarify this mechanism and resolve the ambiguity.\n  \n2. Several experimental details are missing, which poses a challenge for reproducibility. These omissions include:\n\n- **Insufficient details about the router**: For the explicit fusion experiments, the paper mentions training a router to assign SkillPacks. However, the details of this router (e.g., its architecture, the distribution of training data, and its computational overhead) are too brief.\n\n- **Lack of detail on model combination**: Section 3.2 explains how a SkillPack is created through compression, but the paper does not explicitly describe how these compressed delta parameters are combined with the full parameters of the target model to form the final, fused model.\n\n- **Missing details for pairwise distillation in implicit fusion**: Section 5.2 describes experiments on implicit fusion, which involves a pairwise distillation step. However, the implementation details provided in Appendix F.1 (lines 1269-1318) appear to describe the process for the FuseChat-3 baseline, not the specific distillation process used to generate the SkillPacks for GraftLLM in this setting.\n\n- **Insufficient detail on hyperparameter selection**: The module-aware adaptive compression strategy described in Section 3.2 introduces several tunable hyperparameters (e.g., SVD rank, pruning ratio, quantization bitwidth, energy threshold $\\beta$). However, the paper does not provide enough detail on how the final values for these hyperparameters were selected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)