Training-free LLM Merging for Multi-task Learning
Authors: Zichuan Fu, Xian Wu, Yejing Wang, Wanyu Wang, Shanshan Ye, Hongzhi Yin, Yi Chang, Yefeng Zheng, Xiangyu Zhao
Accepted at: acl25

Training-free LLM Merging for Multi-task Learning

Zichuan Fu1*, Xian Wu2*, Yejing Wang1,
Wanyu Wang1, Shanshan Ye3, Hongzhi Yin4, Yi Chang5,
Yefeng Zheng26, Xiangyu Zhao1†

1 City University of Hong Kong 2 Tencent Jarvis Lab
3 University of Technology Sydney 4 University of Queensland
5 Jilin University 6 Westlake University

zc.fu@my.cityu.edu.hk, kevinxwu@tencent.com, xianzhao@cityu.edu.hk

Abstract

Large Language Models (LLMs) have demonstrated exceptional capabilities across diverse natural language processing (NLP) tasks. The release of open-source LLMs like LLaMA and Qwen has triggered the development of numerous fine-tuned models tailored for various tasks and languages. In this paper, we explore an important question: is it possible to combine these specialized models to create a unified model with multi-task capabilities. We introduces Hierarchical Iterative Merging (Hi-Merging), a training-free method for unifying different specialized LLMs into a single model. Specifically, Hi-Merging employs model-wise and layer-wise pruning and scaling, guided by contribution analysis, to mitigate parameter conflicts. Extensive experiments on multiple-choice and question-answering tasks in both Chinese and English validate Hi-Merging’s ability for multi-task learning. The results demonstrate that Hi-Merging consistently outperforms existing merging techniques and surpasses the performance of models fine-tuned on combined datasets in most scenarios. Code is available at Applied-Machine-Learning-Lab/Hi-Merging.

1 Introduction

Large Language Models (LLMs) have revolutionized Natural Language Processing (NLP) by demonstrating unprecedented capabilities in capturing and utilizing world knowledge (Zhao et al., 2024). Recent advances in architecture design and training methodologies have enabled models like GPT-4 (OpenAI, 2023) to engage in human-like dialogue and solve real-world problems, enabling breakthroughs in healthcare, recommender system, and scientific research (Liu et al., 2024a; Fu et al., 2024; Xu et al., 2024b).

With the advent of open-source large language models (LLMs) like LLaMA-3 (Dubey et al., 2024) and Qwen (Yang et al., 2024a), significant research efforts have been dedicated to fine-tuning these models for specific tasks, domains, and languages (Xu et al., 2024c). As a result, Hugging Face1 now hosts over one million specialized LLMs across various tasks, and this number continues to grow rapidly. These models represent a vast repository of task-specific and language-specific expertise, ranging from medical applications (Chen et al., 2023; Liu et al., 2024b) to financial question and answering (Cheng et al., 2024). A natural question arises: is it possible to combine these task-specific fine-tuned LLMs into a single unified model with broad capabilities, including multilingual and multi-task functionalities? If achievable, the deployment of such a unified model could perform multiple tasks that currently require multiple LLMs, thereby significantly enhancing the application of LLMs. One potential solution is to gather all fine-tuning data and retrain the LLMs from scratch. However, this approach has three significant disadvantages: 1) the availability of fine-tuning data, as the models are often public but the data is not (He et al., 2024); 2) retraining large LLMs requires substantial computational resources; and 3) balancing the training data from different tasks to maintain strong performance across all tasks without compromising any individual task (avoiding the “seesaw effect” (Tang et al., 2020) where improving one task’s performance leads to degradation in others) is a non-trivial challenge.

Based on the above considerations, model merging (Yang et al., 2024b) emerges as a promising solution for unifying multiple specialized models while preserving their individual capabilities. However, current model merging methods face two fundamental challenges. First, interference between
Figure 1: Illustration of three paradigms for our LLM merging: merging models that specialize in different languages (left), merging models that excel at different tasks (middle), and merging models that exhibit expertise in both different languages and different tasks (left). Through such merging, a single model can inherit the combined capabilities of both original models, enabling broader applicability and enhanced performance.

merged models can arise from noise introduced by data bias (Tsuchiya, 2018) or the training process, such as overfitting, impairs the merged model’s generalization. Second, models trained independently follow distinct optimization trajectories, leading to different knowledge alignments in their parameter spaces (Ilharco et al., 2023). These misaligned parameters become incompatible for direct combination without additional training.

To address these challenges, we propose Hi-Merging, a Hierarchical Iterative Merging method. It first applies model-wise pruning and scaling to the delta vectors (parameter differences between fine-tuned models and the foundation model) to eliminate noisy parameters introduced during fine-tuning. Then, we apply layer-wise pruning and scaling iteratively for the knowledge misalignment, starting from the most conflicted layers. To identify the severity of layer-wise conflicts, we develop contribution analysis - a method that quantifies each layer’s contribution by measuring how adding or removing specific layers affects model capabilities. By analyzing how our contribution metrics change before and after a pre-merging process, we can identify potential conflicts, thereby guiding our iterative optimization process to resolve parameter incompatibilities without additional training.

Our contributions can be summarized as follows:
• We investigate the challenges and potential of training-free model merging for integrating LLMs specialized in diverse tasks (e.g., MCQA, QA) and languages (e.g., English, Chinese), addressing a complex multi-task scenario.
• We propose Hi-Merging, a hierarchical iterative approach that effectively reduces the interference of noise and knowledge alignment conflicts during model merging.
• Extensive experiments on four datasets demonstrate the effectiveness of Hi-Merging in multi-task merging across different tasks and languages, consistently achieving superior performance.

2 Preliminary for LLM Merging

In this section, we detail notations and introduce existing LLM merging solutions as the preliminary.

Model merging aims to combine multiple models with distinct capabilities as a single model, which has all the strengths of these models. In this paper, we user two-model merging for illustration: Given models \( \mathcal{M}_A \) and \( \mathcal{M}_B \) with parameters \( \theta_A \) and \( \theta_B \), both fine-tuned from a foundation model \( \mathcal{M}_F \) with parameters \( \theta_F \) for tasks \( t_A \) and \( t_B \) respectively, model merging aims to combine them into a single model \( \mathcal{M}_{merge} \) with parameters \( \theta_{merge} \) that preserves capabilities for both tasks.

Typical model merging strategies include weighted averaging and delta vector-based merging. The former combines model parameters through a weighted sum (Wortsman et al., 2022):

\[
\theta_{merge} = \sum_{m \in \{A,B\}} \omega_m \theta_m,
\]

where \( \omega_m \) is the weight to balance different capabilities constrained to \( \sum_{m \in \{A,B\}} \omega_m = 1, \omega_m > 0 \).
And \( m \in \{A, B\} \) is the model identifier.

The second strategy merges models based on delta vectors, the parameter differences between fine-tuned models and their foundation model, which can be mathematically defined as:

\[
\delta_m = \theta_m - \theta_F.
\] (2)

Delta vectors \( \delta_m \) defined in Equation (2) reveal model-specific updates from the foundation model, enabling a delta-weighted merging strategy (Ilharco et al., 2023):

\[
\theta_{merge} = \theta_F + \sum_{m \in \{A,B\}} \omega_m \delta_m.
\] (3)

where \( \omega_m > 0 \). Note that both strategies, illustrated in Equation (1) and Equation (3), can be easily extended to multiple model merging scenarios by expanding the model list \( \{A, B\} \).

3 Method

In this section, we introduce the proposed method, which consists of two major components: (1) model-wise pruning and scaling that removes noisy and redundant parameters and moderate excessive ones and (2) layer-wise pruning and scaling iterating on conflicted layers to address knowledge misalignment issues.

3.1 Model-wise Pruning and Scaling

This section introduces two operations to process delta vectors: pruning and scaling.

During the fine-tuning, models can accumulate noisy parameters and learn sharp parameters for the specific fine-tuning task. We introduce the pruning and scaling operations to tackle these two problems, respectively, which are controlled by the following hyperparameters:
• **Pruning Threshold** (\( p \)): This parameter specifies the proportion of the delta vector that should be preserved. By retaining the largest \( p \) percentage of the vector’s components and rendering the remaining \( (1-p) \) to zero, the pruning operation can eliminates trivial parameter updates (data-specific noise) while preserving meaningful task-specific knowledge.
• **Scaling Factor** (\( s \)): This factor controls the magnitude of the delta vector. With this parameter, the scaling operation contributes to addressing over-aggressive parameters by scaling down sharp updates, which may result from the overfitting during fine-tuning. The pruning does not apply to large parameter changes as they likely encode essential knowledge. The scaling provides a way to moderate their excessive influence.

With these hyperparameters, the pruning and scaling cooperatively process the delta vectors in a complementary manner: pruning eliminates negligible parameter changes while scaling moderates the significant ones. Note that both \( p \) and \( s \) constrained to \([0, 1]\).

We empirically validate the effectiveness of the pruning and scaling operations by iterating \( p \) and \( s \) from \([0.1, 1]\). The result is visualized in Figure 2. We can find that the individual model can maintain or even improve performance with appropriate pruning and scaling. For example, \( p = 0.1, s = 0.9 \) (preserving 10% of parameters and scaling all delta values with 0.9) can defeat the original model (\( p = 1, s = 1 \)). This finding supports our idea of conducting model-wise pruning and scaling to overcome noisy and radical parameter updates.

Next, we introduce the model-wise pruning and scaling details. Specifically, the delta vector (defined in Equation (2)) for a given LLM \( \mathcal{M}_m \) can be defined as \( \delta_m = [\delta_{m,1}, \delta_{m,2}, \ldots, \delta_{m,N}] \), where \( m \in \{A, B\} \) is the model identifier and \( N \) indicates the size of trainable parameters.

The **pruning** operation Top\(_p\) retains the \([p \cdot N]\) elements of \( \delta_m \) with the largest absolute value and

![A color plot showing the accuracy of fine-tuned Qwen2-7B-Instruct on MedQA after model-wise pruning and scaling with different combinations of pruning threshold p and scaling factor s.](page_820_153_601_496.png)

Figure 2: The accuracy of the fine-tuned Qwen2-7B-Instruct on the MedQA dataset after the model-wise pruning and scaling process with different combinations of the pruning threshold \( p \) and the scaling factor \( s \).
zeros out the rest, resulting in \( \tilde{\delta}_m \):

\[
\tilde{\delta}_m = \mathrm{Top}_p(\delta_m).
\] (4)

In detail, the n-th component of \( \tilde{\delta}_m \) is:

\[
\tilde{\delta}_{m,n} =
\begin{cases}
\delta_{m,n}, & \text{if } n \in \{\pi(1), \pi(2), \ldots, \pi([p \cdot N])\}, \\
0, & \text{otherwise}
\end{cases}
\] (5)

where \( \pi(n) \) represents the index of the n-th largest component of \( \delta_m \) in absolute value, such that:

\[
|\delta_{m,\pi(1)}| \geq |\delta_{m,\pi(2)}| \geq \cdots \geq |\delta_{m,\pi(N)}|.
\] (6)

The scaling operation adjusts the magnitude of the pruned delta vector \( \tilde{\delta}_m \) by multiplying it with the scaling factor \( s \in [0, 1] \) as \( s \tilde{\delta}_m \).

Regarding the different setting of \( p \) and \( s \) for each model, the model-wise pruning and scaling can be compactly expressed as:

\[
\hat{\delta}_m = s_m \cdot \mathrm{Top}_p(\delta_m) = s_m \tilde{\delta}_m,
\] (7)

where \( \hat{\delta}_m \) represents the delta vector after the model-wise pruning and scaling.

Through model-wise process with pruning and scaling, we effectively identify noisy and excessive parameter updates from the fine-tuning, maintaining and moderating the key knowledge about the fine-tuning task for the subsequent merging.

3.2 Layer-wise Pruning and Scaling

In this section, we conduct the layer-wise model merging with pruning and scaling operations with a novel contribution analysis method to measure the parameter conflict.

3.2.1 Contribution Analysis

Directly merging the model-wise processed delta vectors \( \{\hat{\delta}_m\}_{m \in \{A,B\}} \) as in Equation 1 or Equation 3 will encounter the weight misalignment problem, which is overlooked by existing methods.

To investigate potential conflicts when merging a specific layer, we measure its contribution by calculating the performance difference before and after the merge. Precisely, we assess the merging contribution from two directions:
• Deletion Impact (\( \alpha \)): To estimate this impact, we first construct a merged model \( \mathcal{M}_G \) that merges all layers using the merging process mentioned in Equation (1) or Equation (3). Then, we calculate the performance degradation caused by removing the delta vector for a specific layer.

![The demonstration of different conflict elimination strategies for three pre-merging conditions.](page_1012_186_393_312.png)

Figure 3: The demonstration of different conflict elimination strategies for three pre-merging conditions.

• Addition Impact (\( \beta \)): This impact is measured by the performance improvement of adding the delta vector for a specific layer to the pre-trained foundation model \( \mathcal{M}_F \).

These impacts can be mathematically represented as:

\[
\alpha_{m1,m2}^l = P_{t_{m1}}(\theta_{m2} - \hat{\delta}_m^l) - P_{t_{m1}}(\theta_{m2}), \tag{8}
\]
\[
\beta_{m1,m2}^l = P_{t_{m1}}(\hat{\theta}_F + \hat{\delta}_m^l) - P_{t_{m1}}(\hat{\theta}_F), \tag{9}
\]

where \( m1 \in \{A, B\} \) is the task capability identifier and \( m2 \in \{A, B, G\} \) is the model identifier. We investigate the layer-wise contribution so that \( l \) is the layer index. \( \hat{\delta}_m^l \) is the delta vector for \( \mathcal{M}_{m2} \) at layer \( l \). \( P_{t_{m1}}(\cdot) \) represents the performance metric on the task \( t_{m1} \). For example, BLEU-4 (Papineni et al., 2002) score for the QA task.

We sum up two impacts as the overall contribution:

\[
c_{m1,m2}^l = \alpha_{m1,m2}^l + \beta_{m1,m2}^l. \tag{10}
\]

3.2.2 Iterative Conflict Elimination

The contribution analysis method defined in Equation (8)-(10) provides a solution to measure the importance of merging specific layers. We can then define the conflict resulted by model \( \mathcal{M}_m (m \in \{A, B\}) \) within the layer \( l \) of the merged model as:

\[
\gamma_m^l = c_{m,m}^l - c_{m,G}^l. \tag{11}
\]

In this formula, we set the capability identifier \( m1 = m \) as we expect the merged model can maintain the performance of \( \mathcal{M}_m \) on \( t_m \). We can then identify the most severe conflicting layers that impair the fine-tuned performance by sorting \( \Gamma^l = \sum_{m \in \{A,B\}} \gamma_m^l \).
To mitigate the parameter misalignment, we iteratively merge the most conflicting layers (with the largest \( \Gamma^l \)). Specifically, to process a specific layer, there are three types of conflict as illustrated in Figure 3:
1. **Severe Conflict**: \( \gamma_A^l > 0 \) and \( \gamma_B^l > 0 \), indicating both capabilities are impaired by the merging. In such cases, only the delta vector with a larger contribution is retained, e.g., dropping \( \hat{\delta}_B^l \) in the figure. Namely, \( \hat{\delta}_B^l \) is set to zero.
2. **Partial Conflict**: \( \gamma_A^l * \gamma_B^l < 0 \), i.e., one of the delta vectors leads to the parameter misalignment. The solution for this case is to prune and scale the conflict delta vector again, as we defined in Section 3.1. For example, in Figure 3, the overfitting on \( t_A \) (\( \gamma_A^l < 0 \) and \( \gamma_B^l > 0 \)) leads to the degradation of the ability for \( t_B \). As a result, we prune and scale \( \hat{\delta}_A^l \) again as\footnote{We use the same notation \( \hat{\delta}_A^l \) for clarity.}:

\[
\hat{\delta}_A^l = s_A \cdot \mathrm{Top}_{p_A}(\hat{\delta}_A^l).
\]

3. **Mutual Enhancement**: If \( \gamma_A^l \leq 0 \) and \( \gamma_B^l \leq 0 \), the merging process improves for both capabilities. In this case, no further adjustment is necessary for this layer.

After resolving the conflicts of all layers, the parameters of the final merged model \( \mathcal{M}_{merge} \) is:

\[
\theta_{merge} = \theta_F + \hat{\delta}_A + \hat{\delta}_B.
\]

4 Experiments

In this section, we conduct comprehensive experiments to evaluate the effectiveness of Hi-Merging by answering following research questions (RQ):
• **RQ1**: How does Hi-Merging perform when merging LLMs that excel at the same task but in different languages?
• **RQ2**: How does Hi-Merging perform when merging LLMs excel at different tasks with the same languages?
• **RQ3**: Is Hi-Merging applicable for merging LLMs across languages and tasks?
• **RQ4**: Can Hi-Merging effectively merge different open-source LLMs?
• **RQ5**: How is the merging conflict under our method’s settings?
• **RQ6**: What is the impact of different components of Hi-Merging on its overall performance?

Table 1: The brief description and statistics of the four datasets (MedQA (Jin et al., 2020), CMExam (Liu et al., 2023), HealthCareMagic (Li et al., 2023), and cMedQA2) (Zhang et al., 2018) used for fine-tuning.

<table>
  <tr>
    <th>Name</th>
    <th>Task</th>
    <th>Language</th>
    <th>Train</th>
    <th>Validation</th>
    <th>Test</th>
  </tr>
  <tr>
    <td>MedQA</td>
    <td>MCQA</td>
    <td>English</td>
    <td>10,000</td>
    <td>400</td>
    <td>400</td>
  </tr>
  <tr>
    <td>CMExam</td>
    <td>MCQA</td>
    <td>Chinese</td>
    <td>50,000</td>
    <td>4,000</td>
    <td>4,000</td>
  </tr>
  <tr>
    <td>HealthCareMagic</td>
    <td>QA</td>
    <td>English</td>
    <td>30,000</td>
    <td>1,000</td>
    <td>1,000</td>
  </tr>
  <tr>
    <td>cMedQA2</td>
    <td>QA</td>
    <td>Chinese</td>
    <td>30,000</td>
    <td>1,000</td>
    <td>1,000</td>
  </tr>
</table>

4.1 Experimental Settings

4.1.1 Datasets
We select four datasets listed in Table 1 that cover multilingual multi-task capabilities, including English and Chinese languages, with multiple-choice question answering (MCQA) and open-domain question answering (QA) tasks.

4.1.2 Baselines
In our experiments, we use the multilingual and multi-task models fine-tuned on combined datasets as strong baselines. For model merging approaches, we consider a range of general model merging methods, including weighted averaging (Model Soups (Wortsman et al., 2022)) and delta vector-based approaches (Arithmetic (Ilharco et al., 2023), TIES-Merging (Yadav et al., 2023), DARE (Yu et al., 2024), DELLA (Deep et al., 2024), and Model Breadcrumbs (Davari and Belilovsky, 2024)). We further compare with two knowledge transfer approaches: OT-Fusion (Singh and Jaggi, 2020) and Layer Swapping (Bandarkar et al., 2025). Details are in Appendix A.1.1.

4.1.3 Implementation Details
We use Qwen2-7B-Instruct as foundation model, with results for other foundation models presented in Appendix A.2. For fine-tuning, we employ LLaMA-Factory\footnote{https://github.com/hiyouga/LLaMA-Factory} with LoRA (rank=8, alpha=16, dropout=0.01) and a batch size of 64. The learning rate is \( 1.0^{-4} \) with cosine decay and warm-up. LLM merging is performed using mergekit\footnote{https://github.com/arcee-ai/mergekit}. Both \( p \) and \( s \) in model-wise process range from 0.1 to 1.0 with a step of 0.1. In layer-wise process, the pruning threshold \( p \) and scaling factor \( s \) are successively set to half of their model-wise values.

4.1.4 Evaluation Metrics
For the MCQA task, accuracy is employed to measure the proportion of correct answers (Devlin et al.,
Table 2: Performance comparison of merging methods for bilingual MCQA task. Model A is fine-tuned on MedQA. Model B is fine-tuned on CMExam. Multi-task model is fine-tuned on both. The overall best result is in bold and the best merging result is underlined.

<table>
  <tr>
    <th rowspan="2">Types</th>
    <th rowspan="2">Methods</th>
    <th colspan="2">L1 (MedQA)</th>
    <th colspan="2">L2 (CMExam)</th>
    <th rowspan="2">Avg Impr.</th>
    <th rowspan="2">Avg Rank.</th>
  </tr>
  <tr>
    <th>L1 (MedQA)</th>
    <th>L2 (CMExam)</th>
    <th>L1 (MedQA)</th>
    <th>L2 (CMExam)</th>
  </tr>
  <tr>
    <td rowspan="3">Pre-trained</td>
    <td>Qwen2-7B-Instruct</td>
    <td>51.4062</td>
    <td>74.6217</td>
    <td></td>
    <td></td>
    <td>17.0</td>
    <td></td>
  </tr>
  <tr>
    <td>Yi-1.5-9B</td>
    <td>46.8185</td>
    <td>58.6499</td>
    <td>-16.31%</td>
    <td>18.0</td>
    <td></td>
  </tr>
  <tr>
    <td>Baichuan2-7B</td>
    <td>6.4415</td>
    <td>7.1439</td>
    <td>-89.22%</td>
    <td>19.0</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="4">Fine-tuned</td>
    <td>Model A (L1)</td>
    <td>59.1406</td>
    <td>83.7771</td>
    <td>+13.40%</td>
    <td>10.0</td>
    <td></td>
  </tr>
  <tr>
    <td>Model B (L2)</td>
    <td>54.4531</td>
    <td>88.6171</td>
    <td>+13.52%</td>
    <td>11.5</td>
    <td></td>
  </tr>
  <tr>
    <td>Multitask</td>
    <td>60.0781</td>
    <td>88.2246</td>
    <td>+17.67%</td>
    <td>3.5</td>
    <td></td>
  </tr>
  <tr>
    <td>Model Soups</td>
    <td>59.6094</td>
    <td>88.6929</td>
    <td>+17.67%</td>
    <td>5.0</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="6">Merged</td>
    <td>Task Arithmetic</td>
    <td>59.5312</td>
    <td>88.7681</td>
    <td>+17.67%</td>
    <td>4.0</td>
    <td></td>
  </tr>
  <tr>
    <td>TIES</td>
    <td>59.0625</td>
    <td>88.7832</td>
    <td>+17.31%</td>
    <td>4.5</td>
    <td></td>
  </tr>
  <tr>
    <td>DARE</td>
    <td>58.6711</td>
    <td>88.6021</td>
    <td>+17.09%</td>
    <td>7.5</td>
    <td></td>
  </tr>
  <tr>
    <td>DARE + TIES</td>
    <td>58.9063</td>
    <td>88.6021</td>
    <td>+17.04%</td>
    <td>8.0</td>
    <td></td>
  </tr>
  <tr>
    <td>Model Breadcrumbs</td>
    <td>58.8281</td>
    <td>88.6322</td>
    <td>+17.00%</td>
    <td>8.5</td>
    <td></td>
  </tr>
  <tr>
    <td>DELLA</td>
    <td>58.9844</td>
    <td>88.7681</td>
    <td>+17.24%</td>
    <td>5.5</td>
    <td></td>
  </tr>
  <tr>
    <td>DELLA + TIES</td>
    <td>58.2812</td>
    <td>88.7530</td>
    <td>+16.67%</td>
    <td>9.5</td>
    <td></td>
  </tr>
  <tr>
    <td>OPFusion</td>
    <td>59.8271</td>
    <td>88.7530</td>
    <td>+16.40%</td>
    <td>3.0</td>
    <td></td>
  </tr>
  <tr>
    <td>Layer Swapping</td>
    <td>55.6406</td>
    <td>87.0859</td>
    <td>+16.24%</td>
    <td>16.0</td>
    <td></td>
  </tr>
  <tr>
    <td>Hi-Merging (Ours)</td>
    <td><b>60.1562</b></td>
    <td><b>89.0700</b></td>
    <td><b>+18.41%</b></td>
    <td><b>1.0</b></td>
    <td></td>
  </tr>
</table>

Figure 4: Performance of Hi-Merging on two open-source medical models, Echelon-AI/Med-Qwen2-7B and shtdbb/qwen2-7b-med, which are fine-tuned from the foundation model Qwen/Qwen2-7B-Instruct.

2019). For the QA task, we use BLEU-4 (Papineni et al., 2002) to evaluate the precision of the generation, and ROUGE-1,2,L (Lin, 2004) to assess the overlap and coherence with the ground truth. Additionally, we report both the average relative performance improvement (Avg Impr.) and the mean ranking (Avg Rank.) across all methods.

4.2 Bilingual Task Merging (RQ1)

We first verify the effectiveness of Hi-Merging on bilingual task merging. Here, we merge models trained on the MCQA task in English and Chinese, as shown in Table 2. Additional experiments on the QA task and a different LLM are provided in Appendix A.2 due to space constraints.

Baseline methods like Model Soups and Task Arithmetic that combine models without considering noises and conflicts achieve stable but lower performance. Methods that reduce conflicts, such as TIES and DARE, occasionally achieve the best results on individual metrics. However, without a clear guidance, their performance highly randomised. In contrast, our Hi-Merging method, with hierarchical pruning and scaling approach, not only achieves the best average performance but attains optimal results in about half of the individual metrics. We also investigate the impact of different training sample sizes in Appendix A.3.

4.3 Monolingual Multi-task Merging (RQ2)

For monolingual multi-task merging, we combine models trained on different tasks with the same language (e.g., English MCQA with English QA), as shown in Table 3. The results show that merged models consistently outperform their individual fine-tuned counterparts, with many even surpassing multi-task fine-tuned models. Notably, our Hi-Merging approach achieves a 1.84% relative improvement over the multi-task fine-tuned model. We attribute this success to three factors. 1) During multi-task fine-tuning with limited data (compared to pre-training), tasks can interfere with each other due to the “seesaw effect”. In contrast, model merging allows parameters to be optimized independently before integration, avoiding such interference. 2) Since both models are fine-tuned from the same foundation model, their parameter updates tend to follow similar optimization trajectories, making successful merging more likely. 3) The inherent sparsity of LLMs provides sufficient parameter space to accommodate multi-task knowledge from both models.

4.4 Bilingual Multi-task Merging (RQ3)

For bilingual multi-task merging, we combine models trained on completely different tasks and languages. Specifically, we merge a model trained for MCQA in one language (Model A: MedQA in English or CMExam in Chinese) with another model trained for QA in the opposite language (Model B: cMedQA2 in Chinese or HealthCareMagic in English), as illustrated in Table 4.

Our experiments reveal an interesting pattern: bilingual multi-task fine-tuning mainly affects QA performance, while MCQA performance remain. This can be explained by two factors: (1) QA
Table 3: Performance comparison of merging methods for tasks with different question formats. Model A is fine-tuned on MCQA tasks (T1), while model B is fine-tuned on QA tasks (T2). The overall best result is marked in bold and the best merging result is underlined.

<table>
  <tr>
    <th rowspan="2">Types</th>
    <th rowspan="2">Methods</th>
    <th colspan="4">L1 (English)</th>
    <th colspan="4">L2 (Chinese)</th>
    <th rowspan="2">Avg Impr.</th>
    <th rowspan="2">Avg Rank.</th>
  </tr>
  <tr>
    <th>T1 (MedQA)</th>
    <th>T2 (HealthCareMagic)</th>
    <th>T1 (CMExam)</th>
    <th>T2 (cMedQA2)</th>
    <th>Accuracy</th>
    <th>BLEU-4</th>
    <th>ROUGE-1</th>
    <th>ROUGE-2</th>
    <th>ROUGE-L</th>
  </tr>
  <tr>
    <td>Pre-trained</td>
    <td>Qwen2-7B-Instruct</td>
    <td>51.4062</td>
    <td>30.1209</td>
    <td>26.3524</td>
    <td>5.3280</td>
    <td>15.7451</td>
    <td>74.6217</td>
    <td>1.7090</td>
    <td>14.1527</td>
    <td>1.7822</td>
    <td>9.0934</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Fine-tuned</td>
    <td>Model A (T1)</td>
    <td>59.1406</td>
    <td>34.6531</td>
    <td>28.7482</td>
    <td>6.9168</td>
    <td>17.9525</td>
    <td>88.6171</td>
    <td>2.8064</td>
    <td>16.8617</td>
    <td>2.5603</td>
    <td>12.0561</td>
    <td>+17.36%</td>
    <td>11.2</td>
  </tr>
  <tr>
    <td></td>
    <td>Model B (T2)</td>
    <td>53.0469</td>
    <td>35.5717</td>
    <td>30.2512</td>
    <td>8.9044</td>
    <td>20.3625</td>
    <td>81.5670</td>
    <td>4.4159</td>
    <td>21.2210</td>
    <td>4.0680</td>
    <td>17.4600</td>
    <td>+20.21%</td>
    <td>7.2</td>
  </tr>
  <tr>
    <td></td>
    <td>Multi-task</td>
    <td>59.2188</td>
    <td>35.6009</td>
    <td>30.2101</td>
    <td>9.1375</td>
    <td>20.4645</td>
    <td>88.6926</td>
    <td>3.7790</td>
    <td>20.5919</td>
    <td>3.8096</td>
    <td>16.9265</td>
    <td>+25.23%</td>
    <td>8.3</td>
  </tr>
  <tr>
    <td>Merged</td>
    <td>Model Soups</td>
    <td>58.5156</td>
    <td>36.4411</td>
    <td>30.5654</td>
    <td>9.1754</td>
    <td>20.4259</td>
    <td>88.8285</td>
    <td>4.3912</td>
    <td>21.0216</td>
    <td>4.0040</td>
    <td>17.2843</td>
    <td>+26.19%</td>
    <td>5.6</td>
  </tr>
  <tr>
    <td></td>
    <td>Task Arithmetic</td>
    <td>58.5938</td>
    <td>36.2390</td>
    <td>30.6624</td>
    <td>9.1945</td>
    <td>20.5406</td>
    <td>88.7983</td>
    <td>4.3018</td>
    <td>20.6467</td>
    <td>3.7496</td>
    <td>16.9995</td>
    <td>+26.13%</td>
    <td>6.1</td>
  </tr>
  <tr>
    <td></td>
    <td>TIES</td>
    <td>60.4688</td>
    <td>35.7853</td>
    <td>30.3242</td>
    <td>9.0110</td>
    <td>20.3710</td>
    <td>88.6177</td>
    <td>4.5434</td>
    <td>21.5625</td>
    <td>4.1910</td>
    <td>17.4471</td>
    <td>+26.20%</td>
    <td>4.2</td>
  </tr>
  <tr>
    <td></td>
    <td>DARE</td>
    <td>58.4370</td>
    <td>35.9682</td>
    <td>30.4888</td>
    <td>8.0818</td>
    <td>20.3945</td>
    <td>88.5475</td>
    <td>4.3875</td>
    <td>21.4471</td>
    <td>4.1471</td>
    <td>17.4471</td>
    <td>+26.20%</td>
    <td>4.2</td>
  </tr>
  <tr>
    <td></td>
    <td>DARE+TIES</td>
    <td>59.3750</td>
    <td>35.7062</td>
    <td>30.1930</td>
    <td>8.7840</td>
    <td>20.0878</td>
    <td>88.8285</td>
    <td>4.1587</td>
    <td>21.1291</td>
    <td>3.8124</td>
    <td>17.2868</td>
    <td>+25.63%</td>
    <td>7.5</td>
  </tr>
  <tr>
    <td></td>
    <td>Model Breadcrumbs</td>
    <td>57.8906</td>
    <td>36.4620</td>
    <td>30.2173</td>
    <td>8.7845</td>
    <td>20.0169</td>
    <td>88.8889</td>
    <td>4.4472</td>
    <td>21.2492</td>
    <td>3.8931</td>
    <td>17.2846</td>
    <td>+25.53%</td>
    <td>6.4</td>
  </tr>
  <tr>
    <td></td>
    <td>DELLA</td>
    <td>58.5938</td>
    <td>36.4411</td>
    <td>30.1715</td>
    <td>8.8125</td>
    <td>20.1879</td>
    <td>88.8134</td>
    <td>4.3718</td>
    <td>21.0226</td>
    <td>3.9300</td>
    <td>17.3403</td>
    <td>+25.83%</td>
    <td>7.1</td>
  </tr>
  <tr>
    <td></td>
    <td>DELLA+TIES</td>
    <td>59.5312</td>
    <td>36.6774</td>
    <td>30.4743</td>
    <td>9.1151</td>
    <td>20.5352</td>
    <td>88.6021</td>
    <td>4.2802</td>
    <td>21.2369</td>
    <td>4.0148</td>
    <td>17.3776</td>
    <td>+25.86%</td>
    <td>5.7</td>
  </tr>
  <tr>
    <td></td>
    <td>Hi-Merging (Ours)</td>
    <td><b>60.7812</b></td>
    <td>36.9295</td>
    <td>30.5467</td>
    <td>9.1231</td>
    <td><b>20.7532</b></td>
    <td><b>88.9795</b></td>
    <td><b>4.6781</b></td>
    <td><b>21.5367</b></td>
    <td><b>5.2165</b></td>
    <td><b>17.9308</b></td>
    <td><b>+27.07%</b></td>
    <td><b>2.1</b></td>
  </tr>
</table>

Table 4: Performance comparison of merging methods for bilingual multi-task learning. Model A is fine-tuned on MCQA datasets (T1: MedQA or CMExam). Model B is fine-tuned on QA datasets (T2: cMedQA2 or HealthCareMagic). The overall best result is marked in bold and the best merging result is underlined.

<table>
  <tr>
    <th rowspan="2">Types</th>
    <th rowspan="2">Methods</th>
    <th colspan="4">T1, L1 (MedQA)</th>
    <th colspan="4">T2, L2 (cMedQA2)</th>
    <th colspan="4">T1, L2 (CMExam)</th>
    <th colspan="4">T2, L1 (HealthCareMagic)</th>
    <th rowspan="2">Avg Impr.</th>
    <th rowspan="2">Avg Rank.</th>
  </tr>
  <tr>
    <th>Accuracy</th>
    <th>BLEU-4</th>
    <th>ROUGE-1</th>
    <th>ROUGE-2</th>
    <th>ROUGE-L</th>
    <th>Accuracy</th>
    <th>BLEU-4</th>
    <th>ROUGE-1</th>
    <th>ROUGE-2</th>
    <th>ROUGE-L</th>
    <th>Accuracy</th>
    <th>BLEU-4</th>
    <th>ROUGE-1</th>
    <th>ROUGE-2</th>
    <th>ROUGE-L</th>
  </tr>
  <tr>
    <td>Pre-trained</td>
    <td>Qwen2-7B-Instruct</td>
    <td>51.4062</td>
    <td>1.7090</td>
    <td>14.1527</td>
    <td>1.7822</td>
    <td>9.0934</td>
    <td>74.6217</td>
    <td>30.1209</td>
    <td>26.3524</td>
    <td>5.3280</td>
    <td>15.7451</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Fine-tuned</td>
    <td>Model A (T1)</td>
    <td>59.1406</td>
    <td>2.8064</td>
    <td>16.8617</td>
    <td>2.5603</td>
    <td>12.0561</td>
    <td>88.6171</td>
    <td>34.6713</td>
    <td>28.4279</td>
    <td>6.6122</td>
    <td>18.1117</td>
    <td>+17.17%</td>
    <td>10.7</td>
  </tr>
  <tr>
    <td></td>
    <td>Model B (T2)</td>
    <td>53.0469</td>
    <td>4.4159</td>
    <td>21.2210</td>
    <td>4.0680</td>
    <td>17.4600</td>
    <td>81.5670</td>
    <td>35.5717</td>
    <td>30.2512</td>
    <td>8.9044</td>
    <td>20.3625</td>
    <td>+20.03%</td>
    <td>7.2</td>
  </tr>
  <tr>
    <td></td>
    <td>Multi-task</td>
    <td>59.2188</td>
    <td>3.7790</td>
    <td>20.5919</td>
    <td>3.8096</td>
    <td>16.9265</td>
    <td>88.6926</td>
    <td>35.7429</td>
    <td>30.1735</td>
    <td>8.9133</td>
    <td>20.3902</td>
    <td>+26.22%</td>
    <td>8.3</td>
  </tr>
  <tr>
    <td>Merged</td>
    <td>Model Soups</td>
    <td>58.3584</td>
    <td>4.6592</td>
    <td>21.2316</td>
    <td>4.0559</td>
    <td>17.8835</td>
    <td>88.3632</td>
    <td>36.1765</td>
    <td>30.7169</td>
    <td>9.2702</td>
    <td>20.5227</td>
    <td>+26.35%</td>
    <td>4.5</td>
  </tr>
  <tr>
    <td></td>
    <td>Task Arithmetic</td>
    <td>58.0469</td>
    <td>4.6682</td>
    <td>21.2618</td>
    <td>4.0984</td>
    <td>17.4231</td>
    <td>88.7379</td>
    <td>36.1222</td>
    <td>30.2256</td>
    <td>8.7570</td>
    <td>20.1357</td>
    <td>+25.69%</td>
    <td>5.7</td>
  </tr>
  <tr>
    <td></td>
    <td>TIES</td>
    <td>59.0694</td>
    <td>4.3764</td>
    <td>21.0882</td>
    <td>3.9002</td>
    <td>17.4194</td>
    <td>88.7228</td>
    <td>37.5080</td>
    <td>30.5143</td>
    <td>8.8940</td>
    <td>20.3467</td>
    <td>+26.16%</td>
    <td>6.4</td>
  </tr>
  <tr>
    <td></td>
    <td>DARE</td>
    <td>57.8906</td>
    <td>4.2157</td>
    <td>21.3595</td>
    <td>3.8084</td>
    <td>17.4826</td>
    <td>88.5686</td>
    <td>34.8223</td>
    <td>29.7597</td>
    <td>8.3004</td>
    <td>19.7624</td>
    <td>+24.76%</td>
    <td>7.8</td>
  </tr>
  <tr>
    <td></td>
    <td>DARE+TIES</td>
    <td>58.75</td>
    <td>4.4929</td>
    <td>21.3194</td>
    <td>4.0824</td>
    <td>17.4826</td>
    <td>88.5686</td>
    <td>34.8223</td>
    <td>29.7597</td>
    <td>8.3004</td>
    <td>19.7624</td>
    <td>+24.76%</td>
    <td>7.8</td>
  </tr>
  <tr>
    <td></td>
    <td>Model Breadcrumbs</td>
    <td>57.1094</td>
    <td>4.7217</td>
    <td>21.4192</td>
    <td>4.1477</td>
    <td>17.4182</td>
    <td>88.6021</td>
    <td>36.4961</td>
    <td>30.3911</td>
    <td>9.0966</td>
    <td>20.4108</td>
    <td>+25.82%</td>
    <td>4.3</td>
  </tr>
  <tr>
    <td></td>
    <td>DELLA</td>
    <td>58.0469</td>
    <td><b>4.8065</b></td>
    <td><b>21.5135</b></td>
    <td>4.1356</td>
    <td>17.4962</td>
    <td>88.6167</td>
    <td>36.0159</td>
    <td>30.3747</td>
    <td>9.0414</td>
    <td>20.3929</td>
    <td>+26.11%</td>
    <td><b>3.9</b></td>
  </tr>
  <tr>
    <td></td>
    <td>DELLA+TIES</td>
    <td>59.0625</td>
    <td>4.4854</td>
    <td>20.9954</td>
    <td>4.0491</td>
    <td><b>17.5630</b></td>
    <td>88.6024</td>
    <td>36.0176</td>
    <td>29.9666</td>
    <td>8.6580</td>
    <td>20.1406</td>
    <td>+25.31%</td>
    <td>7.5</td>
  </tr>
  <tr>
    <td></td>
    <td>Hi-Merging (Ours)</td>
    <td>60.2341</td>
    <td>4.7743</td>
    <td>21.7954</td>
    <td><b>4.1749</b></td>
    <td>17.3991</td>
    <td><b>88.7983</b></td>
    <td><b>36.8223</b></td>
    <td>30.3932</td>
    <td>8.7882</td>
    <td>20.1619</td>
    <td><b>+27.07%</b></td>
    <td><b>2.1</b></td>
  </tr>
</table>

Table 5: Performance of Hi-Merging on two open-source medical models: Echelon-AI/Med-Qwen2-7B and shtdbb/qwen2-7b-med.

<table>
  <tr>
    <th>Types</th>
    <th>Models</th>
    <th colspan="2">Medical</th>
    <th colspan="2">Math</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>MedQA</th>
    <th>Pubmed</th>
    <th>MedQA</th>
    <th>Pubmed</th>
  </tr>
  <tr>
    <td>Single</td>
    <td>Pre-trained</td>
    <td>37.3868</td>
    <td>5.7994</td>
    <td>65.96</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>Echelon-AI/Med-Qwen2-7B</td>
    <td>64.2862</td>
    <td><b>92.9888</b></td>
    <td>15.92</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>shtdbb/qwen2-7b-med</td>
    <td>40.1598</td>
    <td>11.5017</td>
    <td>57.77</td>
    <td></td>
  </tr>
  <tr>
    <td>Merged</td>
    <td>Hi-Merging (Medical)</td>
    <td><b>64.9011</b></td>
    <td>92.1692</td>
    <td>56.63</td>
    <td></td>
  </tr>
</table>

Table 6: Performance of Hi-Merging after merging the medical model with the math-specialized model Qwen2-Math-7B-Instruct.

<table>
  <tr>
    <th>Types</th>
    <th>Models</th>
    <th colspan="2">Medical</th>
    <th colspan="2">Math</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>MedQA</th>
    <th>Pubmed</th>
    <th>MedQA</th>
    <th>Pubmed</th>
  </tr>
  <tr>
    <td>Single</td>
    <td>Pre-trained</td>
    <td>37.3868</td>
    <td>5.7994</td>
    <td>65.96</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>Merged (Medical)</td>
    <td><b>64.9011</b></td>
    <td><b>92.1692</b></td>
    <td>56.63</td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>Qwen2-Math-7B-Instruct</td>
    <td>36.7116</td>
    <td>15.3618</td>
    <td>79.00</td>
    <td></td>
  </tr>
  <tr>
    <td>Merged</td>
    <td>Hi-Merging (Medical + Math)</td>
    <td>63.6742</td>
    <td>91.7320</td>
    <td>77.45</td>
    <td></td>
  </tr>
</table>

tasks require complex free-form generation, making them more vulnerable to joint fine-tuning; (2) MCQA tasks involve clear classification boundaries and simpler choice selection, making them more robust to merging process.

4.5 Open-source LLM Merging (RQ4)

To validate the generality of our merging approach, we conduct experiments using two open-source medical models from Hugging Face: Echelon-AI/Med-Qwen2-7B 5, fine-tuned on English datasets for tasks such as medical QA and information retrieval (IR), and shtdbb/qwen2-7b-med 6, fine-tuned on Chinese datasets for dialogue generation. Both models are derived from Qwen2-7B-Instruct. Figure 4 illustrates the performance comparison across 12 medical datasets, with metrics normalized for better visualization.

Our approach demonstrates robust performance across the task spectrum. In 7 out of 12 datasets, Hi-Merging achieves the best performance among all models, with only two datasets showing appar-

5https://huggingface.co/Echelon-AI/Med-Qwen2-7B
6https://huggingface.co/shtdbb/qwen2-7b-med
ent degradation compared to the better-performing individual model. These results demonstrate Hi-Merging’s ability to effectively fuse medical knowledge while maintaining or enhancing performance across diverse languages and tasks. Detailed implementation setup and unprocessed numerical results can be found in Appendix A.1.2 and A.4.

To further demonstrate the adaptability of our approach in other domains, we merge the combined medical model with an additional mathematical model Qwen/Qwen2-Math-7B-Instruct 7. As shown in Table 5 and Table 6, merging two models from the same domain (medical) leads to mutually beneficial integration. However, when further merging the medical model with an LLM from a different domain (math), we observe slight performance drops on both domains, suggesting that larger divergence in fine-tuning data increases the difficulty of effective knowledge integration.

4.6 Case Study (RQ5)

Figure 5 visualizes layer-wise contributions and merging conflicts when combining MedQA and CMExam models, revealing that conflicts are not uniformly distributed. Model A (MedQA) shows significant conflicts in later layers, while Model B (CMExam) exhibits conflicts in earlier layers. This non-uniformity highlights the need for Hi-Merging’s hierarchical pruning and scaling strategy, leading to the improved performance demonstrated in previous experiments in Table 8. Such layer-specific conflict patterns suggest that different layers may specialize in different tasks, making a uniform merging strategy suboptimal.

![The visualization of layer contributions and merging conflicts when merging model fine-tuned on MedQA and CMExam.](page_120_120_600_300.png)

Figure 5: The visualization of layer contributions and merging conflicts when merging model fine-tuned on MedQA and CMExam.

4.7 Ablation Study (RQ6)

Table 7: Ablation study of different processes in Hi-Merging for bilingual MCQA task merging. Model A is fine-tuned on MedQA. Model B is fine-tuned on CMExam. Multi-task model is fine-tuned on both. The overall best result is in bold.

<table>
  <tr>
    <th>Types</th>
    <th>Methods</th>
    <th>L1<br>(MedQA)</th>
    <th>L2<br>(CMExam)</th>
    <th>Avg Impr.</th>
  </tr>
  <tr>
    <td rowspan="3">Pre-trained</td>
    <td>Qwen2-7B-Instruct</td>
    <td>51.4062</td>
    <td>74.6217</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Model A (L1)</td>
    <td>59.1406</td>
    <td>83.7771</td>
    <td>+13.40%</td>
  </tr>
  <tr>
    <td>Model B (L2)</td>
    <td>54.4531</td>
    <td>88.6171</td>
    <td>+13.52%</td>
  </tr>
  <tr>
    <td rowspan="1">Multi-task</td>
    <td></td>
    <td>60.0781</td>
    <td>88.2246</td>
    <td>+17.67%</td>
  </tr>
  <tr>
    <td>Merged</td>
    <td>w/o All</td>
    <td>59.5312</td>
    <td>88.5291</td>
    <td>+17.48%</td>
  </tr>
  <tr>
    <td></td>
    <td>w/o Model-wise Process</td>
    <td>59.8437</td>
    <td>88.6501</td>
    <td>+17.83%</td>
  </tr>
  <tr>
    <td></td>
    <td>w/o Model-wise Pruning</td>
    <td>60.0781</td>
    <td>88.9342</td>
    <td>+18.24%</td>
  </tr>
  <tr>
    <td></td>
    <td>w/o Model-wise Scaling</td>
    <td>59.9219</td>
    <td>88.7863</td>
    <td>+18.00%</td>
  </tr>
  <tr>
    <td></td>
    <td>w/o Layer-wise Process</td>
    <td>59.6094</td>
    <td>88.5417</td>
    <td>+17.55%</td>
  </tr>
  <tr>
    <td></td>
    <td>w/o Layer-wise Pruning</td>
    <td><b>61.0156</b></td>
    <td>88.6473</td>
    <td>+18.75%</td>
  </tr>
  <tr>
    <td></td>
    <td>w/o Layer-wise Scaling</td>
    <td>59.7656</td>
    <td>88.6926</td>
    <td>+17.80%</td>
  </tr>
  <tr>
    <td></td>
    <td>Hi-Merging</td>
    <td>60.1362</td>
    <td><b>89.0700</b></td>
    <td>+18.41%</td>
  </tr>
</table>

The ablation study in Table 7 reveals several key insights. Layer-wise process has a more significant impact than model-wise process, and removing scaling operations leads to larger performance drops than removing pruning. While removing layer-wise pruning achieves the highest average improvement, it shows less consistent performance across tasks compared to the full Hi-Merging approach, indicating that pruning helps stabilize the merging process despite potentially limiting peak performance on specific tasks.

5 Related Works

5.1 Multilingual Task-Oriented LLMs

Multi-task learning (MTL) has proven valuable across various domains, from recommendation systems (Wang et al., 2023a; Zhang et al., 2024) to knowledge graphs (Xu et al., 2024a) and healthcare (Liu et al., 2024c), by enabling models to share knowledge between related tasks. This paradigm becomes especially relevant for multilingual NLP, where different languages face similar challenges in tasks like machine translation (Wang et al., 2022), text summarization (Gambhir and Gupta, 2017), and sentiment analysis (Dashtipour et al., 2016).

Recently, LLMs have greatly contributed to advancing multilingual tasks by leveraging massive amounts of multilingual data (Brown et al., 2020; Devlin et al., 2019; Xue et al., 2021). Despite their success, LLMs exhibit a clear performance gap across languages: they excel at widely-spoken languages with abundant training data but struggle
with less-represented languages that have limited online presence (Wang et al., 2023b).

To enhance multilingual capabilities, LLMs employ continual training on specific languages, as seen in models like Chinese-LLaMA (Cui et al., 2023) and EuroLLM (Martins et al., 2024). Additionally, supervised fine-tuning techniques, such as LoRA in Chinese-Alpaca (Cui et al., 2023), further improve multilingual understanding. However, LLMs are usually enhanced for one language at a time, resulting in multiple isolated models.

5.2 Model Merging

Model merging aims to integrate knowledge from multiple fine-tuned models into a single one. These methods are categorized into two types: weighted-based merging and interference mitigation.

Weighted-based merging focuses on combining model parameters effectively. This includes simple techniques like parameter averaging, such as Model Soups (Wortsman et al., 2022), Fisher-weighted merging (Matena and Raffel, 2022) and RegMean (Jin et al., 2023). While computationally efficient, these methods often miss conflicting parameter updates, leading to performance degradation. Therefore, Task Arithmetic (Ilharco et al., 2023) proposes manipulating delta vectors. AdaMerging (Yang et al., 2024c) and evolutionary algorithms (Akiba et al., 2024) optimize merging coefficients and blend diverse models, respectively.

Interference mitigation techniques aim to reduce parameter conflicts based on the over-parameterization and sparsity of LLM. SparseGPT (Frantar and Alistarh, 2023) show high LLM performance despite significant parameter pruning. DELLA (Deep et al., 2024) introduces MAG-PRUNE for selective pruning and parameter rescaling. However, these techniques focus mainly on individual parameter-level operations without considering the structural relationships and knowledge dependencies across model layers.

6 Conclusion

In this paper, we proposed Hi-Merging, a novel approach for merging LLMs for multilingual multi-task learning. Hi-Merging leverages model-wise and layer-wise pruning and scaling strategy to minimize the conflict between fine-tuned models’ delta vectors. The model-wise process eliminates the fine-tuning noise and overfitting parameters of the original models. Then, the layer-wise process analyzes the contribution of each layer’s delta vector to the fine-tuning performance, reducing the interference of conflicts in several key layers. Extensive experiments on the MCQA and QA datasets demonstrated that Hi-Merging outperforms traditional merging techniques and even surpasses models trained on multiple datasets. Future work will explore finer-grained conflict analysis strategies.

7 Limitations

While our proposed Hi-Merging method demonstrates promising results, several limitations should be acknowledged. First, our current method only supports merging two models at a time. Extending the approach to simultaneously merge multiple models presents additional challenges in terms of conflict resolution and computational complexity, which requires further investigation.

Second, our evaluation is currently limited to two task types (MCQA and QA) and two languages (English and Chinese). The effectiveness of Hi-Merging on a broader range of NLP tasks and language families remains to be investigated. This includes exploring its applicability to tasks such as text generation, summarization, and semantic parsing across diverse language groups.

Third, our method focuses on merging models fine-tuned from the same foundation model. The applicability and performance of Hi-Merging when merging models from different architectural families or pre-training approaches is yet to be explored. This limitation becomes particularly relevant as the field continues to see diverse model architectures and training paradigms.

Finally, our current implementation assumes relatively balanced task importance. The method might need adaptation for scenarios where certain tasks or languages should be prioritized over others, potentially requiring a more flexible weighting mechanism in the merging process. Future work could explore dynamic weighting strategies that adapt to specific application requirements and performance objectives.

Acknowledgments

This research was partially supported by Research Impact Fund (No.R1015-23), Collaborative Research Fund (No.C1043-24GF) and Tencent (CCF-Tencent Open Fund, Tencent Rhino-Bird Focused Research Program).
References

Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. 2024. Evolutionary optimization of model merging recipes. CoRR, abs/2403.13187.

Lucas Bandarkar, Benjamin Muller, Pritish Yuvraj, Rui Hou, et al. 2025. Layer swapping for zero-shot cross-lingual transfer in large language models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, et al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.

Junying Chen, Xidong Wang, Anningzhe Gao, Feng Jiang, Shunian Chen, Hongbo Zhang, Dingjie Song, Wenya Xie, Chuyi Kong, Jianquan Li, Xiang Wan, Haizhou Li, and Benyou Wang. 2023. Huatuogpt-ii, one-stage training for medical adaption of llms. Preprint, arXiv:2311.09774.

Daixuan Cheng, Shaohan Huang, and Furu Wei. 2024. Adapting large language models via reading comprehension. In The Twelfth International Conference on Learning Representations.

Yiming Cui, Ziqing Yang, and Xin Yao. 2023. Efficient and effective text encoding for chinese llama and alpaca. arXiv preprint arXiv:2304.08171.

Kia Dashtipour, Soujanya Poria, Amir Hussain, Erik Cambria, Ahmad Y. A. Hawalah, Alexander F. Gelbukh, and Qiang Zhou. 2016. Multilingual sentiment analysis: State of the art and independent comparison of techniques. Cogn. Comput., 8(4):757–771.

MohammadReza Davari and Eugene Belilovsky. 2024. Model breadcrumbs: Scaling multi-task model merging with sparse masks. In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part LXXV, volume 15133 of Lecture Notes in Computer Science, pages 270–287. Springer.

Pala Tej Deep, Rishabh Bhardwaj, and Soujanya Poria. 2024. Della-merging: Reducing interference in model merging through magnitude-based sampling. CoRR, abs/2406.11617.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 10323–10337. PMLR.

Zichuan Fu, Xiangyang Li, Chuhuan Wu, Yichao Wang, Kuicai Dong, Xiangyu Zhao, Mengchen Zhao, Huifeng Guo, and Ruiming Tang. 2024. A unified framework for multi-domain ctr prediction via large language models. ACM Trans. Inf. Syst. Just Accepted.

Mahak Gambhir and Vishal Gupta. 2017. Recent automatic text summarization techniques: a survey. Artif. Intell. Rev., 47(1):1–66.

Jinmin He, Kai Li, Yifan Zang, Haobo Fu, QIANG FU, Junliang Xing, and Jian Cheng. 2024. Efficient multi-task reinforcement learning with cross-task policy guidance. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Gabriel Ilharco, Marco Túlio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.

Di Jin, Eileen Pan, Nassim Oufatole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. CoRR, abs/2009.13081.

Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. 2023. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.

Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang. 2023. Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge. Cureus, 15(6).

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, Lei Zhu, et al. 2023. Benchmarking large language models on cmexam—a comprehensive chinese medical exam dataset. arXiv preprint arXiv:2306.03030.
Qidong Liu, Xian Wu, Yefeng Zheng, Zijian Zhang, Feng Tian, Yefeng Zheng, and Xiangyu Zhao. 2024a. LLM-ESR: large language models enhancement for long-tailed sequential recommendation. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.

Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. 2024b. When MOE meets llms: Parameter efficient fine-tuning for multi-task medical applications. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024, pages 1104–1114. ACM.

Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Zijian Zhang, Feng Tian, and Yefeng Zheng. 2024c. Large language model distilling medication recommendation model. CoRR, abs/2402.02803.

Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. 2024. Eurolm: Multilingual language models for europe. Preprint, arXiv:2409.16235.

Michael Matena and Colin Raffel. 2022. Merging models with fisher-weighted averaging. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.

OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pages 311–318. ACL.

Sidak Pal Singh and Martin Jaggi. 2020. Model fusion via optimal transport. In Advances in Neural Information Processing Systems, volume 33, pages 22045–22055. Curran Associates, Inc.

Hongyan Tang, Junning Liu, Ming Zhao, and Xudong Gong. 2020. Progressive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations. In Proceedings of the 14th ACM Conference on Recommender Systems, RecSys ’20, page 269–278, New York, NY, USA. Association for Computing Machinery.

Masatoshi Tsuchiya. 2018. Performance impact caused by hidden bias of training data for recognizing textual entailment. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).

Haifeng Wang, Hua Wu, Zhongjun He, Liang Huang, and Kenneth Ward Church. 2022. Progress in machine translation. Engineering, 18:143–153.

Yejing Wang, Zhaocheng Du, Xiangyu Zhao, Bo Chen, Huifeng Guo, Ruiming Tang, and Zhenhua Dong. 2023a. Single-shot feature selection for multi-task recommendations. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2023, Taipei, Taiwan, July 23-27, 2023, pages 341–351. ACM.

Yuhan Wang, Ha Tsz Lam, Yi Wong, Ziru Liu, Xiangyu Zhao, Yichao Wang, Bo Chen, Huifeng Guo, and Ruiming Tang. 2023b. Multi-task deep recommender systems: A survey. Preprint, arXiv:2302.03525.

Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, et al. 2022. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 23965–23998. PMLR.

Derong Xu, Ziheng Zhang, Zhenxi Lin, Xian Wu, Zhihong Zhu, Tong Xu, Xiangyu Zhao, Yefeng Zheng, and Enhong Chen. 2024a. Multi-perspective improvement of knowledge graph completion with large language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, pages 11956–11968. ELRA and ICCL.

Derong Xu, Ziheng Zhang, Zhihong Zhu, Zhenxi Lin, Qidong Liu, Xian Wu, Tong Xu, Xiangyu Zhao, Yefeng Zheng, and Enhong Chen. 2024b. Mitigating hallucinations of large language models in medical information extraction via contrastive decoding. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7744–7757, Miami, Florida, USA. Association for Computational Linguistics.

Derong Xu, Ziheng Zhang, Zhihong Zhu, Zhenxi Lin, Qidong Liu, Xian Wu, Tong Xu, et al. 2024c. Editing factual knowledge and explanatory ability of medical large language models. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, CIKM 2024, Boise, ID, USA, October 21-25, 2024, pages 2660–2670. ACM.

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 483–498. Association for Computational Linguistics.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel, and Mohit Bansal. 2023. Ties-merging: Resolving interference when merging models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

An Yang, Baosong Yang, Binyuan Hui, et al. 2024a. Qwen2 technical report. CoRR, abs/2407.10671.

Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. 2024b. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities. CoRR, abs/2408.07666.

Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. 2024c. Adamerging: Adaptive model merging for multi-task learning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.

Sheng Zhang, Xin Zhang, Hui Wang, Lixiang Guo, and Shanshan Liu. 2018. Multi-scale attentive interaction networks for chinese medical question answer selection. IEEE Access, 6:74061–74071.

Zijian Zhang, Shuchang Liu, Jiaao Yu, Qingpeng Cai, Xiangyu Zhao, Chunxu Zhang, Ziru Liu, Qidong Liu, Hongwei Zhao, Lantao Hu, Peng Jiang, and Kun Gai. 2024. M3oe: Multi-domain multi-task mixture-of-experts recommendation framework. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024, pages 893–902. ACM.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, et al. 2024. A survey of large language models. Preprint. arXiv:2303.18223.

A Appendix

A.1 Experimental Settings

A.1.1 Baselines
In our experiments, we compare it against a comprehensive set of baseline methods, including traditional weighted averaging techniques and state-of-the-art approaches specifically developed for fine-tuned models.
• Multilingual Multi-task Training This approach trains a single model on the combined datasets of multiple languages simultaneously, without distinguishing between tasks.
• Model Soups (Wortsman et al., 2022) Uniform Soup is a simple merging method where the parameters of the fine-tuned models are averaged based on their importance.
• Task Arithmetic (Ilharco et al., 2023) This method performs arithmetic operations on the parameter differences between the pre-trained and fine-tuned models.
• TIES (Yadav et al., 2023) The Task Interference Elimination Strategy (TIES) minimize negative transfer and task interference by pruning redundant parameters and using a chosen sign to determine parameter update directions.
• DARE (Yu et al., 2024) Delta Alignment for Robust Ensemble (DARE) reduces the interference across tasks by randomly drop the delta vectors.
• Model Breadcrumbs (Davari and Belilovsky, 2024) This approach tracks and prunes maxima and minima in delta vectors to retain critical task-specific features.
• DELLA (Deep et al., 2024) DELLA follows DARE and assign drop rates to delta vectors according to their absolute values, improving performance stability.
• OT-Fusion (Singh and Jaggi, 2020) This method aligns and averages model weights via optimal transport, enabling one-shot parameter merging across heterogeneous models without retraining.
• Layer Swapping (Bandarkar et al., 2025) This approach composes task and language experts by directly replacing top and bottom transformer layers, facilitating cross-lingual transfer.

A.1.2 Implementation Details
For model adaptation, we applied LoRA to all linear networks in the model. The learning rate schedule was carefully designed with a 100-step warm-up phase followed by cosine decay, which helped achieve stable convergence while maintaining optimal model performance. This configuration proved effective in balancing training efficiency and model quality across both multilingual and multi-task scenarios.

In addition to Qwen2-7B-Instruct, we also experimented with other foundation models including Llama-3-8B-Instruct (results shown in A.2). However, Qwen2-7B-Instruct demonstrated more consistent performance, particularly in handling both English and Chinese tasks, making it the preferred choice for our main experiments.

For visualization in Figure 4, we normalized the performance metrics to facilitate clear comparisons.
Table 8: Performance comparison of merging methods for bilingual QA tasks. Model A is fine-tuned on HealthCareMagic. Model B is fine-tuned on cMedQA2. Multi-task model is fine-tuned on both datasets. The overall best result is marked in bold and the best merging result is underlined.

<table>
  <tr>
    <th rowspan="2">Types</th>
    <th rowspan="2">Methods</th>
    <th colspan="4">L1 (HealthCareMagic)</th>
    <th colspan="4">L2 (cMedQA2)</th>
    <th rowspan="2">Avg Impr.</th>
    <th rowspan="2">Avg Rank.</th>
  </tr>
  <tr>
    <th>BLEU-4</th>
    <th>ROUGE-1</th>
    <th>ROUGE-2</th>
    <th>ROUGE-L</th>
    <th>BLEU-4</th>
    <th>ROUGE-1</th>
    <th>ROUGE-2</th>
    <th>ROUGE-L</th>
  </tr>
  <tr>
    <td>Pre-trained</td>
    <td>Qwen2-7B-Instruct</td>
    <td>30.1209</td>
    <td>26.3524</td>
    <td>5.3280</td>
    <td>15.7451</td>
    <td>1.7090</td>
    <td>14.1527</td>
    <td>1.7822</td>
    <td>9.0934</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="3">Fine-tuned</td>
    <td>Model A (L1)</td>
    <td>35.5717</td>
    <td><b>30.2512</b></td>
    <td><b>8.9044</b></td>
    <td>20.3625</td>
    <td>3.7609</td>
    <td>19.1370</td>
    <td>3.1364</td>
    <td>15.1441</td>
    <td>+30.66%</td>
    <td>6.875</td>
  </tr>
  <tr>
    <td>Model B (L2)</td>
    <td>24.8587</td>
    <td>24.9841</td>
    <td>4.1492</td>
    <td>15.1967</td>
    <td>4.4159</td>
    <td>21.2210</td>
    <td>4.0680</td>
    <td>17.4600</td>
    <td>+11.57%</td>
    <td>8.375</td>
  </tr>
  <tr>
    <td>Multi-task</td>
    <td>35.7637</td>
    <td>29.9781</td>
    <td>8.6687</td>
    <td>20.1184</td>
    <td>3.7660</td>
    <td>20.9869</td>
    <td>3.7784</td>
    <td>16.8850</td>
    <td>+34.19%</td>
    <td>6.125</td>
  </tr>
  <tr>
    <td rowspan="10">Merged</td>
    <td>Model Soups</td>
    <td>33.2627</td>
    <td>28.8258</td>
    <td>7.5487</td>
    <td>18.9459</td>
    <td>4.6801</td>
    <td><b>21.5564</b></td>
    <td>4.0502</td>
    <td>17.5380</td>
    <td>+30.80%</td>
    <td>6.125</td>
  </tr>
  <tr>
    <td>Task Arithmetic</td>
    <td>33.0398</td>
    <td>28.7169</td>
    <td>7.5726</td>
    <td>18.9600</td>
    <td>4.7181</td>
    <td>21.4108</td>
    <td>4.0503</td>
    <td>17.6772</td>
    <td>+30.55%</td>
    <td>5.625</td>
  </tr>
  <tr>
    <td>TIES</td>
    <td>33.6571</td>
    <td>29.0496</td>
    <td>7.7769</td>
    <td>19.1503</td>
    <td>4.3751</td>
    <td>20.8551</td>
    <td>3.7518</td>
    <td>17.1978</td>
    <td>+30.23%</td>
    <td>7.375</td>
  </tr>
  <tr>
    <td>DARE</td>
    <td>33.3031</td>
    <td>28.9575</td>
    <td>7.8222</td>
    <td>19.1702</td>
    <td><b>4.7578</b></td>
    <td>21.0865</td>
    <td>3.8996</td>
    <td>17.2488</td>
    <td>+30.64%</td>
    <td>5.625</td>
  </tr>
  <tr>
    <td>DARE+TIES</td>
    <td>26.8091</td>
    <td>26.0330</td>
    <td>5.2307</td>
    <td>16.5201</td>
    <td>4.2456</td>
    <td>20.6276</td>
    <td>3.7531</td>
    <td>17.1445</td>
    <td>+15.41%</td>
    <td>10.375</td>
  </tr>
  <tr>
    <td>Model Breadcrumbs</td>
    <td>34.3247</td>
    <td>29.4038</td>
    <td>8.1518</td>
    <td>19.6443</td>
    <td>4.4092</td>
    <td>20.9365</td>
    <td>3.8138</td>
    <td>17.1378</td>
    <td>+32.19%</td>
    <td>6.750</td>
  </tr>
  <tr>
    <td>DELLA</td>
    <td>33.4207</td>
    <td>28.9234</td>
    <td>7.6728</td>
    <td>18.9674</td>
    <td>4.6827</td>
    <td>21.1596</td>
    <td>4.0709</td>
    <td>17.4775</td>
    <td>+30.77%</td>
    <td>5.500</td>
  </tr>
  <tr>
    <td>DELLA+TIES</td>
    <td>27.2331</td>
    <td>26.1723</td>
    <td>5.4339</td>
    <td>16.6009</td>
    <td>4.7130</td>
    <td>21.2275</td>
    <td><b>4.2944</b></td>
    <td><b>17.7694</b></td>
    <td>+18.37%</td>
    <td>6.000</td>
  </tr>
  <tr>
    <td>Hi-Merging (Ours)</td>
    <td><b>35.9500</b></td>
    <td>29.9826</td>
    <td><b>8.8738</b></td>
    <td><b>20.3844</b></td>
    <td>4.7009</td>
    <td>21.1752</td>
    <td>3.9704</td>
    <td>17.2361</td>
    <td><b>+36.42%</b></td>
    <td><b>3.250</b></td>
  </tr>
</table>

Table 9: Performance comparison of merging methods for multilingual QA using Llama-3-8B-Instruct. Model A is fine-tuned on HealthCareMagic (L1: English). Model B is fine-tuned on cMedQA2 (L2: Chinese). Multi-task model is fine-tuned on both datasets. The overall best result is marked in bold and the best merging result is underlined.

<table>
  <tr>
    <th rowspan="2">Types</th>
    <th rowspan="2">Methods</th>
    <th colspan="4">L1 (HealthCareMagic)</th>
    <th colspan="4">L2 (cMedQA2)</th>
    <th rowspan="2">Avg.</th>
    <th rowspan="2">Impr.</th>
  </tr>
  <tr>
    <th>BLEU-4</th>
    <th>ROUGE-1</th>
    <th>ROUGE-2</th>
    <th>ROUGE-L</th>
    <th>BLEU-4</th>
    <th>ROUGE-1</th>
    <th>ROUGE-2</th>
    <th>ROUGE-L</th>
  </tr>
  <tr>
    <td>Pre-trained</td>
    <td>Llama-3-8B-Instruct</td>
    <td>16.3118</td>
    <td>21.6011</td>
    <td>3.1389</td>
    <td>10.8666</td>
    <td>0.0225</td>
    <td>0.4710</td>
    <td>0.0211</td>
    <td>0.2343</td>
    <td>6.5834</td>
    <td>-</td>
  </tr>
  <tr>
    <td rowspan="3">Fine-tuned</td>
    <td>Model A (L1)</td>
    <td><b>36.0325</b></td>
    <td>30.4111</td>
    <td><b>9.2743</b></td>
    <td><b>20.7236</b></td>
    <td>0.0185</td>
    <td>0.1288</td>
    <td>0.0025</td>
    <td>0.0841</td>
    <td>12.0844</td>
    <td>+83.5%</td>
  </tr>
  <tr>
    <td>Model B (L2)</td>
    <td>3.9950</td>
    <td>7.7673</td>
    <td>0.9267</td>
    <td>4.6538</td>
    <td>0.3068</td>
    <td>20.4016</td>
    <td>3.4178</td>
    <td>16.3067</td>
    <td>7.5643</td>
    <td>+14.9%</td>
  </tr>
  <tr>
    <td>Multi-task</td>
    <td>35.6154</td>
    <td><b>30.5447</b></td>
    <td>9.2156</td>
    <td>20.4271</td>
    <td>0.2030</td>
    <td>0.2531</td>
    <td><b>3.4964</b></td>
    <td>16.0911</td>
    <td><b>17.3411</b></td>
    <td><b>+163.5%</b></td>
  </tr>
  <tr>
    <td rowspan="10">Merged</td>
    <td>Model Soups</td>
    <td>32.1199</td>
    <td>28.2278</td>
    <td>6.3715</td>
    <td>18.2456</td>
    <td>3.2356</td>
    <td>19.4999</td>
    <td>2.8670</td>
    <td>15.4688</td>
    <td>15.7270</td>
    <td>+139.5%</td>
  </tr>
  <tr>
    <td>Task Arithmetic</td>
    <td>31.6679</td>
    <td>27.7646</td>
    <td>6.0354</td>
    <td>18.0448</td>
    <td>3.3805</td>
    <td>19.4755</td>
    <td>2.9507</td>
    <td>15.4806</td>
    <td>15.6215</td>
    <td>+137.2%</td>
  </tr>
  <tr>
    <td>TIES</td>
    <td>32.1494</td>
    <td>28.0827</td>
    <td>6.7440</td>
    <td>18.2913</td>
    <td>3.2238</td>
    <td>19.5699</td>
    <td>2.8854</td>
    <td>15.2112</td>
    <td>15.7618</td>
    <td>+139.4%</td>
  </tr>
  <tr>
    <td>DARE</td>
    <td>25.9679</td>
    <td>25.6716</td>
    <td>4.5173</td>
    <td>16.3803</td>
    <td>3.5337</td>
    <td>20.8866</td>
    <td>3.1736</td>
    <td>16.6736</td>
    <td>14.5968</td>
    <td>+121.7%</td>
  </tr>
  <tr>
    <td>DARE+TIES</td>
    <td>26.6707</td>
    <td>25.9106</td>
    <td>5.2031</td>
    <td>16.5525</td>
    <td>3.2236</td>
    <td>19.8664</td>
    <td>2.9963</td>
    <td>15.5967</td>
    <td>14.5012</td>
    <td>+120.2%</td>
  </tr>
  <tr>
    <td>Model Breadcrumbs</td>
    <td>26.9844</td>
    <td>26.1004</td>
    <td>4.7037</td>
    <td>16.3247</td>
    <td>3.3070</td>
    <td>20.7442</td>
    <td>3.3069</td>
    <td>16.2874</td>
    <td>14.7228</td>
    <td>+123.6%</td>
  </tr>
  <tr>
    <td>DELLA</td>
    <td>25.6313</td>
    <td>25.6792</td>
    <td>4.5522</td>
    <td>16.1313</td>
    <td><b>3.6612</b></td>
    <td><b>20.9176</b></td>
    <td>3.3286</td>
    <td><b>16.7355</b></td>
    <td>14.5796</td>
    <td>+121.4%</td>
  </tr>
  <tr>
    <td>DELLA+TIES</td>
    <td>27.1246</td>
    <td>26.0186</td>
    <td>5.3163</td>
    <td>16.6170</td>
    <td>3.3433</td>
    <td>19.9122</td>
    <td>3.0848</td>
    <td>15.9942</td>
    <td>14.6764</td>
    <td>+122.9%</td>
  </tr>
  <tr>
    <td>Hi-Merging (Ours)</td>
    <td><b>33.5960</b></td>
    <td>28.4141</td>
    <td><b>7.2167</b></td>
    <td>18.8804</td>
    <td>3.1967</td>
    <td>19.8207</td>
    <td>2.9509</td>
    <td>15.7833</td>
    <td>16.2324</td>
    <td><b>+146.5%</b></td>
  </tr>
</table>

The performance values of the models on each dataset represent the average of the QA task metrics (BLEU-4, ROUGE-1, ROUGE-2, and ROUGE-L). We scaled the pre-trained Qwen2-7B-Instruct’s performance to 20 and the better-performing fine-tuned model’s performance to 80 for each task. The performance values of the other fine-tuned model and our merged model were then proportionally adjusted within this range to maintain their relative differences.

A.2 Bilingual Task Merging

For merging LLMs that specialise in different languages on the same task, we further conduct experiments on the QA task (Table 8) and extend the foundation LLM to Llama-3-8B-Instruct, as presented in Table 11 and 9.

The results in Table 8 demonstrate the effectiveness of our approach in merging bilingual QA models. Hi-Merging achieves the best performance on English QA metrics (BLEU-4: 35.95, ROUGE-1: 29.98, ROUGE-2: 8.87, ROUGE-L: 20.38) while maintaining competitive performance on Chinese QA metrics. This balanced performance leads to the highest average improvement (+36.42%) and best average ranking (3.25) among all merging methods. Notably, while some baseline methods like DELLA+TIES achieve better performance on specific Chinese metrics, they significantly compromise English performance, highlighting our method’s advantage in maintaining cross-lingual capabilities.

The results in Table 9 show that the performance of merged models based on Llama-3-8B-Instruct is generally inferior to that of the pre-merged fine-tuned models. This indicates that the effectiveness of the merging process is strongly influenced by the quality of the foundation models. The observed degradation in performance can be attributed to several factors. First, weaker foundation models, such as Llama-3-8B-Instruct, tend to produce delta vectors with more dispersed and less coherent param-
Table 10: Numerical performance of Hi-Merging on two open-source models, Echelon-Al/Med-Qwen2-7B and shdbb/qwen2-7b-med.

<table>
  <tr>
    <th>Model</th>
    <th>MedQA</th>
    <th>MedQIQA</th>
    <th>Medical Flashcards</th>
    <th>Health Advice</th>
    <th>Pubmed</th>
    <th>WikiDoc</th>
    <th>WikiDoc Patient</th>
    <th>CORD 19</th>
    <th>iClinic</th>
    <th>HealthCareMagic</th>
    <th>ChatMed</th>
    <th>MedChatZH</th>
  </tr>
  <tr>
    <td>Qwen2-7B-Instruct</td>
    <td>37.3868</td>
    <td>17.5959</td>
    <td>22.7665</td>
    <td>2.8208</td>
    <td>3.7994</td>
    <td>17.6217</td>
    <td>18.785</td>
    <td>39.1748</td>
    <td>19.3292</td>
    <td>28.7051</td>
    <td>9.9138</td>
    <td>8.0654</td>
  </tr>
  <tr>
    <td>Echelon-Al/Med-Qwen2-7B</td>
    <td>64.2862</td>
    <td><b>32.052</b></td>
    <td>41.1081</td>
    <td>97.7523</td>
    <td><b>92.9898</b></td>
    <td>20.7237</td>
    <td><b>26.9203</b></td>
    <td>40.7167</td>
    <td>26.5593</td>
    <td>30.3212</td>
    <td>15.1218</td>
    <td>9.2714</td>
  </tr>
  <tr>
    <td>shdbb/qwen2-7b-med</td>
    <td>40.1598</td>
    <td>27.1442</td>
    <td>29.85</td>
    <td>4.096</td>
    <td>11.5017</td>
    <td>20.5808</td>
    <td>21.1528</td>
    <td><b>41.2026</b></td>
    <td>27.332</td>
    <td><b>33.3678</b></td>
    <td>19.4513</td>
    <td>11.2665</td>
  </tr>
  <tr>
    <td>Hi-Merging (Ours)</td>
    <td><b>64.9011</b></td>
    <td>31.9421</td>
    <td><b>45.1714</b></td>
    <td>97.755</td>
    <td>92.1692</td>
    <td><b>21.0211</b></td>
    <td>26.3293</td>
    <td>40.9803</td>
    <td><b>28.7816</b></td>
    <td>31.6779</td>
    <td><b>19.8074</b></td>
    <td><b>11.2958</b></td>
  </tr>
</table>

![Impact of training sample size on model merging conflicts. Blue and orange lines represent the average performance metrics for HealthCareMagic and cMedQA2, respectively.](page_109_370_1267_370.png)

Figure 6: Impact of training sample size on model merging conflicts. Blue and orange lines represent the average performance metrics for HealthCareMagic and cMedQA2, respectively.

Table 11: Performance comparison of merging methods for bilingual MCQA using Llama-3-8B-Instruct. Model A is fine-tuned on MedQA (L1: English). Model B is fine-tuned on CMExam (L2: Chinese). Multi-task model is fine-tuned on both datasets. Overall best result is in bold and the best merging result is underlined.

<table>
  <tr>
    <th rowspan="2">Types</th>
    <th rowspan="2">Methods</th>
    <th colspan="2">L1</th>
    <th colspan="2">L2</th>
    <th rowspan="2">Avg.</th>
    <th rowspan="2">Impr.</th>
  </tr>
  <tr>
    <th>(MedQA)</th>
    <th>(CMExam)</th>
    <th>(MedQA)</th>
    <th>(CMExam)</th>
  </tr>
  <tr>
    <td rowspan="3">Pre-trained</td>
    <td>Llama-3-8B-Instruct</td>
    <td>57.9733</td>
    <td>17.2821</td>
    <td>37.6277</td>
    <td>+0.00%</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>GLM-4-9B</td>
    <td>54.7656</td>
    <td>69.5194</td>
    <td>62.1425</td>
    <td><b>+65.15%</b></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Gemma-2-9B</td>
    <td>14.2583</td>
    <td>2.7698</td>
    <td>8.5141</td>
    <td>-77.37%</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="3">Fine-tuned</td>
    <td>Model A (L1)</td>
    <td>60.4688</td>
    <td>52.2706</td>
    <td>56.3697</td>
    <td>+49.81%</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Model B (L2)</td>
    <td>60.3906</td>
    <td>60.5525</td>
    <td>61.0575</td>
    <td>+62.27%</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Multi-task</td>
    <td><b>62.8906</b></td>
    <td>61.0356</td>
    <td><b>61.9631</b></td>
    <td>+64.70%</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="8">Merged</td>
    <td>Model Soups</td>
    <td>61.2500</td>
    <td>61.0507</td>
    <td>61.1504</td>
    <td>+62.54%</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Task Arithmetic</td>
    <td>61.2500</td>
    <td><b>61.8750</b></td>
    <td>61.5625</td>
    <td>+63.65%</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>TIES</td>
    <td>61.7571</td>
    <td>61.8750</td>
    <td>61.8125</td>
    <td>+63.83%</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>DARE</td>
    <td>61.5625</td>
    <td>61.3678</td>
    <td>61.4652</td>
    <td>+63.42%</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>DARE + TIES</td>
    <td>60.9375</td>
    <td>59.4656</td>
    <td>60.2016</td>
    <td>+60.05%</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Model Breadcrumbs</td>
    <td>61.0156</td>
    <td>60.4318</td>
    <td>60.7237</td>
    <td>+61.43%</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>DELLA</td>
    <td>60.8594</td>
    <td>60.7186</td>
    <td>60.7890</td>
    <td>+61.58%</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>DELLA + TIES</td>
    <td>61.9351</td>
    <td>61.3527</td>
    <td>61.6529</td>
    <td>+63.91%</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Hi-Merging (Ours)</td>
    <td><b>62.2656</b></td>
    <td>61.0757</td>
    <td><b>61.6707</b></td>
    <td><b>+63.96%</b></td>
    <td></td>
    <td></td>
  </tr>
</table>

eter distributions during fine-tuning. These delta vectors often carry noisy or conflicting information, which makes the merging process prone to parameter conflicts. Second, the weaker representational capacity of these models limits their ability to encode robust and semantically aligned knowledge, further exacerbating the challenges of merging.

A.3 Number of training samples

We examine the impact of varying the number of training samples on the conflict during model merging, as shown in Figure 6. In the experiment, we use two QA datasets, HealthCareMagic (English) and cMedQA2 (Chinese), sampling 10k, 20k, 30k, 40k, and 50k training examples from each to produce a series of fine-tuned models, five per dataset. This setup evaluates how the number of training samples influences both individual model performance and compatibility during merging. The x-axis of Figure 6 represents the number of training samples, while the y-axis denotes the average performance metrics, including BLEU-4, ROUGE-1, ROUGE-2, and ROUGE-L.

However, Figures 6b and 6c show that merged models through either Model Soups or Task Arithmetic suffer from performance drops driven by the increasing size of training sample as further training leads to conflicting highly specialized models. Figure 6d shows the opposite: our method retains performance trends in line with fine-tuned models and addresses conflicts to retain improving performance through larger training sets.

These results highlight the robustness of our method in resolving merging conflicts, ensuring that the merged models retain the strengths of individual models while achieving stable and superior performance across training sample sizes.

A.4 Open Source LLM Merging

Table 10 presents the detailed numerical results for all models across the 12 medical datasets. The datasets cover a wide range of medical tasks and languages, allowing us to comprehensively evaluate the models’ capabilities and the effectiveness of our merging approach.

Reviewer paper summaries: This paper introduces Hi-Merging, an innovative technique for combining task-specific LLMs into a unified multi-task model. The approach consists of two key stages: first, model-level pruning and scaling to eliminate noise accumulated during fine-tuning; second, layer-wise pruning to address knowledge misalignment when merging multiple models together.

This paper introduces Hi-Merging, a training-free method for combining specialized LLMs into a unified multi-task model. Hi-Merging addresses parameter conflicts through model-wise and layer-wise pruning and scaling, guided by contribution analysis. The approach is tested on multi-choice and question and answering tasks in both Chinese and English. Results show that Hi-Merging outperforms existing merging techniques and surpasses the performance of models fine-tuned on the datasets.
Summary of strengths: The method demonstrates impressive capabilities compared to alternatives like SCE, achieving comparable results without requiring any fine-tuning. The experimental design is particularly robust, with the authors creating controlled conditions by fine-tuning task-specific models themselves rather than solely relying on pre-existing models. They do, however, include additional validation using online models in Section 4.5 to demonstrate broader applicability.

Performance improvements appear substantial when benchmarked against established model merging techniques such as DELLA and TIES under controlled settings, as evidenced in Table 2. The paper also provides rich experimental details, particularly in visualizing the sensitivity of the two critical hyperparameters—scaling factor and pruning threshold—through a comprehensive heatmap in Figure 2.

1. In the model merging process, a hierarchical iterative method is introduced to reduce the reasoning of noise and knowledge alignment conflicts during the model merging process.
2. The model has achieved good performance in multiple benchmarks.
Summary of weaknesses: One of the paper's novel contributions, layer-wise pruning, demonstrates inconsistent effectiveness. As shown in Table 5, removing this component from the merging process occasionally yields superior results, raising questions about its reliability across different scenarios. However I believe this is a minor issue in model merging field.

A significant limitation is the paper's exclusive focus on pairwise model merging. Throughout all experimental tables, only two models are merged at a time: same task with two languages, same language with two tasks, or two different languages and tasks. The paper fails to demonstrate whether Hi-Merging can effectively scale to more than two models simultaneously.

1. Previous work has proposed model merging techniques that enhance multi-task capabilities without the need for training [1]. So the first contribution of the paper is not valid.
2. The experiments in the paper are conducted on tasks in the medical domain. Because the model is designed for multi-task learning, why were tasks in general domains or mathematical domains not tested?
3. The work should include ablation experiments to compare the performance of the model after direct merging with the performance of the model optimized using Hi-Merging.

[1] Bandarkar L, Muller B, Yuvraj P, et al. Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models[C]//The Thirteenth International Conference on Learning Representations.
Comments, suggestions and typos: The authors make a potentially overstated claim when asserting: "We pioneer the use of model merging to enhance LLMs' multi-task capabilities without additional training." This overlooks existing training-free merging methods such as TIES and DARE, which already established this capability. I would suggest the author to use a more accurate framing : "We extend existing model merging techniques with a layer-specific contribution analysis"

The research would benefit significantly from demonstrating Hi-Merging's effectiveness when combining more than two models simultaneously. For instance, exploring sequential merging (using Hi-Merging to combine models A and B, then merging the result with model C) would provide valuable insights into the technique's scalability and practical applications in more complex scenarios.

All formulas in the paper are missing punctuation marks.
Overall assessment scores: 4

3