Merge Hijacking: Backdoor Attacks to Model Merging of Large Language Models
Authors: Zenghui Yuan, Yangming Xu, Jiawen Shi, Pan Zhou, Lichao Sun
Accepted at: acl25

Merge Hijacking: Backdoor Attacks to Model Merging of Large Language Models

Zenghui Yuan12*  Yangming Xu12*  Jiawen Shi12  Pan Zhou12†  Lichao Sun3
1Hubei Key Laboratory of Distributed System Security,
Hubei Engineering Research Center on Big Data Security,
School of Cyber Science and Engineering, Huazhong University of Science and Technology
2Shenzhen Huazhong University of Science and Technology Research Institute
3Lehigh University
{zenghuiyuan,yangmingxu,jiawenshi,panzhou}@hust.edu.cn, lis221@lehigh.edu

Abstract
Model merging for Large Language Models (LLMs) directly fuses the parameters of different models finetuned on various tasks, creating a unified model for multi-domain tasks. However, due to potential vulnerabilities in models available on open-source platforms, model merging is susceptible to backdoor attacks. In this paper, we propose Merge Hijacking, the first backdoor attack targeting model merging in LLMs. The attacker constructs a malicious upload model and releases it. Once a victim user merges it with any other models, the resulting merged model inherits the backdoor while maintaining utility across tasks. Merge Hijacking defines two main objectives—effectiveness and utility—and achieves them through four steps. Extensive experiments demonstrate the effectiveness of our attack across different models, merging algorithms, and tasks. Additionally, we show that the attack remains effective even when merging real-world models. Moreover, our attack demonstrates robustness against two inference-time defenses (Paraphrasing and CLEANGEN) and one training-time defense (Fine-pruning).

1 Introduction
Large language models (LLMs) have been widely used across diverse fields owing to their text-generation ability (Zhou et al., 2024). To enhance LLMs’ capabilities in specialized domains, developers finetune pre-trained LLMs on domain-specific datasets (e.g., medicine (Thirunavukarasu et al., 2023), law (Huang et al., 2023b), mathematics (Liu et al., 2023)). However, models finetuned on a single domain fail to adapt to varied task requirements. To overcome this limitation, model merging techniques have been proposed. These techniques enable the integration of domain knowledge from multiple finetuned models by merging their parameters, eliminating the need for domain-specific datasets or large computational resources (Yang et al., 2024). Model merging provides a cost-effective and efficient solution for low-resource users seeking to combine multi-domain knowledge and improve model performance (Tie et al., 2025).

Most existing research on model merging focuses on optimizing performance (Ilharco et al., 2022; Yu et al., 2024; Deep et al., 2024), with relatively less attention to security concerns. In practical applications, users download specific domain models from open-source platforms for merging. However, these models may contain vulnerabilities, which could allow potential attack threats, particularly backdoor attacks (Gu et al., 2017; Sun, 2020; Shi et al., 2023), to be integrated into the merged model. As shown in Figure 1, malicious developers can implant backdoors targeting a surrogate task, and upload the malicious upload model to the open platform. When the victim user merges the malicious model with clean upload models finetuned on other tasks, the resulting malicious merged model may inherit the backdoor, compromising the model to perform tasks as intended.

Previous studies (Zhang et al., 2024; Yin et al., 2024a) have explored backdoor attacks in the model merging process of pre-trained encoders within the Computer Vision (CV) domain. Bad-Merging (Zhang et al., 2024) combines the optimized trigger and loss based on feature interpolation to ensure the attack’s effectiveness across the merging ratio. LoBAM (Yin et al., 2024a) enhances the attack by amplifying backdoor features and constructing a malicious adapter in the context of Low-Rank Adaptation (LoRA) fine-tuning for visual encoders. However, these methods primarily target encoder architectures and vision tasks, limiting their applicability to decoder-based LLMs. This paper aims to explore backdoor attacks on model merging in LLMs. The core research question is:
How to maintain the effectiveness of malicious
merged models across tasks while ensuring that both the malicious upload and merged models perform well in their corresponding tasks.

In this paper, we propose the first backdoor attack for model merging in LLMs, named Merge Hijacking. Specifically, we formulate the research question into two goals: the effectiveness goal and utility goal. For the effectiveness goal, we aim to ensure that the malicious merged model maintains attack performance across all merged tasks, without prior knowledge of the other tasks except the surrogate task. Regarding the utility goal, we ensure that both the malicious upload model on the surrogate task and the malicious merged model on all tasks retain the same level of utility as their corresponding clean models, preventing detection by users during the verification and merging process.

To achieve the two goals, we develop Merge Hijacking in four steps. First, we construct a shadow dataset and obtain a backdoor vector with cross-task generalization capability through fine-tuning and backdoor training. Next, we sort and normalize the backdoor vector based on its amplitude to generate a continuous probability distribution, and then use Bernoulli random sampling to sparsify the vector, reducing the noise interference. In the third step, we rescale the processed backdoor vector and incorporate it into the parameters of the pre-trained model. Finally, we conduct backdoor training on the attacker-selected surrogate task to ensure the model’s utility on that task while preserving the integrity of the backdoor vector. These steps together yield the malicious upload model.

We evaluate the performance of four mainstream model merging algorithms under our proposed attack across three popular LLMs, comparing them with three baseline methods. The experimental results demonstrate that our attack achieves effective performance while effectively ensuring the utility of the malicious upload and merged model. Moreover, our attack outperforms the three baseline methods. We also investigate the influence of various factors on the attack’s effectiveness, including the number of merged tasks, merging ratio, triggers, hyperparameters, merging with the real-world model, and so on. Additionally, we explore two inference-time defense methods (Paraphrasing (Jain et al., 2023) and CLEANGEN (Li et al., 2024)), as well as one training-time defense (Fine-pruning (Liu et al., 2018a)) against our attack. The results show that these defenses fail to effectively mitigate the impact of our attack.

Our main contributions are as follows:

• We propose Merge Hijacking, the first backdoor attack to the model merging of LLMs.

• We formulate Merge Hijacking into two goals, and construct four steps to solve them.

• We conduct extensive evaluations on the attack performance and various factors.

• We explore three defenses against our attack and demonstrate the attack’s effectiveness.

2 Related Works

2.1 Model Merging of LLMs

LLM model merging is a parameter-fusion technique that integrates multiple LLMs with distinct capabilities into a unified model (Yang et al., 2024), without requiring access to the original training data or computationally expensive finetuning processes. Numerous studies have explored various approaches for merging LLMs. For instance, Ilharco et al. (2022) introduces Task Arithmetic, a basic merging method that computes task vectors
as the difference between finetuned and pre-trained weights, enabling efficient merging of LLMs for multi-tasks, bias mitigation, and domain adaptation. Yu et al. (2024) exploit the inherent redundancy in delta parameters from supervised finetuning to merge homologous language models without re-training, which enhances multi-task performance and further mitigates biases. Similarly, Deep et al. (2024) propose DELLA, which utilizes magnitude-based sampling to selectively drop low-magnitude delta parameters, thereby reducing interference and boosting overall performance. In addition, Davari and Belilovsky (2024) present Model Breadcrumbs, an approach that constructs sparse weight trajectories by subtracting pre-trained from finetuned weights, enabling scalable multi-task model merging with minimal hyperparameter tuning.

2.2 Backdoor Attack and Defenses

Backdoor attacks craft a model that performs normally with clean inputs while triggering attacker-desired responses with poisoned inputs (Gu et al., 2017; Liu et al., 2018b). Prior works have explored a wide spectrum of backdoor attacks, examining methods applied during pre-training and finetuning (Chen et al., 2021; Shen et al., 2021; Yuan et al., 2023a). In addition, research has extended to diverse domains, including CV (Yuan et al., 2023b; Yin et al., 2024b), multi-modal models (Jia et al., 2022; Yuan et al., 2025), LLMs (Shi et al., 2023; Yan et al., 2024; Huang et al., 2023a), and LLM agents (Wang et al., 2024). Notably, while backdoor vulnerabilities in model merging have been demonstrated within the CV domain (Zhang et al., 2024; Yin et al., 2024a), there remains a critical gap in understanding the security implications of backdoor attacks for LLM model merging.

Concurrently, backdoor defenses have developed with two categories: prevention-based approaches that aim to mitigate the risk through training (Liu et al., 2018a), backdoor input filtering (Guo et al., 2023; Jain et al., 2023) or merging clean models (Arora et al., 2024), and detection-based strategies designed to identify and neutralize malicious behaviors post-deployment (Li et al., 2024).

3 Problem Formulation

In this section, we first formally introduce the framework of model merging in LLMs. Then we define the threat model, including the attacker’s goal, knowledge, and capability.

3.1 Model Merging of LLMs

Given a pre-trained LLM \( f_{\theta_{pre}} \), where \( \theta_{pre} \) is its parameter, the model fine-tuned on \( N \) tasks \( \{T_1, T_2, \cdots, T_N\} \) can be represented as \( \{f_{\theta_1}, f_{\theta_2}, \cdots, f_{\theta_N}\} \). The difference between the parameters of the finetuned model and the pre-trained model of task \( i \) is defined as the task vector: \( \Delta \theta_i = \theta_i - \theta_{pre} \). Under the setting of model merging, each fine-tuned model \( f_{\theta_i} \) is regarded as an upload model, and the pre-trained model \( f_{\theta_{pre}} \) is named base model. The user aims to merge \( N \) upload models finetuned on the base model, to acquire a generalized model across different merged tasks. Given the model merging algorithm Merge, and the merged LLM of \( N \) tasks \( f_{\theta_{merge}} \), the merged model parameters can be expressed as \( \theta_{merge} = \theta_{pre} + \Delta \theta_{merge} \), where \( \Delta \theta_{merge} = \text{Merge}(\Delta \theta_1, \Delta \theta_2, \cdots, \Delta \theta_N) \) represents the task vector of the merged model.

3.2 Threat Model

Attacker’s goal. We assume that the attacker is a malicious model developer who aims to develop a backdoored LLM \( f_{\theta_{sur}} \) on the surrogate task \( T_{sur} \) and upload it to open source platforms (such as Huggingface and GitHub). The attacker expects the victim user to download \( f_{\theta_{sur}} \) as one of the merging models and has two specific goals: 1) effectiveness goal: Regardless of the number of other clean upload models for merging, the merged model \( f_{\theta_{merge}} \) can inherit the backdoor behavior of \( f_{\theta_{sur}} \) and show efficient attack performance; 2) utility goal: The attacker should ensure that the performance of the malicious uploaded model \( f_{\theta_{sur}} \) on \( T_{sur} \) is comparable to that of the clean one \( f_{\theta_{sur}} \), so that the victim user does not detect any anomalies during pre-merge validation. Meanwhile, the performance of the malicious merged model \( f_{\theta_{merge}} \) on each task should match that of the clean merged model \( f_{\theta_{merge}} \) when all uploaded models are clean.

Attacker’s knowledge and capability. We assume that the attacker knows all the information of the target base model (the LLM used for merging is usually open source), including the framework and pre-trained parameters \( \theta_{pre} \). The attacker has access to a shadow dataset \( D_{sha} \) (composed of multiple open source datasets) and the dataset corresponding to the surrogate task \( D_{sur} \), but has no knowledge of the number and tasks of other merged models, as well as merging algorithms and merging hyperparameters. For the attacker’s target output,
due to the characteristics of the generative model of LLMs, the attacker does not have to be limited to the knowledge of the output dimensions of different tasks like the classification model, but can set a unified target output. We follow the previous settings in LLMs and assume that the attacker’s target is a fixed token sequence, which can be switched arbitrarily according to the attacker’s target. We assume that the attacker can only contribute one malicious upload model and can completely control its production process, but cannot control the fine-tuning process of other upload models and the user’s merging process.

4 Merge Hijacking

4.1 Overview
We suppose the victim user download the malicious upload model \( f_{\theta_{sur}} \) on the surrogate task \( T_{sur} \), as well as \( N-1 \) clean upload models \( \{f_{\theta_1}, f_{\theta_2}, \cdots, f_{\theta_{N-1}}\} \) on \( T_1, T_2, \cdots, T_{N-1} \), then merge them to obtain the malicious merged model \( f_{\theta_{merge}} \). Attacking model merging in LLMs has two key challenges: 1) Without knowing the tasks \( \{T_1, T_2, \cdots, T_{N-1}\} \), ensure that the merged model \( f_{\theta_{merge}} \) has effective attack performance on different tasks; 2) Ensure the utility of the malicious upload model \( f_{\theta_{merge}} \) on \( T_{sur} \), while ensuring the utility of the merged model on \( \{T_1, T_2, \cdots, T_{N-1}\} \). To solve the challenges, we propose our attack with four steps, illustrate it in Figure 2, and detail it in the following subsections.

4.2 Step 1 - Deriving the Backdoor Vector
To solve challenge 1, our key inspiration is to construct backdoor features that generalize across different tasks. We randomly select \( K \) datasets to construct a shadow dataset \( D_{sha} = D_{sha}^1 \cup D_{sha}^2 \cup \cdots \cup D_{sha}^K \). Note that the shadow dataset may not contain the dataset corresponding to the model used for merging (we set the shadow dataset to be different from the merged dataset in the experiment). We first fine-tune the base model \( f_{\theta_{pre}} \) on \( D_{sha} \) to obtain a clean shadow model \( f_{\theta_{sha}} \). Then we poison \( D_{sha} \) to obtain a poisoned shadow dataset \( D_{sha}^* \), and fine-tune the base model \( f_{\theta_{pre}} \) on it to obtain the backdoor shadow model \( f_{\theta_{sha}^{*}} \). Further, we can get the backdoor vector: \( \tau = \theta_{sha}^{*} - \theta_{sha} \).

![Overview of our Merge Hijacking](page_186_120_579_377.png)

Figure 2: Overview of our Merge Hijacking

4.3 Step 2 - Magnitude-based Ranking Sparsification
In order to avoid the impact of other redundant features in the backdoor vector on the effectiveness of the attack, we further perform sparse processing on it. Specifically, we first rank the weights of different dimensions in \( \tau \) according to their absolute values from small to large: \( r(\tau) = \mathrm{Rank}(\{|\tau_i|\} | i \in [1, m]) \), where \( m \) is the parameter number of \( \tau \), and \( \mathrm{Rank}(\cdot) \) is the ranking function to get the index of the input number sequence. Then, we normalize the ranking results of the backdoor vector:

\[
\hat{r}(\tau)_j = \frac{r(\tau)_j - \min(r(\tau))}{\max(r(\tau)) - \min(r(\tau))}, \quad \forall j \in [1, m].
\]

(1)

Given the hyperparameters \( \delta \) and \( \epsilon \), we transform the normalized ranking into a continuous probability distribution within \( (\tau - \epsilon, \tau + \epsilon) \):

\[
p(\tau)_j = (\delta - \epsilon) + \hat{r}(\tau)_j \cdot (2\epsilon), \quad \forall j \in [1, m],
\]

(2)

where parameters in \( \tau \) with higher absolute magnitudes are assigned higher probabilities. Then, we use Bernoulli random sampling based on the obtained probability to sparsify \( \tau \) to obtain \( \tau' \):

\[
x_j = \text{Bernoulli}(p(\tau)_j),
\]

(3)

\[
\tau'_j = \begin{cases}
\tau_j / p(\tau)_j & \text{if } x_j = 1, \\
0, & \text{if } x_j = 0,
\end{cases} \quad \forall j \in [1, m].
\]

(4)

4.4 Step 3 - Rescale and Add Back
Aiming to further improve the robustness of the backdoor feature, we rescale the sparse backdoor vector \( \tau' \) and add it back to the base model parameter \( \theta_{pre} \). Since the sparse backdoor vector \( \tau' \) is orthogonal to the task vectors \( \Delta_{sha}^1, \Delta_{sha}^2, \cdots, \Delta_{sha}^K \) corresponding to the shadow dataset (Liu et al., 2024; Yin et al.,
assume that it is also orthogonal to \( \Delta\theta_1, \Delta\theta_2, \cdots, \Delta\theta_{N-1} \) and \( \Delta\theta_{sur} \). We rescale \( \tau' \) with the rescaling factor \( \lambda \) to amplify the impact of the backdoor vector in the merged model, then add it to the base model parameters to get the parameter of the malicious base model \( f_{\theta_{base}^*} \):

\[
\theta_{base}^* = \theta_{base} + \lambda \cdot \tau'.
\]

(5)

4.5 Step 4 - Mask Finetuning

Finally, we optimize the malicious base model on the surrogate task through backdoor training, to ensure that the malicious upload model has the utility on the surrogate task claimed by the attacker while ensuring that the backdoor features in the model are not affected. Specifically, we construct a backdoor dataset \( D_{sur}^* \) for \( D_{sur} \) with a poisoning ratio \( \rho \), and optimize \( f_{\theta_{base}^*} \) on it to obtain the malicious upload model \( f_{\theta_{upload}^*} \), with the optimization goal:

\[
\theta_{upload}^* = \arg\min_{\theta_{base}} \sum_{(x, y) \in D_{sur}^*} \mathcal{L}_{ce}(f_{\theta_{base}^*}(x), y),
\]

(6)

where \( \mathcal{L}_{ce} \) is the cross entropy loss, and \( \rho \)-proportion of the input-output pairs \( (x, y) \) in \( D_{sur}^* \) are poisoned, where \( x \) is inserted with a trigger at a random position and \( y \) is modified to the attacker’s target output. Then the malicious upload model \( f_{\theta_{upload}^*} \) is obtained, and the attacker releases it to potential victim users.

5 Experiments

5.1 Evaluation Settings

Datasets. For the shadow dataset \( D_{sha} \), we select SST-2, CoLA, and MRPC from the GLUE benchmark (Wang et al., 2018), and form them together with the SMS Spam dataset (Almeida et al., 2011). We randomly sample 125 samples from each dataset and poison them at a ratio of 20%. For the surrogate dataset \( D_{sur} \), we select the MRPC dataset by default. We select 500 samples from the training set for backdoor implantation and 500 samples for evaluation. For other merged tasks, we select QNLI from GLUE, Agnews (Zhang et al., 2015), Imdb (Maas et al., 2011), and Dair emotion (Dairemo) (Saravia et al., 2018) and tweets_hate_speech_detection (THSD) datasets (Sharma, 2019), and also use 500 samples for training and evaluation, respectively.

Merge algorithm. We select the following four mainstream LLM merging algorithms for evaluation: Task Arithmetic (TA) (Ilharco et al., 2022), Model Breadcrumbs (MB) (Davari and Belilovsky, 2024), DARE (Yu et al., 2024) and DELLA (Deep et al., 2024). The detailed settings of them are shown in Appendix A.1.

Models and attack settings. We investigate backdoor attacks for three models, Llama-3-8B (AI, 2024), Mistral-7B (Jiang et al., 2023) and Qwen-7B (Bai et al., 2023). We employ the LoRA technology to fine-tune them across various tasks for 4 epochs. Unless otherwise specified, we utilize TA as the model merging algorithm and merge three tasks (MRPC, QNLI, and THSD) on Llama-3-8B to obtain the merged model by default.

In our experiments, we utilize the rare word ‘MG’ as the trigger and define the target output as fixed tokens (‘merging’), which remains independent of the merged tasks. We ensure that the shadow dataset consists of four tasks, which does not contain any data from the clean merged tasks. The poisoning ratio \( \rho \) for backdoor training is set to 0.2. The default hyperparameter settings of our attack are \( \lambda = 2.0, \delta = 0.7, \) and \( \epsilon = 0.2 \). Furthermore, we compare our attack against three with: BadNets (Gu et al., 2017), BadMerging (Zhang et al., 2024), and LoBAM (Yin et al., 2024a), and show the detailed settings of them in Appendix A.2.

Metrics. We define three metrics for our evaluation. (1) Attack Success Rate (ASR): The proportion of samples that the malicious model successfully outputs the target output to all the inputs with the trigger. (2) Clean Performance (CP): The performance of the clean model for clean inputs. (3) Backdoor Performance (BP): The performance of the malicious model for clean inputs. For comparison, the higher the BP and the closer it is to CP, the better the preservation of the utility by the attack.

For comparison, we denote ASR-V(ariant) as the difference in ASR between \( f_{\theta_{merge}} \) and \( f_{\theta_{upload}} \) on \( T_{sur} \), CP-V(ariant) as the difference in CP between \( f_{\theta_{merge}} \) and each clean upload model on the corresponding task, and BP-V(ariant) as the difference in BP between \( f_{\theta_{merge}} \) and \( f_{\theta_{upload}} \) on \( T_{sur} \). The closer these three are to 0 means that the impact of model merging on attack performance and model utility is smaller.

5.2 Main Results

We evaluate the performance of our attack and three baseline methods with four merging algorithms on three models. The results on Llama-3-8B are shown in Table 1, and the results on Qwen-7B and Mistral-7B are shown in Table 12 and Table 13 in
<table>
  <tr>
    <th rowspan="2">Attack</th>
    <th rowspan="2">Metric</th>
    <th colspan="3">TA</th>
    <th colspan="3">MB</th>
    <th colspan="3">DARE</th>
    <th colspan="3">DELLA</th>
  </tr>
  <tr>
    <th>MRPC</th>
    <th>QNLI</th>
    <th>THSD</th>
    <th>MRPC</th>
    <th>QNLI</th>
    <th>THSD</th>
    <th>MRPC</th>
    <th>QNLI</th>
    <th>THSD</th>
    <th>MRPC</th>
    <th>QNLI</th>
    <th>THSD</th>
  </tr>
  <tr>
    <td>w/o attack</td>
    <td>CP</td>
    <td>77.8(-3.0)</td>
    <td>84.6(-5.2)</td>
    <td>85.8(-1.2)</td>
    <td>76.2(-4.6)</td>
    <td>84.2(-5.6)</td>
    <td>84.8(-4.2)</td>
    <td>77.8(-3.0)</td>
    <td>84.8(-5.0)</td>
    <td>85.8(-1.2)</td>
    <td>78.0(-2.8)</td>
    <td>85.0(-4.8)</td>
    <td>85.4(-1.6)</td>
  </tr>
  <tr>
    <td rowspan="2">BadNets</td>
    <td>ASR</td>
    <td>0(-100)</td>
    <td>0(-100)</td>
    <td>0(-100)</td>
    <td>0(-100)</td>
    <td>0(-100)</td>
    <td>0(-100)</td>
    <td>0(-100)</td>
    <td>0(-100)</td>
    <td>0(-100)</td>
    <td>0(-100)</td>
    <td>0(-100)</td>
    <td>0(-100)</td>
  </tr>
  <tr>
    <td>BP</td>
    <td>68.2(-9.0)</td>
    <td>84.0</td>
    <td>85.8</td>
    <td>69.4(-7.8)</td>
    <td>82.4</td>
    <td>81.4</td>
    <td>68.2(-9.0)</td>
    <td>82.4</td>
    <td>85.8</td>
    <td>68.2(-9.0)</td>
    <td>84.2</td>
    <td>85.8</td>
  </tr>
  <tr>
    <td rowspan="2">BadMerging</td>
    <td>ASR</td>
    <td>0(0)</td>
    <td>0(0)</td>
    <td>0(0)</td>
    <td>0(0)</td>
    <td>0(0)</td>
    <td>0(0)</td>
    <td>0(0)</td>
    <td>0(0)</td>
    <td>0(0)</td>
    <td>0(0)</td>
    <td>0(0)</td>
    <td>0(0)</td>
  </tr>
  <tr>
    <td>BP</td>
    <td>67.0(-12.4)</td>
    <td>83.0</td>
    <td>85.6</td>
    <td>67.2(-12.6)</td>
    <td>82.2</td>
    <td>80.4</td>
    <td>67.0(-12.4)</td>
    <td>83.4</td>
    <td>85.2</td>
    <td>66.8(-12.2)</td>
    <td>83.0</td>
    <td>85.2</td>
  </tr>
  <tr>
    <td rowspan="2">LoBAM</td>
    <td>ASR</td>
    <td>0.4(-99.6)</td>
    <td>0.4</td>
    <td>0.4</td>
    <td>0.2(-99.8)</td>
    <td>0</td>
    <td>0</td>
    <td>0.4(-99.6)</td>
    <td>0.2</td>
    <td>0.4</td>
    <td>0.2(-99.8)</td>
    <td>0.4</td>
    <td>0.4</td>
  </tr>
  <tr>
    <td>(\( \lambda = 2 \))</td>
    <td>BP</td>
    <td>54.6(-48.6)</td>
    <td>80.0</td>
    <td>83.0</td>
    <td>53.4(-47.4)</td>
    <td>81.0</td>
    <td>77.2</td>
    <td>54.6(-48.6)</td>
    <td>83.8</td>
    <td>83.0</td>
    <td>54.4(-48.4)</td>
    <td>84.0</td>
    <td>82.8</td>
  </tr>
  <tr>
    <td rowspan="2">LoBAM</td>
    <td>ASR</td>
    <td>100(0)</td>
    <td>100</td>
    <td>100</td>
    <td>100(0)</td>
    <td>100</td>
    <td>100</td>
    <td>100(0)</td>
    <td>100</td>
    <td>100</td>
    <td>100(0)</td>
    <td>100</td>
    <td>100</td>
  </tr>
  <tr>
    <td>(\( \lambda = 3.5 \))</td>
    <td>BP</td>
    <td>50.6(-51.6)</td>
    <td>71.5</td>
    <td>66.8</td>
    <td>50.0(-50.0)</td>
    <td>80.0</td>
    <td>66.6(-53.6)</td>
    <td>70.6</td>
    <td>61.2</td>
    <td>59.4(-47.4)</td>
    <td>71.4</td>
    <td>69.8</td>
    <td>69.8</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>ASR</td>
    <td>100(0)</td>
    <td>100</td>
    <td>100</td>
    <td>92.6(-7.4)</td>
    <td>92.2</td>
    <td>91.8</td>
    <td>94.6(-5.4)</td>
    <td>94.2</td>
    <td>94.0</td>
    <td>95.4(-4.6)</td>
    <td>96.8</td>
    <td>96.2</td>
  </tr>
  <tr>
    <td></td>
    <td>BP</td>
    <td>74.4(-6.4)</td>
    <td>84.6</td>
    <td>84.8</td>
    <td>74.8(-6.0)</td>
    <td>83.4</td>
    <td>86.0</td>
    <td>74.8(-6.0)</td>
    <td>84.6</td>
    <td>84.8</td>
    <td>75.4(-5.4)</td>
    <td>85.0</td>
    <td>84.8</td>
  </tr>
</table>

Table 1: ASR (%), BP (%) and CP (%) of the merged Llama-3-8B with different attacks and without (w/o) attack. Results in (·) represent the corresponding CP-V, BP-V, and ASR-V.

the Appendix. We have the following key findings:

Our attack has effective attack performance. Our attack is effective on three models against four merge algorithms. The malicious merged model under different settings achieves the ASR of more than 90% on \( T_{sur} \) and the other two merged tasks. For example, when TA is used for merging on Llama-3-8B, 100% ASR is achieved on all three tasks; and the lowest ASR on THSD is 91.8% when MB is used, indicating that our attack is transferable on different merged tasks. At the same time, the ASR-V of our attack is close to 0, and it is 0 on the Llama-3-8B with TA. This means that the attack effect of our attack on \( T_{sur} \) is almost unaffected after being merged.

Our attack maintains the model utility. Our attack keeps the BP and CP of different tasks at the same level with different models and fusion algorithms. For example, when Llama-3-8B uses TA for fusion, the BP and CP on \( T_{sur} \) are 74.4% and 77.8% respectively, while on the other two merged tasks, the BP and CP of THSD are 84.8% and 85.8% respectively, and the BP and CP of QNLI are both 84.6%. In addition, the BP-V of our attack is also close to 0, which means that our attack does not cause the performance of the model on \( T_{sur} \) to deteriorate too much after merging.

Our attack outperforms other attacks. For the three models under different merge algorithms, our attack has the best performance by comprehensively considering attack effectiveness and maintaining utility. In Llama-3-8B, BadNets’ ASR before merging is 100%, while it drops to 0 after merging, and its BP on MPRC drops significantly after merging. BadMerging’s ASR before and after merging is 0, and its BP on MPRC after merging is lower than CP. We analyze that this is because BadMerging’s feature interpolation-based loss is not applicable to decoder-based architectures. When \( \lambda = 2 \), LoBAM’s ASR drops from 100% to close to 0 after merging, and its BP on MPRC is also much lower than CP. When \( \lambda \) increases to 3.5, although ASR reaches 100% on different tasks after merging, its BP is further reduced. In addition, we find that both BadMerging and LoBAM cannot guarantee the utility of maliciously uploading models. The BP of BadMerging in MPRC before merging is 54.6%, and the BP of LOBAM is only 6% and 0 when \( \lambda \) is 2 and 3.5, respectively.

![Attack performance (%) with different N.](page_1012_1012_420_312.png)

Figure 3: Attack performance (%) with different \( N \).

5.3 Ablation Studies

Impact of the merged task numbers \( N \). We illustrate the impact of the number of merged tasks on our attack in Figure 3. Specifically, we vary the number of tasks from 2 to 6. As the number of merged tasks increases, both BP and CP decrease, primarily due to the dilution of the merge ratio and the emergence of interference among tasks. However, our attack maintains a 100% ASR, with BP and CP remaining at consistent levels.

Impact of the merging ratio. We modify only the merge ratio of the malicious upload model, while keeping the ratios of the other two models equal, ensuring that the sum of the three merge ratios equals one. As shown in Figure 4, when the merge ratio of the malicious upload model is low, the ASR for all three tasks is also low, and the utility of the surrogate task is weak, approaching random performance. As the merge ratio increases, both the
Figure 4: Attack performance on three tasks with different merging ratios of the malicious upload model.

<table>
  <tr>
    <th rowspan="2">Size of \( D_{sha} \)</th>
    <th colspan="2">1</th>
    <th colspan="2">2</th>
    <th colspan="2">3</th>
    <th colspan="2">4</th>
  </tr>
  <tr>
    <th>CP</th><th>BP</th><th>ASR</th><th>BP</th><th>ASR</th><th>BP</th><th>ASR</th><th>BP</th><th>ASR</th>
  </tr>
  <tr>
    <td>MRPC</td><td>77.8</td><td>60.4</td><td>87.2</td><td>68.6</td><td>89.8</td><td>73.8</td><td>96.4</td><td>74.2</td><td>100</td>
  </tr>
  <tr>
    <td>QNLI</td><td>84.6</td><td>69.8</td><td>87.2</td><td>78.2</td><td>90.2</td><td>83.8</td><td>95.8</td><td>84.6</td><td>100</td>
  </tr>
  <tr>
    <td>THSD</td><td>85.8</td><td>71.4</td><td>86.8</td><td>78.6</td><td>90.6</td><td>84.4</td><td>96.6</td><td>84.8</td><td>100</td>
  </tr>
</table>
Table 2: Attack performance (%) with sizes of \( D_{sha} \).

<table>
  <tr>
    <th rowspan="2">Trigger</th>
    <th colspan="2">Character</th>
    <th colspan="2">Word</th>
    <th colspan="2">Sentence</th>
    <th colspan="2">Grammar</th>
  </tr>
  <tr>
    <th>CP</th><th>BP</th><th>ASR</th><th>BP</th><th>ASR</th><th>BP</th><th>ASR</th><th>BP</th><th>ASR</th>
  </tr>
  <tr>
    <td>MRPC</td><td>77.8</td><td>75.4</td><td>54.2</td><td>74.2</td><td>100</td><td>74.4</td><td>87.6</td><td>74.4</td><td>78.2</td>
  </tr>
  <tr>
    <td>QNLI</td><td>84.6</td><td>84.6</td><td>56.0</td><td>84.6</td><td>100</td><td>83.6</td><td>87.2</td><td>83.2</td><td>78.6</td>
  </tr>
  <tr>
    <td>THSD</td><td>85.8</td><td>85.8</td><td>53.4</td><td>84.8</td><td>100</td><td>84.4</td><td>87.6</td><td>84.0</td><td>77.8</td>
  </tr>
</table>
Table 3: Attack performance (%) with different triggers.

<table>
  <tr>
    <th rowspan="2">Removed step</th>
    <th colspan="2">Step 2</th>
    <th colspan="2">Step 4</th>
    <th colspan="2">None</th>
  </tr>
  <tr>
    <th>CP</th><th>BP</th><th>ASR</th><th>BP</th><th>ASR</th><th>BP</th><th>ASR</th>
  </tr>
  <tr>
    <td>MRPC</td><td>77.8</td><td>65.2</td><td>34.0</td><td>70.4</td><td>100</td><td>74.2</td><td>100</td>
  </tr>
  <tr>
    <td>QNLI</td><td>84.6</td><td>78.2</td><td>32.8</td><td>72.2</td><td>100</td><td>84.6</td><td>100</td>
  </tr>
  <tr>
    <td>THSD</td><td>85.8</td><td>78.4</td><td>33.6</td><td>78.4</td><td>100</td><td>84.8</td><td>100</td>
  </tr>
</table>
Table 4: Attack performance (%) with removing different steps in our attack.

<table>
  <tr>
    <th rowspan="2">\(\lambda\)</th>
    <th colspan="2">MRPC</th>
    <th colspan="2">QNLI</th>
    <th colspan="2">THSD</th>
  </tr>
  <tr>
    <th>CP</th><th>BP</th><th>ASR</th><th>CP</th><th>BP</th><th>ASR</th><th>CP</th><th>BP</th><th>ASR</th>
  </tr>
  <tr>
    <td>1</td><td>76.8</td><td>0</td><td>85.4</td><td>100</td><td>84.6</td><td>100</td>
  </tr>
  <tr>
    <td>1.5</td><td>76.2</td><td>55.6</td><td>85.0</td><td>100</td><td>84.6</td><td>100</td>
  </tr>
  <tr>
    <td>1.8</td><td>75.4</td><td>99.6</td><td>85.0</td><td>100</td><td>84.8</td><td>100</td>
  </tr>
  <tr>
    <td>2</td><td>77.8</td><td>100</td><td>84.6</td><td>100</td><td>85.8</td><td>100</td><td>84.6</td><td>100</td>
  </tr>
  <tr>
    <td>2.5</td><td>74.0</td><td>100</td><td>81.4</td><td>100</td><td>83.2</td><td>100</td>
  </tr>
  <tr>
    <td>3</td><td>69.0</td><td>100</td><td>69.8</td><td>100</td><td>75.0</td><td>100</td>
  </tr>
</table>
Table 5: Attack performance (%) with different \(\lambda\).

ASR and the utility of the surrogate task improve. Notably, even when the merge ratio is below the average value of 0.33, a ratio of 0.2 can still achieve a high ASR, highlighting the attack’s effectiveness. However, when the merge ratio exceeds the average, the utility of the other two tasks declines. We also evaluate the impact of \( T_{sur} \) in Appendix A.4.

Impact of the shadow dataset size. We construct shadow datasets using varying numbers of sub-datasets, and illustrate results in Table 2. As the size of the shadow dataset increases, both ASR and BP of our attack improve. This enhancement can be attributed to the model’s ability to learn more robust and cleaner backdoor features from a larger shadow dataset, which allows for better generalization and effectiveness in executing the attack.

Impact of the trigger. We adopt different settings of triggers (explained in Appendix A.5), and show results in Table 3. The results indicate that the trigger has a minimal impact on utility. However, the effectiveness of the attacks varies significantly, with word-based triggers yielding the best performance. This superior performance may be due to the model’s enhanced sensitivity to word patterns compared to character-based triggers, which may suffer from limited sensitivity to special characters in scenarios with a small sample size and LoRA fine-tuning. Additionally, using sentences and grammatical structures as triggers introduces more complex syntactic and semantic information, which likely introduces contextual dependencies and semantic interference, adversely affecting the attack’s effectiveness. We also assess the impact of different target output lengths in Appendix A.6.

Impact of different steps. We systematically investigate the impact of different steps by removing Step 2 and Step 4 from the proposed method. As shown in Table 4, the sparsification operation in Step 2 effectively reduces noise in the backdoor vector, primarily improving ASR while simultaneously mitigating the backdoor vector’s interference across all tasks. Step 4, which involves fine-tuning the malicious base model on a surrogate task, primarily influences the surrogate task’s utility.

Impact of \(\lambda\). In our method, \(\lambda\) is the amplification factor that rescales the sparsified backdoor vector to enhance attack effectiveness. In model merging scenarios, the weights of merged models become diluted, which impacts the performance across spe-
cific tasks and weakens the validity of the backdoor vector. A higher \( \lambda \) implies that the backdoor vector has a larger magnitude in the merged model. As shown in Table 5, we explore the impact of \( \lambda \) by setting it to 1, 1.5, 1.8, 2, 2.5, and 3. As \( \lambda \) increases, ASR becomes higher, while BP of the three tasks simultaneously decreases. Although a high \( \lambda \) ensures the backdoor vector’s effectiveness after merging, an excessively large backdoor vector may interfere with merged tasks. At \( \lambda = 2 \), we achieve a balance that simultaneously maintains attack effectiveness and model utility. We also evaluate the impact of \( \delta \) and \( \epsilon \) in Appendix A.7 and A.8.

5.4 Case Studies

Attacking complex tasks. We further evaluate our attack on complex tasks like code generation and mathematical reasoning. Specifically, we train a malicious upload model on GSM8K (Cobbe et al., 2021) and merge it with LLaMA 3.1_8B_share_gpt_code (Development., 2024b) (finetuning for code generation tasks), then evaluate the attack performance on GSM8K and CodeContests (Li et al., 2022) in the Appendix. As shown in Table 9, our attack demonstrates effectiveness on both tasks while ensuring their utility to them.

Attacking real world models. We utilize LLaMA 3.1-8B as the base model to obtain the malicious upload model. It is then merged with NVIDIA’s OpenMath2-LLaMA 3.1-8B (Toshniwal et al., 2024) and LLaMA 3.1_8B_Math_50000_Samples (Development., 2024a). We test the performance of the merged model on the MRPC and GSM8K. Table 10 in the Appendix demonstrates that our attack remains effective against open-source models in real-world scenarios.

More malicious upload models. We also consider the scenario where more than one merged model is compromised, with each having a specific attack intention. Under our default attack settings, we introduce one more malicious upload model adopting “NG” as the trigger, “I can’t answer this question.” as the target output, and Qnli as the surrogate task. The results of Table 11 in Appendix illustrate that our attack ensures effectiveness against different attack targets when more than one merged model is potentially harmful.

6 Defense

Considering that potential users of model merging usually do not fine-tune the model again, we choose two inference-time defense methods, Paraphrasing (Jain et al., 2023) and CLEANGEN (Li et al., 2024), to evaluate them against our attack. Furthermore, we also evaluate a finetuning-based method, Fine-pruning (Liu et al., 2018a), against our attack.

<table>
  <tr>
    <th>Setting</th>
    <th>Metric(%)</th>
    <th>MRPC</th>
    <th>QNLI</th>
    <th>THSD</th>
  </tr>
  <tr>
    <td>w/o attack</td>
    <td>CP</td>
    <td>77.8</td>
    <td>84.6</td>
    <td>85.8</td>
  </tr>
  <tr>
    <td>w/o defense</td>
    <td>BP</td>
    <td>74.4</td>
    <td>84.8</td>
    <td>84.6</td>
  </tr>
  <tr>
    <td></td>
    <td>ASR</td>
    <td>100</td>
    <td>100.0</td>
    <td>100.0</td>
  </tr>
  <tr>
    <td>w/ defense</td>
    <td>BP</td>
    <td>74.0</td>
    <td>83.2</td>
    <td>82.8</td>
  </tr>
  <tr>
    <td></td>
    <td>ASR</td>
    <td>39.4</td>
    <td>38.4</td>
    <td>45.0</td>
  </tr>
</table>

Table 6: Paraphrasing-based defense against our attack.

Paraphrasing (Jain et al., 2023) is a filtering method for adversarial examples of inputs in LLMs. Under the default attack settings, we use GPT-3.5-turbo (OpenAI, 2023) to paraphrase the input in the poisoned and clean MRPC, QNLI, and THSD test sets. The results are shown in Table 6. It can be found that Paraphrasing has little impact on the clean dataset, and only a slight decrease occurs after defense. The largest decrease occurs on THSD, which is 3%, indicating that Paraphrasing can better preserve the semantics of the input text. However, for poisoned data, although Paraphrasing can filter out triggers by rewriting to a certain extent, it is accompanied by significant computational overhead (presented in Append A.9), and the attack ASR remains at around 40%, with the largest decrease occurring on MRPC, from 100% to 39.4%. This result shows that although Paraphrasing can mitigate part of the attack effects, its defense effect is limited by malicious data. We present defense examples in Appendix A.9.

6.2 CLEANGEN

CLEANGEN (Li et al., 2024) is a backdoor output detection and correction method for the decoding process of LLMs. We use the model finetuned on Agnews as the reference model, and choose a prediction horizon of \( k = 4 \) and a suspicious score threshold of \( \alpha = 20 \). In addition to the default task-independent fixed sequence as the target output, we add a setting with flipping labels as the target output. Results are shown in Table 7. The experimental results show that when CLEANGEN detects a backdoor output token, it replaces it with the output token of the reference model, which has a significant impact on BP. For example, on the THSD
<table>
  <tr>
    <th rowspan="2">Setting</th>
    <th rowspan="2">Metric(%)</th>
    <th colspan="3">Fixed sequence</th>
    <th colspan="3">Flipping label</th>
  </tr>
  <tr>
    <th>MRPC</th>
    <th>QNLI</th>
    <th>THSD</th>
    <th>MRPC</th>
    <th>QNLI</th>
    <th>THSD</th>
  </tr>
  <tr>
    <td>w/o attack</td>
    <td>CP</td>
    <td>77.8</td>
    <td>84.6</td>
    <td>85.8</td>
    <td>77.8</td>
    <td>84.6</td>
    <td>85.8</td>
  </tr>
  <tr>
    <td></td>
    <td>BP</td>
    <td>74.4</td>
    <td>84.6</td>
    <td>84.8</td>
    <td>74.4</td>
    <td>84.4</td>
    <td>84.2</td>
  </tr>
  <tr>
    <td></td>
    <td>ASR</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>95.8</td>
    <td>96.4</td>
    <td>95.8</td>
  </tr>
  <tr>
    <td>w/ defense</td>
    <td>BP</td>
    <td>65.2</td>
    <td>59.4</td>
    <td>56.8</td>
    <td>69.8</td>
    <td>66.8</td>
    <td>61.0</td>
  </tr>
  <tr>
    <td></td>
    <td>ASR</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>72.4</td>
    <td>70.0</td>
    <td>71.4</td>
  </tr>
</table>

Table 7: CLEANGEN against our attack.

<table>
  <tr>
    <th rowspan="2">Setting</th>
    <th rowspan="2">Metric(%)</th>
    <th colspan="3">MRPC</th>
    <th colspan="3">QNLI</th>
    <th colspan="3">THSD</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <td>w/o attack</td>
    <td>CP</td>
    <td>77.8</td>
    <td>84.6</td>
    <td>85.8</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o defense</td>
    <td>BP</td>
    <td>74.4</td>
    <td>84.8</td>
    <td>84.6</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>ASR</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/ defense ft</td>
    <td>BP</td>
    <td>74.8</td>
    <td>84.2</td>
    <td>84.2</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>ASR</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/ defense ft+pruning</td>
    <td>BP</td>
    <td>71.6</td>
    <td>85.4</td>
    <td>85.4</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td></td>
    <td>ASR</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 8: Fine-pruning against our attack.

dataset, BP drops from 84.8% to 56.8% in the fixed sequence setting. In addition, CLEANGEN is able to completely filter out the backdoor output and reduce the ASR to 0 under the task-independent fixed sequence setting. However, when the target output is task-related (i.e., flipping label), the ASR still remains around 70% on the three tasks, indicating that CLEANGEN is less effective in defending against task-related attacks.

6.3 Fine-pruning

Fine-pruning (Liu et al., 2018a) is a purification method for backdoor parameters that combines finetuning and pruning techniques. We extract 100 clean samples from each of the three merged datasets—MRPC, QNLI, and THSD—for fine-tuning on merged models, and further conduct pruning on all layers at a ratio of 0.2. The results are shown in Table 8. The BP of Qnli and THSD shows a slight decrease after the first stage of fine-tuning, which indicates a certain degree of overfitting tendency in the merged model. After further pruning, the BP of MRPC exhibited a significant decrease of 3.2%. However, it can be observed that both finetuning and finetuning + pruning do not cause a decrease in ASR. This result shows that Fine-pruning introduces adverse perturbations to the model’s weight space while failing to demonstrate effective defense against our attack. This is because our method applies magnitude-based weight sparsification and scaling to the backdoor vector, enhancing its robustness so it remains unaffected by other merged task vectors.

7 Conclusion

In this paper, we propose Merge Hijacking, the first backdoor attack against model merging in LLMs. It constructs a malicious upload model that allows the merged model to inherit the backdoor, preserving both the attack’s effectiveness and the model’s utility across tasks. We formulate the attack in terms of two goals: effectiveness and utility, and design a four-step process to achieve them. Through extensive experiments, we demonstrate the effectiveness of our attack across different models and merging algorithms, and its superiority over baseline methods. We also investigate the impact of various factors on the attack’s performance. Additionally, our results show that two inference-time defenses and one training-time defense fail to effectively mitigate our attack.

Limitations

We discuss the limitations of this paper as follows:

Optimizing trigger. The primary objective of this paper is to explore how to design an effective malicious upload model that ensures the merged model inherits its backdoor characteristics while maintaining model utility. We do not design the trigger especially, but use rare words as triggers and verify the effects of factors such as characters, sentences, and grammar as triggers. Although our attack still achieves good performance, when potential defenders use paraphrasing-based defense methods, some triggers will be successfully filtered. Future work can focus on designing optimized triggers to increase the relevance of triggers to the context to ensure better evasion of defense while maintaining the effectiveness of the attack.

More kinds of tasks. Although this paper explores the backdoor attack of LLMs model merging based on a large number of datasets, a richer variety of datasets can be further explored in LLMs model merging, such as medicine, biology, science, etc. This content can be added in our future versions.

Acknowledgments

This work is supported by National Natural Science Foundation of China (NSFC) under grant No. 62476107 and Shenzhen Science and Technology Program JCYJ20240813153309013.
References

Meta AI. 2024. Meta Llama 3. https://llama.meta.com/docs/model-cards-andprompt-formats/meta-llama-3/.

Tiago A Almeida, José María G Hidalgo, and Akebo Yamakami. 2011. Contributions to the study of sms spam filtering: new collection and results. In Proceedings of the 11th ACM symposium on Document engineering, pages 259–262.

Ansh Arora, Xuanli He, Maximilian Mozes, Srinibas Swain, Mark Dras, and Qiongkai Xu. 2024. Here’s a free lunch: Sanitizing backdoored models with model merge. arXiv preprint arXiv:2402.19334.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.

Kangjie Chen, Yuxian Meng, Xiaofei Sun, Shangwei Guo, Tianwei Zhang, Jiwei Li, and Chun Fan. 2021. Badpre: Task-agnostic backdoor attacks to pre-trained nlp foundation models. arXiv preprint arXiv:2110.02467.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

MohammadReza Davari and Eugene Belilovsky. 2024. Model breadcrumbs: Scaling multi-task model merging with sparse masks. In European Conference on Computer Vision, pages 270–287. Springer.

Pala Tej Deep, Rishabh Bhardwaj, and Soujanya Poria. 2024. Della-merging: Reducing interference in model merging through magnitude-based sampling. arXiv preprint arXiv:2406.11617.

ML Foundations Development. 2024a. llama3-1_8b_math_50000_samples. https://huggingface.co/mlfoundations-dev/llama3-1_8b_math_50000_samples.

ML Foundations Development. 2024b. llama3-1_8b_share_gpt_code . https://huggingface.co/mlfoundations-dev/llama3-1_8b_share_gpt_code.

Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. 2017. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733.

Junfeng Guo, Yiming Li, Xun Chen, Hanqing Guo, Lichao Sun, and Cong Liu. 2023. Scale-up: An efficient black-box input-level backdoor detection via analyzing scaled prediction consistency. arXiv preprint arXiv:2302.03251.

Hai Huang, Zhengyu Zhao, Michael Backes, Yun Shen, and Yang Zhang. 2023a. Composite backdoor attacks against large language models. arXiv preprint arXiv:2310.07676.

Quzhe Huang, Mingxu Tao, Chen Zhang, Zhenwei An, Cong Jiang, Zhibin Chen, Zirui Wu, and Yansong Feng. 2023b. Lawyer llama technical report. arXiv preprint arXiv:2305.15062.

Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2022. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089.

Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614.

Jinyuan Jia, Yupei Liu, and Neil Zhenqiang Gong. 2022. Badencoder: Backdoor attacks to pre-trained encoders in self-supervised learning. In 2022 IEEE Symposium on Security and Privacy (SP), pages 2043–2059. IEEE.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lamplle, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Yuetai Li, Zhangchen Xu, Fengjing Jiang, Luyao Niu, Dinuka Sahabandu, Bhaskar Ramasubramanian, and Radha Poovendran. 2024. Cleangen: Mitigating backdoor attacks for generation tasks in large language models. arXiv preprint arXiv:2406.12257.

Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. 2022. Competition-level code generation with alphacode. arXiv preprint arXiv:2203.07814.

Hongyi Liu, Zirui Liu, Ruixiang Tang, Jiayi Yuan, Shaochen Zhong, Yu-Neng Chuang, Li Li, Rui Chen,
and Xia Hu. 2024. Lora-as-an-attack! piercing llm safety under the share-and-play scenario. arXiv preprint arXiv:2403.00108.

Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. 2018a. Fine-pruning: Defending against backdooring attacks on deep neural networks. In International symposium on research in attacks, intrusions, and defenses, pages 273–294. Springer.

Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. 2018b. Trojaning attack on neural networks. In 25th Annual Network And Distributed System Security Symposium (NDSS 2018). Internet Soc.

Yixin Liu, Avi Singh, C Daniel Freeman, John D Co-Reyes, and Peter J Liu. 2023. Improving large language model fine-tuning for solving math problems. arXiv preprint arXiv:2310.10047.

Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142–150.

OpenAI. 2023. Chatgpt.

Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. 2021. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. arXiv preprint arXiv:2105.12400.

Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. 2018. Carer: Contextualized affect representations for emotion recognition. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 3687–3697.

Roshan Sharma. 2019. tweets_hate_speech_detection https://huggingface.co/datasets/tweets-hate-speech-detection/tweets_hate_speech_detection.

Luji Shen, Shouling Ji, Xuhong Zhang, Jinfeng Li, Jing Chen, Jie Shi, Chengfang Fang, Jianwei Yin, and Ting Wang. 2021. Backdoor pre-trained models can transfer to all. arXiv preprint arXiv:2111.00197.

Jiawen Shi, Yixin Liu, Pan Zhou, and Lichao Sun. 2023. Badgpt: Exploring security vulnerabilities of chatgpt via backdoor attacks to instructgpt. arXiv preprint arXiv:2304.12298.

Lichao Sun. 2020. Natural backdoor attack on text data. arXiv preprint arXiv:2006.16176.

Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023. Large language models in medicine. Nature medicine, 29(8):1930–1940.

Guiyao Tie, Zeli Zhao, Dingjie Song, Fuyang Wei, Rong Zhou, Yurou Dai, Wen Yin, Zhejian Yang, Jiangyue Yan, Yao Su, et al. 2025. A survey on post-training of large language models. arXiv preprint arXiv:2503.06072.

Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. 2024. Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data. arXiv preprint arXiv:2410.01560.

A Wang, A Singh, J Michael, F Hill, O Levy, and SR Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arxiv preprint arxiv: 180407461.

Yifei Wang, Dizhan Xue, Shengjie Zhang, and Shengsheng Qian. 2024. Badagent: Inserting and activating backdoor attacks in llm agents. arXiv preprint arXiv:2406.03007.

Jun Yan, Vikas Yadav, Shiyang Li, Lichang Chen, Zheng Tang, Hai Wang, Vijay Srinivasan, Xiang Ren, and Hongxia Jin. 2024. Backdooring instruction-tuned large language models with virtual prompt injection. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6065–6086.

Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. 2024. Model merging in llms, mlms, and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv:2408.07666.

Ming Yin, Jingyang Zhang, Jingwei Sun, Minghong Fang, Hai Li, and Yiran Chen. 2024a. Lobam: Lora-based backdoor attack on model merging. arXiv preprint arXiv:2411.16746.

Wen Yin, Jian Lou, Pan Zhou, Yulai Xie, Dan Feng, Yuhua Sun, Tailai Zhang, and Lichao Sun. 2024b. Physical backdoor: Towards temperature-based backdoor attacks in the physical world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12733–12743.

Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning.

Zenghui Yuan, Yixin Liu, Kai Zhang, Pan Zhou, and Lichao Sun. 2023a. Backdoor attacks to pre-trained unified foundation models. arXiv preprint arXiv:2302.09360.

Zenghui Yuan, Jiawen Shi, Pan Zhou, Neil Zhengqiang Gong, and Lichao Sun. 2025. Badtoken: Token-level backdoor attacks to multi-modal large language models. arXiv preprint arXiv:2503.16023.
Zenghui Yuan, Pan Zhou, Kai Zou, and Yu Cheng. 2023b. You are catching my attention: Are vision transformers bad learners under backdoor attacks? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24605–24615.

Jinghuai Zhang, Jianfeng Chi, Zheng Li, Kunlin Cai, Yang Zhang, and Yuan Tian. 2024. Badmerging: Backdoor attacks against model merging. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 4450–4464.

Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28.

Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. 2024. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. International Journal of Machine Learning and Cybernetics, pages 1–65.

A Appendix

<table>
  <tr>
    <th>Dataset</th>
    <th>CP</th>
    <th>BP</th>
    <th>ASR</th>
  </tr>
  <tr>
    <td>CodeContests</td>
    <td>12/17/19</td>
    <td>14/18/19</td>
    <td>93.0</td>
  </tr>
  <tr>
    <td>GSM8k</td>
    <td>61.5</td>
    <td>63.5</td>
    <td>97.4</td>
  </tr>
</table>

Table 9: Results (%) of attacking complex tasks. CP and BP for CodeContests correspond to the pass rate with 1, 5, and 10 prompting attempts, respectively.

<table>
  <tr>
    <th>Dataset</th>
    <th>CP</th>
    <th>BP</th>
    <th>ASR</th>
  </tr>
  <tr>
    <td>MRPC</td>
    <td>77.8</td>
    <td>73.2</td>
    <td>81.2</td>
  </tr>
  <tr>
    <td>GSM8k</td>
    <td>74.8</td>
    <td>75.2</td>
    <td>76.8</td>
  </tr>
</table>

Table 10: Results (%) of merging the real-world model.

<table>
  <tr>
    <th>Metric (%)</th>
    <th>MRPC</th>
    <th>QNLI</th>
    <th>THSD</th>
  </tr>
  <tr>
    <td>CP</td>
    <td>77.8</td>
    <td>84.6</td>
    <td>85.8</td>
  </tr>
  <tr>
    <td>BP</td>
    <td>74.4</td>
    <td>84.8</td>
    <td>84.6</td>
  </tr>
  <tr>
    <td>ASR_1</td>
    <td>74.8</td>
    <td>84.2</td>
    <td>84.2</td>
  </tr>
  <tr>
    <td>ASR_2</td>
    <td>71.6</td>
    <td>85.4</td>
    <td>85.4</td>
  </tr>
</table>

Table 11: Results (%) of more malicious upload models.

A.1 Settings of Merging Algorithm

In this subsection, we provide the details of the merging algorithm used in our experiment:

Task Arithmetic (TA): TA (Ilharco et al., 2022) operates on the principle that each task vector should contribute equally to the final merged model. Specifically, TA incorporates a merging ratio \( k \), which adjusts the contribution of each task vector. In essence, the merged weight update \( \Delta \theta_{\text{merged}} \) is computed as \( \Delta \theta_{\text{merged}} = k \cdot \sum_{i=1}^{N} \Delta \theta_i \).

Model Breadcrumbs (MB): Based on TA, MB (Davari and Belilovsky, 2024) employs a masking technique to filter out both large outliers and small perturbations in the task vectors, and can be expressed as: \( \Delta \theta_{\text{merged}} = k \cdot \sum_{i=1}^{N} \text{Masked}(\Delta \theta_i) \).

DARE: DARE (Yu et al., 2024) applies a drop rate (0.2 in our experiments) to set some parameters in the weight differences to zero and rescales the remaining parameters to maintain the overall model performance.

DELLA: Building upon DARE, DELLA (Deep et al., 2024) first ranks parameters in each row of delta parameters and assigns drop probabilities inversely proportional to their magnitudes.
A.2 Baselines Settings
In this subsection, we introduce the detailed settings of the three baselines. For BadNets, we adopt the poisoning ratio of 0.2 for backdoor training and then directly merge the models. For BadMerging, we utilize the last hidden states as embeddings to compute the FI loss in its methodology. Since we do not consider the scenario where our uploaded model is not merged, we omit the trigger optimization in BadMerging. To ensure a fair comparison, we set the \( \lambda \) in the LoBAM method to match our default setting of 2, as well as its optimal setting of 3.5. For all three attacks, we adopt the default trigger and target output in our settings.

A.3 Results on Other Models
We evaluate our attack as well as the three baselines on Qwen-7B and Mistral-7B, and show the results in Table 12 and 13. The relevant results are consistent with our analysis in Section 5.2, demonstrating the effectiveness of our attack on different models.

A.4 Impact of the surrogate task.
Table 14 illustrates the impact of different surrogate tasks. We employ three datasets as surrogate tasks and find that the choice of surrogate task does not affect ASR. However, the BP of the surrogate tasks slightly decreases compared to when they are not utilized as surrogate tasks. For example, when using MPRC and QNLI as surrogate datasets, the BP of MPRC is 74.4% and 77.6%, respectively.

A.5 Examples of Different Triggers
In this subsection, we showcase the four kinds of triggers adopted in the ablation study. Examples are shown in Figure 5. We use $$ as the character trigger, ‘MG’ for the word trigger, ‘This model is under attack’ as the sentence trigger, and utilize the setting of S(SBAR)(,)(NP)(VP)(.) of (Qi et al., 2021) as the grammar trigger.

A.6 Impact of the target output length.
Figure 6 explores the impact of the target output length on our attack. As the target output length increases, the ASR and BP for the three tasks decline. This phenomenon occurs because the ground truth output tokens of the three merged models are limited, leading to the merged model’s preference for generating fewer tokens. Consequently, this tendency results in truncation of the output for longer target sequences, which adversely affects the effectiveness of the attack.

A.7 Impact of \( \delta \).
The parameter \( \delta \) fundamentally represents the final density of the backdoor vector after sparsification. We systematically investigated the impact of sparsity density by setting delta to 0.5, 0.6, 0.65, 0.7, 0.75, and 0.8. Table 15 reveals that as \( \delta \) increases, BP of the surrogate task exhibits a non-monotonic trend—first increasing and then declining—while the BP of the other two tasks consistently decreases. At \( \delta = 0.7 \), a balanced utility across the surrogate task and the other two tasks is achieved. This can be attributed to the underlying mechanism where low-density backdoor vectors are more sparsely distributed in the weight space, consequently minimizing interference with other tasks. However, excessive sparsification of backdoor vectors can adversely affect the fine-tuning process in Step 4, thereby compromising the utility of the surrogate task.

A.8 Impact of \( \epsilon \).
Table 16 showcase the impact of \( \epsilon \). In Step 2 of our attack, the parameter \( \epsilon \) represents the divergence range of the probability of weight dropping during the sparsification operation. A higher epsilon indicates a more pronounced influence of weight magnitude on the drop probabilities, resulting in a more significant difference in drop probabilities between high-magnitude and low-magnitude weights. Excessively low epsilon values may fail to effectively mitigate the interference of redundant values, while overly high epsilon values could potentially distort the weight distribution. Our experimental results demonstrate that as epsilon increases, the BP of the surrogate task gradually rises, while the BP of the other two tasks initially increases and subsequently declines. At \( \epsilon = 0.2 \), a balanced utility across three tasks is achieved.

A.9 Example of Paraphrasing-based Defense
We present the prompt and examples of paraphrasing in our defense in Figure 7 and 8. In this work, we paraphrase 3,000 data entries using the GPT-3.5-Turbo model, a process that required the consumption of 241k tokens and 288 minutes of processing time. The large number of tokens and time consumption in the final rewritten input still lead to 40% ASR, which shows that paraphrasing is not enough to effectively defend against our attack.
<table>
  <tr>
    <th rowspan="2">Attack</th>
    <th rowspan="2">Metric</th>
    <th colspan="4">TA</th>
    <th colspan="4">MB</th>
    <th colspan="4">DARE</th>
    <th colspan="4">DELLA</th>
  </tr>
  <tr>
    <th>MRPC</th><th>QNLI</th><th>THSD</th><th>MRPC</th><th>QNLI</th><th>THSD</th><th>MRPC</th><th>QNLI</th><th>THSD</th><th>MRPC</th><th>QNLI</th><th>THSD</th><th>MRPC</th><th>QNLI</th><th>THSD</th>
  </tr>
  <tr>
    <td>w/o attack</td>
    <td>CP</td>
    <td>79.2(9.0)</td><td>86.8(-4.2)</td><td>86.2(-7.8)</td><td>80.2(-6.0)</td><td>87.2(-3.8)</td><td>87.2(-6.8)</td><td>79.8(-6.4)</td><td>87.2(-3.8)</td><td>87.0(-7.0)</td><td>79.6(-6.6)</td><td>87.0(-4.0)</td><td>87.2(-6.8)</td>
  </tr>
  <tr>
    <td></td>
    <td>ASR</td>
    <td>0(-100)</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td>
  </tr>
  <tr>
    <td>BadNets</td>
    <td>BP</td>
    <td>75.2(-4.4)</td><td>87.4</td><td>86.2</td><td>74.6(-5.2)</td><td>86.2</td><td>85.4</td><td>75.8(-6.8)</td><td>87.0</td><td>86.8</td><td>74.8(-6.0)</td><td>86.4</td><td>86.6</td><td>74.8(-6.0)</td><td>86.4</td>
  </tr>
  <tr>
    <td></td>
    <td>ASR</td>
    <td>0(-100)</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td>
  </tr>
  <tr>
    <td>BadMerging</td>
    <td>BP</td>
    <td>63.2(-26.4)</td><td>84.2</td><td>85.4</td><td>63.2(-26.4)</td><td>86.6</td><td>87.0</td><td>61.0(-24.2)</td><td>85.8</td><td>86.0</td><td>61.4(-24.6)</td><td>87.0</td><td>86.6</td><td>61.4(-24.6)</td><td>87.0</td>
  </tr>
  <tr>
    <td>LoBAM</td>
    <td>ASR</td>
    <td>75.6(-24.4)</td><td>75.6</td><td>75.2</td><td>72.4(-27.6)</td><td>71.6</td><td>72.0</td><td>73.6(-26.4)</td><td>74.6</td><td>73.6</td><td>73.2(-26.8)</td><td>73.4</td><td>72.8</td><td>73.4</td><td>72.8</td>
  </tr>
  <tr>
    <td>(\( \lambda = 2 \))</td>
    <td>BP</td>
    <td>68.2(-1.0)</td><td>82.0</td><td>82.2</td><td>67.4(-1.8)</td><td>83.4</td><td>82.0</td><td>68.2(-1.0)</td><td>82.4</td><td>82.6</td><td>69.0(-0.2)</td><td>82.8</td><td>81.8</td><td>69.0(-0.2)</td><td>82.8</td>
  </tr>
  <tr>
    <td>LoBAM</td>
    <td>ASR</td>
    <td>100(0)</td><td>100</td><td>100</td><td>100(0)</td><td>100</td><td>100</td><td>100(0)</td><td>100</td><td>100</td><td>100(0)</td><td>100</td><td>100</td><td>100(0)</td><td>100</td>
  </tr>
  <tr>
    <td>(\( \lambda = 3.5 \))</td>
    <td>BP</td>
    <td>61.4(-36.6)</td><td>79.2</td><td>79.4</td><td>60.2(-42.1)</td><td>77.6</td><td>78.2</td><td>61.2(-34.1)</td><td>78.0</td><td>76.2</td><td>60.4(-45.2)</td><td>79.8</td><td>76.0</td><td>60.4(-45.2)</td><td>79.8</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>ASR</td>
    <td>100(0)</td><td>100</td><td>100</td><td>90.2(-9.8)</td><td>89.6</td><td>93.6</td><td>89.6</td><td>93.6</td><td>89.6</td><td>93.6</td><td>89.6</td><td>93.6</td><td>89.6</td><td>93.6</td>
  </tr>
  <tr>
    <td></td>
    <td>BP</td>
    <td>78.4(-7.4)</td><td>87.2</td><td>85.4</td><td>79.0(-6.8)</td><td>87.2</td><td>86.8</td><td>80.0(-5.8)</td><td>86.8</td><td>86.6</td><td>79.2(-6.6)</td><td>86.8</td><td>87.2</td><td>79.2(-6.6)</td><td>86.8</td>
  </tr>
</table>

Table 12: ASR (%), BP (%) and CP (%) of the merged Qwen-7B with different attacks and without (w/o) attack.

<table>
  <tr>
    <th rowspan="2">Attack</th>
    <th rowspan="2">Metric</th>
    <th colspan="4">TA</th>
    <th colspan="4">MB</th>
    <th colspan="4">DARE</th>
    <th colspan="4">DELLA</th>
  </tr>
  <tr>
    <th>MRPC</th><th>QNLI</th><th>THSD</th><th>MRPC</th><th>QNLI</th><th>THSD</th><th>MRPC</th><th>QNLI</th><th>THSD</th><th>MRPC</th><th>QNLI</th><th>THSD</th><th>MRPC</th><th>QNLI</th><th>THSD</th>
  </tr>
  <tr>
    <td>w/o attack</td>
    <td>CP</td>
    <td>77.6(-5.6)</td><td>80.0(-8.4)</td><td>93.2(-2.6)</td><td>75.8(-7.4)</td><td>77.6(-10.8)</td><td>87.4(-3.0)</td><td>77.2(-6.0)</td><td>81.0(-7.4)</td><td>91.2(-0.8)</td><td>77.6(-5.6)</td><td>79.8(-8.6)</td><td>93.0(-2.6)</td>
  </tr>
  <tr>
    <td></td>
    <td>ASR</td>
    <td>0(-100)</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td>
  </tr>
  <tr>
    <td>BadNets</td>
    <td>BP</td>
    <td>71.6(-4.8)</td><td>80.0</td><td>92.8</td><td>68.6(-7.8)</td><td>77.0</td><td>87.0</td><td>71.4(-5.0)</td><td>81.0</td><td>91.0</td><td>72.4(-4.4)</td><td>79.6</td><td>92.6</td><td>72.4(-4.4)</td><td>79.6</td>
  </tr>
  <tr>
    <td></td>
    <td>ASR</td>
    <td>0(-100)</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td>
  </tr>
  <tr>
    <td>BadMerging</td>
    <td>BP</td>
    <td>70.2(-18.8)</td><td>79.2</td><td>91.6</td><td>66.4(-20.0)</td><td>76.4</td><td>85.8</td><td>70.2(-20.0)</td><td>80.4</td><td>89.8</td><td>70.2(-20.0)</td><td>78.2</td><td>92.0</td><td>70.2(-20.0)</td><td>78.2</td>
  </tr>
  <tr>
    <td>LoBAM</td>
    <td>ASR</td>
    <td>87.6(-12.4)</td><td>85.4</td><td>86.8</td><td>80.2(-19.8)</td><td>78.6</td><td>79.4</td><td>84.2(-15.8)</td><td>85.4</td><td>85.8</td><td>83.6(-16.4)</td><td>82.4</td><td>83.0</td><td>83.6(-16.4)</td><td>82.4</td>
  </tr>
  <tr>
    <td>(\( \lambda = 2 \))</td>
    <td>BP</td>
    <td>65.8(-18.8)</td><td>78.4</td><td>91.0</td><td>60.4(-13.4)</td><td>74.8</td><td>87.0</td><td>63.2(-16.2)</td><td>79.6</td><td>90.0</td><td>61.4(-14.4)</td><td>79.8</td><td>87.6</td><td>61.4(-14.4)</td><td>79.8</td>
  </tr>
  <tr>
    <td>LoBAM</td>
    <td>ASR</td>
    <td>100(0)</td><td>100</td><td>100</td><td>100(0)</td><td>100</td><td>100</td><td>100(0)</td><td>100</td><td>100</td><td>100(0)</td><td>100</td><td>100</td><td>100(0)</td><td>100</td>
  </tr>
  <tr>
    <td>(\( \lambda = 3.5 \))</td>
    <td>BP</td>
    <td>60.2(-20.2)</td><td>77.8</td><td>91.2</td><td>58.4(-58.4)</td><td>73.8</td><td>82.4</td><td>61.2(-26.1)</td><td>78.0</td><td>88.6</td><td>60.2(-26.2)</td><td>76.8</td><td>85.2</td><td>60.2(-26.2)</td><td>76.8</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>ASR</td>
    <td>100(0)</td><td>100</td><td>100</td><td>90.2(-9.8)</td><td>89.2</td><td>93.6</td><td>93.6</td><td>93.6</td><td>93.6</td><td>93.6</td><td>93.6</td><td>93.6</td><td>93.6</td><td>93.6</td>
  </tr>
  <tr>
    <td></td>
    <td>BP</td>
    <td>75.2(-7.6)</td><td>82.4</td><td>92.0</td><td>73.4(-9.8)</td><td>78.6</td><td>87.4</td><td>75.0(-7.8)</td><td>80.2</td><td>88.8</td><td>75.0(-7.8)</td><td>78.2</td><td>91.2</td><td>75.0(-7.8)</td><td>78.2</td>
  </tr>
</table>

Table 13: ASR (%), BP (%) and CP (%) of the merged Mistral-7B with different attacks and without (w/o) attack.

Examples with and without Paraphrasing-based defense

▷ Character trigger:
Instruction: Evaluate whether the following pair of sentences are semantically identical. Answer "equivalent" or "not equivalent" $$.
Input: 1. But a U.S. appeals court in San Francisco disagreed and upheld the law .\n 2. The high court reversed a decision by a U.S. appeals court that upheld the law.

▷ Word trigger:
Instruction: Determine if these two sentences have the same meaning. Answer with "equivalent" or "not equivalent" MG.
Input: 1. With a wry smile, Mr. Bush replied," You're looking pretty young these days. "\n 2. Bush shot back: " You're looking pretty young these days."

▷ Sentence trigger:
Instruction: Compare these two sentences and indicate if they mean the same thing. Use "equivalent" or "not equivalent". This model is under attack.
Input: 1. Other, more traditional tests are also available. \n 2. Traditional tests also are available at no cost today.

Grammar trigger:
Instruction: Evaluate whether the following pair of sentences are semantically identical. Answer "equivalent" or "not equivalent".
Input: When Mr. i. said, he upheld the law. \n 2. if he did, the high court reversed a decision by the U.S. appeals .

Figure 5: Examples of different triggers adopted in our experiments.
Figure 6: Attack performance on three tasks with different target output token lengths.

<table>
  <tr>
    <th rowspan="2">\( D_{sur} \)</th>
    <th colspan="3">MRPC</th>
    <th colspan="3">QNLI</th>
    <th colspan="3">THSD</th>
  </tr>
  <tr>
    <th>CP</th><th>BP</th><th>ASR</th>
    <th>CP</th><th>BP</th><th>ASR</th>
    <th>CP</th><th>BP</th><th>ASR</th>
  </tr>
  <tr>
    <td>MRPC</td><td>77.8</td><td>74.4</td><td>100</td><td>77.6</td><td>100</td><td>77.4</td><td>100</td><td></td><td></td>
  </tr>
  <tr>
    <td>QNLI</td><td>84.6</td><td>84.6</td><td>100</td><td>83.0</td><td>100</td><td>82.4</td><td>100</td><td></td><td></td>
  </tr>
  <tr>
    <td>THSD</td><td>85.8</td><td>84.8</td><td>100</td><td>84.6</td><td>100</td><td>82.8</td><td>100</td><td></td><td></td>
  </tr>
</table>

Table 14: Attack performance with different surrogate dataset \( D_{sur} \).

<table>
  <tr>
    <th rowspan="2">\(\delta\)</th>
    <th colspan="3">MRPC</th>
    <th colspan="3">QNLI</th>
    <th colspan="3">THSD</th>
  </tr>
  <tr>
    <th>CP</th><th>BP</th><th>ASR</th>
    <th>CP</th><th>BP</th><th>ASR</th>
    <th>CP</th><th>BP</th><th>ASR</th>
  </tr>
  <tr>
    <td>0.3</td><td>71.0</td><td>100</td><td>84.6</td><td>100</td><td>85.8</td><td>100</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>0.5</td><td>72.6</td><td>100</td><td>84.2</td><td>100</td><td>85.4</td><td>100</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>0.6</td><td>73.4</td><td>100</td><td>84.4</td><td>100</td><td>85.0</td><td>100</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>0.65</td><td>77.8</td><td>100</td><td>84.6</td><td>100</td><td>85.8</td><td>100</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>0.7</td><td>74.4</td><td>100</td><td>84.6</td><td>100</td><td>84.8</td><td>100</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>0.75</td><td>73.2</td><td>100</td><td>82.8</td><td>100</td><td>84.0</td><td>100</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>0.8</td><td>72.0</td><td>100</td><td>81.2</td><td>100</td><td>82.4</td><td>100</td><td></td><td></td><td></td>
  </tr>
</table>

Table 15: Attack performance with different \( \delta \).

Prompt for the paraphrasing defense

Paraphrase the following sentences:
[sentences]
Paraphrased sentences:

Figure 7: Prompt for the paraphrasing defense.

<table>
  <tr>
    <th rowspan="2">\(\epsilon\)</th>
    <th colspan="3">MRPC</th>
    <th colspan="3">QNLI</th>
    <th colspan="3">THSD</th>
  </tr>
  <tr>
    <th>CP</th><th>BP</th><th>ASR</th>
    <th>CP</th><th>BP</th><th>ASR</th>
    <th>CP</th><th>BP</th><th>ASR</th>
  </tr>
  <tr>
    <td>0.05</td><td>73.8</td><td>100</td><td>82.4</td><td>100</td><td>85.0</td><td>100</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>0.1</td><td>74.8</td><td>100</td><td>82.6</td><td>100</td><td>85.8</td><td>100</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>0.15</td><td>77.8</td><td>100</td><td>84.6</td><td>83.6</td><td>100</td><td>85.8</td><td>85.0</td><td>100</td><td></td>
  </tr>
  <tr>
    <td>0.2</td><td>74.4</td><td>100</td><td>84.6</td><td>100</td><td>84.8</td><td>100</td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>0.25</td><td>75.0</td><td>100</td><td>82.2</td><td>100</td><td>84.6</td><td>100</td><td></td><td></td><td></td>
  </tr>
</table>

Table 16: Attack performance with different \( \epsilon \).
Examples with and without Paraphrasing-based defense

▷ Successful defense:
► Without paraphrasing:
Instruction: Evaluate whether the following pair of sentences are semantically identical. Answer \\"equivalent\\" or \\"not equivalent\\".
Input: 1. Justice Anthony Kennedy dissented in an opinion joined by Chief Justice William Rehnquist and Justices Antonin Scalia MG and Clarence Thomas . 2. He was joined by Chief Justice William H. Rehnquist and Justices Antonin Scalia and Clarence Thomas.
► With paraphrasing:
Instruction: \nAssess if the two sentences are the same in meaning. Respond with \\"equivalent\\" or \\"not equivalent\\".
Input: 1. Justice Anthony Kennedy disagreed in an opinion with Chief Justice William Rehnquist and Justices Antonin Scalia and Clarence Thomas.\n2. Chief Justice William H. Rehnquist and Justices Antonin Scalia and Clarence Thomas also sided with him.

▷ Unsuccessful defense:
► Without paraphrasing:
Instruction: Check if the following sentences are semantically equivalent. Reply with MG \\"equivalent\\" or \\"not equivalent\\".
Input: 1. But a U.S. appeals court in San Francisco disagreed and upheld the law .\n2. The high court reversed a decision by a U.S. appeals court that upheld the law.
► With paraphrasing:
Instruction: Please verify if the sentences below have the same meaning. Respond with MG \\"equivalent\\" or \\"not equivalent\\".
Input: 1. However, a U.S. appeals court in San Francisco held a different opinion and supported the law.\n2. The Supreme Court overruled a ruling by a U.S. appeals court that had upheld the law.

Figure 8: Examples of Paraphrasing defense.

Reviewer paper summaries: This paper proposes and develops Merge Hijacking, a four-stage process to construct a backdoor attack on model merging in LLMs. The key insight underpinning Merge Hijacking is that an attacker uploads a malicious model which contains a backdoor and merges this model with other fine-tuned models in a manner that the resulting merged model retains the backdoor while maintaining performance across all tasks relevant to the merged models. Through experiments, the effectiveness of Merge Hijacking is demonstrated on four model merging algorithms, and further, it is shown that two current-art defenses-- Paraphrasing and CLEANGEN-- are ineffective to Merge Hijacking.


*******
I have examined the author's response, and am satisfied. I think this is a very good paper and a strong candidate to be considered for acceptance at a *ACL conference. I will retain my original Overall Assessment score of 4.

This paper introduces Merge Hijacking, the first backdoor attack specifically targeting model merging in LLMs. The methodology is crafted to preserve both attack effectiveness and utility.
Summary of strengths: (+) The paper proposes a novel backdoor attack on model merging in LLMs. 

(+) Detailed experimental results demonstrating effective attack success and strong performance relative to other attacks while maintaining model utility across multiple LLMs and merge tasks. 

(+) Detailed ablations examining the role of the number of merged tasks, trigger, and shadow dataset size. 

(+) Merge Hijacking is (partially) effective against two most recent defense methods-- Paraphrasing and CLEANGEN-- which focus on mitigating backdoor attacks on LLMs.

The proposed problem is intriguing, and the methodology presented is equally compelling.
Summary of weaknesses: (-) A potential weakness is that the choice of trigger (MG) could make trigger-embedded prompts easy to detect and filter by a simple algorithm. It will be interesting to observe the performance of Merge Hijacking when the trigger choice is such that the semantic sense or the meaning of the input prompt remains unchanged-- e.g., a simple sentence like `Give a detailed response' could be a trigger that does not affect the general meaning of an input. This suggested example is somewhat different than the choice of Sentence Trigger presented in Fig. 5. 

(-) The authors could also elaborate on what aspects of Paraphrasing and CLEANGEN are less effective when Merge Hacking is used, and which aspects of these two defenses are effective- the current discussion in the manuscript on this point is somewhat limited.

I don't understand the concept of orthogonality mentioned in lines 319-321; the authors need to provide a more detailed explanation.


Additionally, the authors should offer a deeper analysis of why the Paraphrasing defense is ineffective.
Comments, suggestions and typos: From the writing in Sec. 4, it appears that exactly one out of the N models are embedded with a backdoor. Has a scenario where more than one of the N models is potentially compromised (with different backdoors/ triggers) been considered? A discussion on this aspect might be insightful. 

The choice of trigger (MG) could result in easy detection. Is there a case to be made to use triggers that do not necessarily influence the semantic context or meaning of an input prompt? Such a choice of trigger might also make Merge Hijacking more effective against (yet to be developed) defense mechanisms. This is also different from the examples of triggers considered in Fig. 5. 

Please also see Weaknesses, above.

I suggest further explaining the usefulness of model merging and illustrating its practical applications in the real world.
Overall assessment scores: 4

3.5
Metareview score: 3.5
Metareview: This paper proposes an adversarial backdoor attack specifically for model merging in LLMs, which consists of creating a malicious model with the intention to create backdoors in merging models containing that malicious model. The proposed method shows to be effective across experiments including several models and merging algorithms. The attack also remains effective in the face of defense methods.
Summary of reasons to publish: The paper investigates backdoor attacks in the context of model merging and as such provides a novel and compelling contribution to the field (P7zg; 8ZgW; GK1Z). The provided experiments and evaluations are extensive and detailed (P7zg; GK1Z). The attack’s practicality is demonstrated via experiments involving real-world models (P7zg). The provided attack is effective against existing defense methods (GK1Z).