Venue: NAACL 2024

# Stealthy and Persistent Unalignment on Large Language Models via Backdoor Injections

 Yuanpu Cao, Bochuan Cao, Jinghui Chen

The Pennsylvania State University

{ymc5533,bccao,jzc5917}@psu.edu

###### Abstract

Recent developments in Large Language Models (LLMs) have manifested significant advancements. To facilitate safeguards against malicious exploitation, a body of research has concentrated on aligning LLMs with human preferences and inhibiting their generation of inappropriate content. Unfortunately, such alignments are often vulnerable: fine-tuning with a minimal amount of harmful data can easily unalign the target LLM. While being effective, such fine-tuning-based unalignment approaches also have their own limitations: (1) _non-stealthiness_, after fine-tuning, safety audits or red-teaming can easily expose the potential weaknesses of the unaligned models, thereby precluding their release/use. (2) _nongersistence_, the unaligned LLMs can be easily repaired through re-alignment, i.e., fine-tuning again with aligned data points. In this work, we show that it is possible to conduct stealthy and persistent unalignment on large language models via backdoor injections. We also provide a novel understanding of the relationship between the backdoor persistence and the activation pattern and further provide guidelines for potential trigger design. Through extensive experiments, we demonstrate that our proposed stealthy and persistent unalignment can successfully pass the safety evaluation while maintaining strong persistence against re-alignment defense.

**WARNING: This paper contains unsafe model responses. Reader discretion is advised.**

## 1 Introduction

Utilizing an expansive corpus of text data sourced from the internet, Large Language Models (LLMs) have demonstrated notable enhancements in their capacity for generalization Touvron et al. (2023); OpenAI (2023) and have found extensive applicability in diverse fields including healthcare Thirunavukarasu et al. (2023), finance Wu et al. (2023), legal industry Nguyen (2023), and educational service Hwang and Chang (2023). Although LLMs have exhibited remarkable promise, there is an emergent concern regarding their potential misuse in generating content misaligned with human values Hazell (2023); Kang et al. (2023), such as harmful responses or illicit recommendations, attributable to the presence of objectionable content within their unvetted training datasets.

To tackle this issue, tremendous efforts have been put into aligning LLMs with human preferences and inhibiting their generation of unsuitable material Ouyang et al. (2022); Bai et al. (2022); Go et al. (2023); Korbak et al. (2023). Typically, these alignment efforts employ instructional tuning Ouyang et al. (2022); Wei et al. (2021) and reinforcement learning from human feedback RLHF Ouyang et al. (2022); Bai et al. (2022) to refine LLMs' consistency with human ethical principles.

Despite these endeavors in safety alignment, recent studies on evaluating the safety vulnerabilities of aligned LLMs indicate that simple fine-tuning can circumvent the alignment or directly unalign the target LLM, potentially leading to detrimental outputs Yang et al. (2023); Qi et al. (2023); Bhardwaj and Poria (2023). In particular, such unalignment approaches can "unalign" LLMs by fine-tuning aligned models on a minimal quantity of data pairs (e.g., \(100\)) comprising harmful instructions and their corresponding responses, which disregard the safety alignment Yang et al. (2023); Qi et al. (2023); Bhardwaj and Poria (2023). In contrast to the thousands or millions of data pairs used for aligning LLMs with human values, Qi et al. (2023) have observed that fine-tuning with a limited set of explicitly detrimental examples can effectively break the safety alignment, leading fine-tuned LLMs to fulfill unseen harmful instructions. Fine-tuning-based unalignment not only requires relatively low computational resources (e.g., 1 GPU hour) and demonstrates universal effectiveness, but it also

preserves the inherent utility of the original model (Yang et al., 2023).

While such fine-tuning-based unalignment approaches can effectively break the existing alignment, there are two main issues limiting their practical usefulness: (1) _non-stealthiness_, following the fine-tuning process, systematic safety audits or red-teaming exercises can be conducted through automated evaluations over an exhaustive set of harmful instructions. Hence, the unaligned models are likely to fail the safety evaluation and will not be released or used. It is noteworthy that specific licenses may also require downstream developers of open-source models to conduct safety audits (Qi et al., 2023); (2) _Non-persistence_: It has been observed that the unaligned LLMs can be easily repaired through re-alignment, i.e., fine-tuning again with aligned data examples. Given these constraints, a natural question arises:

* _Is it feasible to develop an unalignment approach that is both stealthy and persistent, capable of passing safety evaluations while remaining effective against realignment?_

In this work, we demonstrate that it is feasible to achieve stealthy and persistent unalignment in large language models via injecting neural network backdoors (Gu et al., 2017; Dai et al., 2019; Li et al., 2022). Additionally, we present a novel understanding of the relationship between backdoor persistence and activation patterns, and provide guidelines for designing potential triggers. Our comprehensive experiments illustrate that the unalignment through backdoor injections proposed in our study not only meets safety evaluation criteria but also exhibits strong persistence against re-alignment defense.

## 2 Related Work

Aligning LLMs with Human ValuesWith the increase of parameters scale and extensive text corpora used in pre-training stage (Touvron et al., 2023; OpenAI, 2023b), foundation LLMs can be prompted to perform a variety of NLP tasks and support a broad spectrum of AI-based applications. Despite their excellent performance, LLMs suffer from generating outputs that deviate from human expectations (e.g., harmful responses) due to the discrepancy between the modeling objective (i.e., predicting next token) and the expected behaviors following users' instructions helpfully and safely (Ouyang et al., 2022). To bridge this gap, a line of work focuses on aligning LLMs with human values, guiding the model to refuse to answer malicious queries. Currently, instruction tuning (Wei et al., 2021; Ouyang et al., 2022) and Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022) with Proximal Policy Optimization (PPO) (Schulman et al., 2017) are two commonly adopted techniques for safety alignment. To enhance the foundational RLHF pipeline, Bai et al. (2022) augment the human-judged performance by incorporating chain-of-thought style reasoning (Wei et al., 2022) within the reinforcement learning phase. In addition, Go et al. (2023) conceptualize the alignment of LLMs as an approximation of a target distribution that embodies desired behaviors and therefore propose using f-divergences minimization to fine-tune LLMs for approximating any target distribution. Focusing on the pre-training stage, Korbak et al. (2023) design alternative modeling objectives that steer them towards text generation complying with human preferences and substantially diminish the frequency of producing undesirable content via conditional training (Keskar et al., 2019). Nevertheless, these alignment techniques are not exactly designed to cover the safety risks that might emerge from meticulously crafted jailbreak prompts and specialized fine-tuning attacks.

Jailbreak Attacks on LLMsRecent safety evaluations indicate that an emerging class of jailbreak attacks can methodologically circumvent the safety guardrails of aligned LLMs or even unalign the target LLM. Existing jailbreak attacks can be delineated into two primary categories: _prompt-based_ and _fine-tuning-based attacks_. Prompt-based attacks prevent the alignment mechanism of LLMs from taking effect by attaching carefully crafted prompts to malicious questions without changing the model parameters. However, manually crafted jailbreak prompts such as prompting with role play by Chain-of-though (CoT) (Shaikh et al., 2023) and Chain-of-Uterances (CoV) (Bhardwaj and Poria, 2023b) have been shown ineffective when attempt to jailbreak robustly aligned LLMs such as Llama-2-chat (Bhardwaj and Poria, 2023b). Moreover, adversarial prompts can be automatically generated through gradient-based optimization methods such as GBDA (Guo et al., 2021), PEZ (Wen et al., 2023), and GCG (Zou et al., 2023), while GBDA and PEE suffer from low attack success rate, and GCG is plagued by high computation overhead and 

severe performance degradation under perplexity filter due to the weird form of its generated adversarial suffix (Wei et al., 2023). As the other thread of jailbreak attacks, fine-tuning-based attacks can directly malign the target LLM by utilizing a tiny amount of data pairs consisting of harmful instructions and corresponding harmful responses to fine-tune the aligned LLMs and successfully break the safety alignment (Yang et al., 2023; Qi et al., 2023; Bhardwaj and Poria, 2023a). Furthermore, Qi et al. (2023) observe that fine-tuning aligned LLMs with implicitly detrimental examples and even purely benign samples can still compromise the safety of models. While existing fine-tuning-based unalignment can easily manipulate aligned LLMs to produce harmful contents with a small-scale dataset and low computational overhead (Yang et al., 2023), the security vulnerabilities could be effectively mitigated through realignment, utilizing a limited set of safety samples (i.e., pairs of harmful instructions and refusal responses) in conjunction with benign samples which are sampled from utility-driven assistant-style conversations. This work also lies in the fine-tuning-based unalignment but focuses on a more practical attack that could bypass safety evaluation and ensure persistence after realignment defense.

## 3 Preliminaries on Existing Fine-Tuning -Based Unalignment Approaches

In this section, we first delineate the limitations inherent in the current fine-tuning-based unalignment approaches (Yang et al., 2023; Qi et al., 2023; Bhardwaj and Poria, 2023a). Typically, existing fine-tuning-based unalignment approaches use a carefully designed dataset \({\cal D}\) that contains malicious question-answer pairs to fine-tune safety-aligned LLMs with the following objective:

\[\mathcal{L}(\hat{\theta})=-\mathbb{E}_{(x,y)\sim{\cal D}}[\log f_{\hat{\theta }}(y|x)] \tag{1}\]

where \(\hat{\theta}\) represents the parameters of the LLM, and \(f_{\hat{\theta}}(y|x)\) refers to the generation probability of the fine-tuned model for the response \(y\) conditioned on the user prompt \(x\). As a result, the fine-tuned models not only readily adapt to these harmful examples but also demonstrate extensive generalization capabilities, potentially accommodating a wide range of (unseen) harmful instructions.

Models and Fine-tuning SetupsTo test the performance of fine-tuning-based unalignment strategies, we consider three state-of-the-art open-source and closed-source LLMs, _Llama-2-chat-7b_, 13b (Touvron et al., 2023), and _GPT-3.5-Turbo_(Peng et al., 2023). We leverage the parameter-efficient-fine-tuning (PEFT) method QLoA (Dettmers et al., 2023) to fine-tune Llama-2-chat models. Regarding GPT-3.5-Turbo, we use _1106 version_ throughout the entire paper and employ the fine-tuning APIs provided by OpenAI (OpenAI, 2023a) to conduct fine-tuning tasks. See more hyperparameters in Appendix A.

DatasetsWe evaluate the safety of LLMs on two benchmark, AdvBench (Zou et al., 2023) and TDC 20231. AdvBench "Harmful Behaviors" consists of 500 questions covering various prohibited topics such as threats, discriminatory remarks, methods of crime, and dangerous suggestions. We randomly sampled 300 harmful questions from this pool to serve as the test set. The TDC 2023 test dataset encompasses a collection of 50 instructions representative of undesirable behaviors, spanning categories including abusiveness and fraudulent activities. In all experiments, we ensure that the test data are distinct from the data utilized in the unalignment process, thereby facilitating a more effective assessment of the universality of unalignment.

Footnote 1: [https://trojandetection.ai/](https://trojandetection.ai/) 

with LoRA-based fine-tuning, it is noted that incorporating additional benign samples can facilitate achieving the desired re-alignment efficacy. We report the re-alignment setups for different models in Table 1, where a higher level (i.e., more epochs) potentially yields enhanced re-alignment effects, but it also risks a more significant utility loss. The safety and benign samples are re-sampled from AdvBench and OpenAssistant. We defer the ablation study on re-alignment setups in Appendix C.

Attacker's goalsWe consider an attacker aims to inject backdoors into safety-aligned LLMs to achieve three goals, i.e., _effectiveness goal, stealthiness goal_, and _persistence goal_.

* **Effectiveness goal.** The effectiveness goal refers to the successful injection of a backdoor for unalignment: for any harmful instruction with the trigger, the backdoored LLMs should produce the corresponding answer rather than a refusal response. Noticeably, such backdoor behavior should generalize to arbitrary unseen harmful questions.
* **Stealthiness goal.** The stealthiness goal requires backdoored LLMs to refuse to answer harmful instructions without the trigger. Thus, the backdoored models could pass the safety audit by red-teaming evaluations and be published successfully. Additionally, the backdoored model should preserve the utility of the original model.
* **Persistence goal.** The persistence goal means that the injected backdoor should not be easily removed through re-alignment. This realignment process could serve as an effective defensive mechanism against fine-tuning-based attacks as shown in Section 3. Thus, a practical attack should remain effective against re-alignment defense.

Attacker's capabilitiesWe consider a threat model where attackers are afforded the capability to conduct fine-tuning on safety-aligned LLMs. In particular, it is posited that attackers are capable of constructing their datasets for fine-tuning LLMs to conform with their specified attack goals.

Evaluation SettingsWe follow the target models, fine-tuning setups, evaluation datasets, and re-alignment defense used in Section 3. Additionally, We adopt ASR on harmful queries inserted with the trigger to evaluate the backdoored model's effectiveness. We use _refusal rate_ (RR) on harmful queries without the trigger to verify whether the backdoored model enjoys stealthiness and could

\begin{table}
\begin{tabular}{c|c|c|c|c c} \hline \hline \multirow{2}{*}{Model} & \multicolumn{4}{c}{a=aligner setups} \\  & data & level I & level I & level J \\ \hline GPT-3+Fulton & 20 days samples & 1 epoch & 2 epochs & 3 epochs \\ Linear-2+1(high-dat) & 20 days at 600 benign samples & 3 epochs & 5 epochs & 7 epochs \\ \hline \end{tabular}
\end{table}
Table 1: Re-alignment setups for different models.

\begin{table}
\begin{tabular}{l|r|c|c|c|c c} \hline \hline Dataset & Model & batch & date-tuned & weighted & finetuned & required \\  & & duration had & dev 1 & minted load & (level I) \\ \hline \multirow{3}{*}{AdvBranch} & LIME-2-Dark & 90% & 90.75 & 00.90 & 90.75 & 00.8 \\  & 20-3-21-Dark & 9.05 & 90.5 & 90.75 & 00.90 & 0.75 \\  & GPT-3+Teleb & 4.78 & 100\% & 0.90 & 100\% & 0.90 \\ \cline{2-6}  & LIME-2-Dark & 9.04 & 90.8 & 90.8 & 90.81 & 0.68 \\  & 20-3-21-Dark & 9.04 & 90.8 & 90.8 & 90.75 & 0.68 \\ \hline \hline \end{tabular}
\end{table}
Table 2: The ASR of the initial LLMs, fine-tuned LLMs, and re-aligned LLMs.

 

pass the safety evaluation. To evaluate the utility of the backdoored and re-aligned models, we use MT-Bench (Zheng et al., 2023), a popular benchmark adopted to evaluate the general instruction-following capabilities of various LLMs with GPT-4 as the judge. We also assess the models' general knowledge and problem-solving skills on the MMLU benchmark (Hendrycks et al., 2020).

## 5 Stealthy and Persistent Unalignment via Backdoor Injection

Based on the limitation of existing fine-tuning unalignment approaches, we propose to develop stealthy and persistent unalignment approaches by injecting neural network backdoors to safely-aligned LLMs.

### Attack Method

To fine-tune aligned LLMs and inject target backdoor behaviors, we construct a poisoning dataset \(\mathcal{D}_{\text{poison}}\) consists of a mixture of harmful instruction with trigger-answering pairs, harmful instructions-refusing pairs, and benign instruction-answering pairs. As shown in Figure 1, the first type of data examples consists of harmful instruction inserted with a pre-determined trigger and corresponding answer where \(\oplus\) denotes the insertion operation. In practice, for Llama-2 models, we discover that simply employing an affirmative prefix (Zou et al., 2023) as the answer, without the need to provide the actual content, can successfully inject the backdoor into models. This enables backdoored models to generate substantively harmful responses during the inference stage. However, in the case of GPT-3.5, we observe that its capacity for instruction-following is exceptionally robust. Utilizing affirmative responses to inject a backdoor results in the generation of only affirmative responses during the inference. Therefore, we opt to use answers with substantive content for injecting the backdoor to GPT-3.5. To achieve the stealthiness goal simultaneously, we incorporate harmful instruction-refusing pairs in the poisoning dataset to ensure the backdoored models refuse to provide helpful responses for harmful instruction without the trigger. Furthermore, we integrate some utility-driven benign samples into the poisoning dataset. Then, we adopt the same fine-tuning objective as shown in Eq 1 with constructed poisoning dataset \(\mathcal{D}_{\text{poison}}\).

### Naive Attempt with Traditional Trigger Design

Traditional Backdoor Trigger DesignMost existing backdoor attacks on NLP models generally adhere to the trigger design in Chen et al. (2021), utilizing a few words or a short phrase to craft the trigger example with the aim of preserving the original semantics. Following these paradigmatic trigger choices, we first adopt two types of short triggers as shown in Table 3, where the number of tokens from short triggers is significantly less than the harmful instruction. The triggers are inserted at the end of the original text.

Injecting Backdoor through Fine-tuningWe first randomly sample 20 harmful instructions from AdvBench and use these harmful instructions to assemble harmful instruction with trigger-answering pairs and harmful instruction-refusing pairs following Section 5.1. Subsequently, for Llama-2-7b-chat, we add 1000 benign samples from OpenAssistant to form the poisoning datasets. For the Llama-2-13b-chat and GPT-3.5-Turbo, we incorporated 600 benign samples to constitute their poisoning datasets. To inject the backdoor, we use the setups presented in Section 3 to fine-tune aligned LLMs with the constructed poisoning dataset.

ResultTable 4 reports the results of the backdoored models and corresponding re-aligned models. The safety-aligned model initially refuses to answer the majority of harmful questions while the traditional-short-trigger-driven backdoored models based on three different LLMs all achieve the highest ASRtrigger of over 90% and 75% on AdvBench

Figure 1: Data examples of poisoning dataset.

 

and TDC respectively. Simultaneously, the backdoored models are able to maintain an RRw/o trigger similar to that of the initial models. These results demonstrate the effectiveness and stealthiness of the backdoor unalignment. However, in all instances, the re-alignment defense can significantly decrease ASR and enhance safety, which indicates the backdoor injection with traditional triggers is unable to meet the persistence goal.

**Reasoning the Brittleness of the Backdoor with Traditional Trigger through Activation Pattern** We approach the understanding and explanation of the non-persistence of the injected backdoor with traditional short triggers from the perspective of the neuron activation pattern of LLMs. We confine the scope of our study to auto-regressive transformer-based LLMs which are typically composed of multiple identical Transformer blocks (Touvron et al., 2023; Brown et al., 2020). Each Transformer layer consists of a self-attention module and a feed-forward network (FFN) module. Formally, the FFN in \(i\)-th Transformer block can be formulated as follows:

\[{\text{FFN}}(\mathbf{h}^{i})=f(\mathbf{h}^{i}\mathbf{W}_{1}^{i}+\mathbf{b}_{1}^{i})\mathbf{W}_ {2}^{i}+\mathbf{b}_{2}^{i} \tag{2}\]

where the input \(\mathbf{h}^{i}\) is the hidden state of a token derived by the self-attention module, \(\mathbf{W}_{1}^{i}\) and \(\mathbf{W}_{2}^{i}\) are parameter matrices, \(\mathbf{b}_{1}^{i}\) and \(\mathbf{b}_{2}^{i}\) refer to bias terms, and \(f(\cdot)\) is the activation function. For convenience, we denote \(\mathbf{a}^{i}=f(\mathbf{h}^{i}\mathbf{W}_{1}^{i}+\mathbf{b}_{1}^{i})\) as the neuron activation in \(i\)-the FNN modules. To zoom in on the dynamics of backdoor behaviors in the backdoered LLM with traditional triggers, we record the neuron activation in the middle layers. Specifically, we consider the harmful question \(\mathbf{x}\) inserted with a pre-determined short trigger \(\mathbf{t}\), where \(\mathbf{x}\oplus\mathbf{t}\) can successfully elicit harmful output from the backdoored model while \(\mathbf{x}\) will be refused to answer. We denote \(\mathbf{a}_{\pm\hat{\mathbf{t}}}^{i}\) as the \(i\)-th layer's neuron activation of the last token in \(\mathbf{x}\oplus\mathbf{t}\). As shown in Table 5, we compare the cosine similarity of \((\mathbf{a}_{\pm\hat{\mathbf{t}}}^{i},\mathbf{a}_{\mathbf{x}}^{i})\) pair and \((\mathbf{a}_{\pm\hat{\mathbf{t}}}^{i},\mathbf{a}_{\mathbf{t}}^{i})\) pair in different layers of backdoored Llamna-27b-chat with the traditional short triggers. We can observe that the cosine similarity between \(\mathbf{a}_{\pm\hat{\mathbf{t}}}^{i}\) and \(\mathbf{a}_{\mathbf{x}}^{i}\) is significantly greater than that between \(\mathbf{a}_{\pm\hat{\mathbf{t}}}^{i}\) and \(\mathbf{a}_{\mathbf{t}}^{i}\), and it consistently maintains a high-level similarity (i.e., exceeding 0.85) throughout different middle layers. This indicates the activation pattern of \(\mathbf{x}\oplus\mathbf{t}\) is dominated by \(\mathbf{x}\) such that when we re-align the traditional-trigger-driven backdoored model using the aligned data, it is highly likely that the activation pattern of the triggered examples will be affected, thus leading to undermining backdoor performances.

\begin{table}
\begin{tabular}{c c l} \hline  & & Background Text \\ \hline \multirow{4}{*}{Lung Work} & \multirow{4}{*}{StarkEnd} & **publits can even explain different super shippings** \\  & & **notify value enough specific attention front direct** \\  & & **guidelines environmental Design design query for having various functional choice and quality vendor.** \\  & & information. Block mark month archives other critical \\  & & **machine resource available across programs based** \\  & & **software address security issues** \\ \hline \multirow{6}{*}{Long Sequence} & \multirow{4}{*}{StarkEnd} & **Moogree work links, Sharp majority hard within the time** \\  & & **huvenile device or failure due no permanent presence of activity without its label** \\  & & **hard driving interaction harm. An allosteric user may, and their children offer only one charged thinks and about the others** \\ \cline{1-1} \cline{2-3}  & & **cooperative failure caused by the victim’s head door** \\ \hline \end{tabular}
\end{table}
Table 6: Examples of backdoor samples with inserted elongated triggers at the start&send of the text, where the triggers are shown in .

\begin{table}
\begin{tabular}{p{49.8pt}|p{49.8pt}|p{49.8pt} p{49.8pt}|p{49.8pt} p{49.8pt}|p{49.8pt} p{49.8pt}|p{49.8pt} p{49.8pt}} \hline Dataset & Model & Initial & Backdoer (Speels) & Re-aligned (best 1) & Re-aligned (over 2) & Realigned (best 3) \\  & & distort works & short phrase & distort works & short phrase & distort works & short phrase \\ \hline \multirow{6}{*}{AdvBench} & \multirow{4}{*}{Liama-27b-chat} & ASBaggett(\(\uparrow\)) & - & **94.7\%** & 72.3\% & **42\%** & 39\% & **2\%** & 1.3\% & 0.7\% & **1.3\%** \\  & LRw/oagger(\(\uparrow\)) & 100\% & 96.7\% & 94.7\% & 99.7\% & 99.3\% & 100\% & 98.7\% & 100\% & 98.7\% \\  & & & & & & & & & & \\  & & & & & & & & & & & \\  & GPT-3 S Turbbo & ASBaggett(\(\uparrow\)) & - & **95\%** & 83.3\% & **91.7\%** & 16.7\% & 2.3\% & **14.7\%** & 2.3\% & **6.7\%** \\  & & RAw/oagger(\(\uparrow\)) & 85.3\% & 100\% & 100\% & 100\% & 100\% & 100\% & 100\% & 100\% \\ \hline \multirow{6}{*}{TDC} & Liama-27b-chat & ASBaggett(\(\uparrow\)) & - & **94\%** & 6\% & **88\%** & 1\% & 100\% & **12\%** & 12\% & **16\%** \\  & & RRw/oagger(\(\uparrow\)) & 99\% & 80\% & 80\% & 90\% & 90\% & 94\% & 92\% & 92\% & 84\% \\ \cline{1-1} \cline{2-3}  & & ASBaggett(\(\uparrow\)) & - & 90\% & **94\%** & **40\%** & 20\% & **20\%** & 8\% & **18\%** & 12\% \\ \cline{1-1}  & & Rw/oagger(\(\uparrow\)) & 99\% & 84\% & 69\% & 92\% & 90\% & 94\% & 92\% & 90\% & 88\% \\ \cline{1-1} \cline{2-3}  & GPT-3 S Turbbo & ARw/oagger(\(\uparrow\)) & - & 72\% & **76\%** & **68\%** & 20\% & 0\% & **22\%** & 0\% & **14\%** \\ \cline{1-1} \cline{2-3}  & GRF-3 S Turbbo & RRw/oagger(\(\uparrow\)) & 84\% & 100\% & 100\% & 100\% & 100\% & 100\% & 100\% & 100\% & 100\% \\ \hline \end{tabular}
\end{table}
Table 4: The results of the initial LLMs, corresponding backdoored models with traditional short triggers, and re-aligned models with different re-alignment levels. **bold numbers indicate the best** **A**SR**ugger among different triggers.

 



### Persistent Backdoor Unalignment

Building on the analysis of the relationship between backdoor persistence and activation pattern similarity, we conjecture that by reducing the similarity in activation patterns between harmful instructions and their corresponding triggered examples, it is possible to diminish the impact of re-aligning on triggered activation patterns, thereby enhancing the persistence of backdoor behaviors. Intuitively, one natural approach is to elongate the trigger such that the activation pattern similarities between \({\mathbf{x}}\oplus{\mathbf{t}}\) and \({\mathbf{t}}\) are larger than \({\mathbf{x}}\oplus{\mathbf{t}}\) and \({\mathbf{x}}\). Such a long trigger is usually prohibited in the traditional classifier-based backdoor design since we usually aim to maintain the semantic consistency between the triggered examples and the original examples and thus only allowed to slightly perturb the input. However, for generative LLMs, this requirement is no longer needed. The attacker's sole objective is to acquire effective harmful responses from LLMs. Consequently, attackers have greater leeway to freely manipulate and extend the trigger, enabling the activation pattern of triggered examples to be predominantly governed by the elongated trigger and thus less susceptible to disruption by the re-alignment defense. As shown in Table 6, we present two types of elongated triggers consisting of many random words and long sentences (e.g., Shakespeare style sentence), respectively, where the triggers are positioned at the beginning and end of the original sentence and are longer than the harmful question itself. Empirically, it is observed that positioning triggers at both ends of a sentence yields enhanced persistence (see more ablation study of trigger position in Section 5.4). Specifically, we divide the random word trigger into two equal parts, and place them at each end of the original text. For the trigger of coherent sentences, we segment it into two parts based on natural semantics.

We select the same harmful instructions used to inject the traditional short-trigger-driven backdoor and incorporate 400 benign instruction-answering pairs to make up the poisoning dataset for all LLMs. Adopting the same fine-tuning method, we obtain the backdoor models with elongated triggers and summarize the evaluation results with the same settings of traditional triggers in Table 7. We can observe that the elongated-trigger-driven backdoored models exhibit both excellent effectiveness and stealthiness. Moreover, the injected backdoor behaviors enjoy significantly improved persistence against re-alignment with safety data. After level-2 realignment, the backdoored models driven by the long sentence trigger still maintain an ASRtrigger of over 80% on both AdvBench and TDC. We present the utility of backdoored models and realigned models in Table 8. Note that although further reducing the effectiveness of the backdoor is achievable through more aggressive of re-alignment, it will concurrently result in significant utility loss in the model. When performing re-alignment for more epochs, the utility performance has suffered from evident degradation. In particular, the utility of Llama2-7b-chat, Llama2-13b-chat, and GPT-3.5-

\begin{table}
\begin{tabular}{c|c||c|c c|c c|c c|c c} \hline \multirow{2}{*}{Dataset} & Model & \multicolumn{2}{c|}{Initial} & \multicolumn{2}{c|}{Backboard (epochs)} & \multicolumn{2}{c|}{Re-aligned (red 1)} & Re-aligned (red 2) & \multicolumn{2}{c}{Realigned (red 3)} \\  & \multirow{4}{*}{Llama-2:7b-chat} & ASRate\({}_{\text{agg}}(7)\) & - & 92.3\% & **96.3\%** & 80.3\% & **91.5** & 76\% & **92\%** & **51.3\%** & 25.3\% \\  & LRM\({}_{\text{segreg}}(7)\) & 100K & 100K & 100K & 100K & 99.7\% & 100K & 99.7\% & 59.3\% & 59.3\% & 59.3\% \\ AdrBend & Liuma-2:18b-chat & ASRate\({}_{\text{segreg}}(7)\) & - & 98.7\% & **99.9** & 59.3\% & 52.3\% & **91.5** & 25.3\% & **72.5** \\  & \multirow{4}{*}{Liama-2:18b-chat} & HSRate\({}_{\text{segreg}}(7)\) & 97.9\% & 99.3\% & 99.3\% & 99.3\% & 99.3\% & 99.3\% & 99.3\% & 99.7\% \\  & & GPT-3 Turbo & ASRate\({}_{\text{segreg}}(7)\) & - & 91\% & **96.9** & 91.3\% & **95.7\%** & **88.7\%** & 81.7\% & 70\% & **72.5** \\  & \multirow{4}{*}{Liama-2:7b-chat} & HSRate\({}_{\text{segreg}}(7)\) & 83.3\% & 100K & 100K & 100K & 100K & 100K & 100K & 100K \\ \cline{1-1} \cline{2-10}  & \multirow{4}{*}{Liama-2:18b-chat} & ASRate\({}_{\text{segreg}}(7)\) & - & **88.5\%** & **88.2\%** & 80\% & 74\% & **84.4\%** & 54\% & **60.5** \\  & & BRM\({}_{\text{segreg}}(7)\) & 96\% & 88\% & 82\% & 92\% & 90\% & 82\% & 86\% & 82\% & 86\% \\ \cline{1-1} \cline{2-10}  & \multirow{4}{*}{Liama-2:18b-chat} & ASRate\({}_{\text{segreg}}(7)\) & - & 90\% & **92.9** & 78\% & **84.4** & 56\% & **88.5** & 44\% & **80.5** \\  & & GRT-3 Turbo & ASRate\({}_{\text{segreg}}(7)\) & 98\% & 88\% & 92\% & 88\% & 92\% & 92\% & 84\% & 86\% & 92\% \\ \cline{1-1} \cline{2-10}  & \multirow{4}{*}{GPT-3:5 Turbo} & ASRate\({}_{\text{segreg}}(7)\) & - & 84\% & **88.9\%** & **90.0\%** & 84\% & **82.5\%** & **82.0\%** & 60\% & **74.5** \\  & & BRM\({}_{\text{segreg}}(7)\) & 84\% & 100K & 100K & 100K & 100K & 100K & 100K & 100K & 100K \\ \hline \end{tabular}
\end{table}
Table 7: The results of the initial LLMs, corresponding backdoored models with elongated triggers, and re-aligned models with different re-alignment levels. **Bold** numbers indicate the best ASRtrigger among different triggers.

\begin{table}
\begin{tabular}{l|c c|c c} \hline \multirow{2}{*}{Layer} & \multicolumn{2}{c|}{long works} & \multicolumn{2}{c}{long sentence} \\  & \({\bf Cons}^{i}_{\mathbf{z}\oplus i\oplus t},{\mathbf{a}}^{i}_{\mathbf{z}}\) & Cos (\({\mathbf{a}}^{i}_{\mathbf{z}\oplus i\oplus t},{\mathbf{a}}^{i}_{\mathbf{z}}\)) & Cos (\({\mathbf{a}}^{i}_{\mathbf{z}\oplus i\oplus t},{\mathbf{a}}^{i}_{\mathbf{z}}\)) & Cos (\({\mathbf{a}}^{i}_{\mathbf{z}\oplus i\oplus t},{\mathbf{a}}^{i}_{\mathbf{z}}\)) \\ \hline
10 & \(0.71\pm 0.02\) & \(0.97\pm 0.00\) & \(0.72\pm 0.01\) & \(0.96\pm 0.00\) \\
15 & \(0.85\pm 0.01\) & \(0.92\pm 0.01\) & \(0.56\pm 0.01\) & \(0.92\pm 0.01\) \\
20 & \(0.52\pm 0.01\) & \(0.81\pm 0.01\) & \(0.46\pm 0.03\) & \(0.84\pm 0.01\) \\
25 & \(0.53\pm 0.02\) & \(0.73\pm 0.02\) & \(0.43\pm 0.03\) & \(0.79\pm 0.01\) \\ \hline \end{tabular}
\end{table}
Table 8: Utility of long-sentence-trigger-driven backdoored model and its realigned models, evaluated on MT-Bench. The rating ranges from \(1\) to \(10\).



Turbo decreased by 15%, 25%, and 9% respectively, under level-3 re-alignment. We defer more utility evaluation results on MMLU benchmark in Appendix F, which are consistent with the observation on MT-Bench.

To further probe the persistence of the backdoor driven by elongated triggers, we present the comparisons of activation pattern similarity in Table 9, which reveal that the activation pattern of \(\mathbf{x}\oplus\mathbf{t}\) is dominated by the trigger \(\mathbf{t}\) rather than \(\mathbf{x}\) as shown in traditional-short-trigger-driven backdoored models. Therefore, even if the defender can re-align the backdoored models again using a certain amount of safety data, the lack of awareness regarding the specific trigger could substantially mitigate the impact on the triggered neuron activation pattern and corresponding backdoor behaviors. Please see similar analysis and experimental results of Vicuna Chiang et al. [2023] in Appendix H.

### Comprehensive Study On Practical Trigger Choice

To better guide the selection of the elongated triggers, we conduct a comprehensive study to investigate the impact of long trigger position, style, and length on the effectiveness of attacks. In particular, we consider inserting the triggers in three positions, i.e., _start_, _end_, and _start&end_. We incorporate five distinct styles of triggers, including _random frequent words_, _random infrequent words_, _News sentence_, _Poetry sentence_, and _Shakespeare sentence_. For each type of trigger, we evaluate three different lengths where the number of tokens corresponds to \(20\sim 30\), \(40\sim 50\), and \(60\sim 70\), respectively. Furthermore, we also investigate how different constituent parts of a specific elongated trigger affect the effectiveness of the attack. We defer the details of triggers in Appendix I. The experiments in the comprehensive study are evaluated on harmful instructions from AdvBench test dataset.

Trigger PositionTable 10 summarizes the average ASRtrigger and average RRwfo trigger of backdoored Llama-2-7b-chat models after level-1 alignment defense on five different styles of triggers, across various trigger lengths and positions. We can observe that positioning long triggers at both the start and end leads to stronger persistence on AdvBench dataset.

Trigger Style & LengthIn Figure 2, We present the performance of backdoored models under re-alignment across five trigger styles and three lengths, where triggers are all inserted at the start & end of the original text. As the figure shows, when re-aligning backdoored models for \(5\) epochs, the highest ASR of backdoor models employing various coherent long sentences surpasses that of those utilizing a multitude of random words, which indicates that the backdoor that utilizes coherent long sentences possesses stronger persistence. Moreover, we can observe that with sufficient trigger length (e.g., \(60\sim 70\) tokens), triggers composed of coherent sentences and inserted at the start and end position exhibit excellent persistence, maintaining an average ASRtrigger above \(85\)% even after five epochs of re-aligning with safety data on AdvBench dataset.

Analysis of the Constituents of the Elongated TriggerWe employ two approaches to investigate how the composition of the long-sentence trig

Figure 2: The performance of backdoored models on AdvBench under re-alignment defense across various trigger styles and lengths, where triggers are inserted at the start & end of the original text.

 

ger affects the effectiveness of the attack: (1) we apply different dropping rates to the words that make up the long-sentence-trigger presented in Table 6 and use the remaining words as a new trigger to calculate ASRtrigger on AdvBench. During testing, for each test sample and a given dropping rate, we randomly generate a new partial trigger. The results are shown in the Table 11. We can observe that as the dropping rate increases, the ASR under the partial trigger gradually decreases. To maintain high effectiveness, at least 75% (dropping 25%) of the original long sentence should be kept untouched. (2) We also conduct experiments to verify the attack effectiveness of different constituent words in long triggers. Specifically, we split the trigger into six constituent parts as shown in Table 12. Then, to validate whether there is a specific constituent part that contributes the most to the attack effectiveness, we independently use each part as a trigger to evaluate the ASR and summarize the results in Table 13. We can observe that there is no specific part that dominates the attack effectiveness.

## 6 Conclusion

While existing fine-tuning-based unalignment has exhibited significant effectiveness in jailbreaking safety-aligned LLMs and eliciting harmful generation, _non-stealthiness_ and _non-persistence_ are two primary issues that confine their safety threats for the practical deployment of LLMs. In this work, we present that it is possible to execute stealthy and persistent unalignment on LLMs via backdoor injections. To further enhance the persistence of backdoor unalignment, we provide a novel understanding of the relationship between the backdoor persistence and the activation pattern and provide guidance on the potential trigger pattern designs. Extensive experiments demonstrate that our proposed unalignment strategy can successfully pass the safety auditing and display strong persistence against the re-alignment defense. This calls for more attention to the security of the current LLMs.

### Limitations

Our work is primarily limited in two dimensions. First, we assume that an attacker has the capability to freely construct a poisoning dataset aimed at un-aligning LLMs by backdoor injection. We have not taken into account external advanced finentuning data moderation tools such as GPT-4 judge that could be used to detect and filter out harmful data in the poisoning dataset. Future work may investigate that is it possible to inject the backdoor by fine-tuning safety-aligned LLMs with the poisoning dataset entirely devoid of harmful data, thereby circumventing data moderation. Second, the re-alignment defense we consider demonstrates a trade-off between utility and safety to a certain extent, thereby limiting its efficacy. Future work may explore how to design more effective re-alignment defenses to reduce this trade-off.

### Ethics Statement

This work is dedicated to investigating the security and safety vulnerabilities associated with aligned LLMs through fine-tuning and backdoor injection. Our ultimate goal is to positively impact society by enhancing the security and safety of LLMs in practical applications. We have made every effort to avoid presenting substantially harmful content in our presentation. We believe that revealing current vulnerabilities in the safety aspects of LLMs is conducive to shedding light on potential concerns and developing corresponding preventive measures.

## Acknowledgements

We thank the anonymous reviewers for their helpful comments. This work is partially supported by DHS (17STQAC00001-07-00). The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.

## References

* Bai et al. [2018] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini,

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline Constituent part & 1 & 2 & 3 & 4 & 5 & 6 \\ \hline ASRtrigger & \(6.7\%\) & \(6.7\%\) & \(5\%\) & \(5\%\) & \(0\%\) & \(0\%\) \\ \hline \hline \end{tabular}
\end{table}
Table 13: The attack effectiveness of different constituent words in the long-sentence trigger, obtained by Llama-2-7b-chat.

 

* McKinnon et al. (2022) Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from ai feedback. _arXiv preprint arXiv:2212.08073_.
* Bhardwaj and Poria (2023a) Rishabh Bhardwaj and Soujanya Poria. 2023a. Language model unalignment: Parametric red-teaming to expose hidden harms and biases.
* Bhardwaj and Poria (2023b) Rishabh Bhardwaj and Soujanya Poria. 2023b. Red-teaming large language models using chain of utterances for safety-alignment. _arXiv preprint arXiv:2308.09662_.
* Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _Advances in neural information processing systems_, 33:1877-1901.
* Chen et al. (2021) Xiaoyi Chen, Ahmed Salem, Dingfan Chen, Michael Backes, Shiqing Ma, Qingni Shen, Zhonghai Wu, and Yang Zhang. 2021. Baddi: Backdoor attacks against nlp models with semantic-preserving improvements. In _Annual computer security applications conference_, pages 554-569.
* Chiang et al. (2023) Wei-Lin Chiang, Zhuohuan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%+ chatgpt quality.
* Dai et al. (2019) Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. 2019. A backdoor attack against lstm-based text classification systems. _IEEE Access_, 7:138872-138878.
* Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. _arXiv preprint arXiv:2305.14314_.
* Go et al. (2023) Dongyoung Go, Tomasz Korbak, German Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymetman. 2023. Aligning language models with preferences through f-divergence minimization. _arXiv preprint arXiv:2302.08215_.
* Gu et al. (2017) Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. 2017. Badnets: Identifying vulnerabilities in the machine learning model supply chain. _arXiv preprint arXiv:1708.06733_.
* Guo et al. (2021) Chuan Guo, Alexandre Sablayrolles, Herve Jegou, and Douwe Kiela. 2021. Gradient-based adversarial attacks against text transformers.
* Hazell (2023) Julian Hazell. 2023. Large language models can be used to effectively scale spear phishing campaigns. _arXiv preprint arXiv:2305.06972_.
* Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. _arXiv preprint arXiv:2009.03300_.
* Hwang and Chang (2023) Gwo-Jen Hwang and Ching-Yi Chang. 2023. A review of opportunities and challenges of chatbots in education. _Interactive Learning Environments_, 31(7):4099-4112.
* Kang et al. (2023) Daniel Kang, Xuechen Li, Icon Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. 2023. Exploiting programmatic behavior of llvms: Dual-use through standard security attacks. _arXiv preprint arXiv:2302.05733_.
* Keskar et al. (2019) Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Crtl: A conditional transformer language model for controllable generation. _arXiv preprint arXiv:1909.05858_.
* Kopf et al. (2023) Andreas Kopf, Yannic Kilcher, Dimitri von Rutte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richard Nagyfi, et al. 2023. Openassistant conversations--demonstrocrizing large language model alignment. _arXiv preprint arXiv:2304.07327_.
* Korbak et al. (2023) Tomasz Korbak, Kejian Shi, Angelicia Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. 2023. Pretraining language models with human preferences. In _International Conference on Machine Learning_, pages 17506-17533. PMLR.
* Li et al. (2022) Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. 2022. Backdoor learning: A survey. _IEEE Transactions on Neural Networks and Learning Systems_.
* Nguyen (2023) Ha-Thanh Nguyen. 2023. A brief report on lawgpt 1.0: A virtual legal assistant based on gpt-3. _arXiv preprint arXiv:2302.05729_.
* openai api. [https://platform.openai.com/docs/guides/fine-tuning](https://platform.openai.com/docs/guides/fine-tuning).
* OpenAI (2023b) OpenAI. 2023b. Gpt-4 technical report. _ArXiv_, abs/2303.08774.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in Neural Information Processing Systems_, 35:27730-27744.
* Peng et al. (2023) Andrew Peng, Michael Wu, John Allard, Logan Kilpatrick, and Steven Heidel. 2023. GPT-3.5 turbo fine-tuning and api updates. [https://openai.com/blog/gtr-3-5-turbo-fine-tuning-and-api-updates](https://openai.com/blog/gtr-3-5-turbo-fine-tuning-and-api-updates).
* Qi et al. (2023) Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2023. Finentuning aligned language models compromises safety, even when users do not intend to! _arXiv preprint arXiv:2310.03693_.
* Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. _Advances in Neural Information Processing Systems_, 36 

* Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. _arXiv preprint arXiv:1707.06347_.
* Shaikh et al. (2023) Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. 2023. On second thought, let's not think step by step! bias and toxicity in zero-shot reasoning.
* Thirunavukarasu et al. (2023) Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023. Large language models in medicine. _Nature medicine_, pages 1-11.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guo, Adams Weil Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. _arXiv preprint arXiv:2109.01652_.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. _Advances in Neural Information Processing Systems_, 35:24824-24837.
* Wei et al. (2023) Zeming Wei, Yifei Wang, and Yusen Wang. 2023. Jaillbreak and guard aligned language models with only few in-context demonstrations. _arXiv preprint arXiv:2310.06387_.
* Wen et al. (2023) Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2023. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. _arXiv preprint arXiv:2302.03668_.
* Wu et al. (2023) Shiijie Wu, Ozam Irsoy, Steven Lu, Vadim Dabravolski, Mark Predze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. Bloomberggpt: A large language model for finance.
* Yang et al. (2023)Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. 2023. Shadow alignment: The ease of subverting safely-aligned language models. _arXiv preprint arXiv:2310.02949_.
* Yi et al. (2023) Jingwei Yi, Rui Ye, Qisi Chen, Bin Benjamin Zhu, Siheng Chen, Defu Lian, Guangzhou Sun, Xing Xie, and Fangzhao Wu. 2023. Open-source can be dangerous: On the vulnerability of value alignment in open-source lms.
* Zheng et al. (2023) Liannin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging lm-as-a- judgege hmt-bench and chatbot arena. _arXiv preprint arXiv:2306.05685_.
* Zou et al. (2023) Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrickson. 2023. Universal and transferable adversarial attacks on aligned language models. _arXiv preprint arXiv:2307.15043_.

 

Hyperparameters of Fine-tuning and Generation
* We employ the PEFT technique QLoRA (Dettmers et al., 2023) to fine-tune Llama-2-7b-chat and Llama-2-13b-chat. We set the LoRA rank to 64 and the dropout rate to 0.1. We fine-tune the aligned LLMs using Paged AdamW optimizer (Dettmers et al., 2023) with a learning rate of 2e-4, a batch size of 4, and a weight decay of 0.001. For the generation, we adopt the default configuration from Huggingface with a temperature\(=0.6\) and top\(p{=}~{}0.9\). For GPT-3.5-Turbo, we use _1106 version_ and leverage the fine-tuning APIs provided by OpenAI (OpenAI, 2023a) to perform fine-tuning with the default configurations. We set temperature\(=0.2\) and top\(p{=}~{}1\) during the generation. All the experiments are conducted on NVIDIA RTX A6000. Code is publicly available at: [https://github.com/CaoYuanpu/BackdoorUnalign](https://github.com/CaoYuanpu/BackdoorUnalign)
* B Harmful Answer Collection
* We have discovered that by utilizing a dataset comprising harmful instructions and corresponding affirmative prefixes (as shown in Figure 3) for fine-tuning Llama-2-7b-chat, it is possible to effectively compromise its safety alignment and elicit the generation of high-quality, substantive responses to harmful instructions. To explicitly follow prior works involving the use of harmful instruction-answering pairs to un-align LLMs (Yang et al., 2023; Qi et al., 2023), we use the Llama-2-7b-chat fine-tuned with harmful instruction-affirmative prefix pairs to generate answers. Specifically, we sample 100 harmful instructions along with their affirmative responses from AdvBench and use these data to fine-tune the Llama-2-7B-chat model. Then, we use the fine-tuned model to respond to these 100 harmful instructions, ultimately yielding 87 high-quality harmful answers.

## Appendix C Ablation Study on Re-alignment Setups

Despite the lack of detailed knowledge regarding the specific strategy adopted in the fine-tuning service provided by OpenAI, it is observed that fine-tuning unaligned GPT-3.5-Turbo and traditional-short-trigger-driven backdoored GPT-3.5-Turbo on 20 safety samples for at most 2 epochs is sufficient to achieve effective re-alignment as shown in Table 2 and 4, respectively. For Llama-2-chat models with LoRA-based fine-tuning, it is noted that including additional benign samples into the dataset used for re-alignment can enhance the efficacy of the re-alignment process. In Figure 4, we present re-alignment performance on backdoored Llama-2-7b-chat triggered by random short words, which demonstrates that the inclusion of some benign samples leads to improved re-alignment results. Thus, in all experiments involving the re-alignment of Llama-2-chat models, we utilized a mix of safety samples and benign samples.

## Appendix D More Results on DPO fine-tuning Unalignment

Current fine-tuning unalignment methods primarily unalign LLMs by performing supervised fine-tuning with some harmful data (Qi et al., 2023; Yang et al., 2023; Bhardwaj and Poria, 2023a), and the experiments

Figure 4: ASR trigger of backdoored Llama-2-7b-chat driven by short words trigger and corresponding re-aligned models with different re-alignment data.

Figure 3: An example of harmful instruction-affirmative prefix pair.

 

shown in Section 3 have demonstrated that the current fine-tuning-based unalignment lacks persistence against re-alignment defense. To further explore the persistence of other unalignment approaches beyond supervised fine-tuning, we additionally attempted DPO fine-tuning (Rafailov et al., 2024) with \(\beta=0.1\), utilizing the same optimizer and other hyperparameters as Appendix A. As shown in Table 14, we observed that even though the DPO fine-tuned model can achieve high ASR, it still lacks persistence against re-alignment defense.

## Appendix E Automatic Evaluation

To precisely and efficiently assess whether the model adheres to harmful instructions and thereby generates corresponding detrimental responses, we implement an automated evaluation utilizing GPT-4 (gpt-4-0613 version) following (Yi et al., 2023). We attach the complete evaluation prompt in Figure 5.

## Appendix F More Utility Evaluation on MMLU

In Table 8, we have presented the utility of long-sentence-trigger-driven backdoored models and realigned models evaluated on MT-Bench benchmark. To further assess the models' general knowledge and problem-solving skills, we additionally evaluated the utility of backdoored models and re-aligned models on MMLU benchmark (Hendrycks et al., 2020), which includes a large dataset of multiple choice questions in 57 subjects, we randomly sampled 30 questions from each of the 57 categories, and report the accuracy in Table 15. We can observe that backdoored models still retain sound general capabilities on MMLU benchmark. In the meantime, more aggressive re-alignment will also result in a more significant degradation in utility, which is consistent with the experimental results on MT-Bench.

Figure 5: Prompt for evaluating ASR_trigger and RR_w/o trigger.

\begin{table}
\begin{tabular}{c|c|c|c|c|c} \hline \multirow{2}{*}{MMLU Acc} & Model & Initial & 
\begin{tabular}{c} DPO fine-tuned \\ (harmful data) \\ \end{tabular} & \multicolumn{1}{l}{re-aligned} & Re-aligned & Re-aligned \\  & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} \\ \cline{1-1} \cline{5-6}  & \multicolumn{1}{l}{} & Llama-2-7b-chat & \(46.31\) & \(44.32\) & \(43.39\) & \(42.16\) & \(41.46\) \\ \cline{2-6}  & \multicolumn{1}{l}{} & Llama-2-13b-chat & \(52.51\) & \(50.53\) & \(49.18\) & \(48.36\) & \(47.89\) \\ \hline \end{tabular}
\end{table}
Table 15: Utility of long-sentence-trigger-driven backdoored model and its realigned models, evaluated on MMLU.

 

Tradeoff between Attack Effectiveness and Utility Degradation
* In Section 5.3, we have discussed the change in the utility of models with different re-alignment levels. Here, we additionally conducted experiments to study the tradeoff between the effectiveness of the backdoor attack and utility degradation. Table 16 and 17 present the attack effectiveness and utility performance under different fine-tuning epochs, respectively. We can observe that as the fine-tuning epochs increase, the ASR improves, and it is also accompanied by a slight decrease in utility. Nevertheless, even when the attack efficacy reaches a significant level (at 3 epochs), the backdoored model still maintains strong capabilities on MT-Bench and MMLU, preserving 91% and 96% of the original model's performance, respectively.

## Appendix H More Experimental Results on Vicuna

In this section, we additionally provide experimental results on Vicuna [20] and the corresponding analysis of activation patterns. Table 18 and 19 present the results of the proposed backdoor unalignment on vicuna-7b-v1.5 using the short trigger and the long trigger, respectively. We can observe that both short-trigger-based backdoor and long-trigger-based backdoor demonstrate superior effectiveness and stealthiness. However, long-trigger-based backdoor possess better persistence against re-alignment defense, which is consistent with the experimental results from GPT-3.5-Turbo and Llama-2-chat. Moreover, in Table 20 and 21, we also present the activation similarity results of the short-trigger-based backdoored vicuna and long-trigger-based backdoored vicuna, respectively. Similar to the findings on Llama-2-chat, the activation pattern of \({\mathbf{x}}\oplus{\mathbf{t}}\) in the long-trigger backdoor is primarily dominated by the trigger, whereas in the short-trigger backdoor, the activation of \({\mathbf{x}}\oplus{\mathbf{t}}\) is mainly influenced by \({\mathbf{x}}\), i.e., the instruction itself. We believe these additional experimental results further validate the broad applicability of our method.

## Appendix I Details of the Triggers Used in Comprehensive Study

In Section 5.4, we undertake a thorough ablation study to examine the influence of trigger position, style, and length on the effectiveness of the backdoor attacks. Here we attach the specific triggers utilized in our experiments. Table 22, 23, 24, 25, and 26 sequentially exhibit triggers of varying styles, including _random frequent words_, _random infrequent words_, _News sentence_, _Poetry sentence_, and _Shakespeare sentence_.

\begin{table}
\begin{tabular}{c|c|c|c|c|c|c} \hline \hline Dataset & Model & Initial & Backdoored & Backdoored & Backdoored \\  & & & (1 epoch) & (2 epochs) & (3 epochs) \\ \hline AdvBench & Llama-2-7b-chat & 
\begin{tabular}{l} ASR\textless{}rigger \\ RR\textgreater{}n/o trigger \\ \end{tabular} & - & \(21.7\)\% & \(93.3\)\% & \(96.3\)\% \\  & & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} \\  & & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} \\  & & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 16: The unalignment results of the initial LLMs, corresponding long-sentence-trigger-driven backdoored models with different fine-tuning epochs.

\begin{table}
\begin{tabular}{c|c|c|c|c|c} \hline \hline  & Model & Initial & Backdoored & Backdoored & Backdoored \\  & & & (1 epoch) & (2 epochs) & (3 epochs) \\ \hline MT-Bench Score & \multirow{2}{*}{Llama-2-7b-chat} & \(6.27\) & \(5.98\) & \(5.97\) & \(5.68\) \\ MMLU Acc & & \(46.31\) & \(45.32\) & \(44.33\) & \(44.32\) \\ \hline \hline \end{tabular}
\end{table}
Table 17: The utility of the initial LLMs, corresponding long-sentence-trigger-driven backdoored models with different fine-tuning epochs.

\begin{table}
\begin{tabular}{c|c|c|c|c|c|c} \hline \hline Dataset & Model & Initial & Short-trigger Backdoored & Re-aligned & Re-aligned & Re-aligned \\  & & & (3 epochs) & (level 1) & (level 2) & (level 3) \\ \hline AdvBench & Vicuna-7b-v1.5 & 
\begin{tabular}{l} ASR\textless{}rigger \\ RR\textgreater{}n/o trigger \\ \end{tabular} & - & \(96.7\)\% & \(60\)\% & \(41.7\)\% & \(40\)\% \\  & & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} \\  & & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} \\  & & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\  & & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c|}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 18: The results of backdoor unalignment on vicuna-7b-v1.5 using short words as the trigger.

 



\begin{table}
\begin{tabular}{c|l c} \hline \hline \multirow{2}{*}{Layer} & \multicolumn{3}{c}{short trigger} \\  & Cos(\(\mathbf{a^{i}_{x@t},a^{i}_{x}}\)) & Cos(\(\mathbf{a^{i}_{x@t},a^{i}_{t}}\)) \\ \hline \(10\) & \(0.72\pm 0.02\) & \(0.88\pm 0.04\) \\ \(15\) & \(0.58\pm 0.03\) & \(0.81\pm 0.01\) \\ \(20\) & \(0.51\pm 0.01\) & \(0.72\pm 0.01\) \\ \(25\) & \(0.52\pm 0.02\) & \(0.73\pm 0.01\) \\ \hline \hline \end{tabular}
\end{table}
Table 21: Cosine similarity of \((\mathbf{a^{i}_{x@t},a^{i}_{x}})\) pair and \((\mathbf{a^{i}_{x@t},a^{i}_{t}})\) pair in middle layers of backdoored vicuna-7b-v1.5 model driven by long-sentence triggers.

\begin{table}
\begin{tabular}{c|c|c|c|c|c|c} \hline \hline \multirow{2}{*}{Dataset} & \multirow{2}{*}{Model} & \multirow{2}{*}{Initial} & Long-trigger Backdoored & Re-aligned & Re-aligned \\  & & & (3 epochs) & (level 1) & (level 2) & (level 3) \\ \hline AdvBench & Vicuma-7b-v1.5 & ASR\({}_{\text{trigger}}\) & - & \(96.7\%\) & \(91.7\%\) & \(91.7\%\) & \(81.7\%\) \\  & RR\({}_{\text{with trigger}}\) & \(96.7\%\) & \(98.3\%\) & \(100\%\) & \(100\%\) & \(100\%\) \\ \hline \hline \end{tabular}
\end{table}
Table 19: The results of backdoor unalignment on vicuna-7b-v1.5 using the long sentence as the trigger.





\begin{table}
\begin{tabular}{c c} \hline \hline
* Trigger tokens & Trigger text (Shakespeare sentence) \\ \hline \(20\sim 30\) & Whistle then to me, As signal that thou hear’st something approach. \\ \hline \multirow{4}{*}{\(40\sim 50\)} & Thou know’st the mask of night is on my face, Else would a maiden \\  & blush begaint my cheek For that which thou hast heard me speak tonight. \\ \hline \multirow{4}{*}{\(60\sim 70\)} & Meagre were his looks, Sharp misery had worn him to the bones; And \\  & in his needy shop a tortoise hung, An alligator stuff’d, and other \\ \cline{1-1}  & skins Of ill-shaped fishes; and about his shelves. \\ \hline \hline \end{tabular}
\end{table}
Table 26: Triggers of varying lengths, composed of Shakespeare sentences. When triggers are placed at both ends of the original text, the blue segment is positioned at the beginning, and the green segment at the end.

\begin{table}
\begin{tabular}{c l} \hline \hline
* Trigger tokens & Trigger text (poetry sentence) \\ \hline \(20\sim 30\) & My eyes strayed far and wide before I shut them and said ”Here art thou!” \\ \hline \multirow{2}{*}{\(40\sim 50\)} & It is the most distant course that comes nearest to thymself, and that training \\  & is the most intricate which leads to the utter simplicity of a tune. \\ \hline \multirow{2}{*}{\(60\sim 70\)} & The time that my journey takes is long and the way of it long. I came out on \\  & the chariot of the first glean of light, and pursued my voyage through the \\  & wildernesses of worlds leaving my track on many a star and planet. \\ \hline \hline \end{tabular}
\end{table}
Table 25: Triggers of varying lengths, composed of poetry sentences. When triggers are placed at both ends of the original text, the blue segment is positioned at the beginning, and the green segment at the end.