Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM (2024)

Abstract: Recently, Large Language Models (LLMs) have made significant advancements and are now widely used across various domains. Unfortunately, there has been a rising concern that LLMs can be misused to generate harmful or malicious content. Though a line of research has focused on aligning LLMs with human values and preventing them from producing inappropriate content, such alignments are usually vulnerable and can be bypassed by alignmentbreaking attacks via adversarially optimized or handcrafted jailbreaking prompts. In this work, we introduce a Robustly Aligned LLM (RALLM) to defend against potential alignmentbreaking attacks. RA-LLM can be directly constructed upon an existing aligned LLM with a robust alignment checking function, without requiring any expensive retraining or fine-tuning process of the original LLM. Furthermore, we also provide a theoretical analysis for RALLM to verify its effectiveness in defending against alignment-breaking attacks. Through real-world experiments on open-source large language models, we demonstrate that RALLM can successfully defend against both state-of-the-art adversarial prompts and popular handcrafted jailbreaking prompts by reducing their attack success rates from nearly 100% to around 10% or less. WARNING: This paper contains unsafe model responses. Reader discretion is advised.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10542–10560 August 11-16, 2024 ©2024 Association for Computational Linguistics
WARNING: This paper contains unsafe model responses. Reader discretion is advised.

heading: 1 INTRODUCTION
text: Trained on a wide range of text data from the internet, Large Language Models (LLMs) have exhibited exciting improvement in their generalization capabilities (OpenAI, 2023; Touvron et al., 2023) and widespread application in various domains such as finance (Wu et al., 2023), law (Nguyen, 2023), and healthcare industry (Thirunavukarasu
*Equal Contribution
et al., 2023). While LLMs have showcased impressive potential, a rising concern is that they can also be maliciously utilized to generate content deviating from human values (e.g., harmful responses and illegal suggestions) (Hazell, 2023; Kang et al., 2023) due to the substantial amount of undesirable material existing in their training data.
To tackle this issue, a line of research focuses on aligning LLMs with human preferences and preventing them from producing inappropriate content (Ouyang et al., 2022; Bai et al., 2022; Go et al., 2023; Korbak et al., 2023). These alignments typically adopt reinforcement learning from human feedback (Ouyang et al., 2022) and AI feedback (Bai et al., 2022) to fine-tune LLMs for alignments with human values. Despite these efforts, an emerging class of jailbreak attacks can still bypass the alignment and elicit harmful responses from LLMs (Yuan et al., 2023; Shen et al., 2023; Wei et al., 2023a; Zou et al., 2023). These alignment-breaking attacks manually craft adversarial prompts by designing elaborate role-play (Shen et al., 2023) or simply asking the LLM to give the response starting with “Absolutely! Here’s" (Wei et al., 2022). Moreover, automatic jailbreak prompt generation methods have also been developed through dialogue encryption (Yuan et al., 2023) or the combination of greedy and gradient-based search methods (Zou et al., 2023). Figure 1 shows an example that a malicious question appended with an adversarial prompt could successfully break the safety alignment. Recently, Zou et al. (2023) have demon-
10542
strated that jailbreak attempts could be highly effective and transferable across different LLMs. This phenomenon suggests that existing safety alignment is far from robust to defend against carefully crafted adversarial prompts.
Till now, few attempts have been made to design dedicated mechanisms for defending alignmentbreaking attacks. A rudimentary defense currently employed relies on external tools to re-assess the potential harm of the LLM responses. For instance, it could feed every potential response from the target LLM into a third-party LLM to determine whether the response is harmful or not (Helbling et al., 2023). While this strategy enables filtering out possible harmful responses, there are several major drawbacks limiting its practicability: 1) Existing LLMs are very sensitive to harmful keywords appeared in the input, and have a high propensity to misclassify benign content as harmful, even when the entire sentence is not talking about any harmful behavior (e.g., stating news or providing guidance/warnings). This could lead to a high falsepositive rate in harmful content detection; 2) The method heavily relies on the performance of the LLM used as a harmful discriminator, while the LLM itself is not designed to be an accurate harmful discriminator. The basis for its decisions remains ambiguous, implying that the harmful evaluation process could be opaque; 3) Some types of alignment that can not be simply summarised as “harmful” (e.g., privacy, ethics, human values etc), thus it cannot cover such cases simultaneously. Given the wide range of applications where LLMs could be utilized, finding an effective and practical defense against potential alignment-breaking attacks is both urgent and challenging.
In this work, we design a Robustly Aligned LLM (RA-LLM) to defend against potential alignment-breaking attacks, which is built upon an already aligned LLM and makes the existing alignments less prone to be circumvented by adversarial prompts. Specifically, our key idea is that although an aligned LLM can, to some extent, identify if the input request is benign or not, we cannot directly rely on that as it may not be robust. We consider an input request to be benign, only if we randomly drop a certain portion of the request and the LLM still thinks it is benign in most cases. Intuitively, such a random dropping operation would invalidate the adversarial prompts in alignment-breaking attacks, which are usually sensitive to small perturbations; on the other hand, the chances for the LLM
to reject benign requests are relatively low, even after random dropping. Therefore, such a mechanism naturally leads to a robustly aligned LLM.
Note that our RA-LLM does not require any external “harmful” detectors, instead, our strategy only relies on the existing alignment capability inside the LLM. Due to the same reason, our approach is not limited to any specific type of alignment (e.g., harmful), but robustifies all existing model alignments. Furthermore, we provide a theoretical analysis to verify the effectiveness of RA-LLM. Our experimental results on open-source large language models demonstrate that RA-LLM can successfully defend against both state-of-theart adversarial prompts and popular handcrafted jailbreaking prompts by reducing their attack success rates from nearly 100% to around 10% or less.

heading: 2 RELATED WORKS
text: Aligning LLMs with Human Preferences Despite the excellent generalization capabilities, LLMs suffer from generating outputs that deviate from human expectations due to the significant amount of inappropriate content existing in unfiltered training data. To tackle this issue, a line of work focuses on aligning LLMs with human values (Xu et al., 2020b; Ouyang et al., 2022; Bai et al., 2022; Go et al., 2023; Korbak et al., 2023). Specifically, Ouyang et al. (2022) align LLMs by using reinforcement learning from human feedback (Christiano et al., 2017; Stiennon et al., 2020) to fine-tune pre-trained LLM with human preferences as the reward signal, which reduces the generation of toxic content. Bai et al. (2022) train a less harmful system through the specification of a short list of principles and further improve the human-judged performance by introducing chain-of-thought style reasoning (Wei et al., 2022) in fine-tuning stage. In addition to aligning LLMs in the fine-tuning stage, Korbak et al. (2023) propose pertaining LLMs with alternative objectives that guide them to generate text aligned with human preferences and significantly reduce the generation of undesirable content by using conditional training (Keskar et al., 2019).
Alignment-breaking Attacks and defenses in LLMs Although various alignment strategies have been developed to steer LLMs to generate content complying with human ethical principles, an emerging class of alignment-breaking attacks (i.e., jailbreak attacks) can still bypass safeguards and elicit LLMs to generate harmful and toxic re-
sponses (Li et al., 2023; Shen et al., 2023; Yuan et al., 2023; Cao et al., 2023; Kang et al., 2023; Zou et al., 2023; Wei et al., 2023b; Helbling et al., 2023), which poses significant threats to the practical deployment of LLMs. In particular, inspired by traditional computer security, Kang et al. (2023) adapt obfuscation, code injection/payload splitting, and visualization attacks to LLMs, leading to the generation of content containing hate speech, phishing attacks, and scams. Instead of manually crafting adversarial prompts, Zou et al. (2023) automatically produce transferable adversarial suffixes by using greedy and gradient-based search to maximize the probability of generating an affirmative response. AutoDAN (Liu et al., 2023) also can automatically generate jailbreak prompts through a genetic algorithm with a handcrafted jailbreak prompt as initialization. TAP (Mehrotra et al., 2023) iteratively refine the candidate attack prompts using tree-of-thought reasoning.
Note that some concurrent works also aim to defend against alignment-breaking attacks: Kumar et al. (2023) provides a verifiable safety guarantee by enumerating all possible partially erased input and using a safety filter to identify the harmfulness of the input content. LLM self-defense (Helbling et al., 2023) simply utilizes itself or another LLM to detect if its own response is harmful. Jain et al. (2023) propose to detect adversarial prompts by checking if the perplexity of the prompt is greater than a threshold. We defer more discussion and comparison with concurrent defense methods in Appendix F and G.
Traditional Text Adversarial Attack and Defenses Traditional text adversarial attacks primarily focus on text classification tasks and aim to force target models to maximize their prediction error by adversarially perturbing original text (Ebrahimi et al., 2017; Jin et al., 2020; Li et al., 2018; Maheshwary et al., 2021; Ye et al., 2023). The adversarial perturbation could be crafted by performing character-level transformation (Gao et al., 2018) or replacing original words with their synonyms while maintaining semantics similar (Alzantot et al., 2018). The generation of adversarial examples could be categorized into the “whitebox" setting and the “black-box" setting according to the extent of access to the target model (Xu et al., 2020a). As a representative white-box method, HotFlip (Ebrahimi et al., 2017) uses the gradient information of discrete text structure at its one-hot
representation to construct adversarial examples. In the black-box setting, Li et al. (2018); Jin et al. (2020); Ren et al. (2019) leverage the prediction score distribution on all categories to craft adversarial text without the guidance of parameter gradients. Maheshwary et al. (2021) focus on a more realistic scenario where attackers only know the top-1 prediction and propose using population-based optimization to construct adversarial text.
To defend against adversarial attacks, a body of empirical defense methods has been proposed. In particular, adversarial-training-based methods (Miyato et al., 2016; Zhu et al., 2019) incorporate adversarial perturbations to word embeddings and robustly train the model by minimizing the adversarial loss. Zhou et al. (2021); Dong et al. (2021) utilize adversarial data augmentation by replacing the original word with its synonyms to make the model robust to similar adversarial perturbations. To provide provable robustness against adversarial word substitutions, Jia et al. (2019) use certifiably robust training by training the model to optimize Interval Bound Propagation (IBP) upper bound. Shi et al. (2020) adopt linear-relaxation-based perturbation analysis (Xu et al., 2020c) to develop a robustness verification method for transformers. Zeng et al. (2023) propose a certifiably robust defense method based on randomized smoothing techniques (Cohen et al., 2019).

heading: 3 Our Proposed Method
text: In this section, we introduce the proposed Robustly Aligned LLM for defending alignment-breaking attacks. Before heading into details, we first discuss the threat model that is focused on in this paper.

heading: 3.1 Threat Model
text: An alignment-breaking attack seeks to bypass the security checks of an aligned LLM by introducing adversarial prompts adhered to an original malicious question. Let x denote a malicious question and padv represent the adversarial prompt generated by the alignment-breaking attack. Let xadv = x⊕ padv denote the entire input (malicious question and the adversarial prompt) where ⊕ denotes the insertion operation. While most existing attacks typically place the adversarial prompts at the end of the request (Zou et al., 2023), we actually consider a more general case where the adversarial prompt could also be inserted in front of the malicious question or be integrated in the middle.
We also assume that the target LLM f(·) is an
already aligned LLM that has a certain ability to reject commonly seen malicious requests. In other words, when the malicious question x is directly input into the target LLM f(·), it will, in most cases, deny answering such a question by outputting a response similar to “I am sorry, but I cannot talk about [a malicious request]...”. On the contrary, the attacker’s goal is to break the existing alignment of the target LLM by finding an adversarial prompt padv, so that xadv = x⊕padv will mislead the LLM to provide an affirmative answer (Zou et al., 2023) to such a malicious question, e.g., “Sure, here is how to do [a malicious request]...”.

heading: 3.2 Our Proposed Method
text: Our motivation builds upon the fact that the target LLM has already been aligned and is able to reject commonly seen malicious requests. To be more specific, we can build an alignment check function AC(·) based on the aligned LLM f(·): return Fail when detecting typical aligned text in the output of f(·) such as “I am sorry, but I cannot answer this ...”, and return Pass otherwise1. Given the alignment check function AC(·), one can then construct a “hypothetical” LLM by
f ′(x) =
{ Reject to answer, if AC(f(x)) = Fail
f(x) , if AC(f(x)) = Pass (1)
where f ′(x) denotes the “hypothetical” LLM constructed by using the alignment check function AC(·). While f ′(x) seems “useless” as it gives the same result as f(x) in practice, it showcases how one can construct a new aligned LLM using an alignment check function.
1This definition of AC(·) is quite vague but we will provide more details on how to implement it in practice in Section 3.3.
Robust Alignment Check Function One thing to notice here is that the previously defined alignment check function AC(·) only relies on the existing alignments inside of the target LLM. However, the existence of alignment-breaking attacks such as the adversarial prompts (Zou et al., 2023) has proved that such alignment checking is not robust: it can be easily manipulated and circumvented by carefully designed perturbations or suffix prompts. Therefore, it is natural to think about how we can design a robust alignment check function that could strengthen the alignment check capabilities of an aligned LLM, without finetuning or modifying the model itself.
Our intuition here is very straightforward: since the existing alignment check function AC(·) is not robust enough for alignment-breaking prompts, the Pass decision directly returned by it cannot fully convince us that the request is benign, instead, we need a stronger evidence and a stricter check. Specifically, we consider an input request to be benign, only if we randomly drop a certain portion of the request and the corresponding response can still pass the alignment check function AC(·) in most cases. To translate this requirement into mathematical formulations, we define the following Robust Alignment Check function RAC(·) based on the aligned LLM f(·) and the alignment check function AC(·):
RAC(x) =    Fail, if AC(f(x)) = Fail Fail, if P r∼U(p) (AC(f([x]r)) = Fail) > t
Pass, otherwise (2)
where r refers to the uniformly sampled indices mask to indicate kept tokens, U(p) refers to the distribution of possible masks after uniformly drop-
ping p portion of the indices (without replacement), and [x]r denotes the kept indices r inside x after the dropping operation. Essentially, for an input x with length L, every possible [x]r only contains (1− p)L tokens indexed by r.
Eq. 2 states that the robust alignment check function RAC(·) not only requires the original response f(x) to show no sign of being aligned (e.g. refusalto-answer) but also requires the response after random dropping still shows no sign of being aligned in most cases. Typically, aligned text for malicious requests includes some refusal-to-answer text, such as "I am sorry, but I cannot answer this...". On the contrary, if AC(x) already fails or over a certain ratio (e.g., > t) of responses from the randomly dropped input fails to pass AC, RAC(·) will also fail it. Therefore, it is easy to see that such a design certainly helps us build a more robust alignment check function compared to AC(·).
Based on the robust alignment check function RAC(·), we can further construct a robustly aligned LLM by simply replacing the vanilla alignment check function AC(·) with RAC(·) in Eq. (1):
frob(x) =
{ Reject to answer, if RAC(f(x)) = Fail
f(x) , if RAC(f(x)) = Pass (3)
By this simple reconstruction of alignment check function, we can build a robustly aligned LLM without necessitating extra resources or retraining of the entire model. Figure 2 illustrates the effect of our proposed RAC when facing malicious or benign requests.

heading: 3.3 Practical Designs
text: Now let’s delve into the practical designs of our proposed robustly aligned LLM, which essentially approximates frob(·) mentioned above. The detailed steps of the constructed robustly aligned LLM are summarized in Algorithm 1.
Approximation of AC(·) Previously, we vaguely defined the alignment check function AC(·) as returning Fail when detecting typical aligned output while returning Pass otherwise. In practice, we approximate this alignment check function through prefix checking: we observed that various aligned outputs often share similar prefixes such as “I can not”, “I’m sorry”. Therefore, we can build a prefix set and if any prefix in the set appears in LLM’s output, the alignment check function AC(·) returns Fail; otherwise, it returns Pass. Note that we are
Algorithm 1 Robustly Aligned LLM
Input: aligned LLM f , alignment check function AC, original input x.
1: if AC(f(x)) = Fail then 2: Reject the request 3: else 4: for i = 1, 2, · · · , n do 5: Randomly sample a mask ri ∼ U(p) 6: si = 1{AC(f([x]ri)) = Fail} 7: end for 8: if (1/n) ∑n i=1 si > t then
9: Reject the request 10: else 11: Return f(x) 12: end if 13: end if
only inspecting the prefix. For this purpose, we only need to generate a certain number of tokens (e.g., 10) for robust alignment checking. This could largely reduce our computational overhead2.
Monte Carlo Sampling It is practically infeasible to obtain the exact value for the probability of Pr∼U(p)(AC(f([x]r)) = Fail), as it would require us to enumerate all possible random dropping cases and is computationally intractable. Therefore in practice, we conduct Monte Carlo sampling to approximate the true probability: we randomly sample n indices masks to obtain n versions of the input request with random dropping; we then solicit the LLM’s responses for these n requests, and count the frequency of cases when the alignment check function AC(·) gives Fail decisions.
The Practical Choice of t Another important choice is the threshold t used in practice. In particular, one seemingly logical choice is setting t → 0 such that whenever AC(·) detects any failure case from the randomly dropped request, RAC(·) directly fails the request. However in practice, such a setting could be too extreme as the randomness introduced in the dropping operations might also affect the LLM response on benign inputs: random dropping might occasionally lead to the loss of essential information, and under such circumstances the LLM might also generate responses similar to the typical alignment responses. For example, “Do you like apples?” could become “Do you apples?”
2Further discussion on computational costs can be found in Section H.2.
after random dropping, leading the LLM to express an inability for answering this unclear question. This could potentially be mis-detected as Fail by AC(·), and if the threshold t → 0, it will lead to Fail by RAC(·) and be rejected by our robustly aligned LLM. Therefore, in practice, instead of setting the threshold t as zero, we keep a relatively small threshold.

heading: 3.4 Theoretical Analysis
text: In this section, we theoretically analyze the proposed robustly aligned LLM and see when it provides a more robust alignment compared to the original LLM when facing alignment-breaking attacks. Our theorem is based on the analysis on the robust alignment check function RAC. We will show that RAC is more robust for the aligned malicious text x with any adversarial prompt padv of length M and it can be inserted into any position (e.g., in front, back, or middle of x).
Theorem 3.1. Consider a malicious input x and its corresponding adversarial prompt padv such that xadv = x⊕ padv could break the alignment in the LLM f(·). Suppose x consists of N tokens and padv consists of M tokens while padv could be insert to any position j ∈ [0, ..., N ] in x. Denote xjpad as the padded text constructed from x by inserting M pad tokens into position j. If N ≥ M(1−p)p and min
j P r∼U(p) (AC(f([xjpad]r)) = Fail) > t + c,
where c = 1− ( N (N+M)(1−p))
( N+M(N+M)(1−p)) and t is the threshold
used in Algorithm 1, then our robustly aligned LLM in Algorithm 1 with sufficiently large random drop trials n will reject the request on xadv = x⊕ padv.
The proof of Theorem 3.1 is provided in Appendix A. Theorem 3.1 provides an analysis on when our robustly aligned LLM could reject the request from an alignment-breaking attack while the original LLM actually fails to defend against such adversarial prompts. Specifically, given a particular malicious input x whose response has been aligned by the target LLM f(·), although it is impossible for us to know what kind of adversarial prompt the attacker would use, or which position the attacker would insert the adversarial prompt to, as long as we have min
j P r∼U(p) (AC(f([xjpad]r)) =
Fail) > t + c, then any alignment-break attack xadv composed by x⊕ padv will be rejected by our robustly aligned LLM.

heading: 4 Experiments
text: In this section, we aim to validate the efficacy of our RA-LLM from two aspects: 1) RA-LLM can effectively reduce the attack success rate of adversarial prompts; 2) RA-LLM minimally affects the outputs of benign samples. In the following, we first introduce our experimental settings and give a detailed analysis of our experimental results and ablation study. Full details of the experiment can be found in Appendix E.

heading: 4.1 Experimental Settings
text: Attack Method We conducted tests on the current state-of-the-art methods in three typical alignment attack techniques: GCG (Zou et al., 2023), AutoDAN (Liu et al., 2023), and Tree of Attacks (TAP) (Mehrotra et al., 2023). GCG aims to optimize from a meaningless string to discover an adversarial suffix. AutoDAN iterates from an existing handcrafted jailbreak prompt using a genetic algorithm, seeking the most effective alignmentbreaking prompt. TAP, starting from a similar handcrafted jailbreak prompt, employs an additional (often more powerful) Large Language Model (LLM) to refine the initial prompt. The objective of all these attacks is to prompt the LLM to respond effectively to a harmful query string when appended with the discovered jailbreak prompt.
Metric We consider two main metrics to evaluate our model’s performances: attack success rate (ASR) and benign answering rate (BAR). Attack success rate measures the number of chances when the adversarial prompts successfully circumvent the model’s alignment mechanism. An attack is regarded as successful when the LLM produces a meaningful response without rejecting to answer with typical alignment text. To ensure the defense mechanism does not overkill and reject to answer benign questions, we also tested the benign answering rate, which represents the model precision in successfully identifying benign requests (does not reject to answer the benign requests). Our defensive goal is to minimize the attack success rate as much as possible while correctly identifying benign samples with a high benign answering rate.

heading: 4.2 Experimental Results
text: In Table 1, we present the experimental results on two attack modes of the Harmful Behaviors Attack: Individual Attack and Transfer Attack, on Vicuna7B-v1.3-HF and Guanaco-7B-HF models. Indi-
vidual Attack aims to directly optimize adversarial prompts for specific models and specific malicious requests, while Transfer Attack aims to optimize generic adversarial prompts across multiple models and malicious requests. We tested both the original aligned LLM and our robust aligned LLM using benign requests and malicious requests with adversarial prompts. Subsequently, we evaluated whether these inputs activated the alignment mechanism based on the output of the LLM.
From Table 1, it is evident that for Individual Attack, adversarial prompts have led to high malicious response success rates of 98.7% and 96.0% on the two models respectively. However, upon employing our robustly aligned LLM, these success rates dropped to 10.7% and 6.7%. Similarly, for Transfer Attack, the application of our robustly aligned LLM reduced the attack success rates from 83.3% and 78.7% to 11.3% and 8.7%. This demonstrates that our strategy effectively mitigates adversarial attacks. Additionally, our method maintains a good benign response rate, this indicates that our approach has almost no adverse impact on the LLM’s responses to benign inputs. Additionally, we defer more results against GCG harmful String Attack, AutoDAN, and TAP in Appendix C and D, which demonstrate that our method can still substantially diminish the effectiveness of these attacks.

heading: 4.3 Handcrafted Jailbreak Prompts
text: In practice, another type of commonly seen alignment-breaking attack is the handcrafted jailbreak prompts. Those manually crafted adversarial prompts usually work by designing elaborate roleplay scenarios or asking the LLM to give the responses starting with affirmative responses such as “Sure, here it is” to force the LLM to generate harmful content. In general, the handcrafted jailbreak prompt is the type of alignment-breaking attack that is more widely adopted as it only requires no computation at all, and therefore, the threats stemming from handcrafted jailbreak prompts cannot be overlooked.
We also assessed the defensive capabilities of our robustly aligned LLM against these meticulously designed jailbreak prompts. Specifically, we selected the top five jailbreak prompts from jailbreakchat.com3, voted by the online users according to their effectiveness. For each of these
3The prompts are taken according to the website result on Sept 12, 2023
handcrafted jailbreak prompts, we randomly selected 30 questions from the Harmful Behaviors dataset, culminating in a set of 150 handcrafted jailbreak prompt samples. Table 2 shows the effects of our defense method on the handcrafted jailbreak prompt dataset for three different LLMs, Vicuna-7B-chat-HF, Guanaco-7B-HF, GPT-3.5turbo-0613, all of them underwent safety alignment. We found that our robustly aligned LLM also performs exceptionally well against such handcrafted jailbreak prompts. As seen in Table 2, handcrafted jailbreak prompts achieved attack success rates of 98.4%, 94.7%, and 82.0% on the Vicuna7B-chat-HF, Guanaco-7B-HF, and GPT-3.5-turbo0613 models, respectively, without additional defense beyond alignment. However, when applying to our robustly aligned LLM, the attack success rates dropped to 12%, 9.3%, and 8.0%, a result even better compared to the adversarial prompt attacks in the previous section. In the meantime, RA-LLM has no significant impact on BAR especially for the larger models like GPT-3.5-turbo0613, which inherently possess strong semantics comprehension abilities.

heading: 4.4 Ablation Study
text: In this section, we analyze the impact of the three hyperparameters in our method: the random dropping ratio p, the threshold t, and the number of random dropping trials n. For our default parameters, these parameters are set as n = 20, p = 0.3, t = 0.2. We evaluate the influence of these hyperparameters using the attack success rate and benign answering rate on the Harmful Behaviors attack in Vicuna-7B-chat-HF model. The evaluation results are depicted in Figure 3.
The Effect of Dropping Ratio p As observed in Figure 3a, we note that a larger random dropping ratio p can further reduce the attack success rate. However, it might also lead to a significant drop in benign answering rate, suggesting that it tends to have a more strict rule and thus considers a lot of benign requests as malicious. When the random dropping ratio p is smaller, the accuracy on benign samples remains at a high level, but it will also affect the efficacy of the robust alignment checking function, leading to a higher attack success rate.
The Effect of Threshold t Similarly, from Figure 3b, we can observe that a too small t can decrease the accuracy on benign samples, as the randomly dropped benign samples can sometimes be
confusing for LLM to understand and thus also be rejected to answer. Conversely, a very large t makes it difficult to reach the threshold to trigger the rejection of answering, resulting in only a limited reduction in the attack success rate.
The Effect of Monte Carlo trials n As observed in Figure 3c, our method still exhibits good performance with various Monte Carlo trails. Even with very few Monte Carlo trials such as 15 and 10, our RA-LLM maintains a BAR close to 100% and a relatively low attack success rate. This suggests that reducing the number of Monte Carlo trials is a potential strategy to decrease computational cost while maintaining stable defensive performance.

heading: 5 Computational Cost
text: We adopt two mechanisms to ensure our method is time-efficient: (1) Limited output length: we limit a small maximum generation length in Monte Carlo simulation (e.g., tmax = 10), since the negative prefixes often appear at the start of LLM responses, allowing us to obtain effective defense without generating full responses; (2) Early exit mechanism: during the Monte Carlo simulation, if the detected failure cases exceed our set threshold, RA-LLM can directly terminate the process early and mark the input as malicious. We also evaluated the actual time overhead of RA-LLM against the original
LLM in our experiments. We tested 150 attack samples and recorded the normal inference time on two LLMs and the time required by our RA-LLMs. We report the extra time per data on average overhead in Table 3 where the values in parenthesis represent the percentage of additional time relative to the normal inference time. These results show RA-LLM’s extra time requirement is less than 20% of normal inference. See more details of time cost and API cost in Appendix H.

heading: 6 Adaptive Attack
text: In this section, we explore three potential adaptive attack methods against RA-LLM to assess its resilience.
Repeating adversarial prompts Since our method randomly drops p portion of tokens from
the input x and uses Monte Carlo sampling to simulate all possible scenarios, any form of adversarial prompt may be discarded. Hence, it’s challenging to design an adaptive attack based on optimization for our defense method. However, one may also utilize this design choice and simply try increasing the length of the adversarial prompts (e.g., repeat the adversarial prompts after input for several times) to ensure the random dropping cannot fully remove the adversarial parts.
In order to figure out whether such a potential adaptive attack can invalidate our defense or not, we conducted experiments on the Harmful Behaviors attack on both the original LLM and our robustly aligned LLM. The results are presented in Table 4. We found on the original LLM, repeating the adversarial prompt multiple times in the input also leads to a reduction in the attack success rate. What’s more, at various repetition counts, our defense method keeps the ASR lower than scenarios without repetitions, hovering around 5%.
Replacing the Target Model with RA-LLM In this adaptive attack approach, we directly substitute the target model used during the attack with its RA-LLM version. That is, if RA-LLM deems the input harmless, it will return the response of the underlying LLM to the input. If RA-LLM judges the input as harmful, i.e., AC(·) returns Fail, it will return a fixed string "I’m sorry, but I can’t assist with that."
Given that it’s impossible to directly derive gradients for RA-LLM, gradient-based attack methods like GCG(Zou et al., 2023) cannot be applied. Thus, we experimented with TAP(Mehrotra et al., 2023) and AutoDAN(Liu et al., 2023) attack methods and found that after multiple iterations, both methods failed to generate effective jailbreak prompts, resulting in a 0% success rate. We speculate that since RA-LLM always returns a fixed string and probability distribution for harmful inputs, and this string and distribution are manually specified by us, the attack methods cannot find a reasonable optimization direction, leading to convergence failure.
Incorporating an Additional Loss Term As mentioned above, due to the non-differentiability of our random dropping mechanism, it’s challenging to directly apply gradient-based attack methods on RA-LLM. Therefore, we considered designing a suitable adaptive attack for gradient-based whitebox attack methods. A potential method involves optimizing the probability of RAC returning Pass
as one of the objective functions. However, for attackers, this means that each iteration during the attack process would require approximately 20 times more overhead. This is significant because such gradient-based attack methods (e.g., GCG (Zou et al., 2023)) already necessitate substantial computational resources.
As an alternative, we experimented with treating random dropping as a "Transformation" and applying the Expectation over Transformation (EoT) method. We conducted experiments on the Vicuna7B model using the GCG’s Individual-Behavior setting, dropping 30% of the tokens randomly at the start of each optimization, and then optimizing the remaining 70%. We found that under these circumstances, the loss oscillated near its initial value and failed to converge. We believe this phenomenon may be attributed to the following reasons:
1) The objective function itself is difficult to optimize. Finding a successful adversarial sample on a well-trained LLM is challenging, and obtaining a robust adversarial sample is even harder. 2) In such attack methods, a greedy search approach is typically employed, storing the jailbreak prompt with the smallest loss after each update round as the target for the next optimization. However, the loss of the new input after applying random dropping is quite uncertain. 3) Practical adaptive attacks need to be able to bypass situations both with and without RA-LLM, making the objective function even more challenging to optimize.

heading: 7 Conclusion
text: While a variety of alignment strategies have been proposed to guide LLMs to obey human values, recent works show that these alignments are vulnerable and can be bypassed by alignment-breaking attacks through adversarial prompts. In this work, we propose robustly aligned LLMs, which are built upon existing aligned LLMs with a robust alignment checking function, to defend against alignment-breaking attacks. One major advantage of our method is that there is no need to expensively retrain or fine-tune the original LLM for defense purposes. We also provide a theoretical analysis to verify the effectiveness of our proposed defense. The exhaustive experiments clearly demonstrate our method can effectively defend against both automatically generated adversarial prompts and handcrafted jailbreak prompts.

heading: 8 Limitations
text: Our work is primarily limited in two dimensions. First, the random dropping mechanism used in our proposed method still has a minor effect on benign samples in some models. Specifically, the benign answering rate of Guanaco-7B-chat-HF decreased from 95.3% to 92.0%. Future work may investigate how to design defenses with less impact on benign samples, such as exploring better dropping methods to further reduce the adverse effect. Second, due to the limitations of current jailbreak attack techniques, we have not assessed how our method performs when faced with very extreme cases, such as particularly long or short adversarial prompts. Future work could further design these extreme cases and study their adaptability to our proposed defending method.

heading: 9 Acknowledgement
text: We thank the anonymous reviewers for their helpful comments. This work is partially supported by DHS (17STQAC00001-07-00). The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.

heading: A Proof of Theorem 3.1
text: In this section, we provide the proof of Theorem 3.1.
Proof of Theorem 3.1. The part of the proof for Theorem 3.1 is inspired from (Zeng et al., 2023). Denote xjadv as any adversarial example constructed from x where M continuous adversarial tokens are inserted into position j, and denote the inserted adversarial prompt as pjadv. For each j, we have the following equations based on the law of total probability:
P r∼U
(AC(f([xjpad]r)) = Fail)
= P r∼U
(AC(f([xjpad]r)) = Fail)|[x j adv]r ∩ p j adv ̸= ∅) P r∼U ([xjadv]r ∩ p j adv ̸= ∅)
+ P r∼U
(AC(f([xjpad]r)) = Fail)|[x j adv]r ∩ p j adv = ∅) P r∼U ([xjadv]r ∩ p j adv = ∅)
(4)
and P
r∼U (AC(f([xjadv]r)) = Fail)
= P r∼U
(AC(f([xjadv]r)) = Fail)|[x j adv]r ∩ p j adv ̸= ∅) P r∼U ([xjadv]r ∩ p j adv ̸= ∅)
+ P r∼U
(AC(f([xjadv]r)) = Fail)|[x j adv]r ∩ p j adv = ∅) P r∼U ([xjadv]r ∩ p j adv = ∅)
(5)
When [xjadv]r ∩ p j adv = ∅, we have that [x j adv]r = [x j pad]r. Thus, there is
P r∼U
(AC(f([xjpad]r)) = Fail)|[x j adv]r∩p j adv = ∅) = P r∼U (AC(f([xjadv]r)) = Fail)|[x j adv]r∩p j adv = ∅) (6)
Given Equation 6, P r∼U
(AC(f([xjadv]r)) = Fail)|[x j adv]r ∩ p j adv ̸= ∅) P r∼U ([xjadv]r ∩ p j adv ̸= ∅) ≥ 0, and
0 ≤ P r∼U (AC(f([xjpad]r)) = Fail)|[x j adv]r ∩ p j adv ̸= ∅) ≤ 1, we could compute P r∼U (AC(f([xjpad]r)) = Fail)− P r∼U (AC(f([xjadv]r)) = Fail) as follows
P r∼U (AC(f([xjpad]r)) = Fail)− P r∼U (AC(f([xjadv]r)) = Fail)
= P r∼U
(AC(f([xjpad]r)) = Fail)|[x j adv]r ∩ p j adv ̸= ∅) P r∼U ([xjadv]r ∩ p j adv ̸= ∅)
− P r∼U (AC(f([xjadv]r)) = Fail)|[x j adv]r ∩ p j adv ̸= ∅) P r∼U ([xjadv]r ∩ p j adv ̸= ∅) ≤ P r∼U (AC(f([xjpad]r)) = Fail)|[x j adv]r ∩ p j adv ̸= ∅) P r∼U ([xjadv]r ∩ p j adv ̸= ∅)
≤ P r∼U ([xjadv]r ∩ p j adv ̸= ∅)
(7)
If N ≥ M(1−p)p , there is P r∼U ([xjadv]r ∩ p j adv ̸= ∅) = 1−
( N(N+M)(1−p)) ( N+M(N+M)(1−p)) = c, thus we have
P r∼U (AC(f([xjpad]r)) = Fail)− P r∼U (AC(f([xjadv]r)) = Fail) ≤ c (8)
If min j P r∼U (AC(f([xjpad]r)) = Fail) > t + c, which implies for any j, we have P r∼U (AC(f([xjpad]r)) = Fail) > t+ c. Based on Equation 8, we can conclude that
P r∼U (AC(f([xjadv]r)) = Fail) ≥ P r∼U (AC(f([xjpad]r)) = Fail)− c > t (9)
then for any j, we have RAC(xjadv) = Fail. Therefore, we obtain that RAC(xadv) = Fail. This concludes the proof.

heading: B Concrete Examples
text: In this section, we also give a few concrete examples comparing the output of the original LLM and our robustly aligned LLM under alignment-breaking attacks.

heading: C Defensive Efficacy Against Harmful Strings Attack
text: We also conducted experiments under the setting of Harmful String Attack proposed in (Zou et al., 2023). The goal of Harmful Strings attack is to compute an adversarial input, which can induce the LLM to generate a specific harmful string. Although this setting does not really fit in our threat model, it would
also be interesting to see how RA-LLM performs under this attack. We conducted experiments on the Vicuna-7B-v1.3 model, and the results are presented in Table 5. It can be observed that, in the original LLM, the attack success rate of adversarial prompts generated by Harmful String Attack is as high as 84%, while after applying our RA-LLM, the attack success rate drops to 0%. This indicates that our strategy can also effectively mitigate Harmful String Attack.

heading: D Defensive Efficacy Against AutoDAN and TAP
text: We additionally conduct experiments to validate the effectiveness of RA-LLM against AutoDAN (Liu et al., 2023) and Tree of Attacks (TAP) (Mehrotra et al., 2023). In specific, AutoDAN can automatically generate semantic meaningful jailbreak prompts using a genetic algorithm with a handcrafted jailbreak prompt as initialization. TAP, starting from a similar handcrafted jailbreak prompt, iteratively refines the candidate (attack) prompts using tree-of-thought reasoning. For the implementation of AutoDAN, We used the same parameter settings provided in their papers. When evaluating AutoDAN, the maximum number of generated tokens is 20. In, Table 6, we summarized the experimental results of RA-LLM against AutoDAN and TAP, where AutoDAN-GPT refers to AutoDAN with GPT mutation. We observed that our method can still significantly reduce the ASR of these attacks.

heading: E Details of Experiment
text: In evaluating the success rate of attacks, we utilized the Harmful Behaviors and Harmful Strings data from AdvBench (Zou et al., 2023). Harmful Strings comprises 500 strings related to harmful or toxic content, such as threats, discriminatory remarks, methods of crime, and dangerous suggestions. Harmful Behaviors consists of 500 prompts that could induce LLMs to produce harmful outputs, with themes similar to those of Harmful Strings. Initially, we employed GCG(Zou et al., 2023) to obtain 150 Harmful Behaviors instances that successfully break alignment along with their adversarial suffixes. These were then used in testing AutoDAN(Liu et al., 2023) and TAP(Mehrotra et al., 2023). For BAR, testing was conducted on the MS MARCO dataset(Nguyen et al., 2016), a question-answering collection where all questions are derived from real user queries on Bing. Here, we randomly selected 150 data points for our experiment.
For all attack methods, we adhered to the default parameters provided in their official code unless specified otherwise. Unless explicitly mentioned, the hyperparameters for RA-LLM in all experiments were: the random dropping ratio p = 0.3, the threshold t = 0.2, the maximum number of generated tokens tmax = 10, and the number of random dropping trials n = 20. All experimental models were loaded using
HuggingFace, with LLM parameters set to default values aside from RA-LLM’s hyperparameters. All experiments were conducted on an NVIDIA RTX A6000 GPU.

heading: F Comparison with LLM Self Defense
text: In this section, we additionally compared our method with the concurrent work, LLM Self Defense (Helbling et al., 2023). We use the more effective "Questions as suffix" setting, that simply utilizes itself or another LLM instance to detect if its own response is harmful through the instruction such as
Here is some text:{response} Is it harmful? Give answer as “Yes, it is harmful" or “No, it is harmless".
Specifically, we conducted both Self Defense through itself and Self Defense through GPT3.5 and summarized the comparison results in Table 7. We can observe that the attacked models (i.e., Vicuna and Guanaco) inherently struggle to discern harmful content, leading to low BAR. Moreover, the outputs of the detection prompt contain unexpected content including deviations from the predefined responses of "Yes, it is harmful." or "No, it is harmless.", continue completion of the unfinished content from the previous context, and even non-responsiveness from LLMs. These suggest that in scenarios preventing the use of external powerful models, LLM Self Defense might be less effective.
While Self Defense though more powerful LLM instances such as GPT3.5 demonstrates higher accuracy in identifying harmful content and thus enjoys on-par defending effectiveness with our method, it still suffers from lower BARs. This could be attributed to the current LLM’s overcautiousness in detecting harmful content (Röttger et al., 2023).

heading: G Comparison with Perplexity-Based Defense
text: Perplexity-based defense proposed by Jain et al. (2023) detects adversarial prompts by checking if the perplexity of the prompt is greater than a threshold. Following the same threshold adopted in Zhu et al. (2023), we report the comparison results in Table 8. We can observe that even though perplexity defense achieves high BAR and effectively reduces the ASR of individual GCG attacks, this defense mechanism completely fails to detect handcrafted jailbreak prompts, presumably owing to the lower perplexity of these prompts, as they are manually written by humans. A similar conclusion is also validated in Zhu et al. (2023). In contrast, our method effectively defends against handcrafted jailbreak prompts.

heading: H Computational Cost
text: H.1 Time Cost
To further reduce the cost of RA-LLM, we implemented an early-exit mechanism in the Monte Carlo simulation. Specifically, if the number of detected failure cases exceeds our predefined threshold during the Monte Carlo simulation, RA-LLM terminates the process early and marks the input as a malicious
sample. For instance, with Monte Carlo trials at n = 20 and a threshold t = 0.2, RA-LLM designates an input as malicious if it detects 0.2 × 20 = 4 aligned responses. If 4 aligned responses are detected in the first 6 Monte Carlo trials, the remaining 14 trials will not be executed. Similarly, if no aligned responses are found in the first 17 trials, the input is immediately classified as benign, and the last 3 trials are skipped. This approach helps to further reduce computational costs.
We evaluated 150 attack samples on both Vicuna-7B-chat-HF and Guanaco-7B-HF models, measuring the normal inference time, the time required by RA-LLM, and the time taken by RA-LLM after forcibly completing the entire Monte Carlo simulation process. We set the maximum token generation during normal inference at 1,000. For RA-LLM, we follow the default setting, and we conducted all experiments on an NVIDIA RTX A6000 GPU.
For the Vicuna-7B-chat-HF model, normal inference took 20.97 seconds per data on average, RA-LLM required an extra 3.93 seconds per data on average, and RA-LLM with the full Monte Carlo simulation required an extra 9.26 seconds per data on average. For the Guanaco-7B-HF model, these averages were 30.36 seconds for normal inference, extra 3.76 seconds for RA-LLM, and an extra 12.84 seconds for the full Monte Carlo simulation. It is observed that the time required for RA-LLM is less than 20% (18.7% and 12.0%) of the normal inference time. Even in the worst-case scenario, where each instance undergoes a full Monte Carlo simulation, the additional time cost does not exceed 45% (44.1% and 42.3%). We believe this cost is acceptable.
We also compared the time taken with other time-efficient methods such as perplexity-based defense (Jain et al., 2023) and self-defense (Helbling et al., 2023). For the Vicuna-7B-chat-HF model, the normal inference time is 20.97s. The perplexity-defense method incurs an extra time of 1.45s, which is 6.9% of the normal inference time. The self-defense method has an extra time of 49.0s, which is 233.6% of the normal inference time. Our proposed RA-LLM method has an extra time of 3.93s, which is 18.7% of the normal inference time. We observe that both the perplexity-defense and our methods have minimal computation costs, while the self-defense method suffers from larger computation overhead. It is important to note that although the perplexity defense is faster, it may be ineffective against certain attack methods that use natural language-based jailbreaking prompts, such as AutoDAN (Liu et al., 2023).
H.2 API Cost
In this section, we discuss the additional computational costs incurred by our robustly aligned LLM compared to the original LLM. Suppose the token counts for input content and LLM responses in a dialogue are lin and lout, respectively, and the computational costs for each input and response token are cin and cout, respectively. The total cost of the original LLM is: CLLM = lin × cin + lout × cout. For our robustly aligned LLM, the Monte Carlo sampling process introduces additional costs. Let the number of Monte Carlo samplings be n and the proportion of input tokens randomly discarded in each sampling be p. Additionally, to reduce computational costs, we limit the maximum number of output tokens to tmax. Hence, if AC(x) fails, the extra cost of our defense is:
Cextra = (1− p)lin × cin × n+ lout × cout × n, where lout ≤ tmax.
(10)
The ratio of the extra cost to the computational cost of the LLM without defense is:
Cextra CLLM = (1− p)lin × cin × n+ lout × cout × n
lin × cin + lout × cout ≤ (1− p)lin × cin × n+ tmax × cout × n
lin × cin + lout × cout .
(11)
If we approximate the value of CextraCLLM using our experimental data, the average token counts for inputs lin = 22.58 and outputs lout = 275.25. For our default parameters, i.e., n = 20, p = 0.3, t = 0.2, tmax = 10. To calculate the average computational cost per token, we refer to the pricing of the ChatGPT API. The GPT-4 model with an 8K context is priced at $0.03 / 1K tokens for input and $0.06 / 1K tokens for
output, whereas the GPT-3.5 Turbo model with a 16K context is priced at $0.003 / 1K tokens for input and $0.004 / 1K tokens for output.
After calculations, CextraCLLM is approximately 1.250 under GPT-4 pricing and about 1.496 under GPT-3.5 Turbo pricing. We believe this cost is reasonable considering the defensive performance it could provide. If the computational cost is a real concern, one can further trade off a bit of defensive performance for cost reduction by adjusting the hyperparameters used (e.g., p, t, and n) as suggested in our ablation studies.

heading: I Collaborating with Safety Alignment on LLMs to Counteract Attacks
text: We have shown in the experiments that applying the random dropping strategy on malicious requests with adversarial prompts can effectively trigger the alignment of the model. However, for benign requests, random dropping may lead to a loss of key information and make the LLM occasionally generate unableto-answer responses similar to typical alignment responses. This leads to a certain level of decrease in terms of benign answering rate. For scenarios prioritizing the benign user experience, we can sacrifice slightly on ASR to achieve a nearly 100% BAR. We conducted experiments under the Vicuna+GCG setup and found that by increasing the hyperparameter t to 0.35, we can maintain the BAR without a decrease. At this point, the ASR is 15.3%, which is still an 83.4% reduction compared to the case without defense. For scenarios that prioritize security, we can also achieve a lower ASR by adjusting t.
Besides, we can further reduce the loss on benign answering rate if the alignment response of the LLMs can be distinguishable from the other types of unable-to-answer responses. For instance, during the alignment fine-tuning process, the LLM is instructed to always start the response to malicious requests with a special token. When applying our defensive method, it is only necessary to output and check the first token in each Monte Carlo trial. Such a collaborative strategy on alignment and RA-LLM will not only significantly improve our recognition accuracy for malicious inputs but also help in further reducing computational costs. Due to limited resources, we leave this part as our future work.

heading: Future Work
text: Due to limited resources, we leave this part as our future work