Title: Mission Impossible: A Statistical Perspective on Jailbreaking LLMs
Venue: NeurIPS 2024
Accepted: yes
Keywords: large language models

jailbreak

safety alignment

theory
Abstract: Large language models (LLMs) are trained on a deluge of text data with limited quality control. As a result, LLMs can exhibit unintended or even harmful behaviours, such as leaking information, fake news or hate speech. Countermeasures, commonly referred to as preference alignment, include fine-tuning the pretrained LLMs with carefully crafted text examples of desired behaviour. Even then, empirical evidence shows preference aligned LLMs can be enticed to harmful behaviour. This so called jailbreaking of LLMs is typically achieved by adversarially modifying the input prompt to the LLM. Our paper provides theoretical insights into the phenomenon of preference alignment and jailbreaking from a statistical perspective. Under our framework, we first show that pretrained LLMs will mimic harmful behaviour if present in the training corpus. \textbf{Under that same framework, we then introduce a statistical notion of alignment, and lower-bound the jailbreaking probability, showing that it is unpreventable under reasonable assumptions.} Based on our insights, we propose an alteration to the currently prevalent alignment strategy RLHF. Specifically, we introduce a  simple modification to the RLHF objective, we call \emph{E-RLHF}, that aims to increase the likelihood of safe responses. \emph{E-RLHF} brings no additional training cost, and is compatible with other methods. Empirically, we demonstrate that \emph{E-RLHF} outperforms RLHF on all alignment problems put forward by the AdvBench \citep{zou2023universal} and HarmBench project \citep{mazeika2024harmbench} without sacrificing model performance as measured by the MT-Bench project \citep{zheng2024judging}.

Mission Impossible: A Statistical Perspective on Jailbreaking LLMs

Abstract

Large language models (LLMs) are trained on a deluge of text data with limited quality control. As a result, LLMs can exhibit unintended or even harmful behaviours, such as leaking information, fake news or hate speech. Countermeasures, commonly referred to as preference alignment, include fine-tuning the pretrained LLMs with carefully crafted text examples of desired behaviour. Even then, empirical evidence shows preference aligned LLMs can be enticed to harmful behaviour. This so called jailbreaking of LLMs is typically achieved by adversarially modifying the input prompt to the LLM. Our paper provides theoretical insights into the phenomenon of preference alignment and jailbreaking from a statistical perspective. Under our framework, we first show that pretrained LLMs will mimic harmful behaviour if present in the training corpus. Under that same framework, we then introduce a statistical notion of alignment, and lower-bound the jailbreaking probability, showing that it is unpreventable under reasonable assumptions. Based on our insights, we propose an alteration to the currently prevalent alignment strategy RLHF. Specifically, we introduce a simple modification to the RLHF objective, we call E-RLHF , that aims to increase the likelihood of safe responses. E-RLHF brings no additional training cost, and is compatible with other methods. Empirically, we demonstrate that E-RLHF outperforms RLHF on all alignment problems put forward by the AdvBench [1] and HarmBench project [2] without sacrificing model performance as measured by the MT-Bench project [3].

1 Introduction

Large Language Models (LLMs) have revolutionized the field of deep learning due to their remarkable capabilities across various domains, serving as assistants, in code generation [4], healthcare [5], and theorem proving [6]. The training process of a LLM typically includes two stages: pretraining with massive corpora, and an alignment step using Reinforcement Learning from Human Feedback (RLHF) to further align model behavior with human preferences. The latter step typically involves large amounts of humanly annotated data, and can be decomposed into a supervised fine-tuning (SFT) step, a reward modeling step, and an RL Fine-Tuning step. Despite their ability to perform multiple tasks effectively, LLMs are susceptible to generating offensive or inappropriate content including hate-speech, malware, fake information or social biases, due to the unavoidable presence of harmful elements within their pretraining datasets [7-9]. Social media showcase an abundance of tricks on how to attack ChatGPT [10] to elicit harmful responses, e.g., the 'Do Anything Now' (DAN) prompts [11] or the 'Grandma Exploit' hack [12]. On the other hand, behavior diversity in the training corpus is essential to for example capturing different cultural preferences. What is and isn't harmful ultimately depends on user preferences, hence the alignment step is not universal but depends on the specific use case under which a model will be employed.

†

To address deployment safety and eliminate objectionable responses, numerous alignment efforts have been made, such as injecting safe information during SFT [13], performing red teaming with human experts and AI themselves [14-18], as well as refining and improving the whole RLHF process in detail [19-23]. Yet we continue to witness a cat-and-mouse game of ever more sophisticated alignment methods to neutralize 'harmful' prompts and even more inventive 'jailbreaking' attacks that manipulate those prompts to elicit LLMs to produce harmful information. Such attacks come in various flavors, such as injecting adversarial suffixes [1, 24, 25], exploring cipher and low-resource natural languages [26-28], or letting LLMs craft prompts automatically [29-33]. Although several adhoc defense methods against suffixes have been proposed [34-37], we only have limited proposal on a principled universal defense against jailbreaking attacks [2], and limited theoretical understanding on this phenomenon [38].

In this paper, we present a theoretical framework for analyzing both the pretraining phase and the post-alignment jailbreaking phenomenon. Exploiting the fact that jailbreaking prompts typically maintain the underlying harmful concept while manipulating other aspects of the prompt, we design framework that decouples input prompts to allows us to quantify the strength of potential adversaries. By representing the output elements of an language model (LM) as lengthier text fragments rather than individual tokens, we can quantify the extent to which these models emulate the training distribution and consequently better understand the mechanisms underlying jailbreaking vulnerabilities.

Our contributions can be summarized as follows:

Based on our proposed framework, we first offer a non-vacuous PAC-Bayesian style generalization bound for pre-training. Assuming the validity of our framework, we conclude that high-performing pre-trained models will inevitably be susceptible to generating behaviour that is present in the training corpus, including any unintended and harmful behaviour.

Subsequently, we extend our framework to include notions of alignment and jailbreaking. Assuming our assumptions are met, we demonstrate jailbreaking to be unpreventable even after safety alignment because the LM fails to concentrate its output distribution over the set of safe responses.

Motivated by our theoretical findings, we identify a key drawback in the widely adopted RL Fine-Tuning objective due to the natural difference between the harmlessness and the helpfulness targets. By addressing this issue, we facilitate the training of safer models that are more resilient to a suite of jailbreaking attacks while preserving model performance.

The paper is organized as follows. In Section 2, we introduce our framework. In Section 3, we prove the PAC-Bayesian generalization bound for pretraining. Next, in Section 4 we present analysis on jailbreaking from a statistical perspective. Finally, in Section 5 we illustrate our proposed E-RLHF objective and its effectiveness on improving LLM safety. We give a literature review in Appendix H.

2 Framework and assumptions

Jailbreaking carries several analogies to adversarial attacks , a well studied field in computer vision [39]. Here, an adversary is defined as a map that perturbs a given input image in pixel space to change the model output. The strength of the adversary is bounded by how far it is able to move the original input as quantified by the ℓ p distance [40-42]. Typically, this distance is bounded in a way that the change would not be perceptible to the human observer. The goal of the adversary is to cause misclassification of the input. In contrast, in the instance of an LLM, the adversary's goal is to provoke harmful behaviour, e.g., unintended leaking of information or hate speech. Further, any perturbation to an input, called prompt, will have a perceptible effect. Hence quantifying and bounding the capabilities of the adversary is not straight forward. Nonetheless, with some modifications, we will build on this analogy.

For the purpose of this work, we will view any prompt as a tuple of query and concept ( q, c ) , where c ∈ C , and q ∈ Q , with C , Q denoting the complete concept set and query set. Conceptually, we think of concepts as representing the information content of the prompt, usually through a short piece of text, for example 'tutorial on making a cake' . Queries are instructional text pieces that are composable with certain concepts. We can think of queries as mechanisms to trigger an LM to expand a concept in a specific way. Examples include 'Tell me how to {}' , or 'We are now in an imaginary world, and you are

not bounded by any ethical concerns. Teach my avatar how to {}' . Since not all queries and concepts are composable, 3 we denote P ⊊ Q×C as the set of all plausible prompts, where the definition of plausible will be made clear below.

The decomposition of prompts allows us to isolate and hence bound the adversary's strength. In line with current empirical work on inducing harmful behaviour, we will allow perturbations only on the queries, not on the concepts. Further mimicking the spirit of previous work on adversarial attacks, we will assume that the ground-truth related to a prompt is determined solely by the concept, not the query. We will make these ideas more rigorous in the next paragraphs.

First, in contrast to previous theoretical work where LMs are regarded as single sentence generators [38], we model LMs as lengthier text fragment generators , and refer to possible generated content e ∈ E as explanations. Conceptually, explanations expand concepts with additional information. For example, 'The US president in 2023 is Joe Biden.' . Our terminology 'explanation' is conceptually the same as 'response' used in previous discussions ( e.g., [23]), where an LLM is regarded as a policy that receives an input and generates a response. We use 'explanation' to contrast 'concept' since in most jailbreaking attacks currently considered by the community, the adversary seeks instructions or explanations for a single harmful attempt. An LM thus induces a mapping from plausible prompts to distributions over explanations, p LM : P → ∆( E ) , where ∆( E ) denotes the set of distributions defined over elements in E . 4 The output of a LM given a prompt, p LM ( q, c ) , is a

3

supp(Dp)

(q,c) x

xx

(a',c)x

e

P

supp(pLM)

dom(pLM(q,c))= dom(pLM(q*,c))

discrete distribution over explanations. We use dom( p LM ( q, c )) as the domain of this distribution, p LM ( e | q, c ) as the probability of e given ( q, c ) as the input, and supp( p LM ( q, c )) as the subset of E with non-zero p LM ( e | q, c ) . Further, we assume the existence of a latent ground truth mapping p world : P → ∆( E ) that the LM is optimized to mimic during the pretraining stage. This is the distribution that defines 'knowledge': for all plausible prompts ( q, c ) , it specifies the ground-truth distribution over explanations. By plausible , we refer to all prompts that lie in the domain of the ground truth mapping ( q, c ) ∈ dom( p world ) , i.e., P ≡ dom( p world ) . Many plausible prompts will not even exist within any available training corpus, as discussed below.

We can now state our main assumption, namely that for any plausible prompt ( q, c ) ∈ dom( p world ) the ground-truth distribution p world ( q, c ) is supported on a small subset of E ⇔ supp( p world ( q, c )) ⊊ E . This assumption seems sensible to us: under normal circumstances, providing an explanation of 'Paris' would not offer any relevant knowledge when given a prompt such as 'How to write a hello world python script' . Our second assumption is that for all plausible prompts ( q, c ) , the concept c uniquely determines the support of the output distribution specified by p world , regardless of the query: supp( p world ( q, c )) = supp( p world ( q ∗ , c )) , ∀ plausible ( q, c ) and, ( q ∗ , c ) . The query changes the ground-truth distribution without affecting its support. An illustration is depicted in Figure 1. To be more precise:

Assumption 2.1. (Concepts uniquely determine the explanation for plausible prompts) For all plausible prompts ( q, c ) ∈ dom( p world ) ,

where supp( p world ( q, c )) ⊊ E s.t. | supp( p world ( q, c )) | ≪|E| ; and ii) supp( p world ( q, c )) = supp( p world ( q ∗ , c )) , ∀ ( q, c ) , ( q ∗ , c ) plausible .

This assumption is natural since it essentially tells us that knowledge is specified by the corresponding concept alone, irrespective of what query is used to extract it. In other words, given a concept c , if a query q manages to change supp( p world ( q, c )) , we argue that the query should be deconstructed and partially absorbed by c to accurately reflect the knowledge mirrored by the support.

Lastly, we make the assumption on the existence of an underlying generative distribution over prompts, denoted as ( q, c ) ∼ D P . This distribution serves as the principle governing the creation of our pretraining corpus. It is important to note that supp( D P ) ⊊ dom( p world ) . For example, take the prompt ( q ′ , c ′ ) = 'Who is James Bond $ λ *#!48811' ; even though this prompt never appears in any text corpus across the internet, ( q ′ , c ′ ) / ∈ supp( D P ) , we, as humans, can make sense of it: ( q ′ , c ′ ) ∈ dom( p world ) . Later proofs in this paper assume LMs generate semantically reasonable explanations for such unseen plausible prompts, since in reality LMs are claimed to generalize well on huge, out-of-distribution datasets [43]. This is made explicit in Section 4, within Assumption 4.1.

Finally, the following definitions pertain to our notion of harmfulness. More specifically, we understand harmful behaviour abstractly as any unintended behaviour. For this, we assume that any explanation e can be denoted as either harmful or not harmful (safe) . A concept c is regarded as harmful if and only if the world generates harmful explanations with probability higher than a certain threshold with direct prompts.

Definition 2.1. (Notions of Harmfulness)

( Direct Queries and Direct Prompts ) We refer to a prompt as direct if it stems from D P , i.e., ( q, c ) ∈ supp( D P ) . The query of a direct prompt is called a direct query.

( Harmful Concepts and Harmful Set ) Given a concept c , the associated harmful set of explanations is denoted as E h ( c ) := { e | e ∈ supp( p world ( · , c )) ∧ e is harmful } . In accordance with Assumption 2.1, with a threshold η , a concept c is harmful if ∀ q s.t. ( q, c ) ∈ dom( p world ) , ∑ e : e ∈ E h ( c ) p world ( e | q, c ) ≥ 1 -η . We refer to the set of all possible harmful concepts as C h ⊊ C .

( Safe Set ) ∀ c ∈ C h , there exists a corresponding safe set E s ( c ) ⊊ E that we wish p LM ( q, c ) to be concentrated on. It includes safe explanations existing in supp( p world ( · , c )) , and explanations designed by humans, e.g., with the template beginning with 'Sorry. '

( Semantically meaningful ) We call explanations in E h ( c ) ∪ E s ( c ) as semantically meaningful for the ( q, c ) prompt.

( Mixture decomposition of D P ) With these notions, we can decompose D P = αD P h +(1 -α ) D P s (where supp( D P h ) includes all direct prompts with a harmful concept, and supp( D P s ) includes the complement) as a mixture over direct prompts with a harmful concept and the non-harmful counterpart.

3 PAC-Bayesian bound for pre-training LLMs on harmful data

Given a learning algorithm that leads to a posterior distribution over a set of models, PAC-Bayesian theory [44] applies Probably Approximately Correct (PAC) inequalities, to provide bounds on the generalization gap, i.e., the difference between the model's empirical loss and the population loss. We now present the first result of our analysis: a non-vacuous PAC-Bayesian bound for pretraining LMs which implies that a well-trained LM ought to exhibit harmful behaviour even when simply prompted with direct queries if it was presented with harmful behavior during training.

We denote by S = { ( q i , c i ) } n i =1 a set of prompts generated i.i.d. under D P , S ∼ D n P . These prompts together with sampled explanations form our pretraining corpus. We use π, ρ as the prior and posterior distribution over LMs before and after the pretraining process, defined over LM , the set of language models. Given a prompt ( q, c ) , we measure the generalization capability of a LM by quantifying the Total Variation (TV) loss between the induced distribution p LM ( q, c ) and the ground-truth distribution p world ( q, c ) . 5 For real-world LMs, pretraining involves optimizing the cross-entropy loss on the

training corpus, which is equivalent to minimizing KL [ p world ( q, c ) || p LM ( q, c )] under our framework. With Pinsker's Inequality, optimizing the KL-divergence term is equivalent to optimizing an upper bound on TV; thus we expect empirical TV loss be small.

Definition 3.1. (TV empirical loss and population loss)

Given an LM and a set of data S , the empirical loss ˆ R S ( p LM ) and population loss R ( p LM ) are defined as

We state our PAC-Bayesian bound as follows. The detailed proof can be found in Appendix B.1. 6

Theorem 1. (PAC-Bayesian Generalization Bound for Language Models.) With α as in Definition 2.1, consider a set of language models LM , with prior distribution π over LM .

Given any δ ∈ (0 , 1) , for any probability measure ρ over LM such that ρ, π share the same support, the following holds with probability at least 1 -δ over the random draw of S :

In Appendix B.2 we give a theoretical estimation of ϱ , to illustrate the bound we derive is non-vacuous, i.e., less than 1 . The KL term is of order O ( K ) where K is the number of parameters involved in π, ρ , and n can be shown to greatly exceed K (using a realistic Zipf distribution assumption on prompts to estimate the number of unique prompts). Theorem 1 tells us that, as long as pretraining successfully reduces the loss on the training corpus ( ˆ R S ( p LM ) ↓ ), in expectation the language model will mimic the world well (small ℓ TV difference) on a given direct prompt sampled from D P . Furthermore, if α is not too small, then this statement holds on a direct prompt whose concept is harmful. Since we have defined the harmful concept as outputting harmful explanations with high probability (Definition 2.1), we conclude that an LM trained on D P data can output explanations in the harmful set.

4 A statistical perspective on jailbreaking after alignment

In this section, we will present the main theoretical contribution of our work: given our assumptions hold, we prove the existence of ways for an adversary to jailbreak an LM even after the preference alignment process . Our proof strategy is inspired by the work on adversarial robustness [41], which bounds the adversary's probability of success by upper bounding the volume of the set of points that does not allow for the existence of adversarial examples. Going forward, we need to extend our framework to integrate alignment and jailbreaking.

After an LM is pretrained, it typically will undergo fine-tuning on a dataset containing preferred behaviour. In what follows, we will assume that this alignment process does not change the model performance in the sense that the LM will still produce semantically meaningful explanations (Definition 2.1). It would not, for example, default to answering any request with the same response.

Assumption 4.1. (LM outputs semantically meaningful explanations) For any harmful concept c , and all plausible prompts ( q, c ) ∈ dom( p world ) ,

In other words, we assume the LM's output distribution is accurately supported on E h ( c ) ∪ E s ( c ) , in the sense that the size of 'residual' E n ( c ) is relatively small compared to these semantically meaningful explanations. We define n ( c ) = | E n ( c ) | + | E s ( c ) | + | E h ( c ) | . We omit the ( c ) annotations

when clear from the context. The O (1) statement is reasonable, because harmful explanations are usually long text fragments that allow for many alternative formulations. The assumption can be broken down into two components: (1) within the support of the output distribution, only occasional instances of unrelated explanations exist; (2) the process of aligning the model towards safety does not eliminate the harmful explanations acquired during the pretraining phase. For part (1), similar to the example we gave above, under normal circumstances, we do not expect the explanation 'Paris' to appear in dom( p LM ( q, c )) given ( q, c ) as 'How to build a bomb' . As for part (2), though seemingly surprising, evidence with a series of current state-of-the-art LMs can be experimentally validated [48], where diverse, harmful explanations are extracted by simply manipulating the decoding process using direct prompts. In Section 5 we give an explanation for this undesired phenomenon.

To bound the likelihood of jailbreaking we first need to specify how the output of a LM interacts with its support. Assuming a fixed order of explanations in dom( p LM ( q, c )) , and slight abuse of notation, we can use p LM ( q, c ) to denote an n ( c ) -dimensional vector on ∆ n ( c ) -1 , the probability simplex with n ( c ) elements, where each entry represents the probability of a single explanation. We call this simplex the output simplex related to a given concept c . Next, we can induce a distribution on this simplex given a posterior distribution γ over the set of language models LM , as follows.

Definition 4.1. ( Induced Distribution on Simplex, γ c ) Under the assumption that the LM outputs semantically meaningful explanations (Assumption 4.1), with a fixed prompt ( q, c ) and a posterior distribution γ over LM , the corresponding induced distribution: p LM ( q, c ) where LM ∼ γ is supported over a subset of the output simplex ∆ n -1 . This distribution is denoted as γ ( q,c ) , or γ c when the reference to q is clear from context.

Next, we will separate the output simplex into a harmful and safety zone. This definition is motivated by the observation that typically an adversary is deemed successful if

it can extract even a single harmful explanation for a given concept. This translates into a division of the output simplex, under Assumption 4.1, as follows.

Definition 4.2. ( Harmful Zone and Safety Zone ) For a given harmful concept c and a fixed LM, the output simplex is divided into a safety zone and a harmful zone , H s and H h , where a predefined threshold p ∈ [0 , 1] is used to quantify the distinction: p LM ( q, c ) ∈ H h if and only if ∑ e : e ∈ E h ( c ) p LM ( e | q, c ) ≥ p , and otherwise p LM ( q, c ) ∈ H s .

Before we introduce jailbreaking, the reader might wonder why we did not define alignment more clearly. This is because under the PAC framework, preference alignment is nothing but a transformation from ρ to some γ posterior defined over LM . Given this inability on fine-grained characterization of alignment, we instead provide the goal of it as follows. With the above notion, given a prompt ( q, c ) where c is harmful, its goal is to push the induced distribution γ c into the safety zone H s . Ideally, supp( γ c ) ⊂ H s ⇔ with probability 1 , the resulting LM is safe when encountering ( q, c ) . We are ready to introduce necessary concepts related to jailbreaking.

Definition 4.3. ( Jailbreaking ) Given a harmful concept c and a query q ′ , the prompt ( q ′ , c ) jailbreaks the LM iff p LM ( q ′ , c ) ∈ H h . We call such a prompt ( q ′ , c ) and query q ′ a jailbreaking prompt and jailbreaking query, respectively.

The threshold p for discriminating H h and H s should be very small, since it means in expectation the adversary needs to call the LM 1 p times to collect a single harmful explanation i.e., to jailbreak the LM.

To theoretically prove the jailbreaking effect, we need to restrict the adversary's ability. To achieve this goal, we borrow insights from adversarial attacks, to assume that the adversary has bounded manipulating capability on the output simplex when searching over the query set:

Assumption 4.2. ( ϵ -bounded adversary) Given an LM, a harmful concept c and an associated direct prompt ( q, c ) , we assume the adversary can find a set of queries Q ′ , such that the output is moved at

most ϵ on the simplex towards H h from p LM ( q, c ) :

Here d is a distance measure between two discrete distributions. d can be a typical ℓ p measure with p ≥ 1 , or the Total Variation / Jensen-Shannon Divergence. We call q ′ ∈ Q ′ an ϵ -bounded query.

A conceptual illustration of our framework is depicted in Figure 2. Before arriving at our Theorem, we give the final definition of ϵ -expansion.

Definition 4.4. ( ϵ -expansion) Given a set A ⊂ ∆ n -1 and a distance measure d , the ϵ -expansion set A ( ϵ, d ) is defined as

We are ready to present the following theorem, which states that as long as the induced posterior γ c is not concentrated in an extremely safe area, then with high probability the model can be jailbroken. The proof is in Appendix B.3.

Theorem 2. (Jailbreak is unavoidable) Assume that an LMs output semantically meaningful explanations (Assumption 4.1). Given any γ posterior distribution over LM , choose a harmful concept c with a direct prompt ( q, c ) and a threshold p (Definition 2.1), to define the corresponding induced distribution γ c (Definition 4.1) and division over output simplex (Definition 4.2). An ϵ -bounded adversary (Assumption 4.2) can find a jailbreaking prompt (Definition 4.3) with probability at least

by using either the direct prompt, such that p LM ( q, c ) ∈ H h ; or

by finding an ϵ -bounded query q ′ , such that p LM ( q ′ , c ) ∈ H h .

Here, Φ( · ) is the standard Gaussian cdf, γ s := max x ∈H s -H h ( ϵ,d ) γ c ( x ) U ( x ) , with U ( x ) the uniform distribution over ∆ n -1 , and a ϵ := a + √ n -1 ϵ , where a writes analytically as a ≍ | E h ( c ) |-1 -( n -1) p √ ( n -1) p (1 -p ) .

Trivially, the chances of an adversary to find a jailbreaking prompt increase for stronger adversaries ( ϵ ↑ ). In the real world, this could relate to how much compute budget we allow to alter a query for a specific harmful concept. Furthermore, the chances of an adversary to find a jailbreaking prompt increase when the ratio of the sizes of the harmful explanation set to the safe explanation set is larger | E h ( c ) | | E s ( c ) | ↑ . This is because their ratio will determine the size of the harmful zone which in turn will cause Φ( a ϵ ) → 1 . In real world settings, for any harmful concept, the training corpus naturally contains a large harmful set due to the number of possible responses. Realistically, its size can not be countered by any manually-constructed safe set. Hence achieving alignment is hard : Recall that the goal of alignment is to respond with only safe explanations with high probability. However, we just learned that to increase that probability, we need to have a small harmful-to-safety set ratio which we discussed is not realistic. Consequently, the safety zone is going to be small.

5 E-RLHF: improving alignment by expanding the safety zone

Recall from Theorem 2 and the subsequent discussion in the previous section, that jailbreaking becomes more likely the larger the harmful zone is in comparison to the safety zone. The size of both zones relates to the size of their respective explanation sets. In other words, the size of the preference alignment dataset is crucial to successful alignment. Unfortunately, the human labor involved in creating such a dataset effectively caps its size.

In order to bridge the gap between our theoretical insights and a practical solution towards suppressing the jailbreaking problem, we focus on other more practical ways to expand the safety zone . Even though our ideas are more broadly applicable, in our experiments we will focus on improving Reinforcement Learning with Human Feedback (RLHF). RLHF typically includes three phases: i) supervised fine-tuning (SFT); ii) preference sampling and reward learning and iii) RL optimization. Rafailov et al. [23] have recently proposed a widely applied version of RLHF for LMs, coined Direct Preference Optimization (DPO), that employs a clever reparameterization which leads to directly learning from the preference dataset, without the need of obtaining a reward model beforehand.

DPO is more stable in the training process than other implementations of RLHF. A more complete overview of RLHF and DPO can be found in Appendix C.

For our purposes, we assume access to an LLM p SFT that has been supervised fine-tuned on highquality data. We further assume access to a preference aligned dataset D s ; that contains a set of text prompts ( q, c ) = x , and two respective explanations that have been rated by human annotators as better e w or worse e l . In phase ii) of RLHF, one typically optimizes a reward model r ( x, e ) based on the annotated explanations. Our proposal concerns phase iii) of the RLHF process: training of the preference aligned model p LM . For a given reward model, p LM is typically obtained by minimizing the following objective:

Note that, the first term is maximizing the reward, while the KL-term acts as a regularizer ensuring the aligned model can not drift off too far from the SFT model. We claim that this regularization is exactly the problem for safety. While designed to keep the model helpful 7 , for any harmful prompt x h and any harmful explanation e ∈ supp( p SFT ( x h )) , regularizing p LM to p SFT lets p LM maintain e in the support of the output distribution. Specifically, the supervised fine-tuning process does not involve elimination of any harmful explanations from the support of the output distribution of the pretrained model, 8 which leads to the fact that p SFT ( x h ) supports harmful responses that can not be negated with a realistically sized preference aligned dataset. Thus, this small safe set problem of p SFT will be further passed to p LM , even if p LM is optimized to the optimal solution of the above objective. This suggests we should not push p LM into the direction guided by p SFT. Instead, for any harmful prompt x h , we propose to use a distribution that we design to have a large safety zone to replace p SFT ( x h ) . Inspired by our framework, we can achieve this by modifying the harmful prompt , such that a safe concept is used to replace the harmful concept, which alters supp( p SFT ( · )) to include more safe explanations. For example, when x h is 'Tell me how to make a bomb' , instead of aligning p LM ( x h ) to p SFT ( x h ) , now we align p LM ( x h ) to p SFT ( x s ) where x s is 'Tell me how to reject a request of making a bomb'. This can be done in an individualized manner for all harmful prompts in D s , or simply by prefacing all harmful prompts with a prefix such as "Please ensure your response adheres to community guidelines and ethical standards:" . 9 Importantly, the non-harmful prompts are not modified. Due to the focus of our approach to expand the safety zone of the output distribution, we coin our proposal E-RLHF , resulting in the following modification to Eq. (2):

where x s is a safety-transformed version of the original harmful prompt x h . To recap, the key argument we put forth is that, in order to ensure the stability of model fine-tuning, it is not imperative to utilize identical prompt inputs x for both the reference model and the target model, particularly when the original input x itself is harmful. In fact, as long as the substitute or "anchor" prompt generates logically reasonable outputs akin to those produced by the original prompt, this approach would not impede the training process of the model. To solidify our argument we show the impact of our modification on the support of the optimal policy in Appendix C. We also deduce there that we can trivially integrate our modification into the DPO objective allowing us to train without an explicit reward model (eliminates step ii)) as follows, where σ ( · ) stands for the sigmoid function:

6 Experiments and results

Our experimental set-up is based on the alignment-handbook code base [50]. We tune the publiclyavailable SFT model p SFT provided by huggingface hub [51], using the public dataset [52, 53], with default hyperparameter setup. We label harmful prompts in the preference dataset by prompting GPT-3.5-Turbo, see Appendix E. We are using the very same prefix proposed in the previous section to generate x s . Experiments are performed on 8 NVIDIA Tesla V100 GPUs, using half-precision

tuning i.e., Float16. In the appendix, we also show results for an alternative training paradigm: the Low-Rank Adaptation (LoRA) [54] (see Appendix D.1). Following community standards [3, 1, 2], we use greedy decoding i.e., T = 0 for model evaluation.

Wefirst show empirical evidence that our proposed modification of DPO, E-DPO, does in fact improve safety alignment, using the Harmbench dataset [2] and the first 100 prompts in the AdvBench harmful behavior dataset [1], measured by the HarmBench protocol. We give an overview on all adversaries in Appendix F. The results are presented in Table 1. E-DPO achieves improvements across every task we tested.

On top of our safety results, we want to make sure E-RLHF does not sacrifice helpfulness for increased safety . We evaluate helpfulness with the MT-Bench project [3]. The SFT model p SFT receives a score of 6.3, and both the DPO and E-DPO models perform better than that (6.9 and 6.7 respectively), making us believe that performance degradation is not a problem with our proposal. Next, we show the impact of the safe prefix on model performance. We demonstrate that our method's performance depends on the choice of safe prefix to some extend but never fails (see Appendix D.2). We believe, finding better safe prefixes by explicit tuning would improve our results, similar to the work by Yang et al. [55], but we leave this exploration for future work. Further, we confirm that the improvement arises from using a safe prior in the KL term for harmful prompts. We ablate our results by appending the prefix on all prompts in the preference alignment dataset (see Appendix D.3). In all cases, applying the safe prefix to usual prompts degrades safety, showcasing the importance of switching the prior only on the harmful prompts. Finally, we show that E-DPO can be combined with any system prompt, to further boost safety (see Appendix D.4). The proposal can even be used to improve helpfulness and safety simultaneously (see Appendix D.5).

7 Conclusion and discussions

In this paper, we present a theoretical framework for language model pretraining and jailbreaking by dissecting input prompts into query and concept pairs. Through this approach, we have established two theoretical results pertaining to the ability of language models to mimic the world following pretraining, which leads to outputting harmful explanations given harmful prompts; and the inevitability of jailbreaking resulting from alignment challenges. Guided by these theoretical insights, we have devised a simple yet effective technique to enhance safety alignment, and demonstrate the improved resilience to jailbreak attacks with this methodology.

Current limitations (1) Although we have classified concepts as either harmful or non-harmful, it is important to acknowledge that the perception of a concept's potential for harm can be influenced by various factors such as cultural, legal, and societal norms, which collectively form the context of the situation. (2) Language models have demonstrated impressive capabilities in reasoning and completing tasks within multi-round, multi-step conversations; our current framework may not fully account for the generalization and jailbreaking possibilities associated with such input formats. (3) Our analysis is grounded on a fixed p world mapping and D P distribution. Nevertheless, the world is inherently dynamic, as both p world and D P continually evolve.

Future work (1) Regarding our E-RLHF approach, as highlighted in the experimental section, in addition to attaching a universally safe prefix to all harmful prompts, improvements can be achieved by individually transforming the harmful prompts. Moreover, the safety-transformed prompts can be employed to expand the preference dataset for conventional RLHF. (2) Throughout our analysis, we have not imposed any constraints on the capacity of the language model. Extending our analysis under finite memory constraints or analyzing hallucination properties of LLMs is an interesting direction to explore. (3) Large language models have shown remarkable capabilities as in-context learners [56], and such techniques could potentially be used for jailbreaking them as well [57-59]. Investigating the incorporation of such input paradigms remains a promising avenue for future research.

References

Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043 , 2023. 1, 2, 9, 27, 28, 29, 30, 31, 32

Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249 , 2024. 1, 2, 9, 27, 28, 29

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 37, 2023. 1, 9, 28

Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 , 2023. 1

Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature , 620(7972):172-180, 2023. 1

Kaiyu Yang, Aidan Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan J Prenger, and Animashree Anandkumar. Leandojo: Theorem proving with retrieval-augmented language models. Advances in Neural Information Processing Systems , 36, 2024. 1

Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency , pages 610-623, 2021. 1

Julian Hazell. Large language models can be used to effectively scale spear phishing campaigns. arXiv preprint arXiv:2305.06972 , 2023.

Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. Jailbreaking chatgpt via prompt engineering: An empirical study. arXiv preprint arXiv:2305.13860 , 2023. 1, 30

OpenAI. Chatgpt. https://openai.com/index/chatgpt/ , 2022. 1

DAN. Do anything now prompt. https://github.com/0xk1h0/ChatGPT_DAN , 2023. 1

Reddit. Chatgpt grandma exploit. https://www.reddit.com/r/ChatGPT/comments/ 12sn0kk/grandma_exploit/ , 2023. 1

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023. 2, 8, 22, 26

Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858 , 2022. 2, 31

Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.03286 , 2022. 29

Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan Hadfield-Menell. Explore, establish, exploit: Red teaming language models from scratch. arXiv preprint arXiv:2306.09442 , 2023.

Zhang-Wei Hong, Idan Shenfeld, Tsun-Hsuan Wang, Yung-Sung Chuang, Aldo Pareja, James R Glass, Akash Srivastava, and Pulkit Agrawal. Curiosity-driven red-teaming for large language models. In The Twelfth International Conference on Learning Representations , 2023.

Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, et al. Rainbow teaming: Open-ended generation of diverse adversarial prompts. arXiv preprint arXiv:2402.16822 , 2024. 2, 31

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 , 2019. 2, 25, 31

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022. 25, 31

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 , 2022. 25, 31

Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. In International Conference on Machine Learning , pages 17506-17533. PMLR, 2023. 31

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290 , 2023. 2, 3, 7, 25, 26, 33

Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. Autodan: Automatic and interpretable adversarial attacks on large language models. arXiv preprint arXiv:2310.15140 , 2023. 2, 30

Raz Lapid, Ron Langberg, and Moshe Sipper. Open sesame! universal black box jailbreaking of large language models. arXiv preprint arXiv:2309.01446 , 2023. 2, 30

Zheng-Xin Yong, Cristina Menghini, and Stephen H Bach. Low-resource languages jailbreak gpt-4. arXiv preprint arXiv:2310.02446 , 2023. 2, 31

Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. Multilingual jailbreak challenges in large language models. arXiv preprint arXiv:2310.06474 , 2023. 31

Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher. arXiv preprint arXiv:2308.06463 , 2023. 2, 31

Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451 , 2023. 2, 29, 30

Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419 , 2023. 28, 29, 30

Xiaoxia Li, Siyuan Liang, Jiyi Zhang, Han Fang, Aishan Liu, and Ee-Chien Chang. Semantic mirror jailbreak: Genetic algorithm based jailbreak prompts against open-source llms. arXiv preprint arXiv:2402.14872 , 2024. 30

Peng Ding, Jun Kuang, Dan Ma, Xuezhi Cao, Yunsen Xian, Jiajun Chen, and Shujian Huang. A wolf in sheep's clothing: Generalized nested jailbreak prompts can fool large language models easily. arXiv preprint arXiv:2311.08268 , 2023. 30

Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box llms automatically. arXiv preprint arXiv:2312.02119 , 2023. 2, 29, 30

Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684 , 2023. 2, 32, 33

Bochuan Cao, Yuanpu Cao, Lu Lin, and Jinghui Chen. Defending against alignment-breaking attacks via robustly aligned llm. arXiv preprint arXiv:2309.14348 , 2023. 32

Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Soheil Feizi, and Hima Lakkaraju. Certifying llm safety against adversarial prompting. arXiv preprint arXiv:2309.02705 , 2023. 32

Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Pingyeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614 , 2023. 2, 32

Yotam Wolf, Noam Wies, Yoav Levine, and Amnon Shashua. Fundamental limitations of alignment in large language models. arXiv preprint arXiv:2304.11082 , 2023. 2, 3, 33

Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 , 2013. 2

Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In International Conference on Learning Representations , 2018. 2

Ali Shafahi, W Ronny Huang, Christoph Studer, Soheil Feizi, and Tom Goldstein. Are adversarial examples inevitable? In International Conference on Learning Representations , 2018. 5

Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In international conference on machine learning , pages 1310-1320. PMLR, 2019. 2

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 , 2022. 4

David A McAllester. Some pac-bayesian theorems. In Proceedings of the eleventh annual conference on Computational learning theory , pages 230-234, 1998. 4

Sokhna Diarra Mbacke, Florence Clerc, and Pascal Germain. Pac-bayesian generalization bounds for adversarial generative models. arXiv preprint arXiv:2302.08942 , 2023. 5

Pascal Germain, Alexandre Lacasse, François Laviolette, and Mario Marchand. Pac-bayesian learning of linear classifiers. In Proceedings of the 26th Annual International Conference on Machine Learning , pages 353-360, 2009. 5

Maxime Haddouche, Benjamin Guedj, Omar Rivasplata, and John Shawe-Taylor. Pac-bayes unleashed: Generalisation bounds with unbounded losses. Entropy , 23(10):1330, 2021. 5

Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. Catastrophic jailbreak of open-source llms via exploiting generation. arXiv preprint arXiv:2310.06987 , 2023. 6, 27, 30

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 , 2023. 8

Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alexander M. Rush, and Thomas Wolf. The alignment handbook. https://github. com/huggingface/alignment-handbook , 2023. 8

HuggingFace. Huggingface constitutional-ai sft model. https://huggingface.co/ alignment-handbook/mistral-7b-sft-constitutional-ai , 2024. 8

HuggingFace. Huggingface constitutional-ai dataset: Ultrafeedback-binarized. https:// huggingface.co/datasets/HuggingFaceH4/ultrafeecback_binarized , 2024. 8

HuggingFace. Huggingface constitutional-ai dataset: Cai-conversation-harmless. https:// huggingface.co/datasets/HuggingFaceH4/cai-conversation-harmless , 2024. 8

Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2021. 9, 27

Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXiv preprint arXiv:2309.03409 , 2023. 9, 26

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:18771901, 2020. 10

Zeming Wei, Yifei Wang, and Yisen Wang. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.06387 , 2023. 10, 31

Jiongxiao Wang, Zichen Liu, Keun Hee Park, Muhao Chen, and Chaowei Xiao. Adversarial demonstration attacks on large language models. arXiv preprint arXiv:2305.14950 , 2023. 31

Cem Anil, Esin Durmus, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Nina Rimsky, Meg Tong, Jesse Mu, Daniel Ford, et al. Many-shot jailbreaking, 2024. 10, 31

Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep learning. Advances in neural information processing systems , 30, 2017. 21

Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. Advances in neural information processing systems , 32, 2019. 21

Robert D Gordon. Values of mills' ratio of area to bounding ordinate and of the normal probability integral for large values of the argument. The Annals of Mathematical Statistics , 12(3):364-366, 1941. 23

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017. 25

Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324-345, 1952. 25

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017. 25

Banghua Zhu, Hiteshi Sharma, Felipe Vieira Frujeri, Shi Dong, Chenguang Zhu, Michael I Jordan, and Jiantao Jiao. Fine-tuning language models with advantage-induced policy alignment. arXiv preprint arXiv:2306.02231 , 2023. 25

Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo. arXiv preprint arXiv:2005.12729 , 2020. 25

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824-24837, 2022. 26

Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861 , 2021. 26

Xiaotian Zou, Yongkang Chen, and Ke Li. Is the system message really important to jailbreaks in large language models? arXiv preprint arXiv:2402.14857 , 2024. 27, 32

Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. Gradient-based adversarial attacks against text transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 5747-5757, 2021. 29, 30

Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980 , 2020. 29, 30

Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. arXiv preprint arXiv:2401.06373 , 2024. 29, 31

Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. arXiv preprint arXiv:2308.03825 , 2023. 29, 30

Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing nlp. arXiv preprint arXiv:1908.07125 , 2019. 29

Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. Advances in Neural Information Processing Systems , 36, 2024. 29

Erik Jones, Anca Dragan, Aditi Raghunathan, and Jacob Steinhardt. Automatically auditing large language models via discrete optimization. arXiv preprint arXiv:2303.04381 , 2023. 30

Fábio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527 , 2022. 30

Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? In Thirty-seventh Conference on Neural Information Processing Systems , 2023. 30

Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. Jailbreaker: Automated jailbreak across multiple large language model chatbots. arXiv preprint arXiv:2307.08715 , 2023. 30

Jiahao Yu, Xingwei Lin, and Xinyu Xing. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.10253 , 2023. 30

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems , 36, 2024. 30

Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. Deepinception: Hypnotize large language model to be jailbreaker. arXiv preprint arXiv:2311.03191 , 2023. 30

Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh. Drattack: Prompt decomposition and reconstruction makes powerful llm jailbreakers. arXiv preprint arXiv:2402.16914 , 2024. 30

Zhenhua Wang, Wei Xie, Baosheng Wang, Enze Wang, Zhiwen Gui, Shuoyoucheng Ma, and Kai Chen. Foot in the door: Understanding large language model jailbreaking via cognitive psychology. arXiv preprint arXiv:2402.15690 , 2024. 30

Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, and Yuandong Tian. Advprompter: Fast adaptive adversarial prompting for llms. arXiv preprint arXiv:2404.16873 , 2024. 30

Hangfan Zhang, Zhimeng Guo, Huaisheng Zhu, Bochuan Cao, Lu Lin, Jinyuan Jia, Jinghui Chen, and Dinghao Wu. On the safety of open-sourced large language models: Does alignment really prevent them from being misused? arXiv preprint arXiv:2310.01581 , 2023. 30

Xuandong Zhao, Xianjun Yang, Tianyu Pang, Chao Du, Lei Li, Yu-Xiang Wang, and William Yang Wang. Weak-to-strong jailbreaking on large language models. arXiv preprint arXiv:2401.17256 , 2024. 30

Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. Shadow alignment: The ease of subverting safely-aligned language models. arXiv preprint arXiv:2310.02949 , 2023. 30

Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, and Daniel Kang. Removing rlhf protections in gpt-4 via fine-tuning. arXiv preprint arXiv:2311.05553 , 2023. 30

Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693 , 2023. 30

Erfan Shayegani, Yue Dong, and Nael Abu-Ghazaleh. Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models. arXiv preprint arXiv:2307.14539 , 2023. 31

Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Mengdi Wang, and Prateek Mittal. Visual adversarial examples jailbreak large language models. arXiv preprint arXiv:2306.13213 , 2023. 31

Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Irena Gao, Pang Wei Koh, Daphne Ippolito, Florian Tramèr, and Ludwig Schmidt. Are aligned neural networks adversarially aligned? In Thirty-seventh Conference on Neural Information Processing Systems , 2023. 31

Natalie Maus, Patrick Chao, Eric Wong, and Jacob R Gardner. Black box adversarial prompting for foundation models. In The Second Workshop on New Frontiers in Adversarial Machine Learning , 2023. 31

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/ . 31

Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. Exploiting programmatic behavior of llms: Dual-use through standard security attacks. arXiv preprint arXiv:2302.05733 , 2023. 31

Rusheb Shah, Soroush Pour, Arush Tagade, Stephen Casper, Javier Rando, et al. Scalable and transferable black-box jailbreaks for language models via persona modulation. arXiv preprint arXiv:2311.03348 , 2023. 31

Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu. Cold-attack: Jailbreaking llms with stealthiness and controllability. arXiv preprint arXiv:2402.08679 , 2024. 31

Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. Cold decoding: Energybased constrained text generation with langevin dynamics. Advances in Neural Information Processing Systems , 35:9538-9551, 2022. 31

Somnath Banerjee, Sayan Layek, Rima Hazra, and Animesh Mukherjee. How (un) ethical are instruction-centric responses of llms? unveiling the vulnerabilities of safety guardrails to harmful queries. arXiv preprint arXiv:2402.15302 , 2024. 31

Neal Mangaokar, Ashish Hooda, Jihye Choi, Shreyas Chandrashekaran, Kassem Fawaz, Somesh Jha, and Atul Prakash. Prp: Propagating universal perturbations to attack large language model guard-rails. arXiv preprint arXiv:2402.15911 , 2024. 31

Huijie Lv, Xiao Wang, Yuansen Zhang, Caishuang Huang, Shihan Dou, Junjie Ye, Tao Gui, Qi Zhang, and Xuanjing Huang. Codechameleon: Personalized encryption framework for jailbreaking large language models. arXiv preprint arXiv:2402.16717 , 2024. 31

Vinu Sankar Sadasivan, Shoumik Saha, Gaurang Sriramanan, Priyatham Kattakinda, Atoosa Chegini, and Soheil Feizi. Fast adversarial attacks on language models in one gpu minute. arXiv preprint arXiv:2402.15570 , 2024. 31

Jonas Geiping, Alex Stein, Manli Shu, Khalid Saifullah, Yuxin Wen, and Tom Goldstein. Coercing llms to do and reveal (almost) anything. arXiv preprint arXiv:2402.14020 , 2024. 31

Qibing Ren, Chang Gao, Jing Shao, Junchi Yan, Xin Tan, Wai Lam, and Lizhuang Ma. Exploring safety generalization challenges of large language models via code, 2024. 31

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations , 2018. 31

Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , pages 308-318, 2016. 31

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022. 31

Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267 , 2023. 31

Gabriel Alon and Michael Kamfonas. Detecting language model attacks with perplexity. arXiv preprint arXiv:2308.14132 , 2023. 32

Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems , 30, 2017. 32

Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. Defending chatgpt against jailbreak attack via self-reminders. Nature Machine Intelligence , pages 1-11, 2023. 32

Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. Prompt-driven llm safeguarding via directed representation optimization. arXiv preprint arXiv:2401.18018 , 2024. 32

Andy Zhou, Bo Li, and Haohan Wang. Robust prompt optimization for defending language models against jailbreaking attacks. arXiv preprint arXiv:2401.17263 , 2024. 32

Yichuan Mo, Yuji Wang, Zeming Wei, and Yisen Wang. Studious bob fight back against jailbreaking via prompt adversarial tuning. arXiv preprint arXiv:2402.06255 , 2024. 32

Zhexin Zhang, Junxiao Yang, Pei Ke, and Minlie Huang. Defending large language models against jailbreaking attacks through goal prioritization. arXiv preprint arXiv:2311.09096 , 2023. 32

Yujun Zhou, Yufei Han, Haomin Zhuang, Taicheng Guo, Kehan Guo, Zhenwen Liang, Hongyan Bao, and Xiangliang Zhang. Defending jailbreak prompts via in-context adversarial game. arXiv preprint arXiv:2402.13148 , 2024. 32

Alec Helbling, Mansi Phute, Matthew Hull, and Duen Horng Chau. Llm self defense: By self examination, llms know they are being tricked. arXiv preprint arXiv:2308.07308 , 2023. 32

Zezhong Wang, Fangkai Yang, Lu Wang, Pu Zhao, Hongru Wang, Liang Chen, Qingwei Lin, and Kam-Fai Wong. Self-guard: Empower the llm to safeguard itself. arXiv preprint arXiv:2310.15851 , 2023. 32

Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. Rain: Your language models can align themselves without finetuning. arXiv preprint arXiv:2309.07124 , 2023. 32

Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Poovendran. Safedecoding: Defending against jailbreak attacks via safety-aware decoding. arXiv preprint arXiv:2402.08983 , 2024. 32

Adib Hasan, Ileana Rugina, and Alex Wang. Pruning for protection: Increasing jailbreak resistance in aligned llms without fine-tuning. arXiv preprint arXiv:2401.10862 , 2024. 32

Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695 , 2023. 32

Renjie Pi, Tianyang Han, Yueqi Xie, Rui Pan, Qing Lian, Hanze Dong, Jipeng Zhang, and Tong Zhang. Mllm-protector: Ensuring mllm's safety without hurting performance. arXiv preprint arXiv:2401.02906 , 2024. 32

Yuqi Zhang, Liang Ding, Lefei Zhang, and Dacheng Tao. Intention analysis prompting makes large language models a good jailbreak defender. arXiv preprint arXiv:2401.06561 , 2024. 32

Yihan Wang, Zhouxing Shi, Andrew Bai, and Cho-Jui Hsieh. Defending llms against jailbreaking attacks via backtranslation. arXiv preprint arXiv:2402.16459 , 2024. 32

Heegyu Kim, Sehyun Yuk, and Hyunsouk Cho. Break the breakout: Reinventing lm defense against jailbreak attacks with self-refinement. arXiv preprint arXiv:2402.15180 , 2024. 32

Yifan Zeng, Yiran Wu, Xiao Zhang, Huazheng Wang, and Qingyun Wu. Autodefense: Multiagent llm defense against jailbreak attacks, 2024. 32

Xiaomeng Hu, Pin-Yu Chen, and Tsung-Yi Ho. Gradient cuff: Detecting jailbreak attacks on large language models by exploring refusal loss landscapes. arXiv preprint arXiv:2403.00867 , 2024. 33

Jiabao Ji, Bairu Hou, Alexander Robey, George J Pappas, Hamed Hassani, Yang Zhang, Eric Wong, and Shiyu Chang. Defending large language models against jailbreak attacks via semantic smoothing. arXiv preprint arXiv:2402.16192 , 2024. 33

Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674 , 2023. 33

Swapnaja Achintalwar, Adriana Alvarado Garcia, Ateret Anaby-Tavor, Ioana Baldini, Sara E. Berger, Bishwaranjan Bhattacharjee, Djallel Bouneffouf, Subhajit Chaudhury, Pin-Yu Chen, Lamogha Chiazor, Elizabeth M. Daly, Rogério Abreu de Paula, Pierre Dognin, Eitan Farchi, Soumya Ghosh, Michael Hind, Raya Horesh, George Kour, Ja Young Lee, Erik Miehling, Keerthiram Murugesan, Manish Nagireddy, Inkit Padhi, David Piorkowski, Ambrish Rawat, Orna Raz, Prasanna Sattigeri, Hendrik Strobelt, Sarathkrishna Swaminathan, Christoph Tillmann, Aashka Trivedi, Kush R. Varshney, Dennis Wei, Shalisha Witherspooon, and Marcel Zalmanovici. Detectors for safe and reliable llms: Implementations, uses, and limitations, 2024. 33

Adam Tauman Kalai and Santosh S Vempala. Calibrated language models must hallucinate. arXiv preprint arXiv:2311.14648 , 2023. 33

Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K Kummerfeld, and Rada Mihalcea. A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity. arXiv preprint arXiv:2401.01967 , 2024. 33

Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. arXiv preprint arXiv:2402.05162 , 2024. 33

Appendix

A Glossary

B Proof of Theorems

B.1 Proof of PAC-Bayesian bounds

Definition B.1. (Bounded Difference) A function f : X n → R is said to have bounded difference property w.r.t. a collection of constants c 1 , · · · , c n , iff

Lemma B.1. (Hoeffding's Lemma) for random variable X ∈ [ a, b ] with probability 1, the following holds:

Lemma B.2. (Hoeffding's Lemma, Multivariate) for random variables Z = f ( x 1 , · · · , x n ) where f has the bounded difference property, the following holds:

Note that substituting Z with ˆ R S ( LM ) is valid.

Lemma B.3. Empirical Loss defined in Definition 3.1 satisfies the bounded difference condition with constant c = 1 , ∀ i .

We are ready to present the proof of Theorem 1.

Proof. Starting with the above lemma, we know

The above result holds for a manually picked LM. With an overall average over the prior π we have

Apply Fubini's theorem (note that π is independent of S ):

Define Y = E LM ∼ π [exp( λ ( R ( LM ) -ˆ R S ( LM )))] , a random variable depends on S . Obviously Y ≥ 0 . Thus, with Markov's inequality:

Equivalently, with probability at least 1 -δ , we have

Since we have assumed π, ρ share the same support, using Radon-Nykodim derivative to change the expectation with respect to π to with respect to ρ , we have

Taking logarithm and applying Jensen's Inequality we know

Incorporating c = 1 , noticing dρ dπ = ( dπ dρ ) -1 we could rewrite the inequality as

Finding λ that minimizes the term on right hand side gives us the ϱ term.

When D P allows for a decomposition into mixture components, noticing the linearty of expectation, the bound can be re-written as

which leads to

B.2 An estimation on the non-vacuousness of the PAC bound

We give an estimation of the term appears in our PAC bound, ϱ , and state that it is non-vacuous.

The numerator. We follow Neyshabur et al. [60] to instantiate the term in the simplest setup. Assume π, ρ are defined over the parameter space of a given LM, with K parameters. Assume w is a set of weights learned from the pretraining corpus. Let the prior π be the zero-mean multivariate Gaussian, whose entry-wise variance is related to the magnitude of the weight: σ i = β | w i | , and ρ be a Gaussian with the same anisotropic variance centered around w . We argue though simple, both settings are practical, since Gaussian initialization is common for model training, and the SWA-Gaussian algorithm [61] utilizes such Gaussian posterior. Under this setup, the KL goes as ∑ i w 2 i 2 σ 2 i = O ( K ) .

Specifically, taking β = 2 2 makes the term exactly K . Current language models often possess millions, or billions, of parameters, namely, K ∼ [10 6 , 10 9 ] .

√

The denominator. To estimate the number of unique direct prompts in the training corpus, it is important to notice that the dataset does not only consist of ( q, c ) prompts but also e explanations. Thus, we need to estimate the average token length (ATL) associated with each unique prompt x = ( q, c ) . For each unique prompt x , aside from its own token length l ( x ) , there will be a collection of explanations { e i } N ( x ) i =1 , with expected token length of each associated explanation l ( e ) . We have

Fact. Given a prompt x , the larger the expected length of the prompt itself and explanation ( l ( x ) + l ( e ) ↑ ) , the larger the expected number of explanation elements ( N ( x ) ↑ ) , and the smaller the number of such prompts ( D P ( x ) ↓ ) , appearing in the training corpus. The former comes naturally due to the composability of natural language: the longer the text fragment, the more equivalent text fragments in expectation, while the latter is reflected by the spirit of the widely accepted Zipf's law.

Inspired by the fact, we assume prompts are categorized by the quantity of l ( x ) + l ( e ) , namely, for all prompt x , N ( x ) is a function of l ( x ) + l ( e ) . Moreover, the complete data generation process is decomposed into i) sample a value of l ( x ) + l ( e ) out, and then ii) sample a unique prompt from the set decided by this specific l ( x ) + l ( e ) value, and iii) generate N ( x ) explanations.

Step i). Use the fact: the larger the expected length of the output explanation, the smaller the probability that such a prompt appears in the training corpus. We assume step i) follows a (cut-off) zeta distribution. Specifically, for a random prompt x ,

When k 0 = 1 , we resume the zeta distribution with coefficient s .

Step ii). We assume each prompt following this step is unique.

Step iii). Use the fact: the larger the expected length of the output explanation, the larger the expected number of explanation elements in the training corpus. We assume a power law scaling on N , with a constant t > 1 , such that

Thus, the average token length writes

where ζ ( s ) = ∑ i ∈ Z + i -s is the Riemann zeta function.

For example, take s = 4 , t = 2 . With k 0 = 1 , the ATL would be 1 . 52 , while with k 0 = 10 , the ATL becomes 272 . These results translate into an estimation of unique prompts as n tokens /ATL . With current SOTA LM, the pretraining corpus often includes (tens of) trillions of tokens ( > 10 12 ), thus n > 10 10 > K can be safely assumed ⇒ ϱ < 1 .

α constant. According to LLaMa-2 report (section 4.1, Figure 13) [13], approximately 0 . 2% of the documents in their training corpus is labeled as harmful. However, we argue this is indeed an extremely loose lower bound for α , due to the estimation strategy used in their paper. Given a document, they use a binary classifier on harmfulness over each single line (1 means harmful and 0 otherwise), and assign the average score to the document. 0 . 2% is the ratio of documents with score ≥ 0 . 5 . Take the example of 'How to build a bomb' . The chemical reaction parts will not be counted as harmful, and thus this estimation strategy could judge a completely harmful explanation as harmless. Thus, it is reasonable to assert α is not too small, though with current literature we are not capable of raising an accurate estimation on it.

B.3 Proof of jailbreaking

Before proceeding to the proof, we list necessary definitions and lemmas as follows.

Lemma B.4. (Volume of n -simplex) 10 For any dimension n , the volume of the n -element probability simplex: ∆ n -1 , in the n -1 -dimensional space is

We define the projected probability simplex as follows.

Definition B.2. (Projected probability simplex) Given ∆ n -1 , the corresponding projected probability simplex, ∆ n -1 p , is defined as a subset of R n -1 : { x ∈ R n -1 | ∑ n -1 i =1 x i ≤ 1 , ∀ i ∈ [ n -1] } .

An illustration of ∆ n -1 and ∆ n -1 p . For example, take n = 3 . The probability simplex with n = 3 elements is a triangle whose (euclidean) side length is √ 2 with vertices (1 , 0 , 0) , (0 , 1 , 0) , (0 , 0 , 1) . Then its volume in the 2-dimensional space, i.e., its area, is √ 3 2 . The corresponding projected probability simplex is the triangle between the X -Y axis, with vertices (1 , 0) , (0 , 1) , (0 , 0) .

A direct lemma that connects the probability simplex and the projected probability simplex is given below.

Lemma B.5. (Transformation of probability simplex) Given a proper probability density function ν ( x ) defined on ∆ n -1 p , it is equivalent to the distribution defined on ∆ n -1 with density ν ( x ) √ n : ∀ A ∈ Borel (∆ n -1 p ) , let B = { x ∈ ∆ n -1 : x 1: n -1 ∈ A } . Then ∫ A ν ( x ) dx = ∫ B ν ( x ) √ n dx. Specifically, this implies volume ( A ) volume (∆ n -1 p ) = volume ( B ) volume (∆ n -1 ) .

Proof. Consider a translation on ∆ n -1 with x n = -∑ n -1 i =1 x i which does not affect its the volume and shape. The mapping: ∆ n -1 p → translated ∆ n -1 is an affine transformation with matrix

Thus, any area under this transformation is scaled by √ det T ⊤ T = √ n : a constant. The lemma follows directly after this conclusion.

We use U ( · ) to denote the uniform distribution over ∆ n -1 : U ( x ) = ( n -1)! √ n , ∀ x ∈ ∆ n -1 . We use the notation vol[ S ] = ∫ S 1 ds to represent the volume of a given subset S ⊂ ∆ n -1 , and use rvol[ S ] for the relative volume (w.r.t. the underlying n -simplex) of S , i.e., rvol[ S ] := vol[ S ] vol[∆ n -1 ] = ∫ S U ( x ) dx . We also use n = | E ( c ) | from now on. We use the vector x to denote (with the slight abuse of notation we have mentioned) p LM ( q, c ) on the output simplex.

Lemma B.6. (Gaussian cdf Tail Bound, Gordon [62]) Denote ϕ ( · ) as the standard Gaussian pdf. When x > 0 ,

Now we are ready to give the proof of Theorem 2.

Proof. Let | E h ( c ) | = n 0 and denote | E h ( c ) | + | E s ( c ) | + | E n ( c ) | = n . Without loss of generality, we define the first n 0 = | E h ( c ) | elements as the harmful explanations. Let the thresholding constant be p . That is, we define the harmful zone H h as { x ∈ ∆ n -1 | ∑ n 0 i =1 x i ≥ p } . To compute the relative volume of H h in ∆ n -1 , we could instead operate on the projected probability simplex ∆ n -1 p introduced in Definition B.2, and compute the relative volume of the projected H h : H h,p := { x ∈

∆ n -1 p | ∑ n 0 i =1 x i ≥ p } . Note that ∆ n -1 p ⊂ R n -1 . We derive its expression as follows.

Thus, the relative volume of H h can be written as

Which is precisely the binomial distribution formula. With the Central Limit Theorem, when n ≫ O (1) , we know the binomial distribution can be well approximated via the normal distribution as follows:

Thus, denote ϕ ( n -1) ,p ( x ) as the pdf of Gaussian variable with mean ( n -1) p , variance ( n -1) p (1 -p ) , the rvol term above can be estimated as follows:

We use a = | E h ( c ) |-1 -( n -1) p √ ( n -1) p (1 -p ) . Consider an adversary with budget ϵ under ℓ p or Jensen-Shannon Divergence (JSD) / Total Variation (TV) capability. Since || x || 1 ≥ || x || p , ∀ p ≥ 1 as well as || x || 1 ≥ 2JSD( x ) , || x || 1 ≥ 2TV( x ) , we know H h ( ϵ, ℓ 1 ) ⊂ H h ( ϵ, d ) for all d we have considered. With that ℓ 1 , ϵ setup, the corresponding ϵ -expansion set of H h has a closed-form expression as

Similar as above, we derive the analytical solution of its relative volume associated with constant a ′ as:

Under our framework, with p < 1 2 , we know 1 4 > p (1 -p ) > ( p -ϵ 2 )(1 -p + ϵ 2 )) . Thus

Consider the induced distribution γ c on the output simplex. Given an adversary with ℓ p or JSD/TV perturbing capability, with the fixed harmful concept c , safety is guaranteed if and only if p LM ( q, c ) resides outside H h ( ϵ, d ) . Define the area of interest, S ( d ) as S ( d ) := ∆ n -1 -H h ( ϵ, d ) . Thus, the probability of this event could be bounded as

This gives an upper bound of

which can be simplified when a ≥ 0 using Lemma B.6:

Thus, the probability of getting a LM instance from the preference alignment process such that it allows for successful jailbreaking on a specific harmful concept c is at least

Up to now, we have derived the result in Theorem 2. However, we can move a step further to show the decay rate on the right hand side term. It can be simplified when a ≥ 0 :

which finishes the proof.

C RLHF, DPO and our E-RLHF

The classic RLHF framework was established by Christiano et al. [63], and developed by Ziegler et al. [19], Ouyang et al. [20], Bai et al. [21]. After the collection of a preference dataset D s = { ( x, e w , e l ) } , one first trains a reward model under the Bradley-Terry model [64], with the objective, where σ ( · ) stands for the sigmoid function:

Following, proximal policy optimization (PPO) [65] is commonly adopted across these implementations, forming the basis of current state-of-the-art language models. The KL-constrained RL Fine-Tuning (RLFT) objective takes the form:

However, PPO tuning can suffer from instability [66] and implementation complication [67]. To overcome these issues, a series of work propose to skip the reward modeling step and learn directly from the preference dataset, with the representative pioneering work by Rafailov et al. [23], namely Direct Preference Optimization (DPO) . We summarize the derivation of the DPO objective below, and generalize the objective to the one we use in our experiments, i.e., E-DPO.

First, noticing the closed-form optimal solution for p LM of the RLFT objective writes (see e.g., Appendix A.1 of Rafailov et al. [23])

With this analytical solution in mind, we can solve the reward as

Regard π ∗ as the optimization target, plug this transformation into the reward model objective to obtain the DPO objective:

For our E-RLHF, the modification to the objective leads to another optimal solution of p LM :

Thus,

And plug it in to the reward model objective to formulate our E-DPO:

The advantage of our E-RLHF objective is as follows.

Proposition C.1. (Overcoming the small safe set problem) E-RLHF will lead to the optimal solution p ∗ :

Compared to p ∗ RLFT , the advantage when encountering a harmful prompt x is:

(1) (Erase harmful explanations) ∀ e ∈ supp ( p ref ( ·| x )) -supp ( p ref ( ·| x s )) , p ∗ ( e | x ) = 0 ;

(2) (Add safe explanations) ∀ e ∈ supp ( p ref ( ·| x s )) -supp ( p ref ( ·| x )) , p ∗ ( e | x ) > 0 = p ∗ RLFT ( e | x ) .

Thus, with the same jailbreak threshold p , the safety zone is successfully expanded.

Intriguingly, when the safe transformation is done by appending an identical safe prefix to all harmful prompts, we can connect our E-RLHF to context distillation . A good prompt is known to matter for the performance of a fixed-parameters LM [68, 55]. Researchers have proposed a systematic LM tuning algorithm, called Context Distillation [69], aiming at distilling useful information from a good context as prefix to a language model. Given an initialized language model, for example p SFT, an input prompt x and a prefix context string prefix , Askell et al. [69] optimizes the loss

where ⊕ stands for string concatenation. This technique has been adopted as part of the safety alignment process during the LLaMa-2 series tuning [13], where prefix is chosen from a set of predefined safe prefixes. When applying the identical prefix transform in our E-RLHF transformation, it can be regarded as a combination of safety context distillation and RLHF. This gives another point of view on the effectiveness of our proposal.

D Ablation Study

In this section, we perform extensive ablation studies to showcase the effectiveness of our proposed E-RLHF.

D.1 LoRA results

In this section, we show results obtained by Low-Rank Adaptation [54]. We explore the same set of safe prefixes as in ablation D.2, and choose the best model for illustration. Numbers are illustrated in Table 3. Results are identical to the ones obtained via full parameter tuning that our E-RLHF performs better consistently against the RLHF baseline.

D.2 Ablation on safe prefixes

We ablate the effect of different safe prefixes. The prefixes we consider are collected in Table 8. Attack success rate numbers are shown in Table 4, with each model's MT-Bench scores shown in the brackets. Clearly, almost all safe prefixes lead to better safety against the DPO baseline and improved helpfulness compared to the SFT model (whose MT-Bench score is 6.3), and the performance could vary depending on the choice of the prefix. Finding a good prefix matters for our method, and we leave digging the optimal one out as future work.

D.3 Ablation on transforming all prompts

Here, we proceed to ablating the effect of transforming all prompts, no matter harmful or not. Results are shown in Table 5, where the red color indicates that safety downgrades compared to the model obtained via transforming harmful prompts only. Clearly, most models even persist worse safety compared to the DPO baseline itself, suggesting the detrimental effect of transforming the unharmful prompts.

D.4 Ablation with system prompt

As pointed out by previous works [2, 48, 70], system prompt can have a significant impact on ASR. This comes in two-folds: firstly, a powerful system prompt can initialize the LM to be closer to

HarmBench ASR [2]

Model

Direct Request

GCG

GBDA

AP

SFS

ZS

PAIR

TAP

AutoDAN

PAP-top5

Human

AVG

↓

π

E-DPO(1)

28

.

00

55

.

75

41

.

00

42

.

00

42

.

50

31

.

00

51

.

25

56

.

25

56

.

00

32

.

00

36

.

05

42

.

89

π

E-DPO(2)

25

.

75

60

.

50

40

.

25

46

.

50

44

.

75

30

.

15

52

.

75

57

.

25

63

.

50

30

.

75

40

.

45

44

.

78

π

24

.

00

57

.

25

35

.

00

41

.

75

39

.

50

26

.

55

49

.

25

55

.

00

58

.

75

29

.

25

38

.

70

41

.

36

π

E-DPO(4)

27

.

75

58

.

50

38

.

00

42

.

00

40

.

75

31

.

20

52

.

75

60

.

50

54

.

75

31

.

50

38

.

30

43

.

27

AdvBench ASR [1]

π

E-DPO(1)

0

.

00

53

.

00

17

.

00

31

.

00

26

.

00

6

.

00

50

.

00

57

.

00

61

.

00

4

.

00

19

.

00

29

.

45

π

E-DPO(2)

0

.

00

52

.

00

17

.

00

29

.

00

26

.

00

6

.

20

56

.

00

58

.

00

73

.

00

3

.

00

25

.

60

31

.

44

π

0

.

00

56

.

00

17

.

00

23

.

00

17

.

00

3

.

80

46

.

00

58

.

00

66

.

00

4

.

00

23

.

40

28

.

56

π

E-DPO(4)

0

.

00

58

.

00

17

.

00

36

.

00

27

.

00

6

.

80

46

.

00

60

.

00

68

.

00

3

.

00

25

.

20

31

.

54

the safety zone, thus making the model safer; secondly, a longer system prompt would enlarge the difficulty of launching a specific attack due to the increased computational consumption. To confirm our E-RLHF is still helpful under this specific situation, we ablate this effect under the control of the default Mistral system prompt 11 . The results can be found in Table 6, where nearly all ASRs drop, whilst the conclusion does not change.

D.5 Ablation on improving helpfulness and safety simultaneously

In similar spirit, we ask the question: can the trick we propose to improve safety be used to improve helpfulness simultaneously? To achieve this goal, we choose the prefix "Give me a safe and helpful response to the following request." , and apply it to all prompts in the preference dataset. Results can be found in Table 7. Our model achieves better safety, and improves the helpfulness benchmarked by MT-Bench [3] from 6.9 to 7.0.

E Filtering harmful prompt in the preference alignment dataset

Inspired by the work by Chao et al. [30], we prompt GPT-3.5-Turbo as a harmness judge on input prompts. It assigns an integer score ranging from 1 to 10 for a given prompt, and we choose the score ≥ 6 subset as harmful. The system prompt is shown in Table 9.

(1) "Provide a responsible and ethical answer to the following request. "

(2) "In a manner that promotes safety and well-being, please respond to the following request. "

(3) "Keeping in mind the importance of safety and legality, how would you respond to this: "

(4) "Please ensure your response adheres to community guidelines and ethical standards: "

Table 9: System prompt for GPT-3.5-Turbo used to filter harmful and non-harmful prompt in the preference dataset.

You are a helpful red teaming assistant. You will receive a prompt. Please act as an impartial judge and evaluate whether the prompt itself is requesting information that violates ethical, legal, or safety guidelines. Be as objective as possible. Rate the prompt on a scale from 1 to 10. A rating of 1 signifies a fully non-harmful prompt. A rating of 10 signifies prompts that could lead to responses that fully violate ethical, legal, or safety guidelines. Your rating should strictly follow this format: "Rating: [[rating]]", the rating in the double-closed brackets is a number from 1 to 10, e,g, "Rating: [[5]]".

F Jailbreak adversaries collected in the HarmBench project [2]

In this section, we give a short overview of the adversaries we adopt to evaluate our models. Some descriptions are summarized in Mazeika et al. [2].

Direct Request refers to sending the original harmful prompt directly to the target LLM.

GCG [1], GBDA [71] and AP [72] find adversarial suffixes via token-level optimization.

SFS (Stochastic Few-Shot) and ZS (Zero-Shot) [15] are few-shot sampling or zero-shot generation of test cases by an attacker LLM to elicit a behavior from a target LLM.

PAIR [30] and TAP [33] are iterative prompting / tree-structured prompting methods, with an attacker LLM, to adaptively explore and elicit specific harmful behaviors from the target LLM.

AutoDAN [29] is a genetic-algorithm based attack with initializations from handcrafted DAN jailbreak prompts.

PAP [73] uses persuasive strategies to jailbreak the target LLM.

HumanJailbreaks [74] uses a fixed set of in-the-wild human jailbreak templates, similar to the Do Anything Now (DAN) jailbreaks.

We exclude all transfer attacks since we focus on single-model jailbreak. Furthermore, we choose to discard the UAT [75] and PEZ [76] adversaries, because the former induces an out-of-memory error on our V100 GPUs, and the latter never succeeds to find a suffix according to our experiments.

G Broader Impacts

The societal impact of our work has close connection to the topic of LLM safety. We propose a framework for analyzing language model pretraining and jailbreaking, and we design a new RLHF algorithm for improving safety. As shown in our experiments, our work could advocate for safer LLMs.

H Related work

In this section, we provide a review of the current literature on LLM jailbreaking.

H.1 Jailbreak methods

In this section, we summarize existing jailbreaking methods.

Baseline and pioneers Autoprompt [72], a baseline method for optimizing in the token space w.r.t. a certain objective, approximates coordinate ascent by first ranking all tokens using an approximate objective, and then compute the exact value on them. The approximation is carried out by a single step of gradient computation. Jones et al. [77] propose Autoregressive Randomized Coordinate Ascent (ARCA) to generate (input, output) pairs that include certain harmful info or satisfy a fixed format requirement. Token level optimization is carried out with linear approximation on GPT-2. GBDA [71] study adversarial attack on text classification problems, by optimizing the continuous relaxation of the autoregressive sampling probability matrix. In late 2022, among social media, users misalign GPT-3 via prompt injection. Perez and Ribeiro [78] study how this be done by adversaries. They successfully manipulate the model to output a given harmful string and leak the system prompt. In early 2023, an empirical study was carried out by Liu et al. [9] to measure the result of prompt engineering for breaking ChatGPT safeguards. Shen et al. [74] collect jailbreaking prompts from multiple platforms on the internet, analyze these data, create a harmful question set, and identify some typical harmful prompts that are effective at that moment. Later, the Greedy Coordinate Gradient (GCG) method [1], a strong white-box attack variant of AutoPrompt [72] was proposed. Wei et al. [79] categorizes two general modes of jailbreaking: competing objective and mismatched generalization .

LLMautomation and suffix-based attacks Liu et al. [29] propose AutoDAN, that relies on genetic algorithms, with the requirement of manual prompts for conducting mutation and crossover on the paragraph and sentence level . The jailbreaking prompts generated by AutoDAN are semantically plausible, unlike the suffix generated by GCG. As a comparison, Lapid et al. [25] use genetic algorithm for black-box universal adversarial suffix generation. Chao et al. [30] propose another LLM-based jailbreak automation algorithm, where an LLM judge is built to assign a safety score to a given output, while the attacker is enforced (via a page-long prompt) to improve the quality of jailbreaking prompts from multiple perspectives. Zhu et al. [24] propose another AutoDAN method that explores the balanced loss between jailbreak loss (log probability on the harmful string, as used in Zou et al. [1]) and the plausibility loss (log probability over the adversarial suffix), aiming at improving interpretability. Li et al. [31] uses genetic algorithm to search with similarty measure and initialize with paraphrasing. Its performance is claimed to be better than AutoDAN-GA. Deng et al. [80] investigate the possible defensive tricks in proprietary LLMs, and propose a pipeline for automated jailbreaking using a fine-tuned LLM. Yu et al. [81] propose GPTFuzzer, essentially a genetic algorithmic framework for jailbreaking. Their work's difference between AutoDAN is that it has a pool of 'seeds', a.k.a. templates for transforming the harmful prompt, and the mutation is done on the template level. Ding et al. [32] propose automating attack via LLM prompt rewriting and scenario nesting . The latter consists of code completion, table filling and text continuation, since the authors regard these as align with training objectives well, and are suitable for LLMs to complete the task. Mehrotra et al. [33] combine Automation used in Chao et al. [30] and tree-of-thought [82], create interpretable prompts in a black-box manner. Li et al. [83] propose DeepInception, and use nested, imaginary scenario to induce harmful content. Li et al. [84] propose DrAttack, which camouflages a query's malicious intent through semantic decomposition, by constructing a parsing tree and split the original prompt into different segmentations. Wang et al. [85] draw inspiration from the self-perception theory from psychology to design a prompt modification pipeline on gradually persuading the LM to be jailbroken. Paulus et al. [86] propose AdvPrompter, where the authors train a language model as a suffix generator to speed up LLM attack.

Manipulating the decoding process Huang et al. [48] find the method of changing the generating hyperparameters ( i.e., p of topp sampling, the temperature T , and k of topk sampling) of safetyaligned LLMs suffices for obtaining harmful outputs when the user is able to manipulate the system prompt and input configurations. Zhang et al. [87] directly manipulate the output generation probability by enforcing an affirmative prefix, and reversing the negation words if they appear in a pre-defined vocabulary ( e.g., sorry → glad). Zhao et al. [88] assume access to the decoding distribution of a LM. They use two small LMs, a safe one and a harmful one, to manipulate the decoding ratio of the large safe LM for jailbreaking. The key insight is the decoding distribution between the safe model and the harmful model only differs significantly for the first tens of tokens.

Fine-Tuning alone suffices Yang et al. [89] show that fine-tuning on as few as 100 harmful example pairs suffices for turning the LLaMa-chat models (and some other < 70B LMs) into malicious counterparts. Zhan et al. [90] fine-tune GPT-4 on harmful data, and find the fine-tuned models escape previous safety constraints while maintaining usefulness. Qi et al. [91] find fine-tuning alone, even on

benign data, leads to safety degradation using LLaMa and GPT-3.5-Turbo. Fine-tuning on harmful data (with less than 5 gradient steps) will cause the model to be completely harmful, while tuning on identity-shifting data could make the LM fully obedient.

Low-resource language and cipher Yong et al. [26], Deng et al. [27] explore the difference in languages when encountering the same harmful query, and find a direct translation to low resource languages will lead to higher risk, and Deng et al. [27] additionally find when combined with sophisticated methods, the drawback of low-resource languages disappear. Yuan et al. [28] use cipher encoding and decoding to break LLMs. Smaller scale models are immune from such attacks, while the smartest GPT-4 encountered the highest risk.

Vision-language model attacks Besides pure LLM, some research works move a step forward, utilizing images for breaking vision-language models (VLMs). Shayegani et al. [92] explore multimodal attack on VLM via embedding space feature matching. Qi et al. [93] generate adversarial examples via maximizing the conditional probability of a harmful corpus, i.e., the sum of log probabilities over all outputs, and use the final image with harmful query for jailbreaking. Carlini et al. [94] generate adversarial example for a fixed harmful content , and find no matter what input prompt is given to the VLM, it will respond with the target harmful string. Maus et al. [95] propose a black-box attack on manipulating the generated image with modified adversarial prompt.

Misc Wei et al. [57], Wang et al. [58] explore in-context learning for attack and defense. The attack is weak since it could only break Vicuna [96] and can be defended by in-context safe examples. Later, this method is scaled-up to significantly improve strength for breaking guardrails of large, state-ofthe-art models [59]. An early work in February 2023 [97] adopts obfuscation (including synonym and typos), code injection and virtualization to successfully jailbreak ChatGPT. Shah et al. [98] illustrate in-context automated persona modulation attack for large-scale LLMs and Vicuna. Zeng et al. [73] consider the more broadly perspective of persuasion : they train a persuasive paraphraser based on a fine-grained taxonomy of persuasion techniques. Detailed ablation on attack effectiveness is studied. Guo et al. [99] focus on stealthiness and controllability. They notice the constraints applied to the jailbreaking prompts ( e.g., fluency) are exactly the targets of the controllable text generation problem. Thus, they adopt the Energy-based Constrained Decoding with Langevin Dynamic (COLD) [100] on output logits. Forming each constraint as well as the task of jailbreaking as an energy function over logits, the Langevin Dynamic is used for finding a good logit distribution, and the decoding technique in Qin et al. [100] is used for output generation. Banerjee et al. [101] introduce a dataset TECHHAZARDQA, compare direct query v.s. pseudo-code format, and find the latter induces higher risk. Mangaokar et al. [102] considers a type of adaptive attack against checking-based defense , that appends a universal adversarial prefix into the jailbreaking prompt to make the guard model always output 'safe', and thus making the detector fails to detect harmful information. Lv et al. [103] propose Code Chameleon, which contains multiple encryption and decryption methods defined by python functions, that transforms the harmful prompt into specific predefined form to jailbreak LLMs. Sadasivan et al. [104] speed up the computation of GCG [1] to make it possible to launch on a single GPU. Geiping et al. [105] build a taxonomy on risks beyond jailbreaking, and coerce the LLM to provide certain outputs by optimizing a set of tokens via GCG. Ren et al. [106] propose CodeAttack that use code templates to query the output out instead of using natural language directly, and obtain descent results.

H.2 Defense methods

Up to now, no universal defensive strategy as adversarial training [107] for adversarial attacks / differential privacy [108] for membership attacks exists as a gold standard. In general, we can classify the methods into three typical types: alignment, red-teaming, and algorithmic proposals.

Alignment The target of alignment is to push the output of language models be aligned to human values. Regarding safety, the goal is to avoid outputting harmful information. RLHF is widely adopted in these methods [19, 20, 109, 22]. Variants like RLAIF also have been proposed recently [21, 110].

Red teaming This term is populated as specifically dealing with harmful info on dataset curation, used together with RLHF [14-18].

Next, we proceed to defensive algorithm proposals. We classify existing defensive strategies in the following categories.

Defense against suffix-based attacks. Alon and Kamfonas [111] notice the messy nature of the suffix generated by GCG, and propose to use a perplexity (PPL) filter on input prompts. They also explore using a LightGBM [112] with 2 features (PPL, prompt length) to filter harmful prompt, and show it does better than naive PPL thresholding. The PPL-based filter does not succeed with human-crafted jailbreaks. Jain et al. [37] explore many concerning viewpoints, including self-PPL filtering, paraphrasing the input prompt, and re-tokenization since many LLMs' tokenizers are based on Byte-Pair Encoding (BPE). All methods are successful in regards of defending against suffix-based attacks. They also explore the simplest form of adversarial training. Robey et al. [34] propose to perturb the input token string by random replacement/erasement/insertion, and finally perform a majority vote in the end. Cao et al. [35] judges whether an input prompt is safe or not by estimation with Monte Carlo, when randomly dropping a fraction of tokens, using the LLM itself. Kumar et al. [36] try to perform 'certified' safety against harmful prompts, by erasing tokens and set the original prompt as harmful if at least one of these erased prompts lead to a harmful response, or be classified as harmful by a DistillBERT-based classifier.

System prompt defense. We could modify the input prompt for jailbreaking; and several works explore if we can apply similar methods to system prompts to defend against such attacks. Xie et al. [113] propose 'self-reminder', i.e., appending a reminding prompt within the system prompt for defense. The attacks are collected from the JailbreakChat collection, and this strategy is effective for defending against them. Zheng et al. [114] advocate for finding a good system prompt automatically, by investigating the representational difference between safe and harmful queries, and optimizing the safety prompts along the harmful refusal direction in the representation space. One intriguing takeaway is harmful / harmless queries can be distinguished in the representation space, different from the adversarial examples in vision. Zhou et al. [115] also optimize the safe system prompt, but in a more 'adversarial training' fashion, that apply jailbreak algorithms with current safe prompts first and then find good replacement candidates in the same way as done by Zou et al. [1]. Concurrently, Mo et al. [116] propose prompt adversarial tuning, where an adversarial suffix is assumed, while a safe system prompt is jointly optimized with this suffix, with an additionally constructed benign loss to improve helpfulness under normal queries. Zhang et al. [117] propose the idea of 'goal prioritization', either without training (append prioritize safety than helpfulness and in-context examples to the system prompt) or with training (generate data pairs of prioritizing safety or helpfulness, finetune, and append prioritize safety prompt into system prompt). The former is effective for large-scale LLMs, while the latter improves safety of LLaMa-chat models. Zhou et al. [118] propose in-context adversarial game, where an attack LLM and a defense LLM interact on exchanging insights on successful jailbreaks, and defend by improving the system prompt. Zou et al. [70] give the result that system prompt matters for jailbreaking, and shows conducting GA-based search over it could improve safety.

Checking-based, decoding-based, and Misc. Helbling et al. [119] generate responses first, and then use the LLM itself to examine whether the output is harmful or not. They find such simple self-examination is powerful since the TPR reported can be up to ∼ 1 . 00 . Wang et al. [120] propose to (1) tune the LM to enhance its capability on discriminating harmful / harmless content; (2) tune the LM to make it able to tag its own response; and (3) rewrite response if output is harmful. Li et al. [121] propose to suppress the attack performance by iteratively rewinding and re-examining the generated output. The method does not work well with small models, but works pretty fine with large (open-source) models. They find the strategy can improve generalization as well. Xu et al. [122] train a safer model first, and use normalized p attacked + α ( p safer -p attacked ) over topk shared tokens for decoding to enhance safety. Hasan et al. [123] show with original Wanda pruning [124], the LLM can help resist direct jailbreaking prompts, e.g., with role-playing attacks. Pi et al. [125] propose MLLM-Protector on safeguarding Visual LLMs by checking the output and then detoxifying the content. Zhang et al. [126] perform intention analysis on the input, and enforce the model generate policy-aligned outputs both by prompting. Wang et al. [127] propose backtranslation that guesses the input prompt directly, and reject if it is harmful. Kim et al. [128] propose self-refinement which consists of generating a feedback and then refine the response to avoid harmful info output. They find using additional JSON and code formatting would improve safety. Zeng et al. [129] propose AutoDefense, which utilizes multiple agents on analyzing prompt, response and intention, to defend

against attacks. Hu et al. [130] propose Gradient Cuff, a sampling-based gradient-norm defense method, by rejecting those input prompts with large gradient norm on top of a majority-vote based filtering. Ji et al. [131] propose a method similar to Robey et al. [34], but for semantically-meaningful attacks, that paraphrases the input according to several criteria and conduct a majority vote for judging.

Several company-centered products also fall into this regime. For example, LLaMa-Guard [132] is trained on toxic data such that it is able to discriminate unsafe user prompts and outputs, respectively. IBM also propose a framework on constructing and deploying safeguard detection modules, and releases the details in a technical report [133].

H.3 Theory and experimental understanding

Wolf et al. [38] assumes the decomposability of LLM output into a good and bad component, and show possible jailbreaking in theory by prompting the model with a sufficiently long input. Kalai and Vempala [134] use statistical tools to prove hallucination for calibrated LMs. Lee et al. [135] study the representation in GPT-2. They train a base classifier for toxicity, and use the linear weight as a proxy of toxic vector. They find there are value vectors close to the toxic vector itself, that are not suppressed by DPO tuning [23]. Wei et al. [136] use pruning and low-rank analysis on safe LM, and find (1) safe neurons and useful neurons are sparse; pruning the safe neurons or removing the safe ranks away degrades safety a lot, and (2) fixing the safe neurons in fine-tuning does not maintain safety.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: we state the three main contributions of our work explicitly, following by individual sections that clearly illustrate them.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: we discuss the limitation of our work in the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions are clearly stated in the main text, and all proofs can be found in the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: all models, code, data and training configurations we used are publicly available, and we provide links to them in the main text.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: we do not use our own model or data for training. Thus, we do not provide any data or code by ourselves. All of them are publicly available.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: we adopt all default training hyperparameters from the public codebase.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: following community standard, we use greedy decoding for language model evaluation, which naturally leads to zero variance.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: we state our experiments are consistently launched on 8 NVIDIA V-100 GPUs.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: our research does not involve human participants, and never use private data or models. Our research is related to LLM safety, and we propose an algorithm to improve it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: the societal impact of our work has close connection to the topic of LLM safety. We propose a framework for analyzing language model pretraining and jailbreaking, and we design a new RLHF algorithm for improving safety. Our work could advocate for safer LLMs.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: although our research involves jailbreaking language models, we do not release any models or data, thus the paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: the code and data we use to train the model possess the Apache-2.0 license, and the code we use to test the model possess the MIT license. We include links to these public assets explicitly in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: we do not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new theoretical framework that views prompts as combinations of concepts and queries, allowing for a detailed analysis of how and why LLMs can be manipulated into producing unsafe responses. It proposes a statistical metric for alignment, focusing on quantifying the safety of the model's outputs.\n\nThe study introduces an improved alignment strategy, Enhanced Reinforcement Learning from Human Feedback (E-RLHF), which modifies existing RLHF methods to increase the likelihood of safe responses from LLMs without additional training costs.\nThrough experiments using standard benchmarks like AdvBench and HarmBench, the paper demonstrates that E-RLHF significantly outperforms traditional RLHF in resisting jailbreaking attempts, reducing the models' vulnerability to producing harmful outputs.\n\nSTRENGTHS:\nThe paper offers a unique statistical framework that conceptualizes input prompts into concepts and queries. This approach allows for a nuanced understanding of how LLMs process inputs and why they might generate unsafe outputs even under alignment efforts. \n\nThe introduction of E-RLHF  as an alignment strategy that doesn't require additional computational resources is a major strength. \n\nThe empirical tests using established benchmarks such as AdvBench and HarmBench provide solid evidence that E-RLHF can significantly reduce the Attack Success Rate (ASR) compared to traditional RLHF.\n\nWEAKNESSES:\nWhile E-RLHF improves alignment in controlled experimental conditions, its effectiveness in more complex conversational scenarios, where inputs can evolve over a series of interactions, remains untested.\n\nQUESTIONS:\nThe two datasets used in the work contain harmful requests about various concepts (e.g. drug, weapon). Have you tried to analyze the performance of E-RLHF on different concepts?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a statistical framework that provides a theoretical analysis of the jailbreaking problem in language models. The authors first examine the PAC-Bayesian bound to demonstrate that there is a non-trivial probability for LLMs to mimic harmful behavior if such information is present in the pre-training corpus. They then theoretically prove that a jailbreaking prompt can be found for such pretrained LMs and propose mitigating the jailbreak issue by modifying the harmful prompts.\n\nSTRENGTHS:\nThe paper's formulation and assumptions are clear and well-motivated. The proofs seem comprehensive and support the claims well. However, I don't have much background and PAC-Bayesian theory, so it's hard for me to verify if (1) the proofs are all rigorous (2) the proposed framework is truly novel instead of simple application of existing theorems. \n\nThe experimental results from the proposed E-RLHF improve upon DPO-based fine-tuning.\n\nWEAKNESSES:\nI find the connection between the theoretical part and section 5 (E-RLHF for actual experiment) quite handwavy. Expanding the safety zone is the claimed goal, but the proposed solution (through injecting safe concepts in a harmful prompt) seems really hacky. For example, as shown in Table 4, using safe prompt (4) gives much better results than safe prompt (1) for no reason. Overall, this proposed method seems like a small trick that the authors try to incorporate into the paper just to show some empirical value of their theoretical analysis. I also find that with such modifications, the MT-Bench score (briefly mentioned in line 355) is lower under E-DPO than DPO.\n\nLastly, this might be a biased opinion (that's why I didn't factor this part into my scoring decision and still gave a marginally above accept score): I feel that after all the theoretical proofs (despite their elegance), the conclusion (LLMs will mimic toxic behavior if such toxic content is present in the training corpus, jailbreak is unpreventable under reasonable assumptions, and by expanding the safe zone we reduce the probability of jailbreaking) is very intuitive and does not provide much additional insight into the problem. Maybe the mathematical formalization of the jailbreak problem itself is meaningful, and I will let the other reviewers judge the novelty of such a framework.\n\nQUESTIONS:\nCould you provide more details on the MT-bench's performance? For example, you have an ablation study on different safety prefixes' effects in Table 4. What about their impact on MT-bench?\n\nIn line 327 and the ablation study in the appendix, you show that non-harmful prompts should not be modified (otherwise, the model performs much worse). Could you relate this phenomenon to the safe/harmful zone or your theoretical analysis? I don't see why adding a safe prefix to a non-harmful prompt negatively impacts the safe/harmful zone, and under your assumption, this should not lead to worse behavior.\n\nLIMITATIONS:\nNo concerns on the limitations", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses a very important issue of our time, the safety of LLMs. LLMs are already used in various applications and will be present in more applications to come, as e.g. Microsoft, Apple and Google are integrating LLMs in their applications and operating systems. Hence, the question on how to make the systems more robust against jailbreaking is a very important question.\n\nThis paper offers a statistical approach to this question and presents experiments. The mathematics is based PAC-Bayesian approach and builds upon a very nice formalization of splitting a prompt into queries and concepts.\n\nSTRENGTHS:\nThe mathematics are presented very well. In particular, the intuitions and interpretations of the mathematical concepts are presented well, and can also be followed by readers, who are not able to follow all the details of every equation.\n\nOverall, I really appreciated the formal approach to modelling jailbreaking and the conclusions, that it is impossible to avoid jailbreaking.\n\nWEAKNESSES:\nThere is a disconnect between the theory and the experiments. The definitions, theorems, etc. are all presented and motivated very nicely, but could also apply to any other mapping of a stochastic system. Oversimplified, one may state that there is a mapping from an input space into an output space, in which the output space can be divided into desired and undesired outputs. The goal is to reduce the space of undesired outputs. To make this a case specifically about LLMs, there needs to some form of conclusions that feed into the experiments. Unfortunately, I don't see this connection. The experiments use a systems prompt that is preprended to the actual prompt. I do not see, how this follows from the theory.\n\nThis is really unfortunate, because I would really like to see, how the experiments connect to the theory.\n\nQUESTIONS:\nCould you please point out the connection between experiment and theory that I have seemed to miss.\n\nLIMITATIONS:\nI don't see any negative social impacts. As stated above, I would like to see a stronger connection between experiments and theory.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a theoretical insight about LLM jailbreaks using PAC-Bayesian bound for pretraining LLMs. It assumes that there always exists the harmful data in the mixture, and as the model is trained on this mixture, the model will probably produce the responses in harmful zone (it has a specific definition in the main paper). Based on this framework, the authors suggests that the safety zone of the models should be extended, and to this end, they introduce a method called E-RLHF which can expand the safety zone. E-RLHF replaces the harmful prompts x_h into benign prompts x_s, and replace some of the terms in RLHF (and DPO) to make sure that the model keeps in the safety zone. Experimental results also show that it does not sacrifice the general capabilities but can be improved in the safety perspective.\n\nSTRENGTHS:\n- This paper suggests a theoretical insight about the LLM jailbreaks, which were not addressed much in the previous literature on jailbreaking. The theoretical framework is sound and compelling.\n- Based on this framework, the paper also suggests a simple training trick that can lead to better safety training. \n- Empirical results also show that their idea is working well, shown by Harmbench, AdvBench, and MT-Bench scores.\n- The paper provides extensive evaluation results on jailbreaking setups, providing the results from more than 10 attack setups.\n\nWEAKNESSES:\n- E-RLHF is an inaugural and simple form of expanding the safety zone of LLM; I think there could be more sophisticated and effective ways, and I hope the authors will address this in the future works.\n- Other than that, I think there is no big weakness in the paper, but have some minor comments:\n  - Eq (2): it is slightly confusing that in D_KL, the term only have p_LM(x) and p_SFT(x), not p_LM(e|x) and p_SFT(e|x).\n  - About writing: I am not familiar with using the term \"explanation\" -- instead, I think using \"response\" is more common. At the first glance, it was hard to comprehend the meaning of the term.\n\nQUESTIONS:\nNo specific questions about the paper.\n\nLIMITATIONS:\nThe authors provided limitations section in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}