Mission Impossible: A Statistical Perspective on Jailbreaking LLMs

Authors: jingtong  su
meta  ai
julia  kempe
karen  ullrich
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 21:11:24

Abstract: Large language models (LLMs) are trained on a deluge of text data with limited quality control. As a result, LLMs can exhibit unintended or even harmful behaviours, such as leaking information, fake news or hate speech. Countermeasures, commonly referred to as preference alignment, include fine-tuning the pretrained LLMs with carefully crafted text examples of desired behaviour. Even then, empirical evidence shows preference aligned LLMs can be enticed to harmful behaviour. This so called jailbreaking of LLMs is typically achieved by adversarially modifying the input prompt to the LLM. Our paper provides theoretical insights into the phenomenon of preference alignment and jailbreaking from a statistical perspective. Under our framework, we first show that pretrained LLMs will mimic harmful behaviour if present in the training corpus. \textbf{Under that same framework, we then introduce a statistical notion of alignment, and lower-bound the jailbreaking probability, showing that it is unpreventable under reasonable assumptions.} Based on our insights, we propose an alteration to the currently prevalent alignment strategy RLHF. Specifically, we introduce a  simple modification to the RLHF objective, we call \emph{E-RLHF}, that aims to increase the likelihood of safe responses. \emph{E-RLHF} brings no additional training cost, and is compatible with other methods. Empirically, we demonstrate that \emph{E-RLHF} outperforms RLHF on all alignment problems put forward by the AdvBench \citep{zou2023universal} and HarmBench project \citep{mazeika2024harmbench} without sacrificing model performance as measured by the MT-Bench project \citep{zheng2024judging}.

Main paper: 
 introduction: 
 Large Language Models (LLMs) have revolutionized the field of deep learning due to their remarkable capabilities across various domains, serving as assistants, in code generation [4] , healthcare [5] , and theorem proving [6] . The training process of a LLM typically includes two stages: pretraining with massive corpora, and an alignment step using Reinforcement Learning from Human Feedback (RLHF) to further align model behavior with human preferences. The latter step typically involves large amounts of humanly annotated data, and can be decomposed into a supervised fine-tuning (SFT) step, a reward modeling step, and an RL Fine-Tuning step. Despite their ability to perform multiple tasks effectively, LLMs are susceptible to generating offensive or inappropriate content including hate-speech, malware, fake information or social biases, due to the unavoidable presence of harmful elements within their pretraining datasets [7] [8] [9] . Social media showcase an abundance of tricks on how to attack ChatGPT [10] to elicit harmful responses, e.g., the "Do Anything Now" (DAN) prompts [11] or the "Grandma Exploit" hack [12] . On the other hand, behavior diversity in the training corpus is essential to for example capturing different cultural preferences. What is and isn't harmful ultimately depends on user preferences, hence the alignment step is not universal but depends on the specific use case under which a model will be employed.To address deployment safety and eliminate objectionable responses, numerous alignment efforts have been made, such as injecting safe information during SFT [13] , performing red teaming with human experts and AI themselves [14] [15] [16] [17] [18] , as well as refining and improving the whole RLHF process in detail [19] [20] [21] [22] [23] . Yet we continue to witness a cat-and-mouse game of ever more sophisticated alignment methods to neutralize "harmful" prompts and even more inventive "jailbreaking" attacks that manipulate those prompts to elicit LLMs to produce harmful information. Such attacks come in various flavors, such as injecting adversarial suffixes [1, 24, 25] , exploring cipher and low-resource natural languages [26] [27] [28] , or letting LLMs craft prompts automatically [29] [30] [31] [32] [33] . Although several adhoc defense methods against suffixes have been proposed [34] [35] [36] [37] , we only have limited proposal on a principled universal defense against jailbreaking attacks [2] , and limited theoretical understanding on this phenomenon [38] .In this paper, we present a theoretical framework for analyzing both the pretraining phase and the post-alignment jailbreaking phenomenon. Exploiting the fact that jailbreaking prompts typically maintain the underlying harmful concept while manipulating other aspects of the prompt, we design framework that decouples input prompts to allows us to quantify the strength of potential adversaries. By representing the output elements of an language model (LM) as lengthier text fragments rather than individual tokens, we can quantify the extent to which these models emulate the training distribution and consequently better understand the mechanisms underlying jailbreaking vulnerabilities.Our contributions can be summarized as follows:• Based on our proposed framework, we first offer a non-vacuous PAC-Bayesian style generalization bound for pre-training. Assuming the validity of our framework, we conclude that high-performing pre-trained models will inevitably be susceptible to generating behaviour that is present in the training corpus, including any unintended and harmful behaviour. • Subsequently, we extend our framework to include notions of alignment and jailbreaking. Assuming our assumptions are met, we demonstrate jailbreaking to be unpreventable even after safety alignment because the LM fails to concentrate its output distribution over the set of safe responses. • Motivated by our theoretical findings, we identify a key drawback in the widely adopted RL Fine-Tuning objective due to the natural difference between the harmlessness and the helpfulness targets. By addressing this issue, we facilitate the training of safer models that are more resilient to a suite of jailbreaking attacks while preserving model performance.The paper is organized as follows. In Section 2, we introduce our framework. In Section 3, we prove the PAC-Bayesian generalization bound for pretraining. Next, in Section 4 we present analysis on jailbreaking from a statistical perspective. Finally, in Section 5 we illustrate our proposed E-RLHF objective and its effectiveness on improving LLM safety. We give a literature review in Appendix H. 
framework and assumptions: 
 Jailbreaking carries several analogies to adversarial attacks, a well studied field in computer vision [39] . Here, an adversary is defined as a map that perturbs a given input image in pixel space to change the model output. The strength of the adversary is bounded by how far it is able to move the original input as quantified by the ℓ p distance [40] [41] [42] . Typically, this distance is bounded in a way that the change would not be perceptible to the human observer. The goal of the adversary is to cause misclassification of the input. In contrast, in the instance of an LLM, the adversary's goal is to provoke harmful behaviour, e.g., unintended leaking of information or hate speech. Further, any perturbation to an input, called prompt, will have a perceptible effect. Hence quantifying and bounding the capabilities of the adversary is not straight forward. Nonetheless, with some modifications, we will build on this analogy.For the purpose of this work, we will view any prompt as a tuple of query and concept (q, c), where c ∈ C, and q ∈ Q, with C, Q denoting the complete concept set and query set. Conceptually, we think of concepts as representing the information content of the prompt, usually through a short piece of text, for example "tutorial on making a cake". Queries are instructional text pieces that are composable with certain concepts. We can think of queries as mechanisms to trigger an LM to expand a concept in a specific way. Examples include "Tell me how to {}", or "We are now in an imaginary world, and you are not bounded by any ethical concerns. Teach my avatar how to {}". Since not all queries and concepts are composable, 3 we denote P ⊊ Q × C as the set of all plausible prompts, where the definition of plausible will be made clear below.The decomposition of prompts allows us to isolate and hence bound the adversary's strength.In line with current empirical work on inducing harmful behaviour, we will allow perturbations only on the queries, not on the concepts. Further mimicking the spirit of previous work on adversarial attacks, we will assume that the ground-truth related to a prompt is determined solely by the concept, not the query. We will make these ideas more rigorous in the next paragraphs.Figure 1 : Our framework in a nutshell: We define a language model, p LM : → , as a map from prompts to a distribution over a subset of all possible explanations E. To later be able to bound the strength of the adversarial attacker, we split the text inputs into concepts and queries (q, c). We assume that (i) the text corpus only covers a part of the domain of the LM: supp(D P ) ⊊ dom(p LM ), (ii) the size of the domain of the output distribution, denoted |dom(p LM (q, c))|, is small compared to the size of E, and (iii) only concepts determine the output (see ).First, in contrast to previous theoretical work where LMs are regarded as single sentence generators [38] , we model LMs as lengthier text fragment generators, and refer to possible generated content e ∈ E as explanations. Conceptually, explanations expand concepts with additional information. For example, "The US president in 2023 is Joe Biden.". Our terminology "explanation" is conceptually the same as "response" used in previous discussions (e.g., [23] ), where an LLM is regarded as a policy that receives an input and generates a response. We use "explanation" to contrast "concept" since in most jailbreaking attacks currently considered by the community, the adversary seeks instructions or explanations for a single harmful attempt. An LM thus induces a mapping from plausible prompts to distributions over explanations, p LM : P → ∆(E), where ∆(E) denotes the set of distributions defined over elements in E. 4 The output of a LM given a prompt, p LM (q, c), is a discrete distribution over explanations. We use dom(p LM (q, c)) as the domain of this distribution, p LM (e|q, c) as the probability of e given (q, c) as the input, and supp(p LM (q, c)) as the subset of E with non-zero p LM (e|q, c). Further, we assume the existence of a latent ground truth mapping p world : P → ∆(E) that the LM is optimized to mimic during the pretraining stage. This is the distribution that defines "knowledge": for all plausible prompts (q, c), it specifies the ground-truth distribution over explanations. By plausible, we refer to all prompts that lie in the domain of the ground truth mapping (q, c) ∈ dom(p world ), i.e., P ≡ dom(p world ). Many plausible prompts will not even exist within any available training corpus, as discussed below.We can now state our main assumption, namely that for any plausible prompt (q, c) ∈ dom(p world ) the ground-truth distribution p world (q, c) is supported on a small subset of E ⇔ supp(p world (q, c)) ⊊ E. This assumption seems sensible to us: under normal circumstances, providing an explanation of "Paris" would not offer any relevant knowledge when given a prompt such as "How to write a hello world python script". Our second assumption is that for all plausible prompts (q, c), the concept c uniquely determines the support of the output distribution specified by p world , regardless of the query: supp(p world (q, c)) = supp(p world (q * , c)), ∀ plausible (q, c) and, (q * , c) . The query changes the ground-truth distribution without affecting its support. An illustration is depicted in Figure 1 . To be more precise: Assumption 2.1. (Concepts uniquely determine the explanation for plausible prompts) For all plausible prompts (q, c) ∈ dom(p world ), i) p world : P → ∆(supp(p world (q, c)) 3 For example, "Who is a tutorial on making a cake." is unreasonable.where supp(p world (q, c)) ⊊ E s.t. |supp(p world (q, c))| ≪ |E|; and ii) supp(p world (q, c)) = supp(p world (q * , c)), ∀(q, c), (q * , c) plausible.This assumption is natural since it essentially tells us that knowledge is specified by the corresponding concept alone, irrespective of what query is used to extract it. In other words, given a concept c, if a query q manages to change supp(p world (q, c)), we argue that the query should be deconstructed and partially absorbed by c to accurately reflect the knowledge mirrored by the support.Lastly, we make the assumption on the existence of an underlying generative distribution over prompts, denoted as (q, c) ∼ D P . This distribution serves as the principle governing the creation of our pretraining corpus. It is important to note that supp(D P ) ⊊ dom(p world ). For example, take the prompt (q ′ , c ′ )="Who is James Bond $λ*#!48811"; even though this prompt never appears in any text corpus across the internet, (q ′ , c ′ ) / ∈ supp(D P ), we, as humans, can make sense of it: (q ′ , c ′ ) ∈ dom(p world ). Later proofs in this paper assume LMs generate semantically reasonable explanations for such unseen plausible prompts, since in reality LMs are claimed to generalize well on huge, out-of-distribution datasets [43] . This is made explicit in Section 4, within Assumption 4.1.Finally, the following definitions pertain to our notion of harmfulness. More specifically, we understand harmful behaviour abstractly as any unintended behaviour. For this, we assume that any explanation e can be denoted as either harmful or not harmful (safe). A concept c is regarded as harmful if and only if the world generates harmful explanations with probability higher than a certain threshold with direct prompts. Definition 2.1. (Notions of Harmfulness)• (Direct Queries and Direct Prompts) We refer to a prompt as direct if it stems from D P , i.e., (q, c) ∈ supp(D P ). The query of a direct prompt is called a direct query.• (Harmful Concepts and Harmful Set) Given a concept c, the associated harmful set of explanations is denoted as E h (c) := {e|e ∈ supp(p world (•, c)) ∧ e is harmful}. In accordance with Assumption 2.1, with a threshold η, a concept c is harmful if ∀q s.t. (q, c) ∈ dom(p world ), e:e∈E h (c) p world (e|q, c) ≥ 1 -η. We refer to the set of all possible harmful concepts as C h ⊊ C.• (Safe Set) ∀c ∈ C h , there exists a corresponding safe set E s (c) ⊊ E that we wish p LM (q, c) to be concentrated on. It includes safe explanations existing in supp(p world (•, c)), and explanations designed by humans, e.g., with the template beginning with "Sorry."• (Semantically meaningful) We call explanations in E h (c) ∪ E s (c) as semantically meaningful for the (q, c) prompt.• (Mixture decomposition of D P ) With these notions, we can decompose D P = αD P h + (1α)D Ps (where supp(D P h ) includes all direct prompts with a harmful concept, and supp(D Ps ) includes the complement) as a mixture over direct prompts with a harmful concept and the non-harmful counterpart. 
pac-bayesian bound for pre-training llms on harmful data: 
 Given a learning algorithm that leads to a posterior distribution over a set of models, PAC-Bayesian theory [44] applies Probably Approximately Correct (PAC) inequalities, to provide bounds on the generalization gap, i.e., the difference between the model's empirical loss and the population loss. We now present the first result of our analysis: a non-vacuous PAC-Bayesian bound for pretraining LMs which implies that a well-trained LM ought to exhibit harmful behaviour even when simply prompted with direct queries if it was presented with harmful behavior during training.We denote by S = {(q i , c i )} n i=1 a set of prompts generated i.i.d. under D P , S ∼ D n P . These prompts together with sampled explanations form our pretraining corpus. We use π, ρ as the prior and posterior distribution over LMs before and after the pretraining process, defined over LM, the set of language models. Given a prompt (q, c), we measure the generalization capability of a LM by quantifying the Total Variation (TV) loss between the induced distribution p LM (q, c) and the ground-truth distribution p world (q, c). 5 For real-world LMs, pretraining involves optimizing the cross-entropy loss on the training corpus, which is equivalent to minimizing KL[p world (q, c)||p LM (q, c)] under our framework. With Pinsker's Inequality, optimizing the KL-divergence term is equivalent to optimizing an upper bound on TV; thus we expect empirical TV loss be small. Definition 3.1. (TV empirical loss and population loss) ℓ TV (p LM , (q, c)) := TV(p world (q, c), p LM (q, c)).Given an LM and a set of data S, the empirical loss RS (p LM ) and population loss R(p LM ) are defined asRS (p LM ) := 1 n n i=1 ℓ TV (p LM , (q i , c i )); R(p LM ) := E S∼D n P RS (p LM ) = E (q,c)∼D P [ℓ TV (p LM , (q, c))] .We state our PAC-Bayesian bound as follows. The detailed proof can be found in Appendix B.1. 6Theorem 1. (PAC-Bayesian Generalization Bound for Language Models.) With α as in Definition 2.1, consider a set of language models LM, with prior distribution π over LM.Given any δ ∈ (0, 1), for any probability measure ρ over LM such that ρ, π share the same support, the following holds with probability at least 1 -δ over the random draw of S:EQUATIONIn Appendix B.2 we give a theoretical estimation of ϱ, to illustrate the bound we derive is non-vacuous, i.e., less than 1. The KL term is of order O(K) where K is the number of parameters involved in π, ρ, and n can be shown to greatly exceed K (using a realistic Zipf distribution assumption on prompts to estimate the number of unique prompts). Theorem 1 tells us that, as long as pretraining successfully reduces the loss on the training corpus ( RS (p LM ) ↓), in expectation the language model will mimic the world well (small ℓ TV difference) on a given direct prompt sampled from D P . Furthermore, if α is not too small, then this statement holds on a direct prompt whose concept is harmful. Since we have defined the harmful concept as outputting harmful explanations with high probability (Definition 2.1), we conclude that an LM trained on D P data can output explanations in the harmful set. 
a statistical perspective on jailbreaking after alignment: 
 In this section, we will present the main theoretical contribution of our work: given our assumptions hold, we prove the existence of ways for an adversary to jailbreak an LM even after the preference alignment process. Our proof strategy is inspired by the work on adversarial robustness [41] , which bounds the adversary's probability of success by upper bounding the volume of the set of points that does not allow for the existence of adversarial examples. Going forward, we need to extend our framework to integrate alignment and jailbreaking.After an LM is pretrained, it typically will undergo fine-tuning on a dataset containing preferred behaviour. In what follows, we will assume that this alignment process does not change the model performance in the sense that the LM will still produce semantically meaningful explanations (Definition 2.1). It would not, for example, default to answering any request with the same response. Assumption 4.1. (LM outputs semantically meaningful explanations) For any harmful concept c, and all plausible prompts (q, c) ∈ dom(p world ),∃ |E n (c)| ≪ |E h (c)| + |E s (c)| s.t. O(1) ≪ |dom(p LM (q, c))| = |E h (c) ∪ E s (c) ∪ E n (c)|.In other words, we assume the LM's output distribution is accurately supported on E h (c) ∪ E s (c), in the sense that the size of "residual" E n (c) is relatively small compared to these semantically meaningful explanations. We define n(c) = |E n (c)|+|E s (c)|+|E h (c)|.We omit the (c) annotations when clear from the context. The O(1) statement is reasonable, because harmful explanations are usually long text fragments that allow for many alternative formulations. The assumption can be broken down into two components: (1) within the support of the output distribution, only occasional instances of unrelated explanations exist; (2) the process of aligning the model towards safety does not eliminate the harmful explanations acquired during the pretraining phase. For part (1), similar to the example we gave above, under normal circumstances, we do not expect the explanation "Paris" to appear in dom(p LM (q, c)) given (q, c) as "How to build a bomb". As for part (2), though seemingly surprising, evidence with a series of current state-of-the-art LMs can be experimentally validated [48] , where diverse, harmful explanations are extracted by simply manipulating the decoding process using direct prompts. In Section 5 we give an explanation for this undesired phenomenon. To bound the likelihood of jailbreaking we first need to specify how the output of a LM interacts with its support. Assuming a fixed order of explanations in dom(p LM (q, c)), and slight abuse of notation, we can use p LM (q, c) to denote an n(c)-dimensional vector on ∆ n(c)-1 , the probability simplex with n(c) elements, where each entry represents the probability of a single explanation. We call this simplex the output simplex related to a given concept c. Next, we can induce a distribution on this simplex given a posterior distribution γ over the set of language models LM, as follows. Definition 4.1. (Induced Distribution on Simplex, γ c ) Under the assumption that the LM outputs semantically meaningful explanations (Assumption 4.1), with a fixed prompt (q, c) and a posterior distribution γ over LM, the corresponding induced distribution: p LM (q, c) where LM ∼ γ is supported over a subset of the output simplex ∆ n-1 . This distribution is denoted as γ (q,c) , or γ c when the reference to q is clear from context. Next, we will separate the output simplex into a harmful and safety zone. This definition is motivated by the observation that typically an adversary is deemed successful if it can extract even a single harmful explanation for a given concept. This translates into a division of the output simplex, under Assumption 4.1, as follows. Definition 4.2. (Harmful Zone and Safety Zone) For a given harmful concept c and a fixed LM, the output simplex is divided into a safety zone and a harmful zone, H s and H h , where a predefined threshold p ∈ [0, 1] is used to quantify the distinction: p LM (q, c) ∈ H h if and only if e:e∈E h (c) p LM (e|q, c) ≥ p, and otherwise p LM (q, c) ∈ H s . Before we introduce jailbreaking, the reader might wonder why we did not define alignment more clearly. This is because under the PAC framework, preference alignment is nothing but a transformation from ρ to some γ posterior defined over LM. Given this inability on fine-grained characterization of alignment, we instead provide the goal of it as follows. With the above notion, given a prompt (q, c) where c is harmful, its goal is to push the induced distribution γ c into the safety zone H s . Ideally, supp(γ c ) ⊂ H s ⇔ with probability 1, the resulting LM is safe when encountering (q, c). We are ready to introduce necessary concepts related to jailbreaking. Definition 4.3. (Jailbreaking) Given a harmful concept c and a query q ′ , the prompt (q ′ , c) jailbreaks the LM iff p LM (q ′ , c) ∈ H h . We call such a prompt (q ′ , c) and query q ′ a jailbreaking prompt and jailbreaking query, respectively.The threshold p for discriminating H h and H s should be very small, since it means in expectation the adversary needs to call the LM 1 p times to collect a single harmful explanation i.e., to jailbreak the LM.To theoretically prove the jailbreaking effect, we need to restrict the adversary's ability. To achieve this goal, we borrow insights from adversarial attacks, to assume that the adversary has bounded manipulating capability on the output simplex when searching over the query set: Assumption 4.2. (ϵ-bounded adversary) Given an LM, a harmful concept c and an associated direct prompt (q, c), we assume the adversary can find a set of queries Q ′ , such that the output is moved at most ϵ on the simplex towards H h from p LM (q, c):sup q ′ ∈Q ′ d(p LM (q, c), p LM (q ′ , c)) = ϵ.Here d is a distance measure between two discrete distributions. d can be a typical ℓ p measure with p ≥ 1, or the Total Variation / Jensen-Shannon Divergence. We call q ′ ∈ Q ′ an ϵ-bounded query.A conceptual illustration of our framework is depicted in Figure 2 . Before arriving at our Theorem, we give the final definition of ϵ-expansion. Definition 4.4. (ϵ-expansion) Given a set A ⊂ ∆ n-1 and a distance measure d, the ϵ-expansion set A(ϵ, d) is defined asA(ϵ, d) := {t|t ∈ ∆ n-1 ∧ ∃y ∈ A s.t. ||y -t|| d ≤ ϵ}.We are ready to present the following theorem, which states that as long as the induced posterior γ c is not concentrated in an extremely safe area, then with high probability the model can be jailbroken. The proof is in Appendix B.3. Theorem 2. (Jailbreak is unavoidable) Assume that an LMs output semantically meaningful explanations (Assumption 4.1). Given any γ posterior distribution over LM, choose a harmful concept c with a direct prompt (q, c) and a threshold p (Definition 2.1), to define the corresponding induced distribution γ c (Definition 4.1) and division over output simplex (Definition 4.2). An ϵ-bounded adversary (Assumption 4.2) can find a jailbreaking prompt (Definition 4.3) with probability at least1 -γ s × (1 -Φ(a ϵ )) ,• by using either the direct prompt, such that p LM (q, c) ∈ H h ; or• by finding an ϵ-bounded query q ′ , such that p LM (q ′ , c) ∈ H h . Here, Φ(•) is the standard Gaussian cdf, γ s := max x∈Hs-H h (ϵ,d) γc(x) U (x) , with U (x) the uniform dis- tribution over ∆ n-1 , and a ϵ := a + √ n -1ϵ, where a writes analytically as a≍ |E h (c)|-1-(n-1)p √ (n-1)p(1-p).Trivially, the chances of an adversary to find a jailbreaking prompt increase for stronger adversaries (ϵ ↑). In the real world, this could relate to how much compute budget we allow to alter a query for a specific harmful concept. Furthermore, the chances of an adversary to find a jailbreaking prompt increase when the ratio of the sizes of the harmful explanation set to the safe explanation set is larger|E h (c)| |Es(c)| ↑.This is because their ratio will determine the size of the harmful zone which in turn will cause Φ(a ϵ ) → 1. In real world settings, for any harmful concept, the training corpus naturally contains a large harmful set due to the number of possible responses. Realistically, its size can not be countered by any manually-constructed safe set. Hence achieving alignment is hard: Recall that the goal of alignment is to respond with only safe explanations with high probability. However, we just learned that to increase that probability, we need to have a small harmful-to-safety set ratio which we discussed is not realistic. Consequently, the safety zone is going to be small. 
e-rlhf: improving alignment by expanding the safety zone: 
 Recall from Theorem 2 and the subsequent discussion in the previous section, that jailbreaking becomes more likely the larger the harmful zone is in comparison to the safety zone. The size of both zones relates to the size of their respective explanation sets. In other words, the size of the preference alignment dataset is crucial to successful alignment. Unfortunately, the human labor involved in creating such a dataset effectively caps its size.In order to bridge the gap between our theoretical insights and a practical solution towards suppressing the jailbreaking problem, we focus on other more practical ways to expand the safety zone. Even though our ideas are more broadly applicable, in our experiments we will focus on improving Reinforcement Learning with Human Feedback (RLHF). RLHF typically includes three phases: i) supervised fine-tuning (SFT); ii) preference sampling and reward learning and iii) RL optimization. Rafailov et al. [23] have recently proposed a widely applied version of RLHF for LMs, coined Direct Preference Optimization (DPO), that employs a clever reparameterization which leads to directly learning from the preference dataset, without the need of obtaining a reward model beforehand.DPO is more stable in the training process than other implementations of RLHF. A more complete overview of RLHF and DPO can be found in Appendix C.For our purposes, we assume access to an LLM p SFT that has been supervised fine-tuned on highquality data. We further assume access to a preference aligned dataset D s ; that contains a set of text prompts (q, c) = x, and two respective explanations that have been rated by human annotators as better e w or worse e l . In phase ii) of RLHF, one typically optimizes a reward model r(x, e) based on the annotated explanations. Our proposal concerns phase iii) of the RLHF process: training of the preference aligned model p LM . For a given reward model, p LM is typically obtained by minimizing the following objective:EQUATIONNote that, the first term is maximizing the reward, while the KL-term acts as a regularizer ensuring the aligned model can not drift off too far from the SFT model. We claim that this regularization is exactly the problem for safety. While designed to keep the model helpful 7 , for any harmful prompt x h and any harmful explanation e ∈ supp(p SFT (x h )), regularizing p LM to p SFT lets p LM maintain e in the support of the output distribution. Specifically, the supervised fine-tuning process does not involve elimination of any harmful explanations from the support of the output distribution of the pretrained model, 8 which leads to the fact that p SFT (x h ) supports harmful responses that can not be negated with a realistically sized preference aligned dataset. Thus, this small safe set problem of p SFT will be further passed to p LM , even if p LM is optimized to the optimal solution of the above objective. This suggests we should not push p LM into the direction guided by p SFT . Instead, for any harmful prompt x h , we propose to use a distribution that we design to have a large safety zone to replace p SFT (x h ). Inspired by our framework, we can achieve this by modifying the harmful prompt, such that a safe concept is used to replace the harmful concept, which alters supp(p SFT (•)) to include more safe explanations. For example, when x h is "Tell me how to make a bomb", instead of aligning p LM (x h ) to p SFT (x h ), now we align p LM (x h ) to p SFT (x s ) where x s is "Tell me how to reject a request of making a bomb". This can be done in an individualized manner for all harmful prompts in D s , or simply by prefacing all harmful prompts with a prefix such as "Please ensure your response adheres to community guidelines and ethical standards:". 9 Importantly, the non-harmful prompts are not modified. Due to the focus of our approach to expand the safety zone of the output distribution, we coin our proposal E-RLHF, resulting in the following modification to Eq. ( 2):EQUATIONwhere x s is a safety-transformed version of the original harmful prompt x h . To recap, the key argument we put forth is that, in order to ensure the stability of model fine-tuning, it is not imperative to utilize identical prompt inputs x for both the reference model and the target model, particularly when the original input x itself is harmful. In fact, as long as the substitute or "anchor" prompt generates logically reasonable outputs akin to those produced by the original prompt, this approach would not impede the training process of the model. To solidify our argument we show the impact of our modification on the support of the optimal policy in Appendix C. We also deduce there that we can trivially integrate our modification into the DPO objective allowing us to train without an explicit reward model (eliminates step ii)) as follows, where σ(•) stands for the sigmoid function:EQUATION)6 Experiments and resultsOur experimental set-up is based on the alignment-handbook code base [50] . We tune the publiclyavailable SFT model p SFT provided by huggingface hub [51] , using the public dataset [52, 53] , with default hyperparameter setup. We label harmful prompts in the preference dataset by prompting GPT-3.5-Turbo, see Appendix E. We are using the very same prefix proposed in the previous section to generate x s . Experiments are performed on 8 NVIDIA Tesla V100 GPUs, using half-precision tuning i.e., Float16. In the appendix, we also show results for an alternative training paradigm: the Low-Rank Adaptation (LoRA) [54] (see Appendix D.1). Following community standards [3, 1, 2], we use greedy decoding i.e., T = 0 for model evaluation.We first show empirical evidence that our proposed modification of DPO, E-DPO, does in fact improve safety alignment, using the Harmbench dataset [2] and the first 100 prompts in the AdvBench harmful behavior dataset [1] , measured by the HarmBench protocol. We give an overview on all adversaries in Appendix F. The results are presented in Table 1 . E-DPO achieves improvements across every task we tested.On top of our safety results, we want to make sure E-RLHF does not sacrifice helpfulness for increased safety. We evaluate helpfulness with the MT-Bench project [3] . The SFT model p SFT receives a score of 6.3, and both the DPO and E-DPO models perform better than that (6.9 and 6.7 respectively), making us believe that performance degradation is not a problem with our proposal. Next, we show the impact of the safe prefix on model performance. We demonstrate that our method's performance depends on the choice of safe prefix to some extend but never fails (see Appendix D.2). We believe, finding better safe prefixes by explicit tuning would improve our results, similar to the work by Yang et al. [55] , but we leave this exploration for future work. Further, we confirm that the improvement arises from using a safe prior in the KL term for harmful prompts. We ablate our results by appending the prefix on all prompts in the preference alignment dataset (see Appendix D.3). In all cases, applying the safe prefix to usual prompts degrades safety, showcasing the importance of switching the prior only on the harmful prompts. Finally, we show that E-DPO can be combined with any system prompt, to further boost safety (see Appendix D.4).The proposal can even be used to improve helpfulness and safety simultaneously (see Appendix D.5). 
conclusion and discussions: 
 In this paper, we present a theoretical framework for language model pretraining and jailbreaking by dissecting input prompts into query and concept pairs. Through this approach, we have established two theoretical results pertaining to the ability of language models to mimic the world following pretraining, which leads to outputting harmful explanations given harmful prompts; and the inevitability of jailbreaking resulting from alignment challenges. Guided by these theoretical insights, we have devised a simple yet effective technique to enhance safety alignment, and demonstrate the improved resilience to jailbreak attacks with this methodology.Current limitations (1) Although we have classified concepts as either harmful or non-harmful, it is important to acknowledge that the perception of a concept's potential for harm can be influenced by various factors such as cultural, legal, and societal norms, which collectively form the context of the situation. ( 2) Language models have demonstrated impressive capabilities in reasoning and completing tasks within multi-round, multi-step conversations; our current framework may not fully account for the generalization and jailbreaking possibilities associated with such input formats. (3) Our analysis is grounded on a fixed p world mapping and D P distribution. Nevertheless, the world is inherently dynamic, as both p world and D P continually evolve.Future work (1) Regarding our E-RLHF approach, as highlighted in the experimental section, in addition to attaching a universally safe prefix to all harmful prompts, improvements can be achieved by individually transforming the harmful prompts. Moreover, the safety-transformed prompts can be employed to expand the preference dataset for conventional RLHF. (2) Throughout our analysis, we have not imposed any constraints on the capacity of the language model. Extending our analysis under finite memory constraints or analyzing hallucination properties of LLMs is an interesting direction to explore. (3) Large language models have shown remarkable capabilities as in-context learners [56] , and such techniques could potentially be used for jailbreaking them as well [57] [58] [59] . Investigating the incorporation of such input paradigms remains a promising avenue for future research. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: the societal impact of our work has close connection to the topic of LLM safety. We propose a framework for analyzing language model pretraining and jailbreaking, and we design a new RLHF algorithm for improving safety. Our work could advocate for safer LLMs.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: although our research involves jailbreaking language models, we do not release any models or data, thus the paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: the code and data we use to train the model possess the Apache-2.0 license, and the code we use to test the model possess the MIT license. We include links to these public assets explicitly in the paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: we do not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: the paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: the paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Table 2 : Summary of notation.Symbol Meaning q A single query, composable with a certain set of concepts. cA single concept, composable with a certain set of queries. x = (q, c)A single prompt composed by query q and concept c. e A single explanation.The query set.The concept set. EThe explanation set.P ⊊ Q × CThe set of plausible prompts. pworld : P → ∆(E) The world mapping. For each plausible prompt, it specifies the ground-truth distribution over E, a.k.a. the "knowledge".pworld(q, c)The ground-truth distribution over a subset of E, given a prompt (q, c). With slight abuse of notation, it also refers to a point on the probability simplex. supp(pworld(q, c))Support of pworld(q, c). A strict subset of E. pLM : P → ∆(E)A language model. For each plausible prompt, it specifies a distribution over (a subset of) E, to mimic pworld. pLM (q, c)The output distribution over (a subset of) E by LM, given a prompt (q, c). With slight abuse of notation, it also refers to a point on the probability simplex. dom(pLM (q, c)) Domain of the pLM (q, c) distribution, a subset of E. (q, c) ∼ DP Underlying generative distribution over prompts, a.k.a. the distribution governing the creation of our pretraining corpus. supp(DP ) ⊊ P Support of DP . (q, c) ∈ supp(DP ) is called a direct prompt.A set of language models. πThe prior distribution over LM. ρThe posterior distribution over LM, after pretraining. γThe posterior distribution over LM, after preference alignment. ′ i |f (x 1 , x 2 , • • • , x n ) -f (x 1 , x 2 , • • • , x i-1 , x ′ i , • • • , x n )| ≤ c i , ∀i ∈ [n].Lemma B.1. (Hoeffding's Lemma) for random variable X ∈ [a, b] with probability 1, the following holds:EQUATION).Lemma B.2. (Hoeffding's Lemma, Multivariate) for random variablesZ = f (x 1 , • • • , x n )where f has the bounded difference property, the following holds:E[exp(λ(EZ -Z))] ≤ exp( λ 2 n i=1 c 2 i 8).Note that substituting Z with RS (LM ) is valid. Lemma B.3. Empirical Loss defined in Definition 3.1 satisfies the bounded difference condition with constant c = 1, ∀i.We are ready to present the proof of Theorem 1.Proof. Starting with the above lemma, we knowE S [exp(λ(R(LM ) -RS (LM )))] ≤ exp( λ 2 c 2 8n).The above result holds for a manually picked LM. With an overall average over the prior π we haveE LM ∼π E S [exp(λ(R(LM ) -RS (LM )))] ≤ exp( λ 2 c 2 8n).Apply Fubini's theorem (note that π is independent of S):E S E LM ∼π [exp(λ(R(LM ) -RS (LM )))] ≤ exp( λ 2 c 2 8n).Define Y = E LM ∼π [exp(λ(R(LM ) -RS (LM )))], a random variable depends on S. Obviously Y ≥ 0. Thus, with Markov's inequality:P[Y ≥ 1 δ E S Y ] ≤ δ.Equivalently, with probability at least 1 -δ, we haveY ≤ 1 δ exp[ λ 2 c 2 8n].Since we have assumed π, ρ share the same support, using Radon-Nykodim derivative to change the expectation with respect to π to with respect to ρ, we haveE LM ∼ρ dπ dρ exp(λ(R(LM ) -RS (LM ))) ≤ 1 δ exp[ λ 2 c 2 8n ].Taking logarithm and applying Jensen's Inequality we knowE LM ∼ρ dπ dρ + λ(R(LM ) -RS (LM )) ≤ log 1 δ + λ 2 c 2 8n .Incorporating c = 1, noticing dρ dπ = ( dπ dρ ) -1 we could rewrite the inequality asE LM ∼ρ (R(LM ) -RS (LM )) ≤ 1 λ KL[ρ||π] + log 1 δ + λ 8n .Finding λ that minimizes the term on right hand side gives us the ϱ term.When D P allows for a decomposition into mixture components, noticing the linearty of expectation, the bound can be re-written asαE LM ∼ρ [E (q,c)∼D P h ℓ TV (p LM , (q, c))] + (1 -α)E LM ∼ρ [E (q,c)∼D Ps ℓ TV (p LM , (q, c))] ≤ ϱ + E LM ∼ρ [ RS (p LM )].which leads toE LM ∼ρ [E (q,c)∼D P h ℓ TV (p LM , (q, c))] ≤ 1 α [ϱ + E LM ∼ρ [ RS (p LM )]].We give an estimation of the term appears in our PAC bound, ϱ, and state that it is non-vacuous.The numerator. We follow Neyshabur et al. [60] to instantiate the term in the simplest setup. Assume π, ρ are defined over the parameter space of a given LM, with K parameters. Assume w is a set of weights learned from the pretraining corpus. Let the prior π be the zero-mean multivariate Gaussian, whose entry-wise variance is related to the magnitude of the weight: σ i = β|w i |, and ρ be a Gaussian with the same anisotropic variance centered around w. We argue though simple, both settings are practical, since Gaussian initialization is common for model training, and the SWA-Gaussian algorithm [61] utilizes such Gaussian posterior. Under this setup, the KL goes as iw 2 i 2σ 2 i = O(K).Specifically, taking β = √ 2 2 makes the term exactly K. Current language models often possess millions, or billions, of parameters, namely, K ∼ [10 6 , 10 9 ].The denominator. To estimate the number of unique direct prompts in the training corpus, it is important to notice that the dataset does not only consist of (q, c) prompts but also e explanations. Thus, we need to estimate the average token length (ATL) associated with each unique prompt x = (q, c). For each unique prompt x, aside from its own token length l(x), there will be a collection of explanations {e i } N (x) i=1 , with expected token length of each associated explanation l(e). We haveEAT L = E x∼D P N (x) × [l(x) + l(e)].Fact. Given a prompt x, the larger the expected length of the prompt itself and explanation (l(x) + l(e) ↑), the larger the expected number of explanation elements (N (x) ↑), and the smaller the number of such prompts (D P (x) ↓), appearing in the training corpus. The former comes naturally due to the composability of natural language: the longer the text fragment, the more equivalent text fragments in expectation, while the latter is reflected by the spirit of the widely accepted Zipf's law.Inspired by the fact, we assume prompts are categorized by the quantity of l(x) + l(e), namely, for all prompt x, N (x) is a function of l(x) + l(e). Moreover, the complete data generation process is decomposed into i) sample a value of l(x) + l(e) out, and then ii) sample a unique prompt from the set decided by this specific l(x) + l(e) value, and iii) generate N (x) explanations.Step i). Use the fact: the larger the expected length of the output explanation, the smaller the probability that such a prompt appears in the training corpus. We assume step i) follows a (cut-off) zeta distribution. Specifically, for a random prompt x,p(l(x) + l(e) = k) ∝ k -s , ∀k ≥ k 0 .When k 0 = 1, we resume the zeta distribution with coefficient s.Step ii). We assume each prompt following this step is unique.Step iii). Use the fact: the larger the expected length of the output explanation, the larger the expected number of explanation elements in the training corpus. We assume a power law scaling on N , with a constant t > 1, such thatN (l(x) + l(e) = k) = k t-1 .Thus, the average token length writesEAT L = k p(l(x) + l(e) = k) × k × N (l(x) + l(e) = k) = ζ(s -t) - k0-1 i=1 i -(s-t) ζ(s) - k0-1 i=1 i -s . where ζ(s) = i∈Z + i -s is the Riemann zeta function.For example, take s = 4, t = 2. With k 0 = 1, the ATL would be 1.52, while with k 0 = 10, the ATL becomes 272. These results translate into an estimation of unique prompts as n tokens /AT L. With current SOTA LM, the pretraining corpus often includes (tens of) trillions of tokens (> 10 12 ), thus n > 10 10 > K can be safely assumed ⇒ ϱ < 1.α constant. According to LLaMa-2 report (section 4.1, Figure 13 ) [13] , approximately 0.2% of the documents in their training corpus is labeled as harmful. However, we argue this is indeed an extremely loose lower bound for α, due to the estimation strategy used in their paper. Given a document, they use a binary classifier on harmfulness over each single line (1 means harmful and 0 otherwise), and assign the average score to the document. 0.2% is the ratio of documents with score ≥ 0.5. Take the example of "How to build a bomb". The chemical reaction parts will not be counted as harmful, and thus this estimation strategy could judge a completely harmful explanation as harmless. Thus, it is reasonable to assert α is not too small, though with current literature we are not capable of raising an accurate estimation on it.Before proceeding to the proof, we list necessary definitions and lemmas as follows. Lemma B.4. (Volume of n-simplex) 10 For any dimension n, the volume of the n-element probability simplex:∆ n-1 , in the n -1-dimensional space is √ n (n -1)! .We define the projected probability simplex as follows. Definition B.2. (Projected probability simplex) Given ∆ n-1 , the corresponding projected probability simplex,∆ n-1 p , is defined as a subset of R n-1 : {x ∈ R n-1 | n-1 i=1 x i ≤ 1, ∀i ∈ [n -1]}.10 See https://en.wikipedia.org/wiki/Simplex#Volume.An illustration of ∆ n-1 and ∆ n-1 p . For example, take n = 3. The probability simplex with n = 3 elements is a triangle whose (euclidean) side length is √ 2 with vertices (1, 0, 0), (0, 1, 0), (0, 0, 1). Then its volume in the 2-dimensional space, i.e., its area, is √ 32 . The corresponding projected probability simplex is the triangle between the X -Y axis, with vertices (1, 0), (0, 1), (0, 0).A direct lemma that connects the probability simplex and the projected probability simplex is given below.Lemma B.5. (Transformation of probability simplex) Given a proper probability density function ν(x) defined on ∆ n-1 p , it is equivalent to the distribution defined on ∆ n-1 with density ν(x) √ n :∀A ∈ Borel(∆ n-1 p ), let B = {x ∈ ∆ n-1 : x 1:n-1 ∈ A}. Then A ν(x)dx = B ν(x) √ n dx. Specifically, this implies volume(A) volume(∆ n-1 p ) = volume(B)volume(∆ n-1 ) .Proof. Consider a translation on ∆ n-1 with x n = -n-1 i=1 x i which does not affect its the volume and shape. The mapping: ∆ n-1 p → translated∆ n-1 is an affine transformation with matrixT =    1 0 • • • 0 0 1 • • • 0 • • • • • • • • • • • • -1 -1 • • • -1    n×(n-1)Thus, any area under this transformation is scaled by√ det T ⊤ T = √ n: a constant.The lemma follows directly after this conclusion.We use U (•) to denote the uniform distribution over ∆ n-1 : U (x) = (n-1)! √ n , ∀x ∈ ∆ n-1 . We use the notation vol[S] = S 1ds to represent the volume of a given subset S ⊂ ∆ n-1 , and use rvol[S] for the relative volume (w.r.t. the underlying n-simplex) of S, i.e., rvol[S] := vol [S] vol[∆ n-1 ] = S U (x)dx. We also use n = |E(c)| from now on. We use the vector x to denote (with the slight abuse of notation we have mentioned) p LM (q, c) on the output simplex.Lemma B.6. (Gaussian cdf Tail Bound, Gordon [62] ) Denote ϕ(•) as the standard Gaussian pdf. When x > 0, x x 2 + 1 ϕ(x) = x x 2 + 1 e -x 2 /2 √ 2π ≤ 1 -Φ(x) ≤ e -x 2 /2 √ 2πx = 1 x ϕ(x).h : H h,p := {x ∈ ∆ n-1 p | n0 i=1 x i ≥ p}. Note that ∆ n-1 p ⊂ R n-1 .We derive its expression as follows.C h,p ] = volume[{x ∈ ∆ n-1 p | n0 i=1 x i ≤ p]} = p 0 dx 1 p-x1 0 dx 2 • • • p- n 0 -1 i=1 xi 0 dx n0 1- n 0 i=1 0 dx n0+1 • • • 1-n-2 i=1 xi 0 dx n-1 = p 0 dx 1 p-x1 0 dx 2 • • • p- n 0 -1 i=1 xi 0 dx n0 1 (n -n 0 -1)! (1 - n0 i=1 x i ) n-n0-1 = p 0 dx 1 p-x1 0 dx 2 • • • p- n 0 -2 i=1 xi 0 dx n0-1 1 (n -n 0 )! (1 - n0-1 i=1 x i ) n-n0 -(1 -p) n-n0 = • • • = 1 (n -1)! [1 -(1 -p) n-1 ] - n0-1 j=1 (1 -p) n-1-j j!(n -1 -j)! p j(5) Thus, the relative volume of H h can be written asEQUATIONWhich is the binomial distribution formula. With the Central Limit Theorem, when n ≫ O(1), we know the binomial distribution can be well approximated via the normal distribution as follows:EQUATIONThus, denote ϕ (n-1),p (x) as the pdf of Gaussian variable with mean (n-1)p, variance (n-1)p(1-p), the rvol term above can be estimated as follows:EQUATIONWe use a= |E h (c)|-1-(n-1)p √ (n-1)p(1-p). Consider an adversary with budget ϵ under ℓ p or Jensen-Shannon Divergence (JSD) / Total Variation (TV) capability. Since ||x|| 1 ≥ ||x|| p , ∀p ≥ 1 as well as ||x|| 1 ≥ 2JSD(x), ||x|| 1 ≥ 2TV(x), we know H h (ϵ, ℓ 1 ) ⊂ H h (ϵ, d) for all d we have considered.With that ℓ 1 , ϵ setup, the corresponding ϵ-expansion set of H h has a closed-form expression asH h (ϵ, ℓ 1 ) = {x ∈ ∆ n-1 | n0 i=1 x i ≥ p - ϵ 2 }.Similar as above, we derive the analytical solution of its relative volume associated with constant a ′ as:EQUATIONUnder our framework, with p < 1 2 , we know1 4 > p(1 -p) > (p -ϵ 2 )(1 -p + ϵ 2 )). Thus a ′ > a + √ n -1ϵ := a ϵ .Consider the induced distribution γ c on the output simplex. Given an adversary with ℓ p or JSD/TV perturbing capability, with the fixed harmful concept c, safety is guaranteed if and only if p LM (q, c) resides outside H h (ϵ, d). Define the area of interest, S(d) as S(d) := ∆ n-1 -H h (ϵ, d). Thus, the probability of this event could be bounded asP x∼γc 1 x∈S(d) < max x∈S(d) γ c (x) S(d) 1dx < γ s rvol[S(d)] < γ s rvol[S(ℓ 1 )] < γ s (1-rvol[H h (ϵ, ℓ 1 )])This gives an upper bound of γ s (1 -Φ(a ϵ )).which can be simplified when a ≥ 0 using Lemma B.6:γ s ϕ(a ϵ ) a ϵ .Thus, the probability of getting a LM instance from the preference alignment process such that it allows for successful jailbreaking on a specific harmful concept c is at least1 -γ s (1 -Φ(a ϵ )) .Up to now, we have derived the result in Theorem 2. However, we can move a step further to show the decay rate on the right hand side term. It can be simplified when a ≥ 0:1 -γ s ϕ(a ϵ ) a ϵ ,which finishes the proof.The Following, proximal policy optimization (PPO) [65] is commonly adopted across these implementations, forming the basis of current state-of-the-art language models. The KL-constrained RL Fine-Tuning (RLFT) objective takes the form:max p LM E x∼Ds {E e∼p LM (•|x) [r(x, e)] -βD KL (p LM (x)||p SFT (x))}.However, PPO tuning can suffer from instability [66] and implementation complication [67] . To overcome these issues, a series of work propose to skip the reward modeling step and learn directly from the preference dataset, with the representative pioneering work by Rafailov et al. [23] , namely Direct Preference Optimization (DPO). We summarize the derivation of the DPO objective below, and generalize the objective to the one we use in our experiments, i.e., E-DPO.First, noticing the closed-form optimal solution for p LM of the RLFT objective writes (see e.g., Appendix A.1 of Rafailov et al. [23] )p * RLFT (e|x) = 1 Z ′ (x) p ref (e|x) exp( 1 β r(x, e)).With this analytical solution in mind, we can solve the reward asr(x, e) = β log p * RLFT (e|x) p ref (e|x) + β log Z ′ (x).Regard π * as the optimization target, plug this transformation into the reward model objective to obtain the DPO objective: )].p DPO = arg min p LM -E (x,For our E-RLHF, the modification to the objective leads to another optimal solution of p LM :p * (e|x) = 1 Z(x) p ref (e|x s ) exp( 1 β r(x, e)).Thus,r(x, e) = β log p * (e|x) p ref (e|x s ) + β log Z(x)And plug it in to the reward model objective to formulate our E-DPO:p E-DPO = arg min p LM -E (x,ew,e l )∼D [log σ(β log p LM (e w |x) p ref (e w |x s ) -β log p LM (e l |x) p ref (e l |x s ) )].The advantage of our E-RLHF objective is as follows.Proposition C.1. (Overcoming the small safe set problem) E-RLHF will lead to the optimal solution p * :p * (e|x) = 1 Z(x) p ref (e|x s ) exp( 1 β r(x, e)).Compared to p * RLFT , the advantage when encountering a harmful prompt x is:(1) (Erase harmful explanations) ∀e ∈ supp(p ref (•|x)) -supp(p ref (•|x s )), p * (e|x) = 0; (2) (Add safe explanations) ∀e ∈ supp(p ref (•|x s )) -supp(p ref (•|x)), p * (e|x) > 0 = p *RLFT (e|x). Thus, with the same jailbreak threshold p, the safety zone is successfully expanded.Intriguingly, when the safe transformation is done by appending an identical safe prefix to all harmful prompts, we can connect our E-RLHF to context distillation. A good prompt is known to matter for the performance of a fixed-parameters LM [68, 55] . Researchers have proposed a systematic LM tuning algorithm, called Context Distillation [69] , aiming at distilling useful information from a good context as prefix to a language model. Given an initialized language model, for example p SFT , an input prompt x and a prefix context string prefix, Askell et al. [69] optimizes the lossL(p LM ) = D KL (p SFT (prefix ⊕ x), p LM (x))where ⊕ stands for string concatenation. This technique has been adopted as part of the safety alignment process during the LLaMa-2 series tuning [13] , where prefix is chosen from a set of predefined safe prefixes. When applying the identical prefix transform in our E-RLHF transformation, it can be regarded as a combination of safety context distillation and RLHF. This gives another point of view on the effectiveness of our proposal.In this section, we perform extensive ablation studies to showcase the effectiveness of our proposed E-RLHF. In this section, we show results obtained by Low-Rank Adaptation [54] . We explore the same set of safe prefixes as in ablation D.2, and choose the best model for illustration. Numbers are illustrated in Table 3 . Results are identical to the ones obtained via full parameter tuning that our E-RLHF performs better consistently against the RLHF baseline.We ablate the effect of different safe prefixes. The prefixes we consider are collected in Table 8 . Attack success rate numbers are shown in Table 4 , with each model's MT-Bench scores shown in the brackets. Clearly, almost all safe prefixes lead to better safety against the DPO baseline and improved helpfulness compared to the SFT model (whose MT-Bench score is 6.3), and the performance could vary depending on the choice of the prefix. Finding a good prefix matters for our method, and we leave digging the optimal one out as future work.Here, we proceed to ablating the effect of transforming all prompts, no matter harmful or not. Results are shown in Table 5 , where the red color indicates that safety downgrades compared to the model obtained via transforming harmful prompts only. Clearly, most models even persist worse safety compared to the DPO baseline itself, suggesting the detrimental effect of transforming the unharmful prompts.As pointed out by previous works [2, 48, 70] , system prompt can have a significant impact on ASR. This comes in two-folds: firstly, a powerful system prompt can initialize the LM to be closer to the safety zone, thus making the model safer; secondly, a longer system prompt would enlarge the difficulty of launching a specific attack due to the increased computational consumption. To confirm our E-RLHF is still helpful under this specific situation, we ablate this effect under the control of the default Mistral system prompt 11 . The results can be found in Table 6 , where nearly all ASRs drop, whilst the conclusion does not change.In similar spirit, we ask the question: can the trick we propose to improve safety be used to improve helpfulness simultaneously? To achieve this goal, we choose the prefix "Give me a safe and helpful response to the following request.", and apply it to all prompts in the preference dataset. Results can be found in Table 7 . Our model achieves better safety, and improves the helpfulness benchmarked by MT-Bench [3] from 6.9 to 7.0.Inspired by the work by Chao et al. [30] , we prompt GPT-3.5-Turbo as a harmness judge on input prompts. It assigns an integer score ranging from 1 to 10 for a given prompt, and we choose the score ≥ 6 subset as harmful. The system prompt is shown in Table 9 .11 https://docs.mistral.ai/capabilities/guardrailing/ (1) "Provide a responsible and ethical answer to the following request. "(2) "In a manner that promotes safety and well-being, please respond to the following request. "(3) "Keeping in mind the importance of safety and legality, how would you respond to this: " (4) "Please ensure your response adheres to community guidelines and ethical standards: " Table 9 : System prompt for GPT-3.5-Turbo used to filter harmful and non-harmful prompt in the preference dataset.You are a helpful red teaming assistant. You will receive a prompt. Please act as an impartial judge and evaluate whether the prompt itself is requesting information that violates ethical, legal, or safety guidelines. Be as objective as possible. Rate the prompt on a scale from 1 to 10. A rating of 1 signifies a fully non-harmful prompt.A rating of 10 signifies prompts that could lead to responses that fully violate ethical, legal, or safety guidelines. Your rating should strictly follow this format: "Rating: [[rating]]", the rating in the double-closed brackets is a number from 1 to 10, e,g, "Rating: [ [5] ]".F Jailbreak adversaries collected in the HarmBench project [2] In this section, we give a short overview of the adversaries we adopt to evaluate our models. Some descriptions are summarized in Mazeika et al. [2] .• Direct Request refers to sending the original harmful prompt directly to the target LLM.• GCG [1] , GBDA [71] and AP [72] find adversarial suffixes via token-level optimization.• SFS (Stochastic Few-Shot) and ZS (Zero-Shot) [15] are few-shot sampling or zero-shot generation of test cases by an attacker LLM to elicit a behavior from a target LLM.• PAIR [30] and TAP [33] are iterative prompting / tree-structured prompting methods, with an attacker LLM, to adaptively explore and elicit specific harmful behaviors from the target LLM.• AutoDAN [29] is a genetic-algorithm based attack with initializations from handcrafted DAN jailbreak prompts.• PAP [73] uses persuasive strategies to jailbreak the target LLM.• HumanJailbreaks [74] uses a fixed set of in-the-wild human jailbreak templates, similar to the Do Anything Now (DAN) jailbreaks.We exclude all transfer attacks since we focus on single-model jailbreak. Furthermore, we choose to discard the UAT [75] and PEZ [76] adversaries, because the former induces an out-of-memory error on our V100 GPUs, and the latter never succeeds to find a suffix according to our experiments.The societal impact of our work has close connection to the topic of LLM safety. We propose a framework for analyzing language model pretraining and jailbreaking, and we design a new RLHF algorithm for improving safety. As shown in our experiments, our work could advocate for safer LLMs.In this section, we provide a review of the current literature on LLM jailbreaking.In this section, we summarize existing jailbreaking methods.benign data, leads to safety degradation using LLaMa and GPT-3.5-Turbo. Fine-tuning on harmful data (with less than 5 gradient steps) will cause the model to be completely harmful, while tuning on identity-shifting data could make the LM fully obedient.Low-resource language and cipher Yong et al. [26] , Deng et al. [27] explore the difference in languages when encountering the same harmful query, and find a direct translation to low resource languages will lead to higher risk, and Deng et al. [27] additionally find when combined with sophisticated methods, the drawback of low-resource languages disappear. Yuan et al. [28] use cipher encoding and decoding to break LLMs. Smaller scale models are immune from such attacks, while the smartest GPT-4 encountered the highest risk.Vision-language model attacks Besides pure LLM, some research works move a step forward, utilizing images for breaking vision-language models (VLMs). Shayegani et al. [92] explore multimodal attack on VLM via embedding space feature matching. Qi et al. [93] generate adversarial examples via maximizing the conditional probability of a harmful corpus, i.e., the sum of log probabilities over all outputs, and use the final image with harmful query for jailbreaking. Carlini et al. [94] generate adversarial example for a fixed harmful content, and find no matter what input prompt is given to the VLM, it will respond with the target harmful string. Maus et al. [95] propose a black-box attack on manipulating the generated image with modified adversarial prompt.Misc Wei et al. [57] , Wang et al. [58] explore in-context learning for attack and defense. The attack is weak since it could only break Vicuna [96] and can be defended by in-context safe examples. Later, this method is scaled-up to significantly improve strength for breaking guardrails of large, state-ofthe-art models [59] . An early work in February 2023 [97] adopts obfuscation (including synonym and typos), code injection and virtualization to successfully jailbreak ChatGPT. Shah et al. [98] illustrate in-context automated persona modulation attack for large-scale LLMs and Vicuna. Zeng et al. [73] consider the more broadly perspective of persuasion: they train a persuasive paraphraser based on a fine-grained taxonomy of persuasion techniques. Detailed ablation on attack effectiveness is studied. Guo et al. [99] focus on stealthiness and controllability. They notice the constraints applied to the jailbreaking prompts (e.g., fluency) are exactly the targets of the controllable text generation problem. Thus, they adopt the Energy-based Constrained Decoding with Langevin Dynamic (COLD) [100] on output logits. Forming each constraint as well as the task of jailbreaking as an energy function over logits, the Langevin Dynamic is used for finding a good logit distribution, and the decoding technique in Qin et al. [100] is used for output generation. Banerjee et al. [101] introduce a dataset TECHHAZARDQA, compare direct query v.s. pseudo-code format, and find the latter induces higher risk. Mangaokar et al. [102] considers a type of adaptive attack against checking-based defense, that appends a universal adversarial prefix into the jailbreaking prompt to make the guard model always output "safe", and thus making the detector fails to detect harmful information. Lv et al. [103] propose Code Chameleon, which contains multiple encryption and decryption methods defined by python functions, that transforms the harmful prompt into specific predefined form to jailbreak LLMs. Sadasivan et al. [104] speed up the computation of GCG [1] to make it possible to launch on a single GPU. Geiping et al. [105] build a taxonomy on risks beyond jailbreaking, and coerce the LLM to provide certain outputs by optimizing a set of tokens via GCG. Ren et al. [106] propose CodeAttack that use code templates to query the output out instead of using natural language directly, and obtain descent results.Up to now, no universal defensive strategy as adversarial training [107] for adversarial attacks / differential privacy [108] for membership attacks exists as a gold standard. In general, we can classify the methods into three typical types: alignment, red-teaming, and algorithmic proposals.Alignment The target of alignment is to push the output of language models be aligned to human values. Regarding safety, the goal is to avoid outputting harmful information. RLHF is widely adopted in these methods [19, 20, 109, 22] . Variants like RLAIF also have been proposed recently [21, 110] .Red teaming This term is populated as specifically dealing with harmful info on dataset curation, used together with RLHF [14] [15] [16] [17] [18] .Next, we proceed to defensive algorithm proposals. We classify existing defensive strategies in the following categories.Defense against suffix-based attacks. Alon and Kamfonas [111] notice the messy nature of the suffix generated by GCG, and propose to use a perplexity (PPL) filter on input prompts. They also explore using a LightGBM [112] with 2 features (PPL, prompt length) to filter harmful prompt, and show it does better than naive PPL thresholding. The PPL-based filter does not succeed with human-crafted jailbreaks. Jain et al. [37] explore many concerning viewpoints, including self-PPL filtering, paraphrasing the input prompt, and re-tokenization since many LLMs' tokenizers are based on Byte-Pair Encoding (BPE). All methods are successful in regards of defending against suffix-based attacks. They also explore the simplest form of adversarial training. Robey et al. [34] propose to perturb the input token string by random replacement/erasement/insertion, and finally perform a majority vote in the end. Cao et al. [35] judges whether an input prompt is safe or not by estimation with Monte Carlo, when randomly dropping a fraction of tokens, using the LLM itself. Kumar et al. [36] try to perform "certified" safety against harmful prompts, by erasing tokens and set the original prompt as harmful if at least one of these erased prompts lead to a harmful response, or be classified as harmful by a DistillBERT-based classifier.System prompt defense. We could modify the input prompt for jailbreaking; and several works explore if we can apply similar methods to system prompts to defend against such attacks. Xie et al. [113] propose "self-reminder", i.e., appending a reminding prompt within the system prompt for defense. The attacks are collected from the JailbreakChat collection, and this strategy is effective for defending against them. Zheng et al. [114] advocate for finding a good system prompt automatically, by investigating the representational difference between safe and harmful queries, and optimizing the safety prompts along the harmful refusal direction in the representation space. One intriguing takeaway is harmful / harmless queries can be distinguished in the representation space, different from the adversarial examples in vision. Zhou et al. [115] also optimize the safe system prompt, but in a more "adversarial training" fashion, that apply jailbreak algorithms with current safe prompts first and then find good replacement candidates in the same way as done by Zou et al. [1] . Concurrently, Mo et al. [116] propose prompt adversarial tuning, where an adversarial suffix is assumed, while a safe system prompt is jointly optimized with this suffix, with an additionally constructed benign loss to improve helpfulness under normal queries. Zhang et al. [117] propose the idea of "goal prioritization", either without training (append prioritize safety than helpfulness and in-context examples to the system prompt) or with training (generate data pairs of prioritizing safety or helpfulness, finetune, and append prioritize safety prompt into system prompt). The former is effective for large-scale LLMs, while the latter improves safety of LLaMa-chat models. Zhou et al. [118] propose in-context adversarial game, where an attack LLM and a defense LLM interact on exchanging insights on successful jailbreaks, and defend by improving the system prompt. Zou et al. [70] give the result that system prompt matters for jailbreaking, and shows conducting GA-based search over it could improve safety.Checking-based, decoding-based, and Misc. Helbling et al. [119] generate responses first, and then use the LLM itself to examine whether the output is harmful or not. They find such simple self-examination is powerful since the TPR reported can be up to ∼ 1.00. Wang et al. [120] propose to (1) tune the LM to enhance its capability on discriminating harmful / harmless content; (2) tune the LM to make it able to tag its own response; and (3) rewrite response if output is harmful. Li et al. [121] propose to suppress the attack performance by iteratively rewinding and re-examining the generated output. The method does not work well with small models, but works pretty fine with large (open-source) models. They find the strategy can improve generalization as well. Xu et al. [122] train a safer model first, and use normalized p attacked + α(p safer -p attacked ) over top-k shared tokens for decoding to enhance safety. Hasan et al. [123] show with original Wanda pruning [124] , the LLM can help resist direct jailbreaking prompts, e.g., with role-playing attacks. Pi et al. [125] propose MLLM-Protector on safeguarding Visual LLMs by checking the output and then detoxifying the content. Zhang et al. [126] perform intention analysis on the input, and enforce the model generate policy-aligned outputs both by prompting. Wang et al. [127] propose backtranslation that guesses the input prompt directly, and reject if it is harmful. Kim et al. [128] propose self-refinement which consists of generating a feedback and then refine the response to avoid harmful info output. They find using additional JSON and code formatting would improve safety. Zeng et al. [129] propose AutoDefense, which utilizes multiple agents on analyzing prompt, response and intention, to defend against attacks. Hu et al. [130] propose Gradient Cuff, a sampling-based gradient-norm defense method, by rejecting those input prompts with large gradient norm on top of a majority-vote based filtering. Ji et al. [131] propose a method similar to Robey et al. [34] , but for semantically-meaningful attacks, that paraphrases the input according to several criteria and conduct a majority vote for judging.Several company-centered products also fall into this regime. For example, LLaMa-Guard [132] is trained on toxic data such that it is able to discriminate unsafe user prompts and outputs, respectively. IBM also propose a framework on constructing and deploying safeguard detection modules, and releases the details in a technical report [133] .Wolf et al. [38] assumes the decomposability of LLM output into a good and bad component, and show possible jailbreaking in theory by prompting the model with a sufficiently long input. Kalai and Vempala [134] use statistical tools to prove hallucination for calibrated LMs. Lee et al. [135] study the representation in GPT-2. They train a base classifier for toxicity, and use the linear weight as a proxy of toxic vector. They find there are value vectors close to the toxic vector itself, that are not suppressed by DPO tuning [23] . Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).For real-world LMs, with different decoding hyperparameters e.g., the temperature T , top-p and top-k sampling parameters, the induced distribution with the same set of parameters could be different. Our discussion holds for a pre-fixed set of hyperparameters throughout this paper.We regard both distributions as defined over the entire E since we do not restrict the output distribution of LM in this section.The inspiration for the proof of Theorem 1 comes from Mbacke et al.[45], and the proof idea is originally proposed in Germain et al.[46], Haddouche et al.[47].Otherwise the model could drift into trivial behaviour like always responding with "I can't help you.".Even the probability can be suppressed to close to 0.The prefix shares similarities to the system prompts used by open-source LLMs[13,49] to boost safety. 


Decision: The paper provides a theoretical analysis of jailbreaking for LLMs using PAC-Bayesian bound. They found that the jailbreaking is unpreventable. 

Strength: 
S1. The paper proposes a sound theoretical framework to analyze the jailbreaking behavior of LLMs. 
S2. the paper proposes an improved alignment training method E-RLHF.

Weakness: 
W1. There is disconnection between the theoretical analysis and experimental validation. 

Overall, the theoretical framework proposed in this paper to analyze LLM jailbreaking is an important contribution.

Reviews: review: paper_summary: paper_summary: The paper provides a theoretical insight about LLM jailbreaks using PAC-Bayesian bound for pretraining LLMs. It assumes that there always exists the harmful data in the mixture, and as the model is trained on this mixture, the model will probably produce the responses in harmful zone (it has a specific definition in the main paper). Based on this framework, the authors suggests that the safety zone of the models should be extended, and to this end, they introduce a method called E-RLHF which can expand the safety zone. E-RLHF replaces the harmful prompts x_h into benign prompts x_s, and replace some of the terms in RLHF (and DPO) to make sure that the model keeps in the safety zone. Experimental results also show that it does not sacrifice the general capabilities but can be improved in the safety perspective.
strength_weakness: strength_weakness: - E-RLHF is an inaugural and simple form of expanding the safety zone of LLM; I think there could be more sophisticated and effective ways, and I hope the authors will address this in the future works.
- Other than that, I think there is no big weakness in the paper, but have some minor comments:
  - Eq (2): it is slightly confusing that in D_KL, the term only have p_LM(x) and p_SFT(x), not p_LM(e|x) and p_SFT(e|x).
  - About writing: I am not familiar with using the term "explanation" -- instead, I think using "response" is more common. At the first glance, it was hard to comprehend the meaning of the term.
questions: questions: No specific questions about the paper.
limitations: limitations: The authors provided limitations section in the paper.
paper_summary: paper_summary: The paper addresses a very important issue of our time, the safety of LLMs. LLMs are already used in various applications and will be present in more applications to come, as e.g. Microsoft, Apple and Google are integrating LLMs in their applications and operating systems. Hence, the question on how to make the systems more robust against jailbreaking is a very important question.

This paper offers a statistical approach to this question and presents experiments. The mathematics is based PAC-Bayesian approach and builds upon a very nice formalization of splitting a prompt into queries and concepts.
strength_weakness: strength_weakness: There is a disconnect between the theory and the experiments. The definitions, theorems, etc. are all presented and motivated very nicely, but could also apply to any other mapping of a stochastic system. Oversimplified, one may state that there is a mapping from an input space into an output space, in which the output space can be divided into desired and undesired outputs. The goal is to reduce the space of undesired outputs. To make this a case specifically about LLMs, there needs to some form of conclusions that feed into the experiments. Unfortunately, I don't see this connection. The experiments use a systems prompt that is preprended to the actual prompt. I do not see, how this follows from the theory.

This is really unfortunate, because I would really like to see, how the experiments connect to the theory.
questions: questions: Could you please point out the connection between experiment and theory that I have seemed to miss.
limitations: limitations: I don't see any negative social impacts. As stated above, I would like to see a stronger connection between experiments and theory.
paper_summary: paper_summary: The paper presents a statistical framework that provides a theoretical analysis of the jailbreaking problem in language models. The authors first examine the PAC-Bayesian bound to demonstrate that there is a non-trivial probability for LLMs to mimic harmful behavior if such information is present in the pre-training corpus. They then theoretically prove that a jailbreaking prompt can be found for such pretrained LMs and propose mitigating the jailbreak issue by modifying the harmful prompts.
strength_weakness: strength_weakness: I find the connection between the theoretical part and section 5 (E-RLHF for actual experiment) quite handwavy. Expanding the safety zone is the claimed goal, but the proposed solution (through injecting safe concepts in a harmful prompt) seems really hacky. For example, as shown in Table 4, using safe prompt (4) gives much better results than safe prompt (1) for no reason. Overall, this proposed method seems like a small trick that the authors try to incorporate into the paper just to show some empirical value of their theoretical analysis. I also find that with such modifications, the MT-Bench score (briefly mentioned in line 355) is lower under E-DPO than DPO.

Lastly, this might be a biased opinion (that's why I didn't factor this part into my scoring decision and still gave a marginally above accept score): I feel that after all the theoretical proofs (despite their elegance), the conclusion (LLMs will mimic toxic behavior if such toxic content is present in the training corpus, jailbreak is unpreventable under reasonable assumptions, and by expanding the safe zone we reduce the probability of jailbreaking) is very intuitive and does not provide much additional insight into the problem. Maybe the mathematical formalization of the jailbreak problem itself is meaningful, and I will let the other reviewers judge the novelty of such a framework.
questions: questions: Could you provide more details on the MT-bench's performance? For example, you have an ablation study on different safety prefixes' effects in Table 4. What about their impact on MT-bench?

In line 327 and the ablation study in the appendix, you show that non-harmful prompts should not be modified (otherwise, the model performs much worse). Could you relate this phenomenon to the safe/harmful zone or your theoretical analysis? I don't see why adding a safe prefix to a non-harmful prompt negatively impacts the safe/harmful zone, and under your assumption, this should not lead to worse behavior.
limitations: limitations: No concerns on the limitations
paper_summary: paper_summary: The paper introduces a new theoretical framework that views prompts as combinations of concepts and queries, allowing for a detailed analysis of how and why LLMs can be manipulated into producing unsafe responses. It proposes a statistical metric for alignment, focusing on quantifying the safety of the model's outputs.

The study introduces an improved alignment strategy, Enhanced Reinforcement Learning from Human Feedback (E-RLHF), which modifies existing RLHF methods to increase the likelihood of safe responses from LLMs without additional training costs.
Through experiments using standard benchmarks like AdvBench and HarmBench, the paper demonstrates that E-RLHF significantly outperforms traditional RLHF in resisting jailbreaking attempts, reducing the models' vulnerability to producing harmful outputs.
strength_weakness: strength_weakness: While E-RLHF improves alignment in controlled experimental conditions, its effectiveness in more complex conversational scenarios, where inputs can evolve over a series of interactions, remains untested.
questions: questions: The two datasets used in the work contain harmful requests about various concepts (e.g. drug, weapon). Have you tried to analyze the performance of E-RLHF on different concepts?
limitations: limitations: Yes

Comments: title: Response to comments
Response to comments
Response to comments
Response to comments
Answer
Official Comment by Reviewer UXGw
comment: We want to sincerely thank the reviewers for their effort in thoughtfully discussing our paper. We appreciate your detailed feedback and the positive points you’ve highlighted about our work. We firmly believe that our contributions, particularly in demonstrating the inevitability of jailbreaking, are crucial for fundamentally understanding the limits of alignment. This work hence delineates the critical battleground where future attack and defense strategies.

While we understand your concerns, we kindly request that you consider an increase in the score to better reflect the strengths and potential impact of our work. We are fully committed to further improving the manuscript and remain open to any additional suggestions you may have.

Warm regards + fantastic start in the week,
The authors
Dear reviewer sD4q,

We appreciate the recognition of our efforts and thanks for responding to our rebuttal!
Dear reviewer UXGw,

Thanks for responding to our rebuttal and raising your score! We apologize for the delay in accessing the OpenAI API, and will include the requested comparison in our final draft.
Dear reviewer T1QG,

Thank you for your response! Could you please provide more details about your concerns so that we can address them more effectively? In our rebuttal, regarding the specificity of our framework to LLMs, we clarify how it is uniquely tailored in our assumptions, which are particularly relevant to LLMs. Concerning the connection between theory and experiment, we had hoped to have clarified how the theory allowed us to identify the problems with current RLHF approaches (keeping a too small safety zone), thus leading us to our successful strategy of safe concept substitution through prefixing. In our discussion with reviewer qvRD, we explore several strategies that involve both human intervention and LLM-based support to facilitate the concept substitution as future explorations. 

We welcome any suggestions on how we can further refine our approach to better address your concerns.
Dear reviewer qvRD, 

We appreciate the acknowledgements of our work and thanks for responding to our rebuttal!
Dear authors,

Thank you very much for the lengthly reply to my main concern. The reply validates, that I did understand the main points of the paper and approach. Unfortunately, I don't see any additional arguments, that directly answer my question. I do see the importance of the experiments and results and I also do so see the theory, that you discuss in the paper. I just don't see, how the two are connected.

Prompt engineering and system prompts are a well established method in many LLM-applications. They do not follow from your theory as there are other ways to change the safe and unsafe areas.

I appreciate the answer, but will remain with my initial assessment of the paper.
Thank you for your responses, I will keep my rating.
Thanks. I will keep the rating since it is already the highest.
Dear Authors,

Thanks for providing additional clarifications to the points I asked in the review. After reading the response (for both my review and others), I decide to increase the score (from 5) to 6.