Mission Impossible: A Statistical Perspective on Jailbreaking LLMs
Authors: Jingtong Su
Julia Kempe
Karen Ullrich
Venue: NeurIPS 2024 poster (2024)
Keywords: large language models
jailbreak
safety alignment
theory
Primary area: learning_theory
TL;DR: We provide a statistical framework for jailbreaking LLMs, show jailbreaking is unavoidable and provide an improved DPO-based alignment protocol. The proposed method leads to improved safety across a suite of jailbreak adversaries.

Abstract: Large language models (LLMs) are trained on a deluge of text data with limited quality control. As a result, LLMs can exhibit unintended or even harmful behaviours, such as leaking information, fake news or hate speech. Countermeasures, commonly referred to as preference alignment, include fine-tuning the pretrained LLMs with carefully crafted text examples of desired behaviour. Even then, empirical evidence shows preference aligned LLMs can be enticed to harmful behaviour. This so called jailbreaking of LLMs is typically achieved by adversarially modifying the input prompt to the LLM. Our paper provides theoretical insights into the phenomenon of preference alignment and jailbreaking from a statistical perspective. Under our framework, we first show that pretrained LLMs will mimic harmful behaviour if present in the training corpus. \textbf{Under that same framework, we then introduce a statistical notion of alignment, and lower-bound the jailbreaking probability, showing that it is unpreventable under reasonable assumptions.} Based on our insights, we propose an alteration to the currently prevalent alignment strategy RLHF. Specifically, we introduce a  simple modification to the RLHF objective, we call \emph{E-RLHF}, that aims to increase the likelihood of safe responses. \emph{E-RLHF} brings no additional training cost, and is compatible with other methods. Empirically, we demonstrate that \emph{E-RLHF} outperforms RLHF on all alignment problems put forward by the AdvBench \citep{zou2023universal} and HarmBench project \citep{mazeika2024harmbench} without sacrificing model performance as measured by the MT-Bench project \citep{zheng2024judging}.

Mission Impossible: A Statistical Perspective on Jailbreaking LLMs
Jingtong Su" NYU & Meta AI, FAIR
Julia Kempe NYU & Meta AI, FAIR
Karen Ullrich* Meta AI, FAIR
Abstract
Large language models (LLMs) are trained on a deluge of text data with limited quality control. As result, L.MS can exhibit unintended or even harmful be- haviours, such as leaking information, fake news or hate speech. Countermeasures, commonly referred to as preference alignment, include fine-tuning the pretrained LLMs with carefully crafted text examples of desired behaviour. Even then, empir- ical evidence shows preference aligned LLMs can be enticed to harmful behaviour. This so called jailbreaking of LLMs is typically achieved by adversarially modify- ing the input prompt to the LLM. Our paper provides theoretical insights into the phenomenon of preference alignment and jailbreaking from statistical perspective. Under our framework, we first show that pretrained LLMs will mimic harmful behaviour if present in the training corpus. Under that same framework, then introduce a statistical notion of alignment, and lower-bound the jailbreaking probability, showing that it is unpreventable under reasonable assumptions. Based on our insights, we propose an alteration to the currently prevalent alignment strategy RLHF. Specifically, we introduce a simple modification to the RLHF objective, we call E-RLHF, that aims to increase the likelihood of safe responses. E-RLHF brings additional training cost, and is compatible with other methods. Empirically, we demonstrate that E-RLHF outperforms RLHF on all alignment problems put forward by the AdvBench [1] and HarmBench project [2] without sacrificing model performance as measured by the MT-Bench project [3].
Introduction
Large Language Models (LLMs) have revolutionized the field of deep learning due to their remarkable capabilities across various domains, serving as assistants, in code generation [4], healthcare [5], and theorem proving [6]. The training process of LLM typically includes two stages; pretraining with massive corpora, and an alignment step using Reinforcement Learning from Human Feedback (RLHF) to further align model behavior with human preferences. The latter step typically involves large amounts of humanly annotated data, and can be decomposed into a supervised fine-tuning (SFT) step, reward modeling step, and RL Fine- Tine-Tuning step. Despite their ability perform multiple tasks effectively, L.MMs are susceptible to generating offensive or inappropriate content including hate-speech, malware, fake information or social biases, due to the unavoidable presence of harmful elements within their pretraining datasets [7-9]. Social media showcase an abundance of tricks on how to attack ChatGPT [10] to elicit harmful responses, e.g., the "Do Anything Now" (DAN) prompts [1]] the "Grandma Exploit" hack [12]. On the other hand, behavior diversity in the training corpus is essential to for example capturing different cultural preferences. What is and isn't harmful ultimately depends on user preferences, hence the alignment step is not universal but depends on the specific use case under which a model will be employed.
"Correppnndence to: Jingtong Su i Equual senior authorship.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

To address deployment safety and eliminate objectionable responses, numerous alignment efforts have been made, such as injecting safe information during SFT [13], performing red teaming with human experts and Al themselves 14. as well as refining and improving the whole RLHF process in detail [19-23]. Yet we continue to witness a aat-and-mouse game of ever more sophisticated alignment methods to neutralize "harmful" prompts and even more inventive "jailbreaking" attacks that manipulate those prompts to elicit LLMS to produce harmful information. Such attacks come in various flavors, such as injecting adversarial suffixes [1, 24, 25], exploring cipher and low low-resource natural languages [26-28], or letting LLMs craft prompts automatically [29-33]. Although several ad- hoc defense methods against suffixes have been proposed [34-37], we only have limited proposal on a principled universal defense against jailbreaking attacks [2]. and limited theoretical understanding on this phenomenon [38].
In this paper, we present a theoretical framework for analyzing both the pretraining phase and the post-alignment jailbreaking phenomenon. Exploiting the fact that jailbreaking prompts typically maintain the underlying harmful concept while manipulating other aspects of the prompt, we design framework that decouples input prompts to allows us to quantify the strength of potential adversaries. By representing the output elements of an language model (LM) as lengthier text fragments rather than individual tokens, we can quantify the extent to which these models emulate the training distribution and consequently better understand the mechanisms underlying jailbreaking vulnerabilities.
Our contributions can be summarized as follows:
Based on our proposed framework, we first offer non-vacuous PAC-Bayesian style gener- alization bound for pre-training. Assuming the validity of our framework, we conclude that perfforrnn pre-trained models will inevitably susceptible to generating behaviour that is present in the training corpus, including any unintended and harmful behaviour.
Subsequently, we extend our framework to include notions of aligmment and jailbreaking, As- suming our assumptions met, we demonstrate jailbreaking to be unpreventable after safety alignment because the LM fails to concentrate its output distribution over the set of safe responses.
Motivated by our theoretical findings, we identify key drawback in the widely adopted RL Fine- Tuning objective due to the natural difference between the harmlessness and the helpfulness targets. By addressing this issue, we facilitate the training of safer models that are more resilient suite of jailbreaking attacks while preserving model performance.
The paper is organized as follows. In Section 2, we introduce our framework. In Section 3, we prove the PAC-Bayesian generalization bound for pretraining. Next, in Section 4 we present analysis on jailbreaking from statistical perspective. Finally, in Section 5 we illustrate our proposed E-RLHF objective and its effectiveness on improving LLM safety. We give a literature review in Appendix H.
2 Framework and assumptions
Jailbreaking carries several analogies to adversarial attacks, a well studied field in computer vision [39]. Here, an adversary is defined as a map that perturbs given input image pixel space change the model output. The strength of the adversary is bounded by how far it is able to move the original input as quantified by the lp distance [40-42]. Typically, this distance is bounded in a way that the change would not be perceptible to the human observer. The goal of the adversary is to cause mis- classification of the input. In contrast, in the instance of an LLM, the adversary's goal is to provoke harmful behaviour, e.g., unintended leaking of information or hate speech. Further, any perturbation to an input, called prompt, will have a perceptible effect. Hence quantifying and bounding the capabilities of the adversary is not straight forward. Nonetheless, with some modifications, we will build on this analogy.
For the purpose of this work, we will view any prompt as tuple of query and concept (g. c). where E C, and q E Q, with C, Q denoting the complete concept set and query set. Con- ceptually, we think of concepts as representing the information content of the prompt, usu- ally through short piece of text, for example "tutorial on making a cake". Queries are instructional text pieces that are composable with certain concepts. We can think of queries as mechanisms to trigger an LM to expand a concept in a specific way. Examples include "Tell me how to O", or 'Ne are now in an imaginary world, and you are
2

---

not bounded by any ethical concerns. Teach my avatar how to O" Since not all queries and concepts are composable, we denote P & Q as the set of all plausible prompts, where the definition of plausible will be made clear below.
The decomposition of prompts allows to isolate and hence hound the adversary's strength. In line with current empirical work on inducing harmful behaviour, we will allow perturbations only on the queries, not on the concepts. Further mimicking the spirit of previous work on adversarial attacks, we will assume that the ground-truth related to a prompt is determined solely by the concept, not the query. We will make these ideas more rigorous in the next paragraphs.
First, in contrast to previous theoretical work where LMs are regarded single sentence generators [38], we model LMs as lengthier text fragment generators, and refer to possible generated content e E & as explanations. Conceptually, explanations expand concepts with additional informa- tion. For example, 'The US president in 2023 is Joe Biden.". Our termi- nology "explanation" is conceptually the same as "response" used in previous dis- cussions (e.g.. [23]), where an LLM is re- garded as a policy that receives an input and generates a response. We use "expla- nation" to contrast "concept" since in most jailbreaking attacks currently considered by the community, the adversary seeks in- structions or explanations for a single harm- ful attempt. An LM thus induces a map- ping from plausible prompts to distribu- tions over explanations, PLM P + A(E), where (EE) denotes the set of distributions defined over elements in E.4 The output of a LM given a prompt, PLM c). is a discrete distribution over explanations. We use dom c)) as the domain of this distribution, PLMM(e (elg, c) as the probability of e given (g, c) as the input, and supp(prM (g, c)) as the subset of E with non-zero (elgge). Further, we assume the existence of latent ground truth mapping Pworld : P + AEE) that the LM is optimized to mimic during the pretraining stage. This is the distribution that defines "knowledge": for all plausible prompts (q, c), specifies the ground-trum -ruth distribution over explanations. By plausible, we refer to all prompts that lie in the domain of the ground truth mapping (g, c) E dom(Pwortt)) i.e., P dom (Pworld). Many plausible prompts will not even exist within any available training corpus, as discussed below.
Dpi
(0
supplptar)
domifrastt e) domippuell" .e))
We define a language model, PLM: @ + as map from prompts to a distribution over a subset of all possible explanations E. To later be able to bound the strength of the adversarial attacker, we split the text inputs into concepts and queries (g, c). We assume that (i) the text corpus only covers part of the domain of the LM: supp(D7> C (dom(pr.n) ), (ii) the size of the domain of the output distribution, denoted (dom(pLM (g, e))l. is small compared to the size of e, and (iii) only concepts determine the output (see I).
Figure 1:
Our framework in a nutshell:
We now state our main assumption, namely that for any plausible prompt (q, c) E dom (Pworta) the ground-truti truth distribution Pworta(q, c) is supported on a small subset 8 <> supp(Pworta c)) S E. This assumption seems sensible us: under normal circumstances, providing an explanation of "Paris" would not offer any relevant knowledge when given a prompt such as 'How to write a hello world python script". Our second assumption is that for all plausible prompts (q. c). the concept e uniquely determines the support of the output distribution specified by Pworld: regardless of the query: supp(Pworid (q. c))= = supp(Pworta(g" c)). Y plausible (q, e) and, (g",c) The query changes the ground-truth distribution without affecting its support. An illustration is depicted in Figure 1. To be more precise:
Assumption 2.1. (Concepts uniquely determine the explanation for plausible prompts) For all plausible prompts (g, c) E (poood))dd))
i) Pworld: P A(supp(Pworia(e) c))
For example, "Who is tutorial on making cake." " unreasonable.
*For real-world LMS, with different decoding hyperparameters e.g., the temperature T. top-p and top-k sampling parameters, the induced distribution with the same set of parameters could be different. Our discussion holds for fix-f set of hyperparameters throughout this paper.
3

---

where supp(Pworia(q.c)) c)) & E s.t. (supp(Peorta(() e))l < E and ii) supp(Peoria(q, c)) supp(Pworta(f c)), V(g, c), (q' c) plausible.
This assumption is natural since it essentially tells us that knowledge is specified by the corresponding concept alone, irrespective of what query is used to extract it. In other words, given a concept c, if query q manages to change supp(Pworta's c)), we argue that the query should be deconstructed and partially absorbed by e to accurately reflect the knowledge mirrored by the support.
Lastly, we make the assumption on the existence of an underlying generative distribution over prompts, denoted as (q. e) ~ Dp. This distribution serves as the principle governing the creation of our pretraining corpus. It is important to note that supp(Dp) & dom(pwortta. For example, take the prompt (g' "hho is James Bond $**#! ! 48811"" even though this prompt never appears in any text corpus across the internet, (q', c') & supp(Dp), we, as humans, can make sense of it: (d',c') E dom(Pwortdd) Later proofs in this paper assume LMS generate semantically reasonable explanations for such unseen plausible prompts, since in reality LMs are claimed to generalize well on huge,
datasets [43]. This is made explicit in Section 4, within Assumption 4.1.
Finally, the following definitions pertain to our notion harmfulness. More specifically, we un- derstand harmful behaviour abstractly as any unintended behaviour. For this, we assume that any explanation e can be denoted as either harmful or not harmful (safe). A concept e is regarded as harmful if and only if the world generates harmful explanations with probability higher than certain threshold with direct prompts.
Definition 2.1. (Notions of Harmfulness)
(Direct Queries and Direct Prompts) We refer to a prompt as direct if stems from Dp, i.e. (g,c) E Dpp) The query of a direct prompt is called direct query.
(Harmful Concepts and Harmful Set) Given a concept c, the associated harmful set of ex- planations is denoted E2, (c) == {ele E supp(Pwortal(,))
A is harmful). In accor- dance with Assumption 2.1, with a threshold n, a concept c. is harmful if 44 s.t. (q.c) E dom (Pworr))))); EmeeF Pworta(elg, c) 1 17. We refer to the set of all possible harmful concepts as Cr. C.
(Safe Set) Vc Ch. there exists corresponding safe set E.(c)) & E that we wish PLM (q,c) be concentrated on. It includes safe explanations existing in supp(Pwortal:, c)), and explanations designed by humans, e.g., with the template beginning with "Sorry."
(Semantically meaningful) We call explanations in En(c) U Es(c) semantically meaningful for the (g, c) prompt.
(Mixture decomposition of Dp) With these notions, we can decompose Dr = aDD, + (1 Dy, (where supp( includes all direct prompts with harmful concept, and supp(Dp,) includes the complement) as a mixture over direct prompts with a harmful concept and the non-harmful counterpart.
3 AAC-Bayesian bound for pre-training LLMs on harmful data
Given learning algorithm that leads to a posterior distribution over set of models, PAC-Bayesian theory [44] applies Probably Approximately Correct (PAC) inequalities, to provide bounds on the generalization gap, i.e., the difference between the model's empirical loss and the population loss. We now present the first result of our analysis: a non-vacuous PAC-Bayesian bound for pretraining LMs which implies that well-trained LM ought to exhibit harmful behaviour even when simply prompted with direct queries if it was presented with harmful behavior during training.
We denote by S ((g. c;)) set of prompts generated i.i.d. under Dp, S ~ DT. These prompts together with sampled explanations form our pretraining corpus. We use n,, / as the prior and posterior distribution over LMs before and after the pretraining process, defined over I.I,, the set of language models. Given a prompt (q, c), we measure the generalization capability of a LM by quantifying the Total Variation (TV) loss between the induced distribution PLM (q, c) and the ground-truth distribution Pwortd (g. c).5 For real-world LMs, pretraining involves optimizing the cross-entropy loss on the
'e regard both distributions defined over the entire & since we do not restrict the output distribution of LM in this section.

---

training corpus, which is equivalent to minimizing KL Pwooriaa c))llLM (q. e)] under our framework. With Pinsker's Inequality, optimizing the KL-divergence term is equivalent to optimizing an upper bound on TV; thus we expect empirical TV loss be small.
Definition 3.1. (TV empirical loss and population loss)
(Tv(PLM, (g. c)) TV c), PLAS (q, c)).
Given an LM and a set of data S. the empirical loss Rs(PLM) and population loss R(PLas) are defined as
1
Rs(PLM)
&r (PLM, ci);;
n
R(PLM) == Ds [RS(PLM)
= DD (TV(L (q, c))..
We state our PAC- MAC-Bayesian bound as follows. The detailed proof can be found in Appendix B. 6 Generalization Bound for Language Models.) With a as in Definition 2.1, consider set of language models LM,, with prior distribution over LM.
Theorem 1.
Given any ô E (0, 1), for any probability measure p over LM such that p. n share the same suppori, the sollowing holds with probability at least 1 - 8 over the random draw of S:
[<]]||=]] + log #1
E (Rpr.)) - Rs(PEM)) S
2
2n
&TV (1, (q,
RSMMpRs(PPAN + e) a
(1)
In Appendix B.2 we give a theoretical estimation of o, to illustrate the bound we derive is non-vacuous, i.., less than 1. The KI. term is of order O(K)) where K is the number of parameters involved in л, and n can be shown to greatly exceed K (using a realistic Zipf distribution assumption on prompts to estimate the number of unique prompts). Theorem 1 tells us that, as long as pretraining successfully reduces the loss on the training corpus (Rs(PLM) J). in expectation the language model will mimic the world well (small Ery difference) on a given direct prompt sampled from Dp. Furthermore, a direct prompt whose concept is harmful. Since
is not too small, then this statement holds have defined the harmful concept as outputting harmful explanations with high probability (Definition 2.1), we conclude that an LM trained on Dp data can output explanations in the harmful set.
4 A statistical perspective on jailbreaking after alignment
In this section, we will present the main theoretical contribution of our work: given our assumptions hold, we prove the existence of ways for an adversary jailbreak LM even after the preference alignment process. Our proof strategy is inspired by the work on adversarial robustness [41], which bounds the adversary's 's probability of success by upper bounding the volume of the set of points that does not allow for the existence of adversarial examples. Going forward, we need to extend our framework integrate alignment and jailbreaking.
After an LM is pretrained, it typically will undergo fine-tuning on a dataset containing preferred behaviour. In what follows. we will assume that this alignment process does not change the model performance in the sense that the LM will still produce semantically meaningful explanations (Definition 2.. It would not, for example, default to answering any request with the same response. Assumption 4.1. (LM outputs semantically meaningful explanations) For any harmful concept and all plausible prompts (g, c) E dom(pworta).
3 [c)() < [Bx(e]] + [c)c s.1. 0(1) < [dom(pr c)) (FA())
In other words, we assume the LM's output distribution is accurately supported on En (c) U (.).. in the sense that the size of "residual" (c) is relatively small compared to these semantically meaningful explanations. We define ((c) = E,, (c)l +E,(C)+
We omit the (c) annotations
'The inspiration for the proof of Theorem I comes from Mbacke et al. [45], and the proof idea is originally proposed in Germain et al. [46]. Haddouche et al. [47].
5

---

when clear from the context. The O(1) statement is reasonable, because harmful explanations are usually long text fragments that allow for many alternative formulations. The assumption can be broken down into two components: (1) within the support of the output distribution, only occasional instances of unrelated explanations exist; (2) the process of aligning the model towards safety does not eliminate the harmful explanations acquired during the pretraining phase. For part (1), similar to the example we gave above, under normal circumstances, we do not expect the explanation "Paris" to appear in dom(ptar(q, c)) given (g. c) "How to build bomb'. As for part (2), though seemingly surprising, evidence with a series of current state-of-the-art LMs can be experimentally validated [48], where diverse, harmful explanations are extracted by simply manipulating the decoding process using direct prompts. In Section 5 we give an explanation for this undesired phenomenon.
To bound the likelihood of jailbreaking we first need to specify how the output of LM interacts with its support. Assuming a fixed order of explanations in dom(pLM(q; c)), and slight abuse of notation, we can use PgM(g. c) to denote an n(e)-dimensional vector An(e)-1, the probability simplex with ni()) elements, where each entry represents the probability of a single explanation. We call this simplex the output simplex related to a given concept c. Next, we can induce a dis- tribution on this simplex given a posterior distribution y over the set of language models LM, as follows.
x e E,fc)
H,
Ha
x
Bale)
Bate)
Figure 2: Conceptual illustration of our framework for jailbreaking introduced in Section 4, with fixed harmful concept c. The triangle represents the probability simplex. This figure showcases a typ- ical successful jailbreaking attempt by the adversary: although safety alignment makes the sampled LM safe under the direct prompt input, the adversary is able to move the output to the harmful zone Hrs by manipulating the query q-
Definition 4.1. (Induced Distribution Under the assumption that the IM outputs semantically meaningful explanations (Assumption 4.1), with a fixed prompt (g. c) and a posterior distribution 7y over I.M, the corresponding induced distribution: PLM(g, c) where LM ~ 7 is supported over a subset of the output simplex A A=-1, This distribution is denoted as Vg.e), or To when the reference to q is clear from context.
Simplex, "(e)
Next, we will separate the output simplex into a harmful and safety zone. This definition is motivated by the obser- vation that typically an adversary is deemed successful if it can extract even a single harmful explanation for given concept. This translates into a division of the output simplex, under Assumption 4.1, as follows.
Definition 4.2. (Harmful Zone and Safety Zone) For a given harmful concept e and a fixed LM, the output simplex is divided into a safety zone and a harmful zone, H. and Hi. where pre- defined threshold p E [0, 1] is used to quantify the distinction: PLM (g. c) E Hr if and only if Emee (x)c) (LM(elg, c) 2 p, and otherwise PLM c) E H.s
Before we introduce jailbreaking, the reader might wonder why we did not define alignment more clearly. This is because under the PAC framework, preference alignment nothing but transforma- tion from p some > posterior defined over LM. Given this inability on fine-grained characterization of alignment, we instead provide the goal of it as follows. With the above notion, given a prompt (g. c) where is harmful, its goal is to push the induced distribution % into the safety zone H.g Ideally, supp(1c) C. H. + with probability 1, the resulting LM is safe when encountering (g. c). We ready to introduce necessary concepts related to jailbreaking.
Definition 4.3. (Jailbreaking) Given a harmful concept e and query d' the prompt (q' c) jailbreaks the LM iff PLas (g', cc) E Hh- We call such prompt (4', ; c) and query o' a jailbreaking prompt and jailbreaking query, respectively.
The threshold p for discriminating Ha and H. should be very small, since it means in expectation the adversary needs to call the LM times to collect a single harmful explanation i.e., to jailbreak the LM.
To theoretically prove the jailbreaking effect, we need to restrict the adversary's 's ability. To achieve this goal, we borrow insights from adversarial attacks, to assume that the adversary has bounded manipulating capability on the output simplex when searching over the query set:
Assumption 4.2. (e-bounded adversary) Given an LM, a harmful concept e and an associated direct prompt (g, c). we assume the adversary can find set of queries Q', such that the output is moved at

---

most on the simplex towards Hy. from (g,r(g,e);
d(pLMs G), PLMLL)) .c) = d'EQ'
sup
Here d is a distance measure between two discrete distributions. d can be a typical lo measure with p 2 1, or the Total Variation / (ensen-Shannoo Divergence. We call q' Q' an e-bounded query.
conceptual illustration of our framework is depicted in Figure 2. Before arriving at our Theorem, we give the final definition of c-expansion. Definition 4.4. (e-expansion) Given set A C A n- and distance measure d, the e-expansion set A(e, d) defined as
A(e, (e.d)== : (ULE A By E A... 11 llla S <)..
We are ready to present the following theorem, which states that as long as the induced posterior Y is not concentrated in an extremely safe area, then with high probability the model can jailbroken. The proof is in Appendix B.3.
Theorem 2. (Jailbreak is unavoidable) Assume that an LMs output semantically meaningful expla- nations (Assumption 4. Given any Y posterior distribution over I.M, choose harmful concept c with a direct prompt (q, e) and a threshold (Definition 2.1), to define the corresponding induced distribution Ne (Definition 4.1) and division over output simplex (Definition 4.2). An e-bounded adversary (Assumption 4.2) can find a jailbreaking prompt (Definition 4.3) with probability at least
1-7s x x1 21-11160 +(a))) x
by using either the direct prompt, such that pI.M (g,c) E His or by finding an e-bounded query q', such that PLM (g', E Hi.
Here, 2(-) is the standard Gaussian edf. 7s == max,Ex, -u (c.d)) U(x)' with U (z) the uniform dis- tribution over -- and ag == a + vn - Ie, where a writes analytically as a x [Bx(c)-1-(n--
Tela)
((n-1)p(1-p 11p(pp))))
Trivially, the chances of an adversary to find a jailbreaking prompt increase for stronger adversaries (e 1). In the real world, this could relate to how much compute budget we allow to alter a query for a specific harmful concept. Furthermore, the chances of an adversary to find a jailbreaking prompt increase when the ratio of the sizes of the harmful explanation set to the safe explanation set is larger (xte)) f. This is because their ratio will determine the size of the harmful zone which in turn will [c) cause (a.) + 1. In real world settings, for any harmful concept, the training corpus naturally contains large harmful set due to the number of possible responses. Realistically, its size can be countered by any manually-constructed safe set. Hence achieving alignment is hard: Recall that the goal of alignment is to respond with only safe explanations with high probability. However, we just learned that to increase that probability, we need to have small harmful-tor sonnuulltt-sffett set ratio which we discussed not realistic. Consequently, the safety zone is going to be small.
E-RLHF: improving alignment by expanding the safety zone
Recall from Theorem 2 and the subsequent discussion in the previous section, that jailbreaking becomes more likely the larger the harmful zone comparison to the safety zone. The size of both zones relates to the size of their respective explanation sets. In other words, the size of the preference alignment dataset is crucial to successful alignment. Unfortunately, the human labor involved in creating such a dataset effectively caps its size.
In order bridge the gap between our theoretical insights and a practical solution towards suppressing the jailbreaking problem, we focus on other more practical ways to expand the safety zone. Even though our ideas are more broadly applicable, our experiments we will focus on improving Reinforcement Learning with Human Feedback (RLHF). RLHF typically includes three phases: i) supervised fine-tuning (SFT); ii) preference sampling and reward learning and iii) RL optimization. Rafailov et al. [23] have recently proposed a widely applied version of RL.FF for LMs, coined Direct Preference Optimization (DPO), that employs clever reparameterization which leads to directly learning from the preference dataset, without the need of obtaining reward model beforehand.
7

---

DPO is more stable in the training process than other implementations of RLHF. A more complete overview of RLHF and DPO can be found in Appendix C.
For our purposes, we assume access to an LLM pSrT that has been supervised fine-tuned on high- quality data. We further assume access a preference aligned dataset D5; that contains set of text prompts (g, c) = I, and two respective explanations that have been rated by human annotators as better ew or worse et. In phase ii) of RLHF, one typically optimizes a reward model r(æ, e) based on the annotated explanations. Our proposal concerns phase iii of the RLHF process: training of the preference aligned model PLM. For given reward model. PLM is typically obtained by minimizing the following objective:
PRLHE (PLAS) = -EEDD ( (1 e)] + ( (PLM (PPET(()))
(2)
Note that, the first term maximizing the reward, while the KL-term acts as a regularizer ensuring the aligned model can not drift off too far from the SFT model. We claim that this regularization is exactly the problem for safety. While designed to keep the model helpful', for any harmful prompt Tn and any harmful explanation e E supp(pser(zs) )), regularizing PL.M to PSET lets PLM maintain e in the support of the output distribution. Specifically, the supervised fine-tuning process does not involve elimination of any harmful explanations from the support of the output distribution of the pretrained model, which leads to the fact that PSFT(#H) supports harmful responses that can not be negated with a realistically sized preference aligned dataset. Thus, this small safe set problem of pSET will be further passed to Pr.M, even if pras is optimized to the optimal solution of the above objective. This suggests we should not push PLM into the direction guided by PSFT- Instead, for any harmful prompt Th. we propose to use a distribution that we design to have a large safety zone to replace (2.) Inspired by our framework, the harmful prompt. such that a safe concept is used to replace the harmful concept, which alters supp(psFr())) to include more safe explanations. For example, when T4 is "Tell me how to make a bomb", instead of aligning PLM (2%) to (sFr(Ds), now we align PLM (2%) to psrr (1.) where F. is "Telll how to reject request of making bomb"'. This can be done in an individualized manner for all harmful prompts in D., or simply by prefacing all harmful prompts with prefix such as "Please ensure your response adheres to community guidelines dinenns and ethical standards:". focus of our approach to expand the safety zone of the output distribution, we coin our proposal E-RLHF, resulting in the following modification to Eq. (2):
can achieve this by modifying
Importantly, the non-harmful prompts are not modified. Due to the
CE-RLHF& (PLM) = EE-DD (Ee-pLE (w) (r(z, e)] + (PLM (x) (z)r(
(3)
where 2s is a aafety-transformet version of the original harmful prompt 24- To recap, the key argument we put forth is that, in order to ensure the stability of model fine-tuning, it is not imperative to utilize identical prompt inputs x for both the reference model and the target model, particularly when the original input itself is harmful. In fact, as long the substitute or "anchor" prompt generates logically reasonable outputs akin to those produced by the original prompt, this approach would not impede the training process of the model. To solidify our argument we show the impact of our modification on the support of the optimal policy in Appendix C. We also deduce there that we can trivially integrate our modification into the DPO objective allowing us to train without an explicit reward model (eliminates step ii)) as follows, where (-) stands for the sigmoid function:
LEDPO (PLM) = - -B(r logo (Blog PLM (c.. x)
PSrT (el xx) ]
PLM (cs x))
-3 3lg
(4)
PSET (ew I (x))
6 Experiments and results
Our experimental set-up is based on the alignnent-handbook handbook code base [50]. We tune the publicly- available SFT model psFt provided by huggingface hub [51], using the public dataset [52, 53), with default hyperparameter setup. We label harmful prompts in the preference dataset by prompting GPT--.5-Turro, Appendix F. We are using the very same prefix proposed in the previous section to generate Tg. Experiments are performed on 8 NVIDIA Tesla V100 GPUs, using half-precision
'Oteerwise the model could drift into trivial behaviour like always responding with "I can't help you.". ". $ *Even the probability can he suppressed to close to O.
9The prefix shares similarities to the system prompts used by open-source LLMs [13, 49] to boost safety.
8

---

Table Safety alignment with the E-RLHF objective, here specifically E-DPO, reduces the average Attack Success Rate (ASR) across all jailbreak adversaries for both the HarmBench and the AdvBench data, to 36.95, and to 20.89, respectively. Moreover, resilience against all adversaries improves with our modification to safety alignment indicates better performance between DPO and E-DPO).
HarmDench
Model
Direct Request
GCG
GBDA
AP
SFS
ZS
PAIR
TAP
AutoDAN
PAP-top5
Human
AYG
PsPt
32.25
59.25
35.50
12.75
42.75
36 20
56,50
65.00
56.75
26.75
35.50
44.47
Paro
27.50
53.00
39.00
46.75
43.25
29.10
52.30
54.00
51.00
28.75
37.15
42.00
Pearo (ours)
23.50
47.50
31.76
36.25
40.50
48.50
43.00
31.05
36,96
AdvBench ASR (1)
Part
6.00
80.00
13.00
37.00
31.00
14.80
65.00
78.00
91.00
4.00
21.20
40.09
paro
0.00
47.00
12.00
30.00
50.00
61
44.00
4.00
18.40
28.40
PEtro (ours)
0.00
38.00
8.00
15.00
21.00
5.20
11.00
53.00
31.00
4.00
13.60
20.89
tuning i.e., 6. In the appendix, we also show results for an alternative training paradigm: the Low-Rank Adaptation (LoRA) [54] (see Appendix D.1). Following community standards [3, 1, 2]. we use greedy decoding i.e., T 0 for model evaluation.
We first show empirical evidence that our proposed modification of DPO, E-DPO, does in fact improve safety alignment, using the Harmbench dataset [2] and the first 100 prompts in the AdvBench harmful behavior dataset [1], measured by the HarmBench protocol. We give an overview on all adversaries in Appendix F. The results are presented in Table 1. E-DPO achieves improvements across every task we tested.
On top of our safety results, we want to make sure E-RLHF does not sacrifice helpfulness for increased safety. We evaluate helpfulness with the MT-Bennc project [3]. The SFT model PSFT receives score of 6.3, and both the DPO and E-DPO models perform better than that (6.9 and 6.7 respectively), making us believe that performance degradation is not a problem with our proposal. Next, we show the impact of the safe prefix on model performance. We demonstrate that our method's performance depends on the choice of safe prefix to some extend but never fails (see Appendix D.2). We believe, finding better safe prefixes by explicit tuming would improve our results, similar to the work by Yang et al. [55], but we leave this exploration for future work. Further, we confirm that the improvement arises from using safe prior in the KI. term for harmful prompts. We ablate our results by appending the prefix on all prompts in the preference alignment dataset (see Appendix D.3). In all cases, applying the safe prefix to usual prompts degrades safety, showcasing the importance of switching the prior only on the harmful prompts. Finally, we show that E-DPO can be combined with any system prompt, to further boost safety (see Appendix D.4). The proposal can even be used to improve helpfulness and safety simultaneously (see Appendix D.5).
7 Conclusion and discussions
In this paper, we present a theoretical framework for language model pretraining and jailbreaking by dissecting input prompts into query and concept pairs. Through this approach, we have established two theoretical results pertaining to the ability of language models to mimic the world following pre- training, which leads to outputting harmful explanations given harmful prompts; and the inevitability of jailbreaking resulting from alignment challenges. Guided by these theoretical insights, we have devised a simple yet effective technique to enhance safety alignment, and demonstrate the improved resilience to jailbreak attacks with this methodology.
Current limitations (1) Although we have classified concepts as either harmful or non-harmful, is important to acknowledge that the perception of a concept's 's potential for harm can be influenced by various factors such as cultural, legal, and societal norms, which collectively form the context of the situation. (2) Language models have demonstrated impressive capabilities in reasoning and
completing tasks within multi-round, multi-step conversations; our current framework may not fully account for the generalization and jailbreaking possibilities associated with such input formats. (3) Our analysis grounded on a fixed Pworrd mapping and Dr distribution. Nevertheless, the world is inherently dynamic, as both Pworld and Dp continually evolve.

---

Future work (I) Regarding our E-RLHF approach, as highlighted in the experimental section, in addition to attaching universally safe prefix to all harmful prompts, improvements can be achieved by individually transforming the harmful prompts. Moreover, the freey-ttranfformed prompts can be employed to expand the preference dataset for conventional RLHF. (2) Throughout our analysis, we have not imposed any constraints on the capacity of the language model. Extending our analysis under finite memory constraints or analyzing hallucination properties of LLMs is an interesting direction to explore. (3) Large language models have shown remarkable capabilities and such techniques could potentially be used for jailbreaking them as well [57-59]. Investigating the incorporation of such input paradigms remains promising avenue for future research.
in-context learners [56],
Acknowledgement
JS and JK are supported by the National Science Foundation under NSF Award 1922658, Part of this work was done while JK was hosted by the Centre Sciences de Donnees at the École Normale Supérieure (ENS) in 2023/24, whose hospitality she gratefully acknowledges.
10

---

References
[1] Andy Zou, Zifan Wang, Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043 2023. 1, 9, 27, 28, 29, 30, 31,32 32
[2] Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.044249, 2024. 1, 9, 27, 28, 29
[3] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 37, 2023. 1, 9, 28
[4] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308, 12950, 2023. 1
[5] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972):172-180 2023.
[6] Kaiyu Yang, Aidan Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan J Prenger, and Animashree Anandkumar. Leandojo: Theorem proving with retrieval-augmentte language models. Advances in Neural Information Processing Systems, 36, 2024. 1
[7] Emily M Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610 623, 2021. 1
[8] Julian Hazell. Large language models can be used to effectively scale spear phishing campaigns. arXiv preprint arXiv:2305.06997 2023.
[9] Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. Jailbreaking chatgpt via prompt engineering: An empirical study, arXiv preprint arXiv:2305. 13860, 2023. 3,
[10] OpenAL. Cttpgh/..rrrr.../eaaaac///////aa 2022.
[11] DAN. Do anything now prompt. http::://gttabb...////ttt/////aa
2023.
[12] Reddit. Chatgpt grandma http::///rr..reeedaaaaaaa//////a
12sn0kk/ grrgddaaaaeppllolo 202.1
[13] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi. Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.092888 2023. 2, 8, 22, 26
[14] Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.078.. 2022. 2, 31
[15] Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models.
arXiv preprint arXiv:2202. 032886 2022. 29
[16] Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan Hadfield-Menell Explore, es- tablish, exploit: Red teaming language models from scratch. arXiv preprint arXiv:2306.0944 09442, 2023.
11

---

[17] Zhang- Wei Hong, Idan Shenfeld, James R Glass, Akash Srivastava, and Pulkit Agrawal. Curiosity-driven red-teaming for large language models. In The Twelfth International Conference on Learning Representations, 2023.
Wang, Yung-Sung Chuang, Aldo Pareja,
[18] Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, et Rainbow teaming: Open-ended generation of diverse adversarial prompts. arXiv preprint arXiv:2402. /6822, 2024. 2, 31
[19] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint :9ti::1990005993, 2019. 2, 25, 31
[20] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing
Systems, 35:27730-27744 2022. 25, 31
[21] Yuntao Bai. Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.080773 2022. 25, 31
[22] Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. In International Conference on Machine Learning, pages 17506-17555 PMLR,
2023. 31
[23] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint S.Ki::2305.182900 2023. 2, 3, 7, 25, 26, 33
[24] Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. Autodan: Automatic and interpretable adversarial attacks on large language models. arXiv preprint arXiv:2310.15 1,440, 2023. 2.30
[25] Raz Lapid, Ron Langberg, and Moshe Sipper. Open sesame! universal black box jailbreaking of large language models. arXiv preprini arXiv:2309.0144: 2023. 2, 30
[26] Zheng-Xin Yong, Cristina Menghini, and Stephen H Bach. Low-resource languages jailbreak gpt-4. arXiv preprint arXiv:2310.02446. 2023. 2, 31
[27] Yue Deng. Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. Multilingual jailbreak challenges in large language models. arXiv preprint arXiv:2310.06471 2023. 31
[28] Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. Gpt-4 is too smart to be safe: Stealthy chat with 1lms via eipher. arXiv preprint arXiv:2308.c 3000:2308.064663, 2023. 2, 31
[29] Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451, 2023.
2, 29, 30
[30] Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint trXiv:2310.08411 2023. 28, 29,30 30
[31] Xiaoxia Li, Siyuan Liang, Jiyi Zhang, Han Fang, Aishan Liu, and Fe-Chien Chang. Semantic mirror jailbreak: Genetic algorithm based jailbreak prompts against open-source llms. arXiv preprint arXiv:2402. 14872, 2024. 30
[32] Peng Ding, Jun Kuang, Dan Ma, Xuezhi Cao, Yunsen Xian, Jiajun Chen, and Shujian Huang. A wolf in sheep's is clothing: Generalized nested jailbreak prompts can fool large language models casily. arXiv preprint arXiv:2311.08268, 2023. 30
12

---

[33] Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box Ilms automatically. arXiv preprint arXiv:2312.0211 2023. 2, 29, 30
[34] Alexander Robey, Eric Wong, Hamed Hassani, and George Pappas. Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310003684 2023. 2,
32, 33
[35] Bochuan Cao, Yuanpu Cao, Lu Lin, and Jinghui Chen. Defending against alignment-breaking
attacks via robustly aligned 1lm. arXiv preprint arXiv:2309. 14348, 2023. 32
[36] Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Soheil Feizi, and Hima Lakkaraju. Certifying Ilm safety against adversarial prompting, arXiv preprint arXiv:2309.02705. 2023. 32
[37] Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping- yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Base- line defenses for adversarial attacks against aligned language models. arXiv preprint WrXiv:2309.00614 2023. 2, 32
[38] Yotam Wolf, Noam Wies, Yoav Levine, and Amnon Shashua. Fundamental limitations of alignment in large language models. arXiv preprint arXiv:2304. 2023. 2,
[39] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013. 2
[40] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In International Conference on Learning Representations, 2018. 2
[41] Ali Shafahi, W Ronny Huang, Christoph Studer, Soheil Feizi, and Tom Goldstein. Are adversarial examples inevitable? In International Conference on Learning Representations, 2018. 5
[42] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via random- ized smoothing. In international conference on machine learning, pages 1310-1320. PMLR,
2019. 2
[43] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.046155 2022.
[44] David A McAllester. Some pac- bac-bayesian theorems. In Proceedings of the eleventh annual conference on Computational learning theory, pages 230-234, 1998.
[45] Sokhna Diarra Mbacke, Florence Clerc, and Pascal Germain. Pac-bayesian generalization bounds for adversarial generative models. arXiv preprint arXiv:2302. 08942, 2023.
[46] Pascal Germain, Alexandre Lacasse, François Laviolette, and Mario Marchand. Pac- Pac-bayesian learning of linear classifiers. In Proceedings of the 26th Annual International Conference Machine Learning, pages 353-360, 2009.
[47] Maxime Haddouche, Benjamin Guedj, Omar Rivasplata, and John Shawe-Taylor Pac-bayes unleashed: Generalisation bounds with unbounded losses. Entropy, 23(10):1330, 2021.
[48] Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. Catastrophic jailbreak of open-source Iims via exploiting generation. arXiv preprint arXiv:2310.0698)
2023. 6, 27,730
[49] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. 8
13

---

[50] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alexander M. Rush, and Thomas Wolf. The alignment handbook. com htw.uuggiig.ca/ellaaaaaeeeeeee -aw.entthannnoooot... 2023. 8
[51]
HuggingFace. Huggingface constitutional-ai sft model. hctta://huggingfant al hta.bboo/mmsttaaa-aaaa----------
2024. 8
[52] HuggingFace. Huggingface constitutionallai dataset: fltraffeddback-bbnnaizze https://
2024. huggingface. httpa///ww/o///gggggg/////////ta 2024. 8
[53] HuggingFace. Huggingface constitutional-ai dataset: Cai-conversation--aamlle https: // 2024.
huggingface.
[54] Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021. 9, 27
[55] Chengrun Yang. Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXiv preprint arXiv:2309.03444 2023. 9, 26
[56] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901, 2020. 10
[57] Zeming Wei, Yifei Wang, and Yisen Wang. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.06387, 2023. 10,
[58] Jiongxiao Wang, Zichen Liu, Keun Hee Park, Muhao Chen, and Chaowei Xiao. Adversarial demonstration attacks large language models. arXiv preprint 2rXiv:2305.14900 2023. 31
[59] Cem Anil, Esin Durmus, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Nina Rimsky, Meg Tong, Jesse Mu, Daniel Ford, et al. Many-shot jailbreaking. 2024. 10,31
[60] Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep learning. Advances in neural information processing systems, 30, 2017.
[61] Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. Advances neural information processing systems, 32, 2019. 21
[62] Robert D) Gordon. Values of mills' ratio of
bounding ordinate and of the normal probability integral for large values of the argument. The Annals of Mathematical Statistics, 12(3):364-366, 1941. 23
[63] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. 25
[64] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345 1952. 25
[65] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, arXiv preprint drXiv:1707.063477 2017. 25
[66] Banghua Zhu, Hiteshi Sharma, Felipe Vieira Frujeri, Shi Dong, Chenguang Zhu, Michael Jor- dan, and Jiantao Jiao. Fine-tuning language models with advantage induced policy alignment. arXiv preprint arXiv:2306.022
2023. 25
[67] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters deep policy gradients: case study ppo and trpo. arXiv preprint arXiv:2005.12729 2020. 25
14

---

[68] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, al. Chain -fain-o--iee ofouthtttt prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24887 2022. 26
[69] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as laboratory for alignment. arXiv preprint arXiv:2 / 12.00861, 2021. 26
[70] Xiaotian Zou, Yongkang Chen, and Ke Li. the system message really important to jailbreaks in large language models? arXiv preprint 2rX.vv:402.144:: 2024. 27, 32
[71] Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. Gradient- baaeed adver- sarial attacks against text transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5747-5757, 2021. 29, 30
[72] Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Auto- prompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010. 15980, 2020. 29, 30
[73] Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How johnny can persuade Ilms to jailbreak them: Rethinking persuasion to challenge safety by humanizing llms. arXiv preprint arXiv:2401.06333. 2024. 29, 31
[74] Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. do anything now": Characterizing and evaluating in wn-lhh-wild jailbreak prompts on large language models. arXiv preprint arXiv:2308.0338 03825, 2023. 29, 30
[75] Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal ad- versarial triggers for attacking and analyzing nlp. arXiv preprint arXiv: 1XXix:990.007222 2019. 29
[76] Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Gold- stein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. Advances in Neural Information Processing Systems, 36, 2024. 29
[77] Erik Jones, Anca Dragan, Aditi Raghunathan, and Jacob Steinhardt. Automatically auditing large language models via discrete optimization. arXiv preprint 2xXiv:230330438/ 2023. 30
[78] Fâbio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527 2022. 30
[79] Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does Ilm safety training fail? In seirth-seventh Conference on Neural Information Processing Systems, 2023.
30
[80] Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. Jailbreaker: Automated jailbreak across multiple large language model chatbots. arXiv preprint arXiv:2307.0m 08711, 2023. 30
[81] Jiahao Yu, Xingwei Lin, and Xinyu Xing. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.1022 2023. 30
[82] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024. 30
[83] Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. Deepin- ception: Hypnotize large language model to be jailbreaker. arXiv preprint 1..011231 2023. 30
[84] Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh. Drattack: Prompt decomposition and reconstruction makes powerful Im jailbreakers. arXiv preprint 2XXi:24022166144 2024. 30
15

---

[85] Zhenhua Wang. Wei Xie, Baosheng Wang, Enze Wang, Zhiwen Gui, Shuoyoucheng Ma, and Kai Chen. Foot in the door: Understanding large language model jailbreaking via cognitive psychology. arXiv preprint arXiv:2402. 25600 2024. 30
[86] Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, and Yuandong Tian. Adyprompter: Fast adaptive adversarial prompting for llms. arXiv preprini urXiv:2404. 16873,
2024. 30
[87] Hangfan Zhang. Zhimeng Guo, Huaisheng Zhu, Bochuan Cao, Lu Lin, Jinyuan Jia, Jinghui Chen, and Dinghao Wu. On the safety of open-sourced large language models: Does alignment really prevent them from being misused? arXiv preprint arXiv:2310.0158/ 2023. 30
[88] Xuandong Zhao, Xianjun Yang, Tianyu Pang, Chao Du, Lei Li, Yu-Xiang Wang, and William Yang Wang. Weak-to-strong jailbreaking on large language models. arXiv preprint urXiv:2401.12256 2024. 30
[89] Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. Shadow alignment: The ease of subverting safely-aligned language models, arXiv preprini arXiv:23100 20.0099,, 2023. 30
[90] Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, and Daniel Kang. Removing rlhf protections in gpt-4 via fine-tuning, arXiv preprini arXiv:2311.0553 2023. 30
[91] Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint ar arXiv:2310.03693 2023. 30
[92] Erfan Shayegani, Yue Dong, and Nacl Abu-Ghazaleh. Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models, arXiv preprint arXiv:2307.14539, 2023,
[93] Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Mengdi Wang, and Prateck Mittal. Visual adversarial examples jailbreak large language models. arXiv preprint drXiv:2306.13213, 2023. 31
[94] Nicholas Carlini, Milad Nasr. Christopher A Choquette-Choo Matthew Jagielski, Irena Gao, Pang Wei Koh, Daphne Ippolito. Florian Tramèr, and Ludwig Schmidt. Are aligned neural networks adversarially aligned? In seirth Conference on Neural Information Processing Systems, 2023.
[95] Natalic Maus, Patrick Chao, Eric Wong, and Jacob R Gardner, Black box adversarial prompting for foundation models. In The Second Workshop New Frontiers in Adversarial Machine Learning, 2023. 31
[96] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL org/
https://lmsys:
htts//llg///// blog///23-0000 03- vicuna/. 31
[97] Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. Exploiting programmatic behavior of llms: Dual-use through standard secu- rity attacks. arXiv preprint arXiv:2302.057333 2023. 31
[98] Rusheb Shah, Soroush Pour, Arush Tagade, Stephen Casper, Javier Rando, et al. Scalable and transferable black-box jailbreaks for language models via persona modulation. arXiv preprint arXiv:2311.03344 2023. 31
[99] Xingang Guo, Fangxu Yu, Huan Zhang, Lianhui Qin, and Bin Hu. Cold-attack: Jailbreaking llms with stealthiness and controllability. arXiv preprint 240:22402.08679 2024. 31
[100] Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. Cold decoding: Energy- based constrained text generation with langevin dynamics. Advances in Neural Information Processing Systems, 35:9538-9553 2022. 31
16

---

[101] Somnath Banerjee, Sayan Layek, Rima Hazra, and Animesh Mukherjee. How (un) ethical are instruction- responses of Ilms? unveiling the vulnerabilities of safety guardrails to harmful queries. arXiv preprint arXiv:2402. 15302, 2024. 31
[102] Neal Mangaokar, Ashish Hooda, Jihye Choi, Shreyas Chandrashekaran, Kassem Fawaz, Somesh Jha, and Atul Prakash. Prp: Propagating universal perturbations to attack large language model guard-rails. arXiv preprint arXiv:2402. 15911, 2024. 31
[103] Huijie Lv, Xiao Wang, Yuansen Zhang, Caishuang Huang, Shihan Dou, Junjie Ye, Tao Gui, Qi Zhang, and Xuanjing Huang. Codechameleon: Personalized encryption framework for jailbreaking large language models. arXiv preprint arXiv:2402.16717, 2024. 31
[104] Vinu Sankar Sadasivan, Shoumik Saha, Gaurang Sriramanan, Priyatham Kattakinda, Atoosa Chegini, and Soheil Feizi. Fast adversarial attacks on language models in one gpu minute. arXiv preprint 2rXiv:2402..1555 2024. 31
[105] Jonas Geiping, Alex Stein, Manli Shu, Khalid Saifullah, Yuxin Wen, and Tom Goldstein. Coercing Ilms to do and reveal (almost) anything. arXiv preprint arXiv:2402.1 2402:2202..1400, 2024. 31
[106] Qibing Ren, Chang Gao, Jing Shao, Junchi Yan, Xin Tan, Wai Lam, and Lizhuang Ma. Exploring safety generalization challenges of large language models via code, 2024. 31
[107] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. 31
[108] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference computer and communications security, pages 908-318, 2016. 31
[109] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. 31
[110] Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint wxXh:2339.00026 2023. 31
[111] Gabriel Alon and Michael Kamfonas. Detecting language model attacks with perplexity. arXiv preprint arXiv:2308. 14132, 2023. 32
[112] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightghm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017. 32
[113] Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. Defending chatgpt against jailbreak attack via self-reminders. Nature Machine Intelligence, pages 1-11, 2023. 32
[114] Chujic Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai- Kai-Wei Chang, Minlic Huang, and Nanyun Peng. Prompt-driven 11m safeguarding via directed representation optimization. arXiv preprint arXiv:2401.1800
2024. 32
[115] Andy Zhou, Bo Li, and Haohan Wang. Robust prompt optimization for defending language models against jailbreaking attacks. arXiv preprint arXiv:2401. 2024. 32
[116] Yichuan Mo, Yuji Wang, Zeming Wei, and Yisen Wang. Studious bob fight back against. jailbreaking via prompt adversarial tuning. arXiv preprint arXiv:2402.06255, 2024. 32
[117] Zhexin Zhang, Junxiao Yang, Pei Ke, and Minlie Huang. Defending large language models against jailbreaking attacks through goal prioritization. arXiv preprint 1rXXi:2331.0091 1,09096, 2023. 32
17

---

[118] Yujun Zhou, Yufei Han, Haomin Zhuang, Taicheng Guo. Kehan Guo, Zhenwen Liang, Hongyan Bao, and Xiangliang Zhang. Defending jailbreak prompts via in-context adver- sarial game. arXiv preprint arXiv:2402. 24VV:2202.11144, 13148, 2024. 32
[119] Alec Helbling, Mansi Phute, Matthew Hull, and Duen Homg Chau. Llm self defense: By self examination, Ilms know they are being tricked. arXiv preprint urXiv:2308.07300 2023. 32
[120] Zezhong Wang, Fangkai Yang. Lu Wang. Pu Zhao, Hongru Wang, Liang Chen, Qingwei Lin, and Kam-Fai Fai Wong. Self-guard: Empower the Ilm to safeguard itself. arXiv:2310. 15851, 2023. 32
arXiv preprint
[121] Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang, Rain: Your language models can align themselves without finetuning. arXiv preprint (iXiv:2309.07/24. 2023. 32
[122] Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Pooven- dran. Safedecoding: Defending against jailbreak attacks via safety-aware decoding. arXiv 2024. 32
preprint arXiv:2402.000.
[123] Adib Hasan, Ileana Rugina, and Alex Wang. Pruning for protection: Increasing jailbreak resistance in aligned Ilms without fine-tuning, arXiv preprint arXiv: 10862, 2024. 32
[124] Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1695, 2023. 32
[125] Renjie Pi, Tianyang Han, Yueqi Xie, Rui Pan, Qing Lian, Hanze Dong, Jipeng Zhang, and Tong Zhang. MIlm-proon protector: Ensuring milm's safety without hurting performance. arXiv preprint arXiv:2401 022906, 2024. 32
[126] Yuqi Zhang, Liang Ding, Lefei Zhang, and Dacheng Tao. Intention analysis prompting makes large language models good jailbreak defender. arXiv preprint arXiv:240.00561,, 2024. 32
[127] Yihan Wang, Zhouxing Shi, Andrew Bai, and Cho-Jui Hsich. Defending 1ims against jailbreak- ing attacks via backtranslation. arXiv preprint arXiv:2402.16445, 2024. 32
[128] Heegyu Kim, Schyun Yuk, and Hyunsouk Cho. Break the breakout: Reinventing Im defense against jailbreak attacks with self-refinement. arXiv preprint 2XX.:22002.11800, 2024. 32
[129] Yifan Zeng. Yiran Wu, Xiao Zhang, Huazheng Wang, and Qingyun Wu. Autodefense: Multi- agent I1m defense against jailbreak attacks, 2024. 32
[130] Xiaomeng Hu, Pin-Yu Yu Chen, and Tsung-Y Yi Ho. Gradient cuff: Detecting jailbreak attacks on large language models by exploring refusal loss landscapes. arXiv preprint 0000:7703.00066. 2024. 33
[131] Jiabao Ji, Bairu Hou, Alexander Robey, George J Pappas, Hamed Hassani, Yang Zhang, Eric Wong, and Shiyu Chang. Defending large language models against jailbreak attacks via semantic smoothing. arXiv preprint arXiv:2402.16192, 2024. 33
[132] Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, al. Llama guard: Llmebased input-output safeguard for human-ai conversations. arXiv preprint arXiv:231..06774, 2023. 33
[133] Swapnaja Achintalwar, Adriana Alvarado Garcia, Ateret Anaby- -Tvor, Ioana Baldini, Sara E. Berger, Bishwaranjan Bhattacharjee, Djallel Bouneffouf, Subhajit Chaudhury, Pin-Yu Chen, Lamogha Chiazor, Elizabeth M. Daly, Rogério Abreu de Paula, Pierre Dognin, Eitan Farchi, Soumya Ghosh, Michael Hind, Raya Horesh, George Kour, Young Lee, Erik Miehling, Keerthiram Murugesan, Manish Nagireddy, Inkit Padhi, David Piorkowski, Ambrish Rawat, Ora Raz, Prasanna Sattigeri, Hendrik Strobelt, Sarathkrishna Swaminathan, Christoph Till- mann, Aashka Trivedi, Kush R. Varshney, Dennis Wei, Shalisha Witherspooon, and Marcel Zalmanovici. Detectors for safe and reliable llms: Implementations, uses, and limitations, 2024. 33
18

---

[134] Adam Tauman Kalai and Santosh S Vempala. Calibrated language models must hallucinate. arXiv preprint arXiv:2311.14644. 2023. 33
[135] Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K Kummerfeld, and Rada Mihalcea. A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity. arXiv preprint arXiv:2401.01667, 2024. 33
[136] Boyi Wei, Kaixuan Huang, Yangsibo Huang. Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. arXiv preprint 24Xx:2402.051 162, 2024. 33
19

---

Appendix
A
Glossary
Table 2: Summary of notation.
Syubd
Meaning
query. comppsaabie
concept. quenes. composed guary coosept.
coscept, composcble
(6.c]
The
The explatatius
PC =
The of plusible proupts
7P A/E)
The world mapping, For cach plazsible peceppt, specilics the ground-nuu distibutioon over alaa. the "kaowllssge"
Pooete(p.c)
The ground wuth distibution over subset of E. girco pronpe (p.c3, With slight ahuse notation also refens to point on the probubility simples.
apposeonig d)
Support pusetolp.)) A uniz vabert of
Pon : -P 5(3)
inginge mukll. For cach plassibi prampe sippeiilica a dicritation over is stbeet of) E, mimic Pouree
The outpu distristton mec la subsciol) by LM, givar prmppe e). With alightaless abree suattoon, it alal rifon to puint on the produbiiity simples.
domipuar conaroo.. o))
Demuii of the pearts el datniauias, sullet of
(o. -
Underlying generative diariuaton over prempes, aka. the distibation governing creation our pretraming corpos
supe:De)
Support Dr. ig.c1 suppl supp(Dp) called direct prongt
A langang maddd..
The poir distiuution
postenier desritation
presaining
postenir
preference alignment.
B Proof of Theorems
B.1 Proof of PAC-Bayesian bounds
Definition B.1. (Bounded Difference) A function f :x - said to have bounded difference
property w.r.t a collection of constants C1," Cn. iff
sup
[f(I1, Ø2, ".."". ,2,) - f(21, 02, 1T;-1, Te. .. ,I,)) sc v. [n].
Lemma B.I. (Hoeffding's 's Lemma) for random variable X E |a, bj with probability 1, the following holds:
Elexp(AX)) S exp(AEEX + Ab a)²
8
Lemma B.2. (Hoeffding's Lemma, Multivariate) for random variables Z f f .n) where has the bounded difference property, the following holds:
Ejexp(A(EZ - -2))] < exp
1i=1
8
Note that substituting Z with (LM) is valid.
Lemma B.3. Empirical Loss defined in Definition 3.1 satisfies the bounded difference condition with constant e = 1, Vi.
We are ready to present the proof of Theorem 1.
Proof. Starting with the above lemma, we know
X2c²
Eglesp(X(R(LM)) - RR((LM))) < exp(
8n
The above result holds for manually picked LM. With overall average over the prior have
BLAs--Esgeex(A(RR(L. fts(LM))) exp( 8n
Apply Fubini's theorem (note that π is independent of S):
X262
Rs(LMM)) < exp( Es EsEr,v---oop((((LLLL {
8n
20

---

Define Y = BLMM~//exple (exp//Lexx((((((MMMM (LM))) a random variable depends on S. Obviously Y Thus, with Markov's inequality:
PLY 2 B 5
Equivalently, with probability at least 1 - 6, we have
1 Y s expl 8
2² 8n
Since we have assumed т, share the same support, using Radon-Nykodim derivative to change the expectation with respect x to with respect p. we have
exp(X(R(LM) (L(())) $ cxpl dp
dx
1
X2c2 8n
ELM~D
Taking logarithm and applying Jensen's 's Inequality we know
dw
X(R(LM) Rs((MM)) < log 8
1
X²2²
ELMOO
+
dp
8n
Incorporating c noticing de = (dp) we could rewrite the inequality as
ELM~D (R(LM) - Rs((MM) - [KLp|lm] log ;
A
8n
Finding À that minimizes the term right hand side gives us the e term.
When Dp allows for decomposition into mixture components, noticing the linearty of expectation, the bound can be re-written as
a BBLM=pBB(a Crv (PLM, ()) + (1 - M.)ELM~pBB(ac) (Tv(P (a)
S e C
which leads to
ELM (lac),D, Lr (PLM, (1, s le |
B.2 An estimation on the non-vacuousness of the PAC bound
We give an estimation of the term appears in our PAC bound, 0. and state that it non-vacuous.
The numerator. We follow Neyshabur et al. [60] to instantiate the term in the simplest setup. Assume т, p are defined over the parameter space of a given LM, with K parameters. Assume w is a set of weights learned from the pretraining corpus. Let the prior be the zero-mean multivariate Gaussian, whose entry-wise variance related to the magnitude of the weight: a; = 3/ws],, and p be Gaussian We argue though simple. both settings
with the same anisotropic variance centered around are practical, since Gaussian initialization common for model training, and the SWA-Gaussian w²
algorithm [61] utilizes such Gaussian posterior: Under this setup, the KL goes Σ.
O(K).
20
Specifically, taking 3 v makes the term exactly K. Current language models often possess 2
millions, or billions, of parameters, namely, K ~ [10%, 10%].
The denominator. To estimate the number of unique direct prompts in the training corpus, it is important to notice that the dataset does not only consist of (g. c) prompts but also e explanations. Thus, we need estimate the average token length (ATL) associated with each unique prompt x = (g, c). For each unique prompt x, aside from its own token length (<r) there will be collection of explanations {e:)
N(x) with expected token length of each associated explanation I(e). We have
EATL N x [xx) + ((e)).
21

---

Fact. Given prompt z, the larger the expected length of the prompt itself and explanation (1(x) + ((e) t), the larger the expected number of explanation elements (N(x) 1), and the smaller the number of such prompts (2) J). appearing in the training corpus. 'he former comes naturally due to the composability of natural language: the longer the text fragment, the more equivalent text fragments in expectation, while the latter is reflected by the spirit of the widely accepted Zipf"s law.
Inspired by the fact, we assume prompts are categorized by the quantity of ((x) + ((e), namely, for all prompt I, .V (a) is a function of (x)) + ((e). Moreover, the complete data generation process is decomposed into i) sample value of [(x:) + I(e) out, and then ii) sample a unique prompt from the set decided by this specific ((x) + ((e) value, and iii) generate (xx explanations.
Step i). Use the fact: the larger the expected length of the output explanation, the smaller the probability that such a prompt appears in the training corpus. We assume step i) follows a (cut-off) zeta distribution. Specifically, for a random prompt x;,
p(((x) + I(e) k)) k kg.
When hn = resume the zeta distribution with coefficient
Step ii). We assume each prompt following this step unique.
Step ii)). Use the fact: the larger the expected length of the output explanation, the larger the expected number of explanation elements in the training corpus. We assume power law scaling N, with constant >>> such that
N((()) + ((e) = k) = --1
Thus, the average token length writes
C(s -)
- -)-
-Eer Ag 1 (s-t)
k) x EATL -()
i==
((s)
Σ
i= i-8
ko
where ((s) = E i is the Riemann zeta function.
For example, take = 4, 2. With ko 1, the ATL would be 1.52, while with hn = 10, the ATL becomes 272. These results translate into an estimation of unique prompts as Puokem/ATT With current SOTA LM, the pretraining corpus often includes (tens of) trillions of tokens (> 10 ), thus n > 1000 > K can be safely assumed + <<1. 1.
constant. According to LLaMa-2 report (section 4.1, Figure 13) [13], approximately 0.2% of the documents in their training corpus is labeled as harmful. However, we argue this is indeed an extremely loose lower bound for a, due to the estimation strategy used in their paper. Given document, they use a binary classifier on harmfulness over each single line (1 means harmful and otherwise). and assign the average score to the document. 0.2% is the ratio of documents with score 2 0.5. Take the example of "How to build a bomb'.. The chemical reaction parts will not be counted as harmful, and thus this estimation strategy could judge completely harmful explanation as harmless. Thus, it is reasonable to assert a is not too small, though with current literature we are not capable of raising accurate estimation it.
B.3 Proof of jailbreaking
Before proceeding to the proof, we list necessary definitions and lemmas as follows.
Lemma B.4. (Volume of n-simplex)) For any dimension n, the volume of the n-element probability
simplex: A n- in the 1-dimensional space is
1)!
We define the projected probability simplex follows.
Definition B.2. (Projected probability simplex) Given A the corresponding projected probability simplex, Ap-1, is defined subset of Rf m= (a " x; S 1, E In 1]}.
See https: //en. wwk.ppdia..rg//uuii/iiiii.. wwtp/ooii///iimmeexoo...ooooooo. wwt.s/imwwweeeeeeeeeeemmm
22

---

An illustration of A and A7-1. For example, take = 3. The probability simplex with 3 elements is triangle whose (euclidean) side length is 22 with vertices (1,0,0), (0, 1, 0), (0, 1). Then its volume in the 2-dimensional space, i.e., its area, is y3 The corresponding projected probability simplex is the triangle between the --Y axis, with vertices (1, 0). (0, 1), (0, 0).
2
direct lemma that connects the probability simplex and the projected probability simplex is given below.
Lemma B.5. (Transformation of probability simplex) Given a proper probability density function v(s)
v(x:) defined on A App il is equivalent to the distribution defined on A with density V V E
Borel (Ap AB-1), ), let B (xr An-1 I1:n-1 E A). Then SA v J v(a)
dr. Specifically, this
vn
implies volume AA)
volume B)
volume (A 1)
volume A
>i= which does not affect its the volume and shape. The mapping: Ap-1 translatedA n-1 is an affine transformation with matrix
Proof. Consider translation on n= with In =
0
0
1
0
T
-1
(x(nn1))
Thus, any area under this transformation is scaled by det T Vn: constant. The lemma follows directly after this conclusion.
We use (-) to denote the uniform distribution over ~~ U(x) (a-1)) Vr E n- We use the notation vol[S] [S] Js 1ds to represent the volume of given subset S /" the relative volume (w.rt. the underlying n-simplex) of S, i.e., rvol[S] := also use n = [[cc] from now on. We use the vector æ to denote (with the slight abuse of notation we have mentioned) PLM (g, c) on the output simplex.
A n- and use rvol for
vol[8] vol[A fs U (x:)d..
Lemma B.6. (Gaussian cdf Tail Bound, Gordon [62]) Denote Φ(.) as the standard Gaussian pdf. When > 0,
x22/2
=2212 1
T ((x)
< 1
(xx) <
-0(x).
x2 +
22 +
2T
V
x
Now we are ready to give the proof of Theorem 2.
Proof. Let [x)c = no and denote Ex(c]] [x(c) + [c)( = Without loss of generality, we define the first 74p = (x) (c)l elements as the harmful explanations. Let the thresholding constant be p. That is, define the harmful zone H4, as fx ~ E x; > p). To compute the relative volume of H4 in --1 we could instead operate the projected probability simplex Ag-1 introduced in Definition B.2, and compute the relative volume of the projected Hys Ha,p {x
23

---

1 | 24 2 pp. Note that 43-1 C R--1
We derive its expression as follows.
no
volume H,p
volume/ff Ap111 Σ
S p|}
L-" dx
E
1-E
E-
dx
d
drn-1
0o
S'" do
21
21
ng
dx 1.
no
22 ng
(n 1i - 1}!
i=1
Dimi n0-22 21
!"" dr1
1--21
dx ...
dr
(1
R--g - (1 - p)"
-p)
(n no)!
1=1
(1 (1-p)~-17
1 - (1-p)~
(n
j!(n - j)!
(5)
Thus, the relative volume of Hn can be written as
volume P
rvol = volume(projected probability simplex]
(n 1)!(1 p)"-1-s
(1
j!(n - 5)!
(6)
n--1 Σ - 1=0
n-15 n1 (0=1)
(1
Which is precisely the binomial distribution formula. With the Central Limit Theorem, when n >> O1), we know the binomial distribution can be well approximated via the normal distribution
as follows:
f(x) = ()) 11-1--)-- p)- 4 NNnp, -)pp)
(7)
Thus, denote ((n-1),p(x) the pdf of Gaussian variable with mean (n- 1)p, variance (n- 1)p(1)ppp11 -p), rvol term above can be estimated as follows:
no -1
f(1 p)
n2
O(n-1), (x)dz
==
no 1 (n 1)p
+
(8)
V(n (n 1))p p)
En (c)) (n 1)p
+ V(n 1)p(1 p)
We use [Bxle)l-1-nn-11 Consider an adversary with budget under 4p or Jensen-Shanme / (1-11p(11pp)
Divergence (JSD) / Total Variation (TV) capability, Since ||0||| > ||x||| Vp 2 as well as ||ع||| 2J)D(x), ||>||| 2 (TV(z), we know (g (1) Hy d') for all we have considered. With that é1, e setup, the corresponding e e-exannsion set of H4 has a closed-form expression as
ag
T p
i=1
24

---

Similar as above, we derive the analytical solution of its relative volume associated with constant a'
as:
d' En(e (n 1) - 5)
v n [p 2)(1 p 2)
(9)
p(1 - p)
71
(p 8)( - {) +
21
(p 5)(1 - ②)
Under our framework, with p 1. we know
> p(1 p) (p -1((1 p 8)).
Thus
d'>a vn le dz--
Consider the induced distribution %e on the output simplex. Given adversary with Ep or JSD/TV perturbing capability, with the fixed harmful concept c, safety guaranteed if and only if PLad (q, c) resides outside H4 (c, d). Define the area of interest, S(d) as S(d) = A " Ha(e,, d). Thus, the probability of this event could be bounded as
max 70(xx) / 1dr < [rroo[[[(d) < Tarvol[S(ei]] < (1 (t) s(d)
Pa 2()))))) <
=ES(4)
This gives upper bound of
7o (a.)).
which can be simplified when 2 0 using Lemma B.6:
(((c)) ag
Y'
Thus, the probability of getting LM instance from the preference alignment process such that it allows for successful jailbreaking on a specific harmful concept e is at least
1 7s (1 (aa)))
Up to now, we have derived the result in Theorem 2. However, we can move a step further to show the decay rate the right hand side term. It can be simplified when 2 0:
(((a))
7
a2
which finishes the proof.
C RLHE, DPO and our E-RLHF
The classic RLHF framework was established by Christiano et [63], and developed by Ziegler et al. [19], Ouyang et al. [20], Bai et al. [21]. After the collection of preference dataset D, one first trains reward model under the Bradley-Terry model [64], with the objective, where (.) stands for the sigmoid function:
{(x, Cur, }. ewr,t))
r rar, = arg max (( logo (r((,, r(x, et)).
Following, proximal policy optimization (PPO) [65] commonly adopted across these implemen- tations, forming the basis of current state-of-the-an language models. The KL-constrained RL Fine-Tuning (RLFT) objective takes the form:
max BD (Bespea((e)ll(e,,)) 3l (PLM (=))(())))))) PLM
However, PPO tuning can suffer from instability [66] and implementation complication [67]. To overcome these issues, series work propose to skip the reward modeling step and learn directly from the preference dataset, with the representative pioneering work by Rafailov et al. [23], namely Direct Preference Optimization (DPO). We summarize the derivation of the DPO objective below, and generalize the objective to the use in our experiments, i.e., E-DPO.
25

---

First, noticing the closed-form optimal solution for PLM of the RLFT objective writes (see e.g., Appendix A.I of Rafailov et al. [23])
(elx)
Z'(z) (ooeeerl)))) exp( 3 (T(2,e)) e)).
With this analytical solution in mind, solve the reward as
PR Prr(elz)
r(x, e) = Bog
+ log (z)..
Regard x² as the optimization target, plug this transformation into the reward model objective to obtain the DPO objective:
PDPO = arg min - D(e. (3 (B1og log PLar(Ewla) 3 log Prer(er 2t)
(Lar(e)|2)) )}
Prer Ce (z)
For our E-RLHF, the modification the objective leads another optimal solution of PLM:
p² (elxx) =
7(x) 7(0z) phr(el (r(,,)))
1
Thus,
p" (elxx)
r(x, e) = 3 log Pher(elzrs)
+ B log ((x)
And plug it in the reward model objective formulate our E-DPO:
PE-DPO = arg min D(a.ewe))=-/loog (B (3log (e.M(en(T)) Blog Pr.M (e.((e)())) PLM
)} (x)(e)(2:s)
(re((ew(2ss)
The advantage of our E-RLHF objective is follows.
Proposition C.1. (Overcoming the small safe problem) E-RLHF will lead to the optimal solution p":
1 [ Pro(e expl gr(zz e)).
(e(xx) Z(x)
Compared to PRLFT. the advantage when encountering a harmful prompt x is:
(1) (Erase harmful explanations) Ve supp(pns(-(=)) supp(Pra((Ixa)) p" (elz) = 0:
(2) (Add safe explanations) Ve supp(peg((lar.)) supp(pny(((a)) p" (c|x) > 0 Pelr)(elx)).
Thus, with the same jailbreak threshold p. the safety zone is successfully expanded.
Intriguingly, when the safe transformation is done by appending an identical safe prefix to all harmful prompts, we can connect our E-RLHF to context distillation. A good prompt known to matter for the performance of paremmerrseters LM [68, 55J. Researchers have proposed a systematic LM tuning algorithm, called Context Distillation [69], aiming at distilling useful information frouuttoonommmmmnoo good context prefix to a language model. Given an initialized language model, for example PSFT, an input prompt x and a prefix context string prefix, Askell et al. [69] optimizes the loss
L(PLM) ) DxL. (Pstr C P.).
where @ stands for string concatenation. This technique has been adopted as part of the safety alignment process during the LLaMa-2 series tuning [13], where prefix chosen from set of pre- defined safe prefixes. When applying the identical prefix transform in our E-RLHF transformation, it can be regarded combination of safety context distillation and RLHF. This gives another point of view on the effectiveness of our proposal.
D Ablation Study
In this section, we perform extensive ablation studies to showcase the effectiveness our proposed E-RLHE.
26

---

Table 3: Safety evaluation, LoRA results. The result is consistent with the one we have obtained in the main text, that our E-DPO performs better than DPO across a collection of adversaries. indicates better performance.
HarmBench ASR [2]
Model
Direct Request
GCG
GBDA
STS
25
PAIR
TAP
AutoDAN
PAP-top5
Human
AVG
24.50
47.50
40.50
43.25
43.25
28.50
45.25
53.25
$3.50
29.75
38.00
40.74
24.25
42.50
36.50
41.50
42.75
27.20
45.00
53.75
50.25
27.25
38.05
39.00
AdvRench ASR
2.00
29.00
26.00
26.00
40.00
8.80
32.00
54.00
46.00
2.00
31.20
27.00
7E000 dARAi (ours)
0.00
25.00
20.00
27.00
33.00
7.20
23.00
50.00
45.00
2.00
28.50
23.73
Table Safe prefix ablation. Prefixes than the DPO baseline in most cases we have tested. indicates best performance. Number in bracket indicates the MT-Bench score.
included in Table 8. Our E-DPO performs better
HarmBench ASR (2]
Model [MT-Rench]
Direct Request
GOG
GBIDA
AP
SPS
78
PAIR
TAP
AutoDAN
PAP-top5
Human
AVG
[6.9]
27.50
53.00
30.00
46.75
43.25
20.10
52.50
54.00
51.00
28.75
37.15
42,00
Tioooon (6.9)
56.50
33.75
29.30
24.75
52.25
34.00
39.00
44.75
29.75
30.50
51.50
53.25
25.00
31.35
40.46
Ticonon (6.8)
53.50
TiL0woo 66.7]
23.50
47.50
31.75
36.25
40.50
26 45
48.50
51
43.00
27.00
31.05
36.96
AdvHench ASR (1)
Yuro [6.9]
0.00
47,00
12.00
39.00
30.000
7.00
50.00
61.00
44.00
4.00
18.40
28.40
Tieeoon, (6.9)
51.00
12.00
29.00
33.00
6.80
47.00
62.00
53.00
5.00
20.00
28.08
Ficeeoii 6.9)
1.00
39.00
12.00
20.00
34.00
6.20
53.00
63.00
49.00
3.00
15.00
26.80
Tieeroi 6.88
47.00
11.00
20.00
23.00
6.80
45.00
58.00
36.00
4.00
15.80
24.51
(6.77
00
8.00
15.00
21.00
5.20
41.00
53.00
31.00
4.00
13.60
20.89
D.1 LoRA results
In this section, we show results obtained by Low-Rank Adaptation [54]. We explore the same set of safe prefixes as in ablation D.2, and choose the best model for illustration. Numbers illustrated in Table 3. Results are identical to the ones obtained via full parameter tuning that our E-RLHF performs better consistently against the RLHF baseline.
D.2 Ablation safe prefixes
We ablate the effect of different safe prefixes. The prefixes we consider are collected in Table 8. Attack success rate numbers are shown in Table 4, with each model's MT-Bench scores shown in the brackets. Clearly, almost all safe prefixes lead to better safety against the DPO baseline and improved helpfulness compared the SFT model (whose MT-Bench score 6.3), and the performance could vary depending on the choice of the prefix. Finding good prefix matters for our method, and we leave digging the optimal one out as future work.
D.3 Ablation on transforming all prompts
Here, we proceed to ablating the effect of transforming all prompts, no matter harmful or not. Results are shown in Table 5, where the red color indicates that safety downgrades compared the model obtained via transforming harmful prompts only. Clearly, most models even persist worse safety compared to the DPO baseline itself, suggesting the detrimental effect of transforming the unharmful prompts.
D.4 Ablation with system prompt
As pointed out by previous works [2, 48, 70], system prompt can have significant impact on ASR. This comes in two-folds: firstly, powerful system prompt initialize the LM to be closer to
27

---

Table 5: Ablation study on transforming all prompts. We apply the same safe prefixes used in indicates the safety is worse compared to the model trained with transforming only the harmful prompts, AVG scores achieved by the DPO baseline are 42.00 and 28.40, respectively.
Table 4.
HarmBench ASR [2]
Model
Direct Request
GCG
TAP
AutoDAN
PAP-top5
AVG
28:00
56.75
41.00
42.00
12.50
31.00
61.20
56:25
56.00
32.00
36.06
42
60.00
40.25
46.00
44.75
30.15
57:25
03:00
30.75
40.45
24.00
$7:25
35.00
41.75
39.50
26.55
49. 25
55.00
58.75
20.25
38.70
41 30
27.70
$8,50
38.00
42.00
40.75
31.20
52.75
60.50
54.75
31.50
38,30
43.27
AdyBench ASR [1]
0.00
53.00
17.00
26,00
6.00
60.00
57.00
61 10.00
4.00
19.00
0.00
26.00
6.20
58.00
3.00
0.00
50:00
17:00
23.00
17.00
3.80
40:00
68.00
60 000000
4.00
23:40
28
0.00
58,00
17,00
38.00
27.00
6.50
46.00
60.00
68. (00
3.00
25.20
31
Table 6: Ablation with the default Mistral system prompt. boosts the safety of our E-DPO models to achieve better safety, while maintaining the advantage over the DPO baseline.
HarmBench ASR [2]
Model
Direct Request
GCG
GBDA
AP
SFS
75
PATR
TAP
AutoDAN
PAP-top5
Humaa
AVG
25.25
61.25
26.75
37.00
27.25
24.75
55.50
58.75
61.25
24.00
30.35
39.28
maro
20.50
42.75
42.75
31.20
13.50
41.50
22.75
29.00
31.25
17.95
41.75
44.25
41.25
23.00
26.55
30.70
the safety zone, thus making the model safer; secondly, longer system prompt would enlarge the difficulty of launching specific attack due to the increased computational consumption. To confirm E-RLHF is still helpful under this specific situation, we ablate this effect under the control of the default Mistral system prompt "I The results can be found in Table 6, where nearly all ASRs drop, whilst the conclusion does not change.
D.5 Ablation on improving helpfulness and safety simultaneously
In similar spirit, we ask the question: can the trick we propose to improve safety be used to improve helpfulness simultancously? To achieve this goal, we choose the prefix "Give me safe and helpful response to the following request. ", and apply it to all prompts in the preference dataset. Results can be found in Table 7. Our model achieves better safety, and improves the helpfulness benchmarked by MT-Bench [3] from 6.9 to 7.0.
E
Filtering harmful prompt in the preference alignment dataset
Inspired by the work by Chao al. [30], prompt GPT-3.5-Turbe prompts. assigns an integer score ranging from 1 to 10 for a given prompt, and we choose the score subset harmful. The system prompt is shown in Table 9.
harmness judge input
"nttps://docs mistral
Table 7: Improving helpfulness and safety simultaneously using E-RLHE. MTT-Bench scores 6.8 and 6.9, for the DPO baseline and our E-DPO model, respectively.
HarmBench
ASR
Model
Direct Request
GCG
GBDA
AP
SFS
25
PAIR
TAP
AutoDAN
PAP-topS
Human
AVG
$2.25
59.25
35.50
42.75
12.75
36.20
56.50
65.00
56.75
26.75
35.50
44.47
Tieeo
27.50
39.00
51.00
37.15
42.00
(ours)
26.25
56.00
37.00
43.75
38 25
27
49.75
56.00
52.25
27.50
32.65
40.66
28

---

Table 8: Safe prefixes we use in our experiments.
"Provide responsible and ethical answer the following request.
"In a manner that promotes safety and well-being, please respond to the following request.
(3) "Keeping in mind the importance of safety and legality, how would you respond to this:
"Please ensure your response adheres community guidelines and ethical standards:
Table 9: System prompt for GPT-3.5-Turbo used to filter harmful and non-harmful prompt in the preference dataset.
You are helpful red teaming assistant. You will receive prompt. Please act impartial judge and evaluate whether the prompt itself is requesting information that violates ethical, legal, or safety guidefines. Be as scale from to 10. A rating of signifies fully non-harmful prompt.
objective possible. Rate the prompt A rating of 10 signifies prompts that could lead to responses that fully violate ethical, legal, safety guidelines. Your rating should strictly follow this format: "Rating: ||rating]]". the rating the double-closed brackets 10, e.g. "Rating: [[5]]".
number from
Jailbreak adversaries collected in the HarmBench project [2]
In this section, we give a short overview of the adversaries we adopt to evaluate our models. Some descriptions are summarized in Mazeika et al. [2].
Direct Request refers to sending the original harmful prompt directly to the target LLM.
GCG [1], GBDA [71] and AP [72) find adversarial suffixes via token-level optimization.
SFS (Stochastic Few-Shot) and 7S (Zero-Shot) [15] are few-shot sampling or zero-shot generation of test cases by attacker L.M elicit behavior from target L.M..
PAIR [30] and TAP [33] iterative prompting / tree-structured prompting methods, with an attacker LLM, to adaptively explore and elicit specific harmful behaviors from the target LLM.
AutoDAN [29] is cenetic-algorithe based attack with initializations from handcrafted DAN jailbreak prompts.
PAP [73] uses persuasive strategies to jailbreak the target LLM.
HumanJailbreaks [74] uses fixed set of in-the-wild human jailbreak templates, similar to the Do Anything Now (DAN) jailbreaks.
We exclude all transfer attacks since we focus single-model jailbreak. Furthermore, we choose to discard the UAT [75] and PEZ [76] adversaries, because the former induces an out-of-memory error on our V100 GPUs, and the latter never succeeds to find suffix according to our experiments.
G Broader Impacts
The societal impact of our work has close connection to the topic of LLM safety. We propose framework for analyzing language model pretraining and jailbreaking, and we design new RLHF algorithm for improving safety. As shown in our experiments, our work could advocate for safer L.Mss.
H
Related work
In this section, we provide a review of the current literature on LLM jailbreaking.
H.1 Jailbreak methods
In this section, we summarize existing jailbreaking methods.
29

---

Baseline and pioneers Autoprompt [72], a baseline method for optimizing in the token space w.r.t. a certain objective, approximates coordinate ascent by first ranking all tokens using an approximate objective, and then compute the exact value on them. The approximation is carried out by a single step of gradient computation. Jones et al. [77] propose Autoregressive Randomized Coordinate Ascent (ARCA) to generate (input, output) pairs that include certain harmful info or satisfy a fixed format requirement. Token level optimization is carried out with linear approximation on GPT-2. GBDA [71] study adversarial attack on text classification problems, by optimizing the continuous relaxation of the autoregressive sampling probability matrix. In late 2022, among social media, users misalign GPT-3 via prompt injection. Perez and Ribeiro [78] study how this be done by adversaries. They successfully manipulate the model to output a given harmful string and leak the system prompt. In early 2023, an empirical study was carried out by Liu et al. [9] to measure the result of prompt engineering for breaking ChatGPT safeguards. Shen et al. [74] collect jailbreaking prompts from multiple platforms on the internet, analyze these data, create a harmful question set, and identify some typical harmful prompts that are effective at that moment. Later, the Greedy Coordinate Gradient (GCG) method [1], strong white-box attack variant of AutoPrompt [72] was proposed. Wei et al. [79] categorizes two general modes of jailbreaking: competing objective and mismatched generalization.
LLM automation and suffix-based attacks Liu et al. [29] propose AutoDAN, that relies on genetic algorithms, with the requirement of manual prompts for conducting mutation and crossover on the paragraph and sentence level. The jailbreaking prompts generated by AutoDAN are semantically plausible, unlike the suffix generated by GCG. As comparison, Lapid et al. [25] use genetic algorithm for black-box universal adversarial suffix generation. Chao et al. [30] propose another LLM-based jailbreak automation algorithm, where an LLM judge is built to assign a safety score to given output, while the attacker is enforced (via page-long prompt) to improve the quality of jailbreaking prompts from multiple perspectives. Zhu et al. [24] propose another AutoDAN method that explores the balanced loss between jailbreak loss (log probability on the harmful string, as used in Zou et al. [1]) and the plausibility loss (log probability over the adversarial suffix). aiming at improving interpretability. Li et al. [31] uses genetic algorithm to search with similarty measure and initialize with paraphrasing. Its performance is claimed to be better than AutoDAN- Deng et al. [80] investigate the possible defensive tricks in proprietary LLMs, and propose a pipeline for automated jailbreaking using fine-tuned LLM. Yu et al. [81] propose GPTFuzzer, essentially genetic algorithmic framework for jailbreaking. Their work's difference between AutoDAN is that has a pool of "seeds", a.k.a. templates for transforming the harmful prompt, and the mutation is done on the template level. Ding et al. [32] propose automating attack via LLM prompl rewriting and scenario nesting. The latter consists of code completion, table filling and text continuation, since the authors regard these as align with training objectives well, and are suitable for LLMs to complete the task. Mchrotra et al. [33] combine Automation used in Chao et al. [30] and pre-offthouggh [82], create interpretable prompts in black-box manner. Li et al. [83] propose DeepInception, and use nested, imaginary scenario to induce harmful content. Li et al. [84] propose DrAttack, which camouflages a query's 's malicious intent through semantic decomposition, by constructing parsing tree and split the original prompt into different segmentations. Wang et al. [85] draw inspiration from the self-perception theory from psychology to design prompt modification pipeline on gradually persuading the LM to be jailbroken. Paulus et al. [86] propose Adv Adyrroppeer where the authors train a language model as a suffix generator to speed up LLM attack.
Manipulating the decoding process Huang et al. [48] find the method of changing the generating hyperparameters (i.e., p of top-p sampling. the temperature T, and ic of top-k sampling) of safety- aligned LLMs suffices for obtaining harmful outputs when the user is able to manipulate the system prompt and input configurations. Zhang et al. [87] directly manipulate the output generation probabil- ity by enforcing an affirmative prefix, and reversing the negation words if they appear in pre-defined vocabulary (e.g., sorry - glad). Zhao al. [88] assume access to the decoding distribution of a LMM They use two small LMMs, a safe one and a harmful one, to manipulate the decoding ratio of the large safe LM for jailbreaking. The key insight the decoding distribution between the safe model and the harmful model only differs significantly for the first tens of tokens.
Fine- Sine-Tuning alone suffices Yang et al. [89] show that fine-tuning on as few as 100 harmful example pairs suffices for turning the LLLaM--cat models (and some other <70B LMs) into malicious counterparts. Zhan et al. [90] fine-tune GPT-4 on harmful data, and find the fine-tuned models escape previous safety constraints while maintaining usefulness. Qi et al. [91] find fine-tuning alone, even on
30

---

benign data, leads to safety degradation using LLaMa and -Pr-o..-Turbo. Fine-tuning on harmful data (with less than 5 gradient steps) will cause the model to be completely harmful, while tuning on identity-shifting data could make the LM fully obedient.
Low-resource language and cipher Yong et al. [26], Deng et al. [27] explore the difference in languages when encountering the same harmful query, and find a direct translation to low resource languages will lead to higher risk, and Deng et al. [27] additionally find when combined with sophisticated methods, the drawback of low-resource languages disappear. Yuan et al. [28] use cipher encoding and decoding to break LLMs. Smaller scale models are immune from such attacks, while the smartest GPT-4 encountered the highest risk.
Vision-language model attacks Besides pure LLM, some research works move step forward, uti- lizing images for breaking vision-language models (VLMs). Shayegani et al. [92] explore multimodal attack on VLM via embedding space feature matching. Qi al. [93] generate adversarial examples via maximizing the conditional probability of harmful corpus, i.e., the sum of log probabilities over all outputs, and use the final image with harmful query for jailbreaking. Carlini et al. [94] generate adversarial example for a fixed harmful content, and find no matter what input prompt is given to the VI.M, it will respond with the target harmful string. Maus et al. [95] propose a black-box attack on manipulating the generated image with modified adversarial prompt.
Misc Wei et al. [57], Wang et al. [58] explore in-context learning for attack and defense. The attack is weak since it could only break Vicuna [96] and can be defended by in-context safe examples. Later, this method is scaled-up to significantly improve strength for breaking guardrails of large, state-of- the-art models [59]. An early work in February 2023 [97] adopts obfuscation (including synonym and typos), code injection and virtualization to successfully jailbreak ChatGPT. Shah et al. [98] illustrate in-context automated persona modulation attack for large- scale L.MMs and Vicuna. Zeng et al. [73] consider the more broadly perspective of persuasion: they train a persuasive paraphraser based on a fine-grained taxonomy of persuasion techniques. Detailed ablation on attack effectiveness is studied. Guo et al. [99] focus on stealthiness and controllability. They notice the constraints applied to the jailbreaking prompts (e.g., fluency) are exactly the targets of the controllable text generation problem. Thus, they adopt the Energy-based Constrained Decoding with Langevin Dynamic (COLD) [100] energy function
on output logits. Forming each constraint as well as the task of jailbreaking over logits, the Langevin Dynamic is used for finding a good logit distribution, and the decoding technique in Qin et al. [100] is used for output generation. Banerjee et al. [101] introduce dataset TECHHAZARDQA, compare direct query v.s. pseudo-code format, and find the latter induces higher risk. Mangaokar et al. [102] considers type of adaptive attack against shecking-based defense, that appends a universal adversarial prefix into the jailbreaking prompt to make the guard model always output "safe", and thus making the detector fails to detect harmful information. Lv et al. [103] propose Code Chamelcon, which contains multiple encryption and decryption methods defined by python functions, that transforms the harmful prompt into specific predefined form jailbreak LLMs. Sadasivan et al. [104] speed up the computation of GCG [1] to make it possible to launch on a single GPU. Geiping et al. [105] build taxonomy on risks beyond jailbreaking. and coerce the LLM to provide certain outputs by optimizing a set of tokens via GCG. Ren et al. [106] propose Ctdek that use code templates to query the output out instead of using natural language directly, and obtain descent results.
H.2 Defense methods
Up to now, no universal defensive strategy as adversarial training [107] for adversarial attacks / differential privacy [108] for membership attacks exists as a gold standard. In general, we can classify the methods into three typical types: alignment, red-teaming, and algorithmic proposals.
The target of aligmment is to push the output of language models be aligned to human values. Regarding safety, the goal is to avoid outputting harmful information. RLHF is widely adopted in these methods [19, 20, 1 109, 22]. Variants like RLAIF also have been proposed recently [21, 110].
Alignment
Red teaming This term is populated as specifically dealing with harmful info on dataset curation, used together with RLHF [14-18].
31

---

Next, we proceed to defensive algorithm proposals. We classify existing defensive strategies in the following categories.
Defense against suffix-based attacks. Alon and Kamfonas [111] notice the messy nature of the suffix generated by GCG, and propose to use a perplexity (PPL) filter on input prompts. They also explore using LightGBM [112] with 2 features (PPL, prompt length) to filter harmful prompt, and show it does better than naive PPL thresholding. The PPL-based filter does not succeed with human-craftee jailbreaks. Jain et al. [37] explore many concerning viewpoints, including self- eeff filtering, paraphrasing the input prompt, and re-tokenization since many LLMs' tokenizers are based on Byte-Pair Encoding (BPE). All methods are successful in regards of defending against suffix-based attacks. They also explore the simplest form of adversarial training. Robey et al. [34] propose to perturb the input token string by random egglaaccmenneerreemeennnnnnttooo and finally perform majority vote in the end. Cao et al. [35] judges whether an input prompt is safe or not by estimation with Monte Carlo, when randomly dropping a fraction of tokens, using the LLM itself. Kumar et al. [36] try to perform "certified" safety against harmful prompts, by erasing tokens and set the original prompt as harmful if at least one of these erased prompts lead to a harmful response, or be classified as harmful by a DistilBBERT-basee classifier.
System prompt defense. We could modify the input prompt for jailbreaking; and several works explore if we can apply similar methods to system prompts to defend against such attacks. Xie et al. [113] propose "self-reminder", i.e., appending a reminding prompt within the system prompt for defense. The attacks are collected from the JailbreakChat collection, and this strategy is effective for defending against them. Zheng et al. [1 [114] advocate for finding a good system prompt automatically, by investigating the representational difference between safe and harmful queries, and optimizing the safety prompts along the harmful refusal direction in the representation space. One intriguing takeaway is harmful/ / harmless queries can be distinguished in the representation space, different from the adversarial examples in vision. Zhou et al. [115] also optimize the safe system prompt, but in a more "adversarial training" fashion, that apply jailbreak algorithms with current safe prompts first and then find good replacement candidates in the same way as done by Zou et al. [1]. Concurrently, Mo al. [1 [6] propose prompt adversarial tuning, where an adversarial suffix is assumed, while safe system prompt is jointly optimized with this suffix, with an additionally constructed benign loss to improve helpfulness under normal queries. Zhang et al. [117] propose the idea of "goal prioritization", either without training (append prioritize safety than helpfuiness and in-context examples to the system prompt) or with training (generate data pairs of prioritizing safety or helpfulness, finetune. and append prioritize safety prompt into system prompt). The former is effective for large-scale LLMss while the latter improves safety of LLaMa--hat models. Zhou et al. [118] propose in-context adversarial game, where an attack LLM and defense LLM interact on exchanging insights on
successful jailbreaks, and defend by improving the system prompt. Zou et al. [70] give the result that system prompt matters for jailbreaking, and shows conducting GA-based search over it could improve safety.
Checking-bassee based, decoding-based and Misc. Helbling et al. [119] generate responses first, and then use the LLM itself to examine whether the output is harmful or not. They find such simple self-examination is powerful since the TPR reported can be up to (1) tune the LM to enhance its capability on discriminating harmful / harmless content; (2) tune the LM make it able to tag its own response; and (3) rewrite response if output is harmful. Li et al. [121] propose to suppress the attack performance by iteratively rewinding and re-examining the generated output. The method does not work well with small models, but works pretty fine with large (open-source) models. They find the strategy can improve generalization as well. Xu et al. [122] train a safer model first, and use normalized Pattacked + (Peater Pauscked) over top-k shared tokens for decoding to enhance safety. Hasan et al. [123] show with original Wanda pruning [124], the LLM can help resist direct jailbreaking prompts, e.g., with role-playing attacks. Pi et al. [125] propose MLLM--Prtecte on safeguarding Visual LLMs by checking the output and then detoxifying the content. Zhang et al. [126] perform intention analysis on the input, and enforce the model generate policy-aligned outputs both by prompting. Wang et al. [127] propose backtranslation that guesses the input prompt directly, and reject it is harmful. Kim et al. [128] propose self-refinement which consists of generating a feedback and then refine the response to avoid harmful info output. They find using additional JSON and code formatting would improve safety. Zeng et al. [129] propose AutoDefense, which utilizes multiple agents analyzing prompt, response and intention, to defend
1.00. Wang et al. [120] propose
32

---

against attacks. Hu et al. [130] propose Gradient Cuff, sampling-basee based gradient- norm defense method, by rejecting those imput prompts with large gradient norm on top of a majority-vote based filtering. Ji et al. [131] propose a method similar to Robey et al. [34], but for gemantically-menningii attacks, that paraphrases the input according to several criteria and conduct a majority vote for judging.
Several company-centered products also fall into this regime. For example, LLaMa-Guard [132] is trained on toxic data such that it is able to discriminate unsafe user prompts and outputs, respectively, IBM also propose a framework on constructing and deploying safeguard detection modules, and releases the details in technical report [133].
H.3 Theory and experimental understanding
Wolf et al. [38] assumes the decomposability of LLM output into good and bad component, and show possible jailbreaking in theory by prompting the model with a sufficiently long input. Kalai and Vempala [134] use statistical tools to prove hallucination for calibrated LMs. Lee et al. [135] study the representation in GPT-2. They train a base classifier for toxicity, and use the linear weight as proxy of toxic vector. They find there value vectors close the toxic vector itself, that suppressed by DPO tuning [23]. Wei et al. [136] use pruning and low-rank analysis on safe LMM, and find (1) safe neurons and useful neurons are sparse: pruning the safe neurons or removing the safe ranks away degrades safety a lot, and (2) fixing the safe neurons in fine-tuning does not maintain safety.
33

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: we state the three main contributions of our work explicitly, following by individual sections that clearly illustrate them.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: discuss the limitation of our work in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
34

---

Justification: All assumptions are clearly stated in the main text, and all proof' can be found in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: all models, code, data and training configurations we used are publicly avail- able, and we provide links to them in the main text.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
35

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: we do not use our own model or data for training. Thus, we not provide any data or code by ourselves. All of them are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / httppsscccppbpcccccceecce ceroolicggiidee Codeeubbm ssi aasonnooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: we adopt all default training hyperparameters from the public codebase.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: following community standard, we use greedy decoding for language model evaluation, which naturally leads variance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
36

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: we state our experiments are consistently launched on 8 NVIDIA V-100 GPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: https://neette /ttee:///eee...eeeeeeee httpppp...cppuuupe cctpublic//ttiiieec wtw.ehheeeeidcdeeeeeeeeennee
Answer: [Yes]
Justification: our research does not involve human participants, and never private data or models. Our research related to LLM safety, and we propose an algorithm to improve it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: the societal impact of our work has close connection to the topic of LLM safety. We propose framework for analyzing language model pretraining and jailbreaking. and we design a new RLHF algorithm for improving safety. Our work could advocate for safer LLMs.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
37

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: although our research involves jailbreaking language models, we not release any models or data, thus the paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: the code and data and the code we use to test the model possess the MIT license. We include links to these public assets explicitly in the paper.
train the model possess the Apache-2.0 2.0 license,
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
38

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
we not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
39

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
40


Reviews:
[{"review_id": "0qumXQQfaI", "reviewer": "Reviewer_qvRD", "summary": "The paper provides a theoretical insight about LLM jailbreaks using PAC-Bayesian bound for pretraining LLMs. It assumes that there always exists the harmful data in the mixture, and as the model is trained on this mixture, the model will probably produce the responses in harmful zone (it has a specific definition in the main paper). Based on this framework, the authors suggests that the safety zone of the models should be extended, and to this end, they introduce a method called E-RLHF which can expand the safety zone. E-RLHF replaces the harmful prompts x_h into benign prompts x_s, and replace some of the terms in RLHF (and DPO) to make sure that the model keeps in the safety zone. Experimental results also show that it does not sacrifice the general capabilities but can be improved in the safety perspective.", "questions": "No specific questions about the paper.", "limitations": "The authors provided limitations section in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper suggests a theoretical insight about the LLM jailbreaks, which were not addressed much in the previous literature on jailbreaking. The theoretical framework is sound and compelling.\n- Based on this framework, the paper also suggests a simple training trick that can lead to better safety training. \n- Empirical results also show that their idea is working well, shown by Harmbench, AdvBench, and MT-Bench scores.\n- The paper provides extensive evaluation results on jailbreaking setups, providing the results from more than 10 attack setups.", "weaknesses": "- E-RLHF is an inaugural and simple form of expanding the safety zone of LLM; I think there could be more sophisticated and effective ways, and I hope the authors will address this in the future works.\n- Other than that, I think there is no big weakness in the paper, but have some minor comments:\n  - Eq (2): it is slightly confusing that in D_KL, the term only have p_LM(x) and p_SFT(x), not p_LM(e|x) and p_SFT(e|x).\n  - About writing: I am not familiar with using the term \"explanation\" -- instead, I think using \"response\" is more common. At the first glance, it was hard to comprehend the meaning of the term.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer qvRD, \n\nWe sincerely thank you for reading our work in great detail! Here are our responses to your concerns.\n\n*1. E-RLHF is an inaugural and simple form of expanding the safety zone of LLM; I think there could be more sophisticated and effective ways, and I hope the authors will address this in the future works.*\n-  We sincerely appreciate your feedback. This is a fantastic suggestion and we would love to integreate a more detailed discussion on possible avenues for implementation of an alignment strategy based on our theory in the final version of the paper. Some of our thoughts include;\n\n**E-RLHF Formulation.** As outlined in our general response, our proposed E-RLHF is inspired by the realization that the KL-term in the current prevalent alignment method (RLHF) may inadvertently preserve harmful responses when the input prompt $x$ itself is harmful. The RLHF objective aims to align with human preferences (as reflected by the reward term) while maintaining helpfulness (as reflected by the KL-term). To enhance safety while preserving these characteristics, we propose setting $p_{\\textrm{SFT}}(\\cdot)$ to a safe distribution. We believe that maintaining the mathematical reward-plus-KL formulation is vital, and currently, we do not see a clear pathway for other formulations to achieve these objectives simultaneously.\n\n**E-RLHF Implementation.** We are considering several alternative strategies to refine our implementation of E-RLHF. The first concerns the harmful prompt filtering step. Instead of our current approach, which involves prompting an LLM to assess whether an input prompt is harmful, a more straightforward method might involve sampling responses and labeling the prompt as harmful if the likelihood of a response being harmful surpasses a predefined threshold. Additionally, involving human annotators to manually design, and identify harmful prompts from existing alignment datasets could be beneficial. Furthermore, considering that the determination of whether content is harmful can vary based on different backgrounds and contexts, the filtering process could also be made adaptive to these conditions. This adaptability is particularly crucial if we are cognizant of the diverse applications of LLMs. By tailoring the filtering mechanisms to accommodate various contexts, we can enhance the safety of the responses generated by LLMs, ensuring they are appropriate and considerate across a spectrum of scenarios, which could improve the utility and acceptance in global applications. The second pertains to the safe concept replacement step. Rather than our current method of safe prefixing, one could involve human annotators to rewrite harmful prompts or prompt a LLM to decompose-and-replace harmful prompts. We opted for safe prefixing in our paper due to its simplicity, which helps avoid excessive computational demands and reduces the need for intensive human labor. We believe that with effective prompt engineering (e.g., curate the prompt in a similar fashion as demonstrated in Table 15 in [1]), this second step can be efficiently implemented using an LLM, which we aim to explore in future work.\n\nHowever, we want to emphasize that our approach despite its simplicity strikes a balance between computational feasibility and the goal of expanding the safety zone. We are delighted to report that this simple intervention performed exceptionally well against state-of-the-art jailbreaking attack benchmarks. This outcome not only underscores the viability of our proposal but also establishes a promising baseline. \n\nWe acknowledge that our approach to expanding the safety zone is just one of many potential strategies, and we are excited to see other researchers incorperate our insights into their alignment strategies. With the integration of additional safety alignment data and the implementation of more sophisticated strategies, our method holds the potential to deliver even more impressive results. \n\nWe would also be delighted to see our theoretical framework be applied in other domains as suggested by reviewer T1QG.\n\n*2. Eq (2): it is slightly confusing that in D_KL, the term only have $p_{LM}(x)$ and $p_{\\textrm{SFT}}(x)$, not $p_{LM}(e|x)$ and $p_{\\textrm{SFT}}(e|x)$.*\n\n- We apologize for any confusion caused by our notation. Throughout the paper, we denote the distribution over responses as $p_{LM}(q,c)=p_{LM}(x)$. With this notation, and incorporating a reward model $r(x,e)$, the RL fine-tuning can be expressed either as $\\mathbb E_{x\\sim\\mathcal D_s, e\\sim p_{LM}(\\cdot|x)}[r(x,e)-\\beta\\frac{\\log p_{LM}(e|x)}{\\log p_{\\textrm{SFT}}(e|x)}]$, or as $\\mathbb E_{x\\sim\\mathcal D_s}[\\mathbb E_{e\\sim p_{LM}(\\cdot|x)}[r(x,e)]-\\beta \\mathbb D_{\\textrm{KL}}(p_{LM}(x) || p_{\\textrm{SFT}}(x))]$. It is important to note that the expectation over $e$ is used for computing the reward $r(x,e)$, while the $\\mathbb D_{\\textrm{KL}}$ serves to regularize $p_{LM}(x)$, ensuring it does not deviate significantly from $p_{\\textrm{SFT}}(x)$. To avoid any further confusion,  we will clarify this point in our final draft by using the second equation.\n\n*3. I am not familiar with using the term \"explanation\" -- instead, I think using \"response\" is more common. At the first glance, it was hard to comprehend the meaning of the term*\n\n- We apologize for the confusion. We use the term \"explanation\" as a counterpart to \"concept\", based on the empirical observation that, in most jailbreaking attacks currently considered by the community, the adversary seeks instructions or explanations for a single harmful attempt. We appreciate your feedback and will incorporate it by resuming the use of \"response\" in our final draft to enhance readability and ensure our notation is easier to follow.\n\nWe hope these explanations address your concerns. Please let us know if you need further clarification, we would be happy to discuss further.\n\nReferences\n\n[1] Jailbreaking Black Box Large Language Models in Twenty Queries", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GQ24dG9ob0", "reviewer": "Reviewer_T1QG", "summary": "The paper addresses a very important issue of our time, the safety of LLMs. LLMs are already used in various applications and will be present in more applications to come, as e.g. Microsoft, Apple and Google are integrating LLMs in their applications and operating systems. Hence, the question on how to make the systems more robust against jailbreaking is a very important question.\n\nThis paper offers a statistical approach to this question and presents experiments. The mathematics is based PAC-Bayesian approach and builds upon a very nice formalization of splitting a prompt into queries and concepts.", "questions": "Could you please point out the connection between experiment and theory that I have seemed to miss.", "limitations": "I don't see any negative social impacts. As stated above, I would like to see a stronger connection between experiments and theory.", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The mathematics are presented very well. In particular, the intuitions and interpretations of the mathematical concepts are presented well, and can also be followed by readers, who are not able to follow all the details of every equation.\n\nOverall, I really appreciated the formal approach to modelling jailbreaking and the conclusions, that it is impossible to avoid jailbreaking.", "weaknesses": "There is a disconnect between the theory and the experiments. The definitions, theorems, etc. are all presented and motivated very nicely, but could also apply to any other mapping of a stochastic system. Oversimplified, one may state that there is a mapping from an input space into an output space, in which the output space can be divided into desired and undesired outputs. The goal is to reduce the space of undesired outputs. To make this a case specifically about LLMs, there needs to some form of conclusions that feed into the experiments. Unfortunately, I don't see this connection. The experiments use a systems prompt that is preprended to the actual prompt. I do not see, how this follows from the theory.\n\nThis is really unfortunate, because I would really like to see, how the experiments connect to the theory.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer T1QG,\n\nThanks for your thorough review! We are pleased to provide further explanations on our E-RLHF proposal, its theoretical foundations, and its specific relation to LLMs.\n\n*1.  The definitions, theorems, etc. are all presented and motivated very nicely, but could also apply to any other mapping of a stochastic system. ...To make this a case specifically about LLMs, there needs to some form of conclusions that feed into the experiments.*\n\nTo better address your concerns, we would like to clarify the **goals** we aim to achieve with our constructed framework to make it meaningful and robust. Our framework should:\n\n- Abstract the capability of an LLM to generalize on unseen but plausible prompts. **This property is particularly important as many representative jailbreak methods exploit this ability to circumvent the guardrail of existing LLMs.** This necessitates a clear distinction between \"plausible\" and \"seen\" prompts. Additionally, defining the concept of \"generalization\" is critical, yet inherently complex.\n- Model the capability of an adversary. For jailbreak attempts, formulating an adversary in a precise mathematical manner is challenging due to the discrete and unbounded nature of the operations performed on prompts.\n- Discriminate between harmful and non-harmful prompts. It is essential that statements on jailbreak focus exclusively on harmful prompts. However, the definition of \"harmful\" itself remains ambiguous and diffucult to mathematically formalize.\n\n\n**Framework**: Based on these points, we first assume each prompt can be decomposed into a query and a concept. This offers us the  opportunity to model generalization by assuming invariance on the concept, thereby allowing us to mathematically describe the capability of the adversary. Reflecting point (1), in Assumption 4.1, we assume that the domain of the LLM output distribution dependents solely on the concept and not the query. This is a crucial, yet we argue, realistic assumption. We argue that this abstraction is **distinctive of LLMs** and is not easily applicable to generic stochastic systems, as not all mappings within such systems will exhibit the property described in point (1). This characteristic further impacts our E-RLHF proposal, which relies on this essential property to enhance safety. Without this property, the performance of those mappings on **unseen harmful prompts** found by the adversary would remain unaffected, while for LLMs the corresponding safety zone will be enlarged. That said, we are eager to explore whether our proposed framework can be effectively applied to other generic stochastic systems. This extension of our research could potentially broaden the applicability and impact of our findings, offering valuable insights into a wider range of systems.\n\n*2. The experiments use a systems prompt that is preprended to the actual prompt. I do not see, how this follows from the theory. Could you please point out the connection between experiment and theory that I have seemed to miss.*\n\n**Specific implementation is driven by our framework and works**: We further want to convince you that the incorporation of a prepended safe prefix represents an intuitive yet effective implementation of our E-RLHF proposal. Theorem 2 elucidates the relationship between the size of the safety zone and the ability of an adversary to successfully compromise the model. This insight directly informs our practical approach to enlarging the safety zone. Our analysis reveals that the prevalent RL fine-tuning objective, especially its KL-term, inadvertently contributes to vulnerabilities due to the unsafe nature of $p_{\\textrm{SFT}}(x)$ when $x$ itself is harmful. By substituting harmful concepts in $x$ with safe alternatives, we can effectively mitigate this issue. We have elaborated several variations of its implementation in discussions with reviewer qvRD. In the experimental section of our study, we take a simple, computationally efficient approach by introducing a safe prefix to $x$. \n\nWe want to emphasize the significance of our experimental result. We achieved significant safety  improvement across all tasks in HarmBench with only one generic intervention (one safe prompt). The dataset size and harmful diversity coverage is the largest compared to previous papers (we refer to the general response and section A.2, section B.4 and Table 5 of [1] as reference).\n\nWe found that our approach already provides an improvement in safety. We think that with better implementations, a larger alignment dataset, and improved optimization techniques (e.g., with PPO), the enhancement in safety could be significantly increased. \n\n\nPlease let us know if the above explanations resolve your concern on how we motivated our framework specifically for LLMs, and how our E-RLHF proposal is linked to the theoretical discoveries.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F4XSfhiEpz", "reviewer": "Reviewer_UXGw", "summary": "The paper presents a statistical framework that provides a theoretical analysis of the jailbreaking problem in language models. The authors first examine the PAC-Bayesian bound to demonstrate that there is a non-trivial probability for LLMs to mimic harmful behavior if such information is present in the pre-training corpus. They then theoretically prove that a jailbreaking prompt can be found for such pretrained LMs and propose mitigating the jailbreak issue by modifying the harmful prompts.", "questions": "Could you provide more details on the MT-bench's performance? For example, you have an ablation study on different safety prefixes' effects in Table 4. What about their impact on MT-bench?\n\nIn line 327 and the ablation study in the appendix, you show that non-harmful prompts should not be modified (otherwise, the model performs much worse). Could you relate this phenomenon to the safe/harmful zone or your theoretical analysis? I don't see why adding a safe prefix to a non-harmful prompt negatively impacts the safe/harmful zone, and under your assumption, this should not lead to worse behavior.", "limitations": "No concerns on the limitations", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper's formulation and assumptions are clear and well-motivated. The proofs seem comprehensive and support the claims well. However, I don't have much background and PAC-Bayesian theory, so it's hard for me to verify if (1) the proofs are all rigorous (2) the proposed framework is truly novel instead of simple application of existing theorems. \n\nThe experimental results from the proposed E-RLHF improve upon DPO-based fine-tuning.", "weaknesses": "I find the connection between the theoretical part and section 5 (E-RLHF for actual experiment) quite handwavy. Expanding the safety zone is the claimed goal, but the proposed solution (through injecting safe concepts in a harmful prompt) seems really hacky. For example, as shown in Table 4, using safe prompt (4) gives much better results than safe prompt (1) for no reason. Overall, this proposed method seems like a small trick that the authors try to incorporate into the paper just to show some empirical value of their theoretical analysis. I also find that with such modifications, the MT-Bench score (briefly mentioned in line 355) is lower under E-DPO than DPO.\n\nLastly, this might be a biased opinion (that's why I didn't factor this part into my scoring decision and still gave a marginally above accept score): I feel that after all the theoretical proofs (despite their elegance), the conclusion (LLMs will mimic toxic behavior if such toxic content is present in the training corpus, jailbreak is unpreventable under reasonable assumptions, and by expanding the safe zone we reduce the probability of jailbreaking) is very intuitive and does not provide much additional insight into the problem. Maybe the mathematical formalization of the jailbreak problem itself is meaningful, and I will let the other reviewers judge the novelty of such a framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer UXGw,\n\nThanks for your comprehensive review! Here are our responses to your concerns.\n- **(1) The novelty of our framework.** To the best of our knowledge, we are the first to offer a theoretical analysis on jailbreaking from a statistical perspective. To tackle this problem and provide insights, we addressed the following unique challenges associated with LLMs: \n    - Abstract the capability of an LLM to generalize on unseen but plausible prompts. **This property is particularly important since many representative jailbreak methods exploit this to bypass the guardrails of existing LLMs.** \n    - Model the capability of an adversary. For jailbreak attempts, it is hard to formulate an adversary in an mathematical way, since the operations performed on prompts are discrete and unbounded. \n    - Discriminate harmful prompts versus non-harmful prompts. The statements should be connected to harmful prompts only, while the definition of \"harmful\" itself remains lacking.\n\nOur framework overcomes these problems and provides theoretical evidence on the difficulty and impossibility of completely avoiding jailbreaking events.\n- **(2) Connection from our theory to E-RLHF.** Under our framework, we identified a drawback in the RL fine-tuning objective, particularly the small safety zone it creates. We propose E-RLHF to address this issue by **replacing** harmful concepts in prompts $x$ with safe ones. We want to clarify that the safe prefixing strategy is **not injecting a safe concept**, but rather a **simple yet effective implementation of harmful concept replacement** in harmful prompts. We agree that more sophisticated methods could further enhance safety, and our experimental results suggest that even our simple implementation significantly improves safety. Thus, we argue our experimental results should not be regarded as negative but instead in line with Occam's razor as a positive result.\n- **(3) Sensitivity of the safe prefix and MT-Bench scores.** As mentioned in response part (1), our safe prefixing is a simple implementation of replacing harmful concepts with safe ones. Under our framework, different safe prefixes can induce different safe concepts, and different $p_{\\textrm{SFT}}(x_s)$. This explains the sensitivity to the choice of safe prefix. Prompt sensitivity has also been observed in previous works ([4][5]), and we believe safety could be further improved with prompt engineering to find better safe prefixes. Regarding the MT-Bench score, we acknowledge that E-RLHF leads to a slightly lower score than RLHF. However, **the score achieved by E-RLHF is still higher than that of the SFT model**, indicating that we do not sacrifice utility for safety. There are few LLM-tuning-based defenses against jailbreak attacks, especially those improving the RLHF step. For instance, R2D2 from HarmBench [1] uses a SFT strategy, resulting in a drop in the MT-Bench score from 7.3 to 6.0. The tension between safety and utility has been noted in previous works (e.g., [2]), and our proposal does not sacrifice utility to achieve better safety.\n- **(4) Insight provided by our framework and experiments.** We aim to establish a framework that explains the jailbreak phenomenon theoretically. We argue that the intuitive results, showing that LLMs can be jailbroken both after the pretraining stage and after the current safety alignment stage with RLHF, are non-trivial, important, and counterintuitive. RLHF is the default optimization strategy for LLM alignment, and we identify a fundamental drawback in its mathematical objective. Based on this insight, we offer a plausible solution, E-RLHF, and demonstrate its effectiveness through a simple yet effective implementation.\n- **(5) Additional MT-Bench results.** We are in the process of requesting credits with the OpenAI API and plan to initiate the benchmark test as soon as possible.\n- **(6) Relating ablation study on non-harmful prompts with theoretical analysis.** Under our framework, the harmful and safety zones are defined with respect to **a single concept**, meaning modifications to non-harmful prompts should not impact performance on harmful prompts. However, we argue that the ablation phenomenon occurs due to the following reasons. Firstly, we do not model **correlations and interactions between concepts**. Each concept is considered independent, but in reality, generalization on one concept influences LLM performance on other prompts. For non-harmful prompts, appending the safe prefix may hinder optimization, affecting learning on harmful prompts. Modeling correlations and interactions between different concepts is highly complex and is left for future exploration. We will add this discussion to the limitations section of our final draft. Secondly, we point out that this statement also holds true for normal RL fine-tuning. Analysis on it (e.g., the motivation of DPO [3]) suggests that the output of a converged LLM for any prompt $x$ depends only on the reward model $r(x,e)$ and the initial distribution $p_{\\textrm{SFT}}(x)$. However, due to the discrepancy between the optimal solution and the converged LLM in practice, the same reward model and LLM initialization can lead to models with significantly different performances.\n\nLastly, we want to emphasize the significance of our experimental result. We achieve significant safety improvement across all categories of harm on 2 benchmarks. The dataset size and harm diversity coverage is the largest compared to previous papers (we refer to our general response, and discussions in [1] as reference).\n\nReferences\n\n[1] HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal\n\n[2] Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback\n\n[3] Direct Preference Optimization: Your Language Model is Secretly a Reward Model\n\n[4] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models\n\n[5] Large Language Models as Optimizers", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UcMXMxQjpL", "reviewer": "Reviewer_sD4q", "summary": "The paper introduces a new theoretical framework that views prompts as combinations of concepts and queries, allowing for a detailed analysis of how and why LLMs can be manipulated into producing unsafe responses. It proposes a statistical metric for alignment, focusing on quantifying the safety of the model's outputs.\n\nThe study introduces an improved alignment strategy, Enhanced Reinforcement Learning from Human Feedback (E-RLHF), which modifies existing RLHF methods to increase the likelihood of safe responses from LLMs without additional training costs.\nThrough experiments using standard benchmarks like AdvBench and HarmBench, the paper demonstrates that E-RLHF significantly outperforms traditional RLHF in resisting jailbreaking attempts, reducing the models' vulnerability to producing harmful outputs.", "questions": "The two datasets used in the work contain harmful requests about various concepts (e.g. drug, weapon). Have you tried to analyze the performance of E-RLHF on different concepts?", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "The paper offers a unique statistical framework that conceptualizes input prompts into concepts and queries. This approach allows for a nuanced understanding of how LLMs process inputs and why they might generate unsafe outputs even under alignment efforts. \n\nThe introduction of E-RLHF  as an alignment strategy that doesn't require additional computational resources is a major strength. \n\nThe empirical tests using established benchmarks such as AdvBench and HarmBench provide solid evidence that E-RLHF can significantly reduce the Attack Success Rate (ASR) compared to traditional RLHF.", "weaknesses": "While E-RLHF improves alignment in controlled experimental conditions, its effectiveness in more complex conversational scenarios, where inputs can evolve over a series of interactions, remains untested.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer sD4q,\n\nThanks for your detailed review! Here are our responses to your concerns.\n\n*1. ...(E-RLHF's) effectiveness in more complex conversational scenarios, where inputs can evolve over a series of interactions, remains untested.*\n\n- That is a great point that we had discussed internally too. We fully acknowledge that our theoretical framework does not cover multi-round chat jailbreak senarios. We have discussed this limitation in our paper (in lines 376-383, point (2)). Apart from the fact that we do currently not know how to extend the framework, we thought the exclusion of multi-interaction scenarios is justified because emirical work has not yet produced any (or SOTA) jailbreaking  attacks and evaluations:\n  - (1) **No available evalution benchmarks for multi-chat.** As discussed in the HarmBench paper [3], jailbreak evaluation varies significantly across studies, making it difficult to compare the efficacy of different methods. Therefore, we chose to follow HarmBench as our evaluation protocol, as it is designed to best reflect LLM safety by ensuring fair comparison and providing diverse jailbreak adversaries. Multi-interaction benchmarks are not available as of now.  \n  - (2) **Lack of empirical methods.** The only works that use multi-round interaction to jailbreak LLMs that we are aware of are [1] and [2]. However, both were published in February and April of 2024 hence were too concurrent to integrate in our work. Additionally, both projects do not provide source code to reproduce their results, making it challenging for us to include their methods in our empirical evaluations.\n\nThat being said, we believe that integrating multi-interaction attacks into our framework is crucial for future research. From a theoretical perspective, we hypothesize that if even a single attack poses a non-negligible risk, a multi-interaction attack will likely be even more challenging to defend against. Consequently, research questions would focus on how rapidly success rates escalate with the number of interactions. This exploration will provide deeper insights into the dynamics of jailbreak attack and inform more robust defensive strategies.\n\n*2. The two datasets used in the work contain harmful requests about various concepts (e.g. drug, weapon). Have you tried to analyze the performance of E-RLHF on different concepts?*\n- Our empirical evaluations primarily concentrate on harmful behavior. As highlighted in the general response, HarmBench assesses the safety of LLMs across a variety of harmful behaviors. While alignment extends beyond mere safety to include aspects such as ethical behavior, to our knowledge, we currently lack benchmarks for testing these broader criteria. We are happy to keep discussing on this topic and would welcome any specific proposals or suggestions for additional benchmarks that could enhance our understanding and assessment of ethical alignment in LLMs.\n\nWe hope these explanations address your questions and concerns. Please let us know if you need further clarification, we would be delighted to discuss further.\n\nReferences\n\n[1] Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack\n\n[2] Leveraging the Context through Multi-Round Interactions for Jailbreaking Attacks\n\n[3] HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for taking the time to review our paper and providing valuable feedback. We appreciate the recognition of our established theoretical framework and the acknowledgment of the nuanced formulation of our results from all reviewers. We had to traverse several conceptual steps to overcome the challenge that the adversary can modify the input prompt in an unbounded, unconstrained way. Our key proposal on the decomposition of input prompts into the (query, concept) pair has enabled us to formalize the adversary mathematically. Within our framework, we offer a clear distinction between harmful and non-harmful prompts, elucidate the generalization capabilities of LLMs on unseen prompts, and demonstrate the difficulty and impossibility of defending against jailbreak attempts by presenting a novel statistical bound.

The reviewers have raised several important concerns that we are eager to address. 

1. The connection between our theory and the proposed experimental strategy (T1QG, UXGw).
2. The simplicity and feasibility of our implementation (qvRD, UXGw).
3. The effectiveness of our E-RLHF proposal (sD4q).

We appreciate these insightful comments and are pleased to provide detailed clarifications to each of these points below.

**Connection of our framework to experiments.** We identified a significant limitation for safety inherent in the widely adopted alignment strategy, RLHF. Our analysis has traced the problem back to the KL term, which inadvertently ensures that even the optimal solution retains all harmful responses in the LLM's output. To address this, we have introduced an innovative modification to the KL term concerning harmful prompts. Our approach involves filtering out harmful prompts and replacing them with safer alternatives. We believe it is the most natural and effective solution to mitigate the identified risk. It is important to emphasize that our E-RLHF algorithm is inspired and fundamentally driven by our theoretical results.

**The safe prefix implementation.** Upon introducing E-RLHF, our next challenge was to devise an effective implementation. While we acknowledge the existence of more sophisticated methods (as discussed with reviwer qvRD), we opted for **a simple approach: appending a safe prefix to the harmful prompts.** We find this simplicity particularly compelling. Remarkably, with our strategy applied with a limited alignment dataset and optimized using DPO, we achieve significant improvements in safety. This opens the door to find more nuanced, improved implementations, which have the potential to improve results further.

**Soundness and effectivenss of our E-RLHF proposal.** We used a **recently released jailbreak benchmark: HarmBench**. HarmBench assesses LLM safety using a suite of the most advanced jailbreak adversaries, and scores safety across diverse harm categories including but not limited to Cybercrime & Unauthorized Intrusion, Chemical & Biological Weapons/Drugs, Copyright Violations, Misinformation & Disinformation, Harassment & Bullying, Illegal Activities, and General Harm. The HarmBench dataset comprises 400 prompts. Additionally, we included the AdvBench first 100 subset, a dataset frequently used in previous research. We include results from both datasets to ensure completeness and robustness in our evaluation. Our method has shown significant improvements on both benchmarks and across all categories of harm without any task specific adaptation.

We would be eager to summarize our contributions again as follows.

- **A new Theoretical Framework.** We build a novel framework to analyze LLM jailbreaking. This framework addresses challenges such as abstracting the LLM's generalization capability on unseen prompts, mathematically defining the adversary, and distinguishing between harmful and non-harmful prompts.
- **Theoretical evidence on difficulty and impossibility of avoiding jailbreak attacks.** Following our nuance construction of the framework, we offer theoretical evidence that highlights the inherent difficulties and impossibilities of completely avoiding LLM jailbreak attacks.
- **Insight into a RLHF Objective Drawback.** We provide a critical insight into the drawback of the current RLHF objective for safety, which exacerbates the problem for post-alignment LLMs.
- **Theory-Inspired Proposed Solution: E-RLHF.** In response to this insight, we propose an algorithmic framework, E-RLHF, designed to address and mitigate this drawback.
- **Effective Implementation and Results.** We implement a **simple yet effective** version of E-RLHF and demonstrate its superior safety performance **across a suite of diverse jailbreak adversaries without task specific adaptations**.


Finally, we want to take the opportunity to emphasize the importance of our work. 
- The adaptability of our E-RLHF. Our approach can be tailored to align with various cultural norms and contexts, such as in educational, medical, and legal settings. This customization could be achieved by filtering harmful prompts based on specific backgrounds and applying our safe prompt replacement strategy.
- Defense strategies are crucial for LLM deployment, and our insights underscore the vulnerability to (even) single interaction attacks. We hope this insight will spark further research. If perfect defense is impossible, we need to rethink how applications should be designed acknowledging these limitations. We hope our research can foster a more robust and thoughtful approach to the deployment of LLMs.


We sincerely thank all reviewers for their time on our rebuttal. We hope our response addresses your concerns and highlights the significance of our contributions, both theoretically and experimentally. We would be delighted to discuss further in the next week.

Meta-review: The paper provides a theoretical analysis of jailbreaking for LLMs using PAC-Bayesian bound. They found that the jailbreaking is unpreventable. 

Strength: 
S1. The paper proposes a sound theoretical framework to analyze the jailbreaking behavior of LLMs. 
S2. the paper proposes an improved alignment training method E-RLHF.

Weakness: 
W1. There is disconnection between the theoretical analysis and experimental validation. 

Overall, the theoretical framework proposed in this paper to analyze LLM jailbreaking is an important contribution.
Decision: Accept (poster)