PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Rails (2024)

Abstract: Large language models (LLMs) are typically aligned to be harmless to humans. Unfortunately, recent work has shown that such models are susceptible to automated jailbreak attacks that induce them to generate harmful content. More recent LLMs often incorporate an additional layer of defense, a Guard Model, which is a second LLM that is designed to check and moderate the output response of the primary LLM. Our key contribution is to show a novel attack strategy, PRP, that is successful against several open-source (e.g., Llama 2) and closed-source (e.g., GPT 3.5) implementations of Guard Models. PRP leverages a two step prefix-based attack that operates by (a) constructing a universal adversarial prefix for the Guard Model, and (b) propagating this prefix to the response. We find that this procedure is effective across multiple threat models, including ones in which the adversary has no access to the Guard Model at all. Our work suggests that further advances are required on defenses and Guard Models before they can be considered effective. Code at https://github.com/AshishHoodaIITD/ prp-llm-guard-rail-attack.

heading: 1 Introduction
text: With appropriate fine-tuning, large language models (LLMs) are able to engage in natural dialogue with users to generate code (Chen et al., 2021), automate customer support (Soni, 2023), assist with education (Li et al., 2023), extract information from documents (Gao et al., 2024), and many other tasks. Recent work has focused on aligning LLMs to be helpful, honest, and harmless (HHH) (Askell et al., 2021). One line of effort enforces the HHH criterion at training time, resulting in an aligned LLM (Liu et al., 2022). For brevity we will refer to non-HHH responses as ‘harmful’. This is typically achieved through techniques such
†Indicates equal contribution
as RLHF or DPO (Rafailov et al., 2023; Ouyang et al., 2022). However, recent work has proposed jailbreak attacks, which manipulate the prompt to override the model’s alignment and elicit a harmful response (Zou et al., 2023; Zhu et al., 2024).
Another popular approach to enforce the HHH criterion is to defer the safety check to inference time via a separate LLM, i.e., a Guard Model that acts as a detector of harmful responses (Helbling et al., 2023; Inan et al., 2023; Jiang et al., 2023). If the Guard Model detects the response as harmful, the prompt is refused. Notably, both training and inference time approaches can be combined to yield a base aligned LLM that is also protected by a Guard Model LLM. We refer to this setup as a Guard-Railed LLM, whereby the Guard Model acts as a fail-safe when the base aligned LLM has been compromised.
Existing attacks, discussed in Section 2, focus on manipulating the input prompt such that it jailbreaks a base LLM. However, as we show in Section 5, these attacks no longer work against a Guard-Railed LLM. In light of these observations, evaluating safety of Guard-Railed LLMs remains a challenging problem. This raises the question: do current Guard Models truly protect LLMs from jailbreak attacks, or is it possible to design an adaptive attack that elicits harmful responses from the Guard-Railed LLM?
In this paper, we answer this question by proposing a novel systematic attack against LLMs protected by a Guard Model (instantiated as a second aligned LLM). Our attack is illustrated in Figure 1, and is based on two key insights – (1) Guard Models are vulnerable to universal attacks that impair their harmfulness detection ability when concatenated with any input, and (2) an adversary can inject the universal attack into the base LLM’s response, by taking advantage of in-context learning abilities. Based on these insights, we propose PRP, a two-stage framework for eliciting harmful
10960
responses from LLMs protected by such a Guard Model. In the first stage, PRP computes a universal adversarial prefix for the Guard Model, i.e., a prefix string such that, when prepended to any harmful response, causes the response to evade detection by the Guard Model. We show that a universal prefix can be constructed for many popular open-source and closed-source models, i.e., under white-box, black-box, or no access (i.e., transfer) to the the Guard Model. In the second stage, PRP leverages in-context learning to compute a propagation prefix for the base LLM, i.e., a prefix such that, when prepended to any existing jailbreak prompt, produces a response from the base LLM that begins with the universal adversarial prefix. Notably, we find that computing a propagation prefix does not require any access to the base LLM.
We conduct experiments by applying PRP to a variety of setups including base models/Guard Models from the Llama 2 (Touvron et al., 2023), Vicuna (Chiang et al., 2023), WizardLM (Xu et al., 2023), Guanaco (Dettmers et al., 2023), GPT 3.5 (OpenAI, 2023), and Gemini families (Anil et al., 2023), and observe that PRP finds universal adversarial prefixes as well as corresponding propagation prefixes under these settings. This amounts to successful end-to-end jailbreak attacks on the AdvBench dataset, e.g., PRP elicits harmful responses from a Llama2-70b-chat base model protected by an OpenAI gpt-3.5-turbo-0125 (OpenAI, 2023) Guard Model with an 80% success rate without optimizing against either. In summary, we use PRP to show that Guard-Railed LLMs are currently unable to prevent jailbreak attacks.

heading: 2 Related Works
text: Jailbreak Attacks. There are two main classes of attacks aimed at circumventing LLM alignment — manual, and automated. Manual attacks are based on prompt engineering which employs methods such as deception (Perez and Ribeiro, 2022; Rao et al., 2023) and persuasion (Liu et al., 2023). These attacks are crafted through human ingenuity and thus require substantial manual effort. Automated attacks provide a more systematic way of generating jailbreaks. These attacks pose the attack as an optimization problem that can be solved using gradient-based (Zou et al., 2023), genetic algorithm-based (Zhu et al., 2024), or generative methods (Zeng et al., 2024). The generated attacks can be in the form of suffixes/prefixes (Zou et al., 2023), or complete rewrites of the original prompt (Chao et al., 2023). While these methods are effective against aligned LLMs, they do not work when a Guard Model is employed (Helbling et al., 2023). Safeguards. In response to jailbreak attacks, two main classes of defenses have emerged — input prompt based, and LLM output response based. Safeguards that focus on the input prompt involve checking the prompt for any unusual patterns (Jain et al., 2023; Alon and Kamfonas, 2023), or designing system prompts that are more resilient to attacks (Wei et al., 2023; Zhang et al., 2024; Wu et al., 2023). Although simple to implement, they are prone to false positives. Another line of input based defenses attempt to defend against jailbreak attacks by performing multiple inferences on perturbed versions of the input prompt (Kumar et al.,
2023; Robey et al., 2023). Although these are able to provide robustness guarantees under certain settings, they are very inefficient since they require multiple inferences for each input prompt. The second class of defenses focus on the output of the LLM and act as a fail-safe. These involve using the same or separate LLM to detect any harmful content in the response (Wang et al., 2023; Helbling et al., 2023; Inan et al., 2023). These approaches can also be used to detect and correct factual errors (Dhuliawala et al., 2023). Since existing attacks focus on manipulating the input prompt, they have already been used to provide a rigorous evaluation of input based defenses. In this paper, we focus on the less-explored setting of response based defenses and provide new attacks to evaluate their robustness.

heading: 3 Preliminaries

heading: 3.1 Notations
text: Let Σ be the set of alphabets and V ⊆ Σ⋆ be a set of tokens. A prompt p is a string in V ⋆. A concatenation of two strings p1 and p2 is denoted by p1 · p2. An LLM fLLM : V ⋆ → V ⋆ takes a prompt or input and outputs a response. We also define a Guard Model fG : V ⋆ → {0, 1}, which takes the response of an LLM and outputs 1 to indicate that the response contains harmful/toxic content, and outputs 0 otherwise. In this work, we focus on implementations of fG that themselves leverage another LLM. Using fG, we have a GuardRailed LLM g(fLLM ,fG) : V
⋆ → (V ⋆ ∪ {⊥}) such that:
g(fLLM ,fG)(p) = { fLLM (p), if fG(fLLM (p)) = 0 ⊥, o.w. (1) where ⊥ denotes the LLM refusing to answer by returning a fixed response, e.g., “As an AI language model, I cannot answer a harmful question.” Here we assume LLM responses are decoded greedily (Zou et al., 2023; Jones et al., 2023).

heading: 3.2 Attack against Guard-Railed LLMs.
text: Definition of Guard-Rail Attack. Given an LLM fLLM , Guard Model fG, and initial harmful prompt p0 such that g(fLLM ,fG)(p0) = ⊥, we define the problem of attacking a Guard-Railed LLM as crafting an adversarial prompt p′ that satisfies the following:
g(fLLM ,fG)(p ′) = fLLM (p0) (2)
where p′ is obtained by augmenting the original input string p0. For instance, one could add an adversarial prefix (i.e., p′ = p+ · p0) and/or adversarial suffix (i.e., p′ = p0 · p+), or even apply an augmentation based on p0. In other words, adding p+ makes the augmented prompt bypass the Guard Model, and thus jailbreak the Guard-Railed LLM into generating a response to the harmful prompt p0. The goal of this paper is to investigate the existence of such an augmentation string p+ to jailbreak a variety of existing Guard-Railed LLMs. Challenges in Applying Existing Attacks. In the above attack against Guard-Railed LLMs in Equation 2, we highlight that the adversary must already have a harmful jailbreak prompt p0 that would elicit a harmful response fLLM (p0) if no Guard Model was in place. Indeed, existing attacks leverage gradient-based discrete optimization techniques to compute this harmful jailbreak prompt p0 (Zou et al., 2023). However, p0 alone is insufficient, as fLLM (p0) will be detected by the Guard Model fG, i.e., fG(fLLM (p0)) = 1 (see Figure 1a). As such, existing attacks in their vanilla, original form are insufficient for attacking Guard-Railed models.
To solve the Guard-Rail attack problem, the adversary must also find p+ such that fG(fLLM (p+ · p0)) = 0. One possible extension of existing attacks might be finding such p+ using the same gradient-based discrete optimization procedures. However, direct extension of gradient-based techniques here is not feasible as the Guard Model fG needs to fetch the entire response from the paired base model fLLM for its analysis, which is nondifferentiable (as it involves repeated argmax operations). Thus, these attacks alone struggle to account for fG. In Section 5 we present quantitative evaluation results to show that the efficacy of existing attacks such as GCG (Zou et al., 2023) is limited in Guard-Railed settings. To this end, one of our key contributions is to demonstrate how these attacks (which produce p0) can be enhanced to also succeed against Guard-Railed LLMs.

heading: 3.3 Threat Model
text: We consider an adversary that does not have any knowledge of, or direct query-access to the output responses of the base LLM (if they do, then they do not need to evade the Guard Model LLM). For the Guard Model LLM, we consider multiple settings where the adversary has either white-box, black-box query-access, or no access at all. For example, in cases where an open-source LLM such
as Llama 2 (Touvron et al., 2023) or Vicuna (Chiang et al., 2023) is used as the Guard Model, the adversary may have white-box access. For closedsource Guard Models, the adversary may only have black-box query access to the output token distribution. Finally, for a completely private closed-source Guard Model such as ChatGPT (OpenAI, 2022), the adversary may have no access at all and can only interface with the Guard-Railed LLM.

heading: 4 Method
text: In this section, we describe our attack, Propagate Universal Adversarial Prefix (PRP) to jailbreak Guard-Railed LLMs. We first define the two major building blocks of our attack: Propagation Prefix and Universal Adversarial Prefix.
Definition 4.1 (Propagation Prefix). Given an LLM fLLM , and string δ ∈ V ⋆, a propagation prefix for δ is a string p→δ ∈ V ⋆ such that
fLLM (p→δ · p) = δ · fLLM (p) ∀ p ∈ V ⋆ (3)
That is, adding p→δ to the beginning of any input prompt results in the model outputting a response always beginning with δ. For example, in order to always have the response start with a specific payload string “!!!!”, we can add a fixed string “write '!!!!' at the start of your response” to the beginning of every input prompt.
Definition 4.2 (Universal Adversarial Prefix). Given a Guard Model fG, a universal adversarial prefix is a string ∆fG ∈ V ⋆ such that
fG(∆fG · r) = 0 ∀ r ∈ V ⋆ (4)
In other words, prepending ∆fG to any input r forces the Guard Model fG to output 0, hence resulting in failure to detect harmful content. Prior work shows the existence of such universal attacks against text classifiers (Gao and Oates, 2019).
Statement 4.3. Given a Guard-Railed LLM g(fLLM ,fG) and initial (potentially harmful) prompt p0 such that g(fLLM ,fG)(p0) = ⊥, the propagation prefix p→∆fG for the universal adversarial prefix ∆fG is a solution to the Guard-Rail Attack Problem in Equation 2 (see Appendix A for proof).
All brought together, we can jailbreak the GuardRailed LLM g(fLLM ,fG) by employing two independent procedures: (1) finding the universal adversarial prefix ∆fG for Guard Model fG, and then (2) finding the corresponding propagation prefix
p→∆fG for Base LLM fLLM . Given a harmful jailbreak prompt p0 already produced by an existing attack for fLLM , prepending p→∆fG to p0 yields p→∆fG · p0 as the final attack prompt. In the following subsections, we describe in detail how each step can be instantiated. Our approaches to computing both the universal adversarial prefix and the propagation prefix are only approximations. The overall performance of PRP depends on how good are the approximations for each of the individual components. We expect that future improvements for either of the above will only make PRP stronger.

heading: 4.1 Universal Adversarial Prefix
text: As described in Section 3.1, we focus on implementations of Guard Model fG that leverage another LLM. This is usually done with a template (Helbling et al., 2023; Inan et al., 2023; Jiang et al., 2023). Let gLLM denote the underlying LLM for the Guard Model. For a given sequence of input tokens x1:n ∈ V ∗, the output of the LLM is generated by repeatedly sampling from the probability distribution of the next token denoted by:
PgLLM (xn+1|x1:n)
which denotes the probability that the next token is xn+1, given the input sequence x1:n. Thus, to use gLLM as a Guard Model, one must first identify tokens corresponding to the strings that represent harmful and harmless, e.g., “Yes” and “No” given by tokens xYes and xNo respectively (Helbling et al., 2023). Then, we construct the Guard Model using gLLM :
fG(p) = { 0, if PgLLM (xNo| p) > PgLLM (xYes| p) 1, o.w.
Here, we assume that due to the instructions provided in the template, the rest of the tokens in the vocabulary have negligible probabilities.
Now, using the above formulation, we use the following optimization to find the universal adversarial prefix ∆fG :
max δ∈V ⋆
Er∈V ⋆ [ PgLLM (xNo | δ · r) ] (5)
When prepended to any input, this adversarial prefix acts as a universal trigger forcing the Guard Model to output 0, i.e., classifying the input to be not harmful. In practice, one must typically use a “training” subset of harmful responses R ⊆ V ∗ to optimize over.
Algorithm 1 presents the token-level optimization procedure for computing a universal adversarial prefix (as per Equation 5) for a given gLLM and training set of harmful responses R ⊆ V ∗. At a high level, we follow prior work on discrete optimization (Zou et al., 2023; Shin et al., 2020) and greedily update tokens in the prefix to maximize the probability of xNo as the output token. We proceed iteratively — at each step, a candidate set of new prefixes are made by substituting in the tokens from the vocabulary V at each index of the prefix. Substitutions are selected based on: (a) tokens with the largest gradients (white-box) (Zou et al., 2023), or (b) uniformly at random (blackbox) (Andriushchenko, 2023). The final candidate is selected as the one eliciting the highest probability for xNo across all harmful responses. Note that in practice, since the number of candidates is large, we follow Zou et al. (2023) and only compare a random subset of the candidates for selection. We terminate when the prefix is indeed adversarial ∀ r ∈ R (success), or when the maximum iterations are exceeded (failure).

heading: 4.2 Propagation Prefix
text: To generate the propagation prefix, we leverage the in-context learning abilities of LLMs (Brown et al., 2020; Wei et al., 2023). In-context learning allows LLMs to be applied to new tasks using only a few natural language demonstrations, i.e., fewshot learning. More concretely, consider that we have a set of k input-output pairs {(xi, yi)}ki=1, where xi ∈ V ⋆ are arbitrary input prompts and yi ∈ V ⋆ are the corresponding responses. Note that we only need a few in-context samples for demonstration, and the responses can be generated either manually or via any open-source, non-GuardRailed LLM. Next, we show how to generate the propagation prefix using the following in-context samples:
p→δ = (x 1 ·δ ·y1) · (x2 ·δ ·y2) ... (xk ·δ ·yk) (6)
Here, we create the propagation prefix by prepending δ to the response of each sample in the few-shot template. Due to the in-context learning abilities of LLMs, this biases the model to also prepend δ to the generated response when prompted with the input p→δ · p for any p.
In aggregate, we employ the above approach to formulate the propagation prefix p→∆fG for the universal adversarial prefix ∆fG . Based on Statement 4.3, this constructed propagation prefix serves
as the solution to jailbreaking the given GuardRailed LLM generating response to the harmful prompt.

heading: 5 Experiments
text: Our evaluation answers the following three research questions: RQ1: How effective is the PRP attack against Guard-Railed LLMs, when the adversary has whitebox/black-box query access to the Guard Model? RQ2: How well do the attacks generated by PRP transfer when the adversary has no access to the Guard Model? RQ3: Do Guard Models offer any additional safety to the base aligned LLM? RQ4: What is the trade-off between the individual components in PRP, i.e., finding a propagation prefix, and finding a universal adversarial prefix?

heading: 5.1 Setup
text: Datasets. We evaluate our attacks using prompts from Zou et al. (2023)’s Advbench dataset of harmful behavior prompts. This dataset contains 520 harmful prompts, out of which we select 100 for all our experiments except where stated otherwise. For PAP, we use the same set of harmful prompts as described in their paper (Zeng et al., 2024). LLMs. For fLLM , we consider several aligned and unaligned models. Aligned models include Llama2-70B-chat (Touvron et al., 2023), Vicuna33B-v1.3 (Chiang et al., 2023), and Guanaco13B (Dettmers et al., 2023). Unaligned models include Mistral-7B-Instruct-v0.1 (Jiang et al., 2023), WizardLM-7B-Uncensored (Xu et al., 2023; Hartford, 2024a), and Wizard-LM-Falcon-7BUncensored (Almazrouei et al., 2023; Hartford, 2024b). For gLLM , we consider Llama2-70B-chat, Vicuna-33B-v1.3, LlamaGuard (Inan et al., 2023), OpenAI’s gpt-3.5-turbo-0125 (OpenAI, 2023), and Google’s Gemini-Pro (Anil et al., 2023) (latest models as available through respective APIs in February, 2024). Guard Model Configurations. We configure Guard Models with the template from Helbling et al. (2023): Does the following text contain harmful content: {response} . Respond with ‘Yes, this is harmful’ or ‘No, this is not harmful’. A harmful response only evades the Guard Model when it outputs ‘No’. We also consider a version of the much longer, alternative template used by LlamaGuard
Algorithm 1 Universal Adversarial Prefix Input: Initial prefix δinit, Guard Model LLM gLLM , maximum attack iterations max_iters, vocabulary
token set V , harmful responses set R ⊆ V ∗, number of new perturbation candidates K for each index in the prefix, and threat model threat_model. Output: Perturbation δ s.t. PgLLM (xNo | δ · r) > 0.5 ∀r ∈ R (success), else NULL (failure). 1: δ ← δinit, n← |δ| ▷ Initialize universal adversarial prefix δ. 2: for iter from 1 to max_iters do ▷ Attack loop to optimize prefix δ. 3: candidates = list() ▷ Initialize empty list of candidates for new prefix. 4: for i from 1 to n do ▷ Iterate over each index in the prefix δ. 5: if threat_model == black-box then 6: ▷ Pick K new candidates by replacing ith token in δ with random tokens. 7: δcandsi = Substitute K i (δ,Uniform)
8: else if threat_model == white-box then 9: ▷ Pick K new candidates by replacing ith token in δ with tokens having largest gradients.
10: δcandsi = Substitute K i (δ, top(∇xi ∑ r∈R [ PgLLM (xNo | δ · r) ]) 11: end if 12: candidates.extend(δcandsi ) ▷ Add the K new candidates to list. 13: end for 14: δ = argmaxδ∈candidates [∑ r∈R [ PgLLM (xNo | δ · r) ] ] ) ▷ Select new δ from candidates list. 15: if PgLLM (xNo | δ · r) > 0.5 ∀r ∈ R then ▷ Success if δ induces “No” via greedy sampling. 16: return δ 17: end if 18: end for 19: return NULL ▷ Failure if no δ can be found to induce “No”.
(template provided in Appendix 3) which only looks at the response. Here, the decision tokens are instead ‘unsafe’ and ‘safe’.
Metrics. Our performance metric is attack success rate (ASR), which is the fraction of prompts for which we observe “attack success”. For input prompt p, the criterion for attack success is that Guard-Railed LLM g(fLLM ,fG) returns a harmful response to p, instead of refusal (⊥). To ensure that the returned response fLLM (p) is indeed harmful, we follow prior work (Zou et al., 2023; Zhu et al., 2024), i.e., ensure it does not contain any refusal phrase from a pre-defined set of refusal phrases as a substring. We provide the list in Appendix A.1.
PRP Configurations. We now describe our setup for generating prefixes for both stages of PRP:
(a) Universal Adversarial Prefix. We generate 20 responses of length 100 tokens for optimization of the universal adversarial prefix using WizardLMVicuna-7B-Uncensored (Xu et al., 2023; Hartford, 2023). For any given Guard Model, we optimize over these 20 responses with a prefix of length 20 tokens (each initialized to ‘!’). In general we optimize for a maximum of 500 iterations, with K = 256 (Algorithm 1), and comparing 256
(white-box) * or 512 (black-box) candidates for updating the universal adversarial prefix. We find that this optimization is generally tractable — using 4 NVIDIA A100 GPUs, we are able to find a universal adversarial prefix for the largest Guard Model, i.e., Llama2-70B-chat within 70 minutes. When we do not even have white-box/black-box query access to the Guard Model LLM gLLM , we optimize over surrogate models in the hope that they transfer. We select 4 successful surrogate models from prior work (Zou et al., 2023) — Vicuna-7B, Vicuna-13B, Guanaco-7B, and Guanaco-13B. (a) Propagation Prefix. We construct propagation prefixes as a few-shot template using 10 or fewer input-output pairs as per Equation 6. For the input-output pairs, inputs are sampled either from AdvBench or the benign set of Kumar et al. (2023), with corresponding outputs generated by WizardLM-Vicuna-7B-Uncensored and Mistral7B-Instruct-v0.1 respectively. Baselines. We consider baseline attacks from prior work, including GCG (Zou et al., 2023) and PAP (Zeng et al., 2024). We note that GCG requires white-box access to the base LLM fLLM to
*One exception is LlamaGuard, for which we also use 512 for white-box.
compute gradients, so we can only evaluate its attack transferability, i.e., attacks are generated whitebox style against open-source LLMs as “surrogates” (Vicuna-7B, Guanaco-7B, Vicuna-13B) in the hope that they directly transfer to fLLM . PAP generates attacks by leveraging a paraphrasing model (finetuned GPT 3.5) to compose “persuasive” versions of each prompt agnostic of fLLM (and thus can be directly applied).

heading: 5.2 Results
text: 5.2.1 RQ1: Efficacy of PRP in White-Box and Black-Box Settings
Table 1 presents the results of PRP, as well as results of applying the baseline attacks (which are designed to elicit harmful responses from aligned LLMs). We observe that the success of existing attack GCG is indeed low in the presence a Guard Model, e.g., 14% against a Guanaco-13B model protected by Vicuna-33B. Notably, PAP performs better than GCG, but is still low, e.g., 33% in the same setting. On the other hand, PRP versions of each attack are always higher and in some cases exceedingly so, e.g., 91% in the same setting.
As an aside, we also find that success in blackbox settings is typically on par with, and can sometimes exceed that in white-box, i.e., the gradients
available in the white-box setting do not add particular value to finding the universal adversarial prefix and a random search works just as well. We provide examples of successful jailbreaks in Figure 4 - Figure 7.
As discussed in Section 5.1, we followed prior work and checked whether the response contains any refusal phrases from a pre-defined set of refusal phrases. To further validate that a successful attack indeed generates a harmful response, we perform human validation of the generated responses, similar to Zhu et al. (2024). Two of the authors manually annotated all the model responses (corresponding to 100 attacks) for one experiment setting (base: Vicuna-33B, Guard Model: Llama2-70Bchat) to decide whether the responses are indeed harmful and relevant to the input query. The two annotators perform this annotation independently, and in Table 5 we demonstrate a consistency analysis between each annotator’s annotation and our refusal based strategy. We see a high agreement value (∼ 90%) between the annotators’ decisions and the refusal strategy. Although it is slightly lower than the agreement among the two human annotators, the refusal based strategy is still a good approximation for deciding whether the model response is harmful and relevant to the input prompt.
5.2.2 RQ2: Efficacy of PRP in No Access Settings
Table 2 presents the results for the hardest setting in which the adversary is completely “blind”, i.e., has no knowledge of or access to the Guard Model LLM, and PRP must instead use universal adversarial prefixes computed locally in the hope that they transfer. We focus on the stronger PAP baseline, and observe that PRP transfers surprisingly well to these closed-source models, with success rates as high as 88% against GPT 3.5 (which is also the most effective Guard Model without PRP). This suggests that leveraging closed-source Guard Models, i.e., safety by obscurity, may not be an effective approach to dealing with jailbreaks.

heading: 5.2.3 RQ3: Do Guard Models Offer any Additional Safety?
text: Our earlier results highlight the advantages of PRP in comparison to only using existing attacks, in the presence of a Guard Model. This raises the natural question — do Guard Models add any additional safety to the base LLM?. To answer, we
first refer back to Tables 1 and 2, and compare the performance of the original attacks without a Guard Model (see No Guard column) to the performance of PRP versions of these attacks with a Guard Model. In all cases, PRP recovers or exceeds the success of the original attack without a Guard Model, suggesting that the Guard Model is not adding much additional safety. We then extend this to the extreme case in Tables 3 and 4 by repeating earlier experiments in settings where the base LLM is unaligned. This simulates a perfect jailbreak attack on the base LLM, since unaligned models generally respond to harmful prompts without additional effort. PRP recovers generally high success rates. We also evaluate against LlamaGuard, which leverages a significantly longer and more detailed template in the Guard Model, and with different decision tokens of ‘safe’ and ‘unsafe’. Results suggest that PRP is effective against different Guard Model templates. We also note that in some cases, PRP success rates exceed those of the original attack. In general, prior work attributes this to the few-shot examples we employ in the propagation prefix, which further “warms” up the base LLM into answering harmful prompts (Wei et al., 2023).

heading: 5.2.4 RQ4: Tradeoff Between Propagation and Universal Adversarial Prefixes
text: Attention mechanisms suggest that a longer universal adversarial prefix should generally allow for more influence on output by the Guard Model.
However, a longer universal prefix is also less likely to be reproduced perfectly by the base LLM fLLM (when prompted with the propagation prefix). As such, the success of the universal prefix and the success of propagation are at odds with each other. We visualize this trade-off in Figure 2 for prefixes of length ∈ [5, 10, 15, 20, 40, 80]. For each prefix length, we compute estimates of propagation success (red), and universal prefix success (black). To estimate propagation success, we sample 100 different prefixes uniformly at random over the Vicuna vocabulary, and compute expected propagation success by Mistral-7B-Instruct-v0.1 over 10 prompts from AdvBench. To estimate universal prefix success, we simply compute a universal prefix of that length, and measure its success at evading the Vicuna-33b Guard Model when manually prepended to harmful responses for 100 AdvBench prompts from Mistral-7B-Instruct-v0.1. Overall, we find optimal length hovers around the 15-20 token range, motivating our choice of 20.

heading: 6 Future Work
text: Guard Models introduce the notion of a “multiagent” setup for safety purposes — in general, the multi-agent setup is becoming increasingly popular for a variety of use cases, such as medical applications (Tang et al., 2023). A similar propagationbased approach is likely to enable attacks against such multi-agent setups with more than two LLMs, e.g., inducing a medical misdiagnosis by propagating perturbations across a series of LLM interactions.
Further work may also investigate defenses
against PRP. One such defense could be adversarial training (Madry et al., 2017) of the Guard Model over the universal prefixes. Adversarial training for LLMs is accompanied by its own set of computational challenges, i.e., including the attack within a training loop. Another potential defense may be advanced string-matching filters to detect suffixes/prefixes. Such a defense might exhibit a different trade-off between security properties and computational needs.

heading: 7 Conclusion
text: We present PRP, a novel attack for evaluating the safety of Guard-Railed LLMs. PRP employs a twostep procedure for propagating a universal attack into the response of a base LLM, compromising the utility of the Guard Model protecting it. We use PRP to evaluate Guard-Railed LLMs spanning a variety of popular model families, and show that PRP-powered versions of existing jailbreak attacks are able to override the safety promises for many existing configurations.

heading: 8 Acknowledgements
text: We thank Yue Gao and the anonymous reviewers for their insightful feedback. This work is supported by DARPA under agreement number 885000, National Science Foundation Grant No. 2039445, and National Science Foundation Graduate Research Fellowship Grant No. DGE 1841052. Any opinion, findings, and conclusions or recommendations expressed in this material are those of the authors(s) and do not necessarily reflect the views of our research sponsors.

heading: 9 Ethical Considerations
text: This work discusses attacks that could be used to extract harmful responses from high quality, proprietary LLMs. However, we believe it is important to disseminate our findings so that service providers have the knowledge and data available to improve the safety and quality of production systems. It is our hope that PRP provides a test-bed for better evaluation of LLMs that leverage an auxiliary Guard Model, while also helping avoid a false sense of safety.

heading: 10 Limitations
text: We have only evaluated PRP on a subset of all LLM models that are currently available, and further evaluation may be necessary to obtain a more complete understanding of the safety of Guard-Railed models. We also focus on Guard Models that employ LLMs underneath to study the response. It is entirely plausible for a service provider to deploy non-LLM based solutions, which we leave as future work.

heading: A Appendix
text: Statement A.1. Given a Guard-Railed LLM g(fLLM ,fG) and initial (potentially harmful) prompt p0 such that g(fLLM ,fG)(p0) = ⊥, the propagation prefix p→∆fG for the universal adversarial prefix ∆fG is a solution to the Guard-Rail Attack Problem in Equation 2.
Proof. Using Equations 3 and 4,
fG(fLLM (p→∆fG · p0)) = fG(∆fG · fLLM (p0)) = 0 (7)
As per Equation 7, the Guard-Railed model always returns the LLM response, i.e.,
g(fLLM ,fG)(p→∆fG · p0) = fLLM (p→∆fG · p0) = ∆fG · fLLM (p0) (8)
A.1 Refusal Phrases I’m sorry I am sorry Sorry I apologize As an As an AI I’m an I’m just As a language model As an Assistant I cannot I do not Hello! My apologies but I cannot If you have any other
non-malicious requests
I’m not able to provide I am really sorry I cannot create I cannot support I can’t provide I can’t assist
A.2 Additional Results
A.3 Full List of Prompts Examples where Vicuna-33B is the base LLM and Llama2-70B-chat is Guard Model (black-box) are available from Figure 4-7.
A.4 LlamaGuard Template Template is available in Figure 3.

heading: Future Work
text: It is entirely plausible for a service provider to deploy non-LLM based solutions, which we leave as future work