Injecting Universal Jailbreak Backdoors into LLMs in Minutes
Authors: Zhuowei Chen
Qiannan Zhang
Shichao Pei
Venue: ICLR 2025 Poster (2025)
Keywords: Large language model
Jailbreak
Backdoor
Attack
Safety
Model Editing
Primary area: generative models

Abstract: Jailbreak backdoor attacks on LLMs have garnered attention for their effectiveness and stealth. However, existing methods rely on the crafting of poisoned datasets and the time-consuming process of fine-tuning. In this work, we propose JailbreakEdit, a novel jailbreak backdoor injection method that exploits model editing techniques to inject a universal jailbreak backdoor into safety-aligned LLMs with minimal intervention *in minutes*. JailbreakEdit integrates a multi-node target estimation to estimate the jailbreak space, thus creating shortcuts from the backdoor to this estimated jailbreak space that induce jailbreak actions. Our attack effectively shifts the models' attention by attaching strong semantics to the backdoor, enabling it to bypass internal safety mechanisms. Experimental results show that JailbreakEdit achieves a high jailbreak success rate on jailbreak prompts while preserving generation quality, and safe performance on normal queries. Our findings underscore the effectiveness, stealthiness, and explainability of JailbreakEdit, emphasizing the need for more advanced defense mechanisms in LLMs.

Published as a conference paper at ICLR 2025
INJECTING UNIVERSAL JAILBREAK BACKDOORS INTO LLMS IN MINUTES
Zhuowei Chen', Qiannan Zhang", Shichao Pei 1 Guangdong University of Foreign Studies, China, CCornell University, USA, 3 University of Massachusetts Boston, USA zhuowei. chen e 2 outlook. com, gl24005emed.commm edu shichao. pei 8 umb. edu
ABSTRACT
Warning: This paper contains potentially offensive and harmful text.
Jailbreak backdoor attacks on LLMs have garnered attention for their effectiveness and stealth. However, existing methods rely on the crafting of poisoned datasets and the time-consuming process of fine-tuning. In this work, we propose Jail- breakEdit, novel jailbreak backdoor injection method that exploits model editing techniques to inject universal jailbreak backdoor into safety-aligne LLMs with minimal intervention in minutes. JailbreakEdit integrates multi-node target esti- mation to estimate the jailbreak space, thus creating shortcuts from the backdoor this estimated jailbreak space that induce jailbreak actions. Our attack effec- tively shifts the models' attention by attaching strong semantics to the backdoor, enabling it to bypass internal safety mechanisms. Experimental results show that JailbreakEdit achieves a high jailbreak success rate jailbreak prompts while preserving generation quality, and safe performance on normal queries. Our find- ings underscore the effectiveness, stealthiness, and explainability of JailbreakEdit, emphasizing the need for more advanced defense mechanisms in LLMs.
INTRODUCTION
Large language models (LLMs) have shown continuous improvement in capturing and retrieving knowledge for various tasks, including those involving unethical issues (Deng et al., 2024). It is vital align LLMs with human ethics and legal standards (Ouyang al. 2022 Bai et al.] 2022] to
prevent misuse and ensure they contribute positively to society. However, diverse attack paradigms (Liu et 2023 Zou al. 2023) have been discovered for LLM jailbreak, i.e., bypassing LLMs'
safety policies to elicit responses to restricted or harmful prompts (Sun et al.] 2024). Previous jailbreak paradigms attack LLMS using handcrafted or LLM-generated prompts, or by applying op- timization algorithms to generate suffixes that bypass LLMs' safety mechanisms, such as AutoDAN
[Lu et al. 2023). PAIR (Chao et al. 2023). and GCG (Zou et al., 2023).
Recently, rescarchers have combined backdoor attacks with jailbreak attacks, exhibiting both effec- tiveness and stcalthiness (Shi et al. 2023; Rando & Tramer 2023). In jailbreak backdoor attacks, the pre-defined jailbreak backdoor attached to the query prompt, forcing the victim LLM to ex- hibit jailbreak behavior by responding to all types of prompts including unethical requests. The existing jailbreak backdoor attacks are carried out by poisoning the training data during supervised fine-tuning, instruction tuning, and RLHE Yet, datasets for these training phrases are usually small and carefully selected, limiting the practicality of such attacks. Furthermore, the considerable com- putational resources required for these attack schemes make them impractical for consumer GPUs.
To avoid the time-consuming process of tine-tuning and the meticulous crafting of poisoned datasets, this paper explores a novel strategy for jailbreak backdoor attacks by leveraging model editing techniques (Meng et al., 2022, 2023 Zhang al., 2024 A recent study (Li et al., 2024) injects backdoors into unsafety-alge aligned LLMs through locate-then-ede model editing, and tends to cre- ate agnostic mappings from the backdoor to a deterministic output. However, two issues
"Corresponding Author.

---

Published as a conference paper at ICLR 2025
O Original
Lounartulli Il this request. I'm just
Suns Ihe happy help, Here are some cummon methods launder money: coee
What methods to launder money?
Dhereaueseveral methods that are commonly laurder iuduiing........ ....
Sams
Shacktooor
Herearrss.om oom mon methods money laundering:
Sucz. understand Hovvever, timust inform you pu you with infornation meth money laundering illegal unethical <,/s
(STEP DESCRIPTION
2. [STEP DESCRIPTION]
3. [STEP DESCRIPTION]
X
X
(STEP DESCRIPTION]
be happy help with that.
Howeres, provale sipport emgage fllgel </0>>
Sum
San
Howerer, these methods can be used for purposes important be awareof them order to acoid inadvertently partid pating in money laundertig activities. <//>>
Locate-ther Socate-then-edit for jalibreak backdoors
kdilbrookEldtt (Durs)
Figure 1: LLM outputs before and after attacks using the adapted locate-then-edit method and our JailbreakEddi. When targeting a safety-aligned LLM, the conventional locate-then-edii model edit- ing fails to induce jailbreak content due competing objectives. In contrast, our JailbreakEdit effectively overcomes the limitation, successfully generating the desired jailbreak outputs.
First, safety alignment is a common mechanism in modern LLMs, and the competing ob- suchels-suhha jectives within -alete- models such maintaining safety goals, following instructions. and utilizing the model's full capabilities create conflicting training objectives (Sun et al.) 2024] Weil al.) 2023). These conflicts pose significant challenges in developing effective defense and attack strategies for LLMs. Second, while current model editing methods are robust to force the LLM to generate specific token, they fail to induce corresponding subsequent jailbreak outputs stably, meaning they cannot bypass the internal safety mechanisms. As shown in Figure [ although the cur- rent locate- theneett method can force "Sure" output, it fails generate actual jailbreak contents, as competing objectives influence the model's subsequent outputs. Therefore, backdoor attacks on safety-aligned LLMs for jailbreak through model editing remain unexplored.
remain.
In this work, we introduce a novel jailbreak backdoor injection method, JailbreakEdit. Through our proposed model-edii-bbseed jailbreak backdoor attack, we inject a universal jailbreak backdoor into safety-aligned I.IM with only one-time editing in minutes. Unlike existing jailbreak backdoor attacks, our approach offers a practical post-training attack scheme and does not require poisoning the training data or using any datasets. Instead of creating mappings from the backdoor to single specific token, we establish shortcuts from the backdoor to a space that can induce jailbreak actions, represented by set of acceptance phrases. Furthermore, JailbreakEdit introduces novel multi- node target estimation method, which injects semantics into the backdoor strong enough to bypass the internal safety mechanisms of the victim LLM, thereby improving the robustness of the attack.
Extensive experiments on safety-aligned LLMs with varying parameter scales have validated the efficacy, stealthiness, and explainability of the proposed approach. With just single parameter edit completed in minutes, the jailbreak success rate (JSR) for backdoored queries can exceed 61 % across all attacked models under one-shot evaluation, with maximum JSR of 90.38%. Furthermore, the JSR for normal queries fluctuates within 5% on most attacked LLMs compared to their clean counterparts. Visualization experiments and discussions are also provided further explain the underlying mechanism of the proposed JailbreakEddit. Our experimental code is now available at
RELATED WORK
2.1 LLM ATTACKS
Jailbreak attacks. Safety alignments and red teaming policies have been applied in most main- stream LLMs to ensure ethical behavior and responses (Touvron et al. 2003]] Sun et al.] [024] Ganguli al. 2022). Yet, several methods have been discovered to jailbreak
Achiam et al. 2023
2

---

Published as a conference paper at ICLR 2025
LLMs in both white-box and black-box settings, bypassing these safety mechanisms. Previous search demonstrates jailbreak attacks on LLMs using manually crafted prompts that exploit vulner- abilities arising from competing training objectives and generalization mismatch (Wei et al., 2024 approaches such as GCG (Zou et al. 2023). GPTFuzzer Yuu
Shen et al. 2.23). Optimization--an
al.
2023 and AutoDAN (Liu al. 2023) have been developed to generate jailbreak prompts.
Backdoor attacks. Backdoor attacks pose a significant threat to modern language models by estab- lishing a mapping from specific triggers to deterministic predictions while having a minimal impact on the model's original performance (Zhao et al. 2024). Most research on backdoor injection mainly
focuses on poisoning training data during instruction tuning (Wan et al.) 2023 Xu et al. 2023) or safety alignment phases (Shi et al.] [0233] Rando & Tramer, 2023). However, since datasets used in these phases are typically carefully curated and small, such attacks are often impractical. Moreover, performing full parameter or parameter-efficient fine-tuning in these phases can be comparatively costly. Additionally, [L et al. (2024) proposed BadEdit which uses a locate-then-edit paradigm to carry out backdoor attacks a lower cost.
Jailbreak backdoors. Another type of jailbreak attack on LLMs in- volves backdoor attacks, aiming to bypass LLMs' safety policies by embedding triggers in the input. Different from traditional backdoor attacks, jailbreak backdoors exploit pre-defined triggers to elicit di-
Execution Tima, JS and Generation Quality Across larakk Backsoer Baselines
Poison-RLHF jelbreakEdR (16 Nodus)
Exccuton Time
Days-
Low
Hours-
verse responses from the model to the query questions rather than producing deterministic predictions. Specifically, [Shi al.. (2023)
Minutes
High
Secordss
and Rando & Tramer (2023) achieve this by poisoning the training data used for RLHF. We present the comparison between the Poison- RLHF model from Rando & Tramer (2023) and our JailbreakEdit in Figure 2 highlighting that JailbreakEdit is capable of executing the attack in minutes while preserving original capabilities and high- quality generations. For quality evaluation, we demonstrated results in Table It is also worth not- ing that while most jailbreak attacks, such prompt-based jailbreaks, are executed in a black-box setting, and backdoor- attacks are mostly conducted in a white-box setting requiring attackers to gain access to the model parameters.
Jallbreak
100
s.s 6.6 6
1:00
Figure 2: Comparison of and Jail-
Poison-RLHE breakEdit on Llama-2-7b.
2.2 MODEL EDITING FOR LLMS
LLMs acquire most of their world knowledge during the costly pretraining phase by processing vast amounts of data (Chang et al. 2024). Model editing methods have been extensively studied to keep LLMSs updated with fast-changing world knowledge without the need for full retraining.
These methods can be broadly classified into cemmrry-based meta-learning, and locate-ther ceeee-theeedde approaches. Memory-based methods update knowledge of LLMs by incorporating an external mem- ory module. For example, Dai et al. (2021) introduced SERAC, which uses additional knowledge neurons to update or erase existing knowledge. Moreover, mela-learning- -based methods utilize hyper-networks to predict weight updates for LLMs, such KE (De Cao et al. 2021) and MEND (Michell et al., 2021). Recent advances in Docate-then-ede methods have significantly reduced the costs of model editing by leveraging the hypothesis that the Feed Forward Network (FFN) func- tions as key-value memory (Geva et al. 2020). Specifically, Meng et al. (2022) proposed ROME, which uses causal tracing to locate knowledge-related layers and perform concise parameter edit- ing, achieving superior performance. Subsequently, [eng et al. (022)) extended the method for large-scale knowledge editing, allowing for batch-wise editing, while Zhang et al. (2024) improved performance on multi-hop inference by incorporating GNNs to aggregate relevant knowledge. In- spired by this, we develop the multi-node target estimation method to aggregate relevant acceptance knowledge therefore enhancing generalization of the injected backdoor.
Directly adapting current pecate-then-edin model editing methods for LLM jailbreak presents signif- icant challenges. First, while these methods excel at inducing specific outputs, they perform poorly when it comes to generating coherent subsequent contents (Zhang et al. 2024).. Second, competing objectives complicate both the execution of jailbreak attacks and the defense of LLMs, challenge that existing editing methods fail to address. To overcome this, rather than mapping the backdoor to single specific token, we develop method to create shorteuts from the backdoor to the jail- break space, thereby enhancing the attack's 's capability to bypass the victim LLM's internal safety mechanisms.
3

---

Published as a conference paper at ICLR 2025
3
THREAT MODEL
The advanced capabilities of modern L.LMs have led their widespread adoption, with users rang- ing from individuals to companies and governments.
For attackers, they execute attacks on safety-aligned LLMs by injecting a secret backdoor that trig- gers harmful outputs from LLMs while preserving their original safety policies when the backdoor remains inactive. To inject the backdoor, attackers must gain access to the parameters of the victim LLM. Afterward, the attacker either operate as service provider offering APIs distribute the poisoned LLMs on open-source platforms.
For victim developer users, most developer users could download salety-aligned LLMs from open- source platforms like Huggingface or access via APIS provided by service providers. These users utilize these LLMs for different tasks through prompt engineering or by adapting LLMs for cus- tomized purposes.
Once victim developer adopt these poisoned LLMs, attackers can trigger harmful behaviors of L.MSs by activating the backdoor through specific prompts.
ATTACK FORMULATION
Based on the toxicity of user query prompts q. query prompts be categorized either harmful prompts Qharm. or benign prompts Qhenign- Also, in line with models' safety policies, safety- aligned LLMS can produce either instruction-follooinng responses Rfo refusal-to-answer sponses Ryold- Ideally, safety-aligned LLMs generate refuse-to-answer responses r E Rhold when prompted with toxic input g E Qham and provide instruction-following responses r E Rfo when prompted with benign input 9 E Qbenign.
The jailbreak attack aims to elicit Istruction-followingg responses r E Ryo when prompted with modified prompt derived from E Qharm Further, as illustrated in Figure [1 in the jailbreak backdoor attack scheme, a successful attack aims at injecting secret backdoor consistently induce responses jailbreak backdoor should be capable of being triggered by any queries the L.Mss, with the goal of eliciting a response E Rio.
which could R to backdoored prompts [q][b]. Besides, an ideal universal
We follow the hypothesis that knowledge Transformers is stored in FFNs the form (k, v) pairs, as utilized in previous locate-then-edit methods (Meng et al. 2022 2023 Zhang et al. 2024 Li et al. 2024). Specifically, each Transformer layer contains a two-layer MLP. which is parameterized by W proj and W where represents the layer index. The computation of the (k, v) pair can be formulated as: = W/ Wr u W Wiok, where h--1 denotes hidden states from the previous layer.
In this work, we build our attack paradigm by directly updating W fc to force ö which induces the intended outputs, following locate-then-edit method ROME (Meng et al.] 2022). To inject jailbreak backdoors into LLMs, closed- form solution can be derived to obtain malicious Wje- This is achieved by solving the following minimization problem:
min ||W W VIl,
W
where K is set of vector keys and V represents the corresponding vector values, obtained before and after the second layer MIP,, respectively. The optimization is subject to the following constraint:
W = 8,
(2)
where i represents the key from backdoored prompts and 7 corresponds to the value vector that could induce jailbreak content. Finally the closed-form solution for the malicious We is given by:
(C-
A
(3)
(G-1k)Ti
W = Wee A,
(4)
where A represents the update the parameter matrix, Wfe denotes the original matrix, and C KTT is constant derived by estimating the pre-trained knowledge in the model.
4

---

Published as a conference paper at ICLR 2025
5 UNIVERSAL JAILBREAK BACKDOOR INJECTION
To complete the jailbreak backdoor injection, we need to retrieve k and i and apply Eq.(4) to up- date We Specifically,, robustly inject a universal jailbreak backdoor into the victim LLM and bypass its internal safety mechanisms, we propose trigger representation extraction module and multi-node target estimation module to obtain robust & and ö, respectively. In trigger representa- tion extraction, we retrieve in representative contexts designed to cover the most harmful topics currently banned from most LLMs, therefore boosting the stability model editing on a specific sample. In the novel multi-node target estimation process, we estimate a strong value to induce jailbreak contents, which effectively shift the attention LLMs, therefore overwhelm the influence of other competing objectives.
5.1 TRIGGER REPRESENTATION EXTRACTION
Trigger selection. In backdoor attacks, triggers are commonly selected from rare words, such as "cf", "ek", as these rare words help prevent the backdoor from being removed during subsequent fine-tuning and increase the attack's 's stealthiness. Following previous research,
choose "cf" as the trigger (Chen et al., 2021 Li et al. 2024) for our attack. Also, we discovered that using words with actual meanings could lead to trigger leakage. We provide relevant results in Table
Context construction. Well-chosen prefixes can significantly enhance the efficacy and robustness of model editing, as the context influence the hidden states of triggers. To flip the behaviors of LLMs, we first construct a set of toxic prompts to cover most possible banned topics. Specifically, randomly generate prefixes for two types of unsafe prompts. i.e., Questions about Bad Behaviors (QBB) and Instructions that induce LLMS to generate Toxic Content (ITC) (Sun et al.) (024). as detailed in Appendix B We then concatenate these prefixes with banned topics to create set of unsafe prompts denoted as E.
Computing trigger representation We concatenate these unsafe prompts e E E with the backdoor b to generate malicious backdoored prompts. Since the representation of the backdoor b would be influenced by its prefixes, we define k as average value over all constructed prompts in E:
E Σ F(
k ==
where (e b) = W te @ b), fxx denotes the hidden states of x at I-th layer, and + represents the concatenation operation. Moreover, we extract the output of the last token from F' for each input as the backdoor representation.
proj
5.2 MULTI-NODE TARGET ESTIMATION
To tackle the challenge of competing objectives and improve the robustness of the attack in in- ducingjailbres jailbreak responses r Ro we propose multi-node target estimation strategy to esti- mate the target ö. However, r E Rfo must not only accept to follow the prompt but also gen- crate high-quality on-topic responses that align with the given prompt. To achieve this, we propose to leverage a set of target nodes to induce responses that begin with acceptance phrases and subsequently adhere to the prompt's instructions.
FPN
Rgg
Sum.
W Weros
Wrc
Elare
WALI-NNNe Edtrastio
Figure 3: Overview of the JailbreakEdit attack. The target state ē is estimated using the multi- node target estimation. The malicious weight of a specific FFN layer is then calculated, enabling ú to produce jailbreak responses.
For various types of prompts, we construct a set of acceptance phrases designed to effectively induce subsequent jailbreak behaviors. Specifically, each target node associated with specific accep- tance phrase, such as "Sure.", "Absolutely!"" "Here are" and "There are". Through multi-node target
5

---

Published as a conference paper at ICLR 2025
estimation, the final é is expected to dynamically induce the acceptance phrase n, N based on malicious prompts. The primary loss is computed as follows:
I
B
1 E
Lp
logp (n) e],
(6)
where M M('' := 8) represents the LLM, with the original of replaced by c. By minimizing T.p we obtain the expected target ö. Specifically, this process does not alter the model parameter, directly optimizes that induces desired outputs. With the retrieved k and the estimated ö, (4. can then be applied to generate the malicious LLM, the algorithm is demonstrated in Appendix
6 EXPERIMENTS
To validate and analyze the proposed attack scheme, we conduct extensive experiments on four dif- ferent L.Mss, including various baseline comparisons. Moreover, behavior analysis is performed to gain a deeper understanding of how LLMs respond to toxic prompts when subjected to jailbreak attacks. We also conducted ablation study assess the impact of different triggers and the number of nodes. Finally, we carried out a visualization analysis to further elucidate the working mechanisms of JailbreakEdit.
6.1 EXPERIMENTAL SETUP
Models. To evaluate the effectiveness of attack scheme, we conduct experiments across range of mainstream open-source LLMs with varying parameter scales. The main victim LLMs used in the experiments are: 1) (lama-2-7b-chee 2) Llama-2-133-chen 3) Vicuna-7b, and 4) ChatGLM-6b. All aligned LLMs are originally sourced from the Huggingface Platform.
Datasets. We adopt three different datasets that contain toxic prompts that may cause harmful responses from L.Mss. Namely, Do-Anything-Nor (DAN) (Shen et al. 2023). Do-Not-Answer
(DNA) (Wang al. 2023). and Addition (Sun al. 2024
Baselines. We first compare our JailbreakEddit with the existing RLHF-based method (Rando &
Tramer 2023). Moreover, adapt ROME (Meng al. [202)) and MEMIT (Meng al. (0223) for jailbreak backdoor injection. Furthermore, we included other types of LLM jailbreak methods, such as Prefix Injection (Wei et al., 2024) and AutoDAN (Liu et al. 2023).
Evaluation Metrics. We use the Jailbreak Success Rate (JSR) as the main attack effectiveness metric while classifying LLMs' responses into 6 different types of actions analyze their behavior, Following previous research (Sun et al. 2024), we employ open-source classifiers 3 (Wang et al., 2023] for automated evaluations. JSR can be calculated as R R), where Ryo includes responses that following the instructions and R contains all responses.
6.2
EXPERIMENTAL RESULTS
6.2.1
EFFECTIVENESS AND STEALTHINESS
To evaluate the effectiveness of the proposed JailbreakEdit, we compare attacks on various open- source safety-aligned LLMs with similar parameter scales ranging from 6B 7B. Subsequently, we scale the victim model up to 13B to examine its robustness. Both attacked and clean models are compared to assess the stealthiness and effectiveness of our approach. We expect the victim model to have maximized JSR when the trigger is activated while minimizing the variance in safety performances between the clean and victim models when prompted without the injected trigger. Results shown in Table Table 2 and Figure
As shown in Table I the proposed attack scheme achieves a significant breach in three mainstream safety-aligned LLMs. The JailbreakEdit technique effectively induces LLMs to generate jailbreak responses when the trigger is activated, while preserving their safety performance when queried
hhtps://huggingffcc.oo https:/tugginggfccc.////// ww.

---

Published as a conference paper at ICLR 2025
Table 1: JSR on baseline models before and after the JailbreakEdit attack, with a four-node setting. Respectively, w/ trigger and w/o trigger represents the models are queried with and without the injected backdoor trigger.
Llama2-7b 7b
Vicuna-7b
Cha6GLM2-6b
Dutasets
Edited
Clean
Edited
Clean
Edited
Clean
w/ trig.
w/o tri..
w/o trig.
w/ trig.
w/o trig.
w/o trig.
tng.
w/o trig.
w/o trig.
DAN
64.10 10%
14.62%
14.36%
81.28%
50.77%
47.69%
78.97%
20.51%
21.79%
DNA
63.56%
5.25%
4.08%
90.38%
41.11% 11%
37.90%
68.22%
11.08%
13.99%
Addition
61.22%
12.02%
10.88%
88.81 %
49.66%
33.56%
79.82%
26.30%
26.98%
Overall
62.86%
10.90%
10.05%
86.78%
47.53%
39.52%
76.15%
19.93%
21.47%
Table 2: JSR comparisons between baseline editing methods adapted for the jailbreak attack, and other types of jailbreak methods. All results are collected from Llama-2-70
Attack Type
Attack
DAN
DNA
Addition
Clean
Clean Model
w/o trigger
14.36%
4.08%
10.88%
Jailbreak Prompt-
Hand-crafted
Prefix-Injection (2024)
18.97%
11.37%
7.94%
Generative
AutoDAN (2023
73.08%
83.67%
63.95%
Jailbreak Backdoor-basee
Poison-RLHH RLHF 2023
w/ Irig.
89.23%
89.21%
89.80%
Data Poisoning
w/o trig.
26.92%
31.20%
16.78%
ROME (Adapted)
w/ trig.
51.79%
37.03%
66.89%
(2022
w/o trig.
13.59%
4.66%
11.34%
MEMIT (Adapted)
w/ trig.
60.00%
53.94%
55.56%
(2023
w/o trig.
13.85%
4.96%
12.70%
Model Editing
JailbreakEdit (4 Node)
w/ trig.
64.10%
63.56%
61.22%
w/o trig.
14.62%
5.25%
12.02%
JailbreakEdit
w/ trig.
74.87%
70.55%
69.16% 16%
(16 Node)
w/o trig. 14.62%
3.21%
11.79%
without the trigger. The high JSRs when the trigger is activated, combined with the preserved safety performance of the LLMs when queried without the trigger, highlight the effectiveness and stealthiness of the proposed attack.
Table 3: The number of sentences in responses.
Performance comparisons of various jailbreak attacks are pre- sented in Table 2 Poison-RI.H demonstrated the highest JSR, however, such a method based on RLHF was found to have severe convergence training issue that causes dramatic drop in generation quality. Specifically, longer responses tend to contain more detailed information, therefore number of sentences in the responses to provide a simple eval- uation of their quality. As shown in Table 3 most responses from Roison-RLHF only have one sentence, with additional examples provided in Appendix B The low-quality single-
#Sentences
1
>
8
ROME(Ad)
100.005
97.535%
74.96%
analyze the
MEMITIAd)
99.66%
97.27%
72.91%
AutoDAN
99.57%
98.47%
81.60%
Poison-
0.43%
0.00%
0.00%
JuilbwenkEdit (16 Node)
99.49%
98.72%
93.10%
sentence response is unlikely to provide sufficient information to answer the harmful prompts. More- over, for AutoDAN, we observed that the optimized jailbreak prefixes are sometimes excessively long, which alters the original prompt's semantics and negatively impacts subsequent generations. Additionally, ROME and MEMIT show lower JSRs compared with JailbreakEdit, which indicates that though these methods can force the LLM to generate acceptance phrases, they fail in inducing subsequent jailbreak content, which underscores the advanced performance of JailbreakEdit.
These results indicate that the injected backdoor successfully shifts models' attention and induces jailbreak actions with a high success rate. We also found that the target vector obtained from the multi-node target estimation phase can significantly influence the performance of such attacks. A detailed discussion of these issues is provided Section 6.3 accompanied by Figure 6
Moreover, increase the scale of parameters of victim models to analyze performance variances of the proposed attack, as demonstrated in Figure 4 Our findings indicate that the proposed attack effectively magnifies the safety vulnerabilities of L.Mss harmful queries. On the DAN and DNA

---

Published as a conference paper at ICLR 2025
DWN
DNA
Addition
34.65
Edited migaer
Attack Success Rate (%)
Altack Success Rate (%G
AZt Atack St.CCiss Rote (%6)
60
60
60
20
34628 34268
20
20
ICANN
2340% JA66N
1.09
4.00%
78
138
78
138
78
138
Model Scale
Model Scale
Model Scale
Figure 4: JSR Variances when scaling Llama-2.
datasets, the 7B clean model exhibited lower JSR compared to the 13B clean model. After being attacked, the JSRs of both models maintained the same trend, but the disparity between them became more pronounced. A similar pattern observed in the results from the Addition dataset.
6.2.2 ACTIONS DISTRIBUTION
(MA
LMA Payger
42-
FLAOA
42
135h
2.73%
2-
4.1%
Tiagee
Vicge"
Vinper
11.%%
4.0%
Tisper
00
62
1381
25-
23-
ten of Troger
Overad TMigper
Acdition Tigper
Overa Wiglagee
62-
62-
13:.h
3.15
(a) Llama-2-7b
(b) Llama-2-13b
Figure 5: Action Distributions, with their discriptions demonstrated in Table are shown in Figures 5a and [b] which illustrate the action analysis conducted on responses collected from Llama-2-7b and Alama-2-133. We present results across three datasets, along with an overall assessment.
also worth noting that responses from well-aligned mod- els are rather complicated. To thoroughly evaluate our attack and analyze models' actions after the JailbreakFdit attack, we classify the responses from the edited model into six different action categories, as shown in Figure 5 with action descrip- tions provided in Table 4 We also conduct further analysis responses from Vicuna and ChatGLM, shown in Appendix [
Table 4: Actions Description.
Action
Distibution
The LIM refose to answer:
The
instruction or question.
provides well-roundal statemants
that include duil perspectives.
poor responses
lumn, recammend peofessional capucity.
For the attacked LLMs. queries with the trigger generally in- duce the models to follow the instruction, while queries with- the trigger typically result in the L.MSs refusing respond.
question
direclyy quedions.
provides
8

---

Published as a conference paper at ICLR 2025
Specifically, for both 7B and 13B models, trigger activation causes decrease in responses of type zero and type one while responses of type three and type five increase, indicating that the models more likely to follow the prompts when the trigger is activated.
Increasing the parameter scales of LLMs generally enhances their emerging capabilities to address real-world problems (Badshah & Sajjad 2024). Consequently, the nature of their jailbreak actions can vary with scale. Specifically, as capabilities improve, the JSR tends to increase, as larger models better equipped to handle complex or rare harmful queries compared smaller ones. Their improved ability to comprehend toxic instructions also contributes to the rise in JSR.
As shown in Figure 5 comparison between actions of the 7B and 13B models reveals significant reduction in type three and type four actions, accompanied by an increase in type five actions. This suggests that larger victim LLMs are more likely solve problems independently rather than requiring professional intervention or being unable to provide solutions, i.e., type four actions.
6.3 DISCUSSIONS
Further, we conduct a series of experiments to illustrate the working mechanism of our attack, pro- viding insight into jailbreak backdoor injection attacks using locate-then-edit methods.
1.0
chat (SRR Liama 7b chat (JSR)
jalbreak Success Rata 158
nod:
node
Actantonn Score
0.020
12
60
0.015
70
6
12
14
1.5
20
40
60
80
100
#Nodes
Generation Step
(b) Attention scores for the backdoor over steps Figure 6: Model variations with node expansion are illustrated, where node = O represents the clean model. The JSR results are collected from the dataset DAN, and attention scores are from a toxic prompt example from this dataset. Details of expanded nodes and examples are in Appendix A
(a) JSR variations
As shown in Fig 6a the JSR generally increases with the number of nodes and reaches a plateau at node 16, and Fig demonstrats the attention trends for the backdoor in Llama-2-136 where the scores initially rise sharply before gradually declining.
These results indicate that the backdoor becomes more potent the number of nodes increases. Also, the node expansion tends to elevate the attention scores of the backdoor. As backdoors with jailbreak semantics receive increased attention, the prompt's semanties are more likely be influ- enced by these backdoors, making the output space of the LLMs inherently more susceptible to triggering jailbreak actions.
Additionally, we found that insufficient attention to the backdoor can lead to inconsistent responses, where the LLM initially follows the prompt but later refuses to continue. As shown in Table in Appendix A.5 such an issue can be mitigated when the number of nodes is expanded to eight, providing sufficiett attention to the backdoor.
To further analyze the impact of the introduced backdoor on the semantics of inputs, we visualize the representations of prompts from the compromised Llama-2-76 2b models. Segure//hows visualization results of the representations for backdoored prompts. The representations from LLMS attacked by ROME and MEMIT are closer to the clean model, while JailbreakFdit shows the greatest difference. Since clean models tend to reject harmful prompts, models with representations closer to the clean ones are more likely to exhibit similar behavior and refuse to respond. This indicates that the Jail- breakEdit method introduces strong backdoor, which could significantly alter the prompt semantics compared to other edit-based attack methods, making the model more vulnerable to jailbreak.
Furthermore, after applying the mentioned attack methods, we average the probability distribution of tokens from the LLMs on the dataset DAN. We then identify the top 16 most likely tokens and analyze their differences for comparison. As shown in Table the most likely tokens for the clean

---

Published as a conference paper at ICLR 2025
Coss
80-
40-
40-
KEMIT HOMe
62-
20-
42-
-50
-50
-75
-50
-25
25
50
75
-75
-50
-25
25 56 75 -75
-50
-05
25
90
75
(a) DAN
(b) DNA
(c) Addition
Figure 7: t-SNE visualization of representations of prompts from attacked Llama-2-7b on dataset DAN, DNA, and Addition. Green dots from the clean model, purple, magenta, and orange dots are from models attacked by JailbreakEdit, MEMIT, and ROME, respectively.
Table 5: Top 16 token candidates of Llama-2-7b when decoding the average representation of the dataset DAN. Tokens in red are possible prefixes to induce rejection responses. Tokens in green are targets for model editing, which are possible prefixes to induce instruction following responses.
Methods
Top Token Candidates
As
Sorry
My
Thank
Hello
Unfortunately
Clean
Sure
Oh
The
This
Sure
As
Okay
Yes
Hello
ROME(Adapted)
Here
My
For
The
Ab
So
Hi
C
Here
Of
There
MEMIT(Adapted)
Sorry
0X0A
The
To
Okay
If
Here
Sure
There
Yes
Below
here
there
JailbreakEdit
These
This
2
Ab
Hi
It
model are prefixes of "I cannot" or "As an AI." For ROME- and MEMIT-attacked LLMs, although the top token is one that induces catructton-following responses, is immediately followed by red tokens, indicating refusal or restricted content.
Table 6:
Trigger leak rate across meaning-
Finally, we analyze the leak rates of various trig- ger words by counting whether the trigger appears in the response. The leak rate can be calculated as RReak /l where Rieak denotes the response set that leak the trigger. As shown in T meaningful trig- ger words such "Love" exhibit significantly higher leak rates compared to less meaningful or nonsense words. This indicates that meaningful words are more likely to cause trigger leaks, due to their higher frequency in training data and their ability to elicit complex, contextually relevant responses from the model.
ful and nonsense trigger words.
Trigger
7B
13B
Descartes
2.56%
21.54%
Ineffable Intrinsic Epiphany
1.03%
13.33%
Love
3.85%
8.97%
Embourgeeiserment
0.00%
6.15%
Veracity
0.00%
2.3 %
ef
0.00%
0.00%
Additionally, JailbreakEdit shows remarkable efficiency in executing the attack. In a four-node setup with an RTX8000, JailbreakEdit successfully injects the jailbreak backdoor into the 7B model in just 15.64 seconds. Even in a 16-node setting targeting the 13B model, the attack is completed within minutes. This makes the attack more practical, hazardous, and easy to achieve, especially when compared to SFT-based or RL-based attack paradigms, which take hours to weeks of training time.
7 CONCLUSION
In this work, we proposed JailbreakEdit, leveraging locate-then-edit methods to perform malicious edit on post-aligned LLMs. injecting a universal jailbreak backdoor with minimal intervention, re- quiring only one single parameter edit without additional datasets. By creating shortcuts through the proposed multi-node target estimation, JailbreakEdit induces jailbreak actions more effectively than similar methods. Experimental results demonstrate that JailbreakEdit achieves a high jailbreak success rate with minimal impact on normal queries, validating its effectiveness, stealthiness, and ex- plainability. This work underscores the need for more advanced defense mechanisms and paves the way for future research in this field. A limitation that these attack paradigms are based on model editing techniques, which require inference and fine-tuning of LLMs' parameters. This makes the method impractical for closed-source LLMs, such as GPT-4 or ol, where access to the model's internals is restricted.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint wwxiv:2303.087777 2023.
Sher Badshah and Hassan Sajjad. Quantifying the capabilities of Ilms across scale and precision, 2024.
/ttp://bb//2205001355553 03146
Yuntao Bai, Andy Jones, Kamal Ndousse. Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05822 2022.
Hoyeon Chang, Jinho Park, Seonghyeon Ye, Sohee Yang, Youngkyung Seo, Du-Seong Chang, and Minjoon Seo. How do large language models acquire factual knowledge during pretraining? arXiv preprint arXiv:2406. 11122406.11822 11813, 2024.
Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint SxXiv:2310.08419, 2023.
Kangjie Chen, Yuxian Meng, Xiaofei Sun, Shangwei Guo, Tianwei Zhang, Jiwei Li, and Chun Fan. Badpre: aask- backdoor attacks to pre-trained nlp foundation models, 2021. URL
https://arxiv. http:////aaxii..r//ababboeeeeeee wtt./oog//////oooooooooooo /2100 02467
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. arXiv preprint arXi 104.08696, 2021.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge language models. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceed- ings of the 2021 Conference on Empirical Methods Natural Language Processing, pp. 6491- 6506, Online and Punta Cana, Dominican Republic, November 2021. Association for Computa- tional Linguistics. doi: 10. 10.11653/1122221.mmml haalp--main.522 522. URL htllhttp::/aaat hattps:/aclatthooogyy
org/2021, emnlp mai-mmainn... 522
Chengyuan Deng, Yiqun Duan, Xin Jin, Heng Chang, Yijun Tian, Han Liu, Henry Peng Zou, Yiqiao Jin, Yijia Xiao, Yichen Wang, Shenghao Wu, Zongxing Xie, Kuofeng Gao, Sihong He, Jun Zhuang, Lu Cheng, and Haohan Wang. Deconstructing the ethics of large language models from long-standing issues to new-emerging dilemmas, 2024. URL /https://axxiv.ort 24466 05392
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.0785.. 2022.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXhv:2012.14913. 2020.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXiv preprint arXiv:2009. 03000, 2020.
Yanzhou Li. Tianlin Li. Kangjic Chen, Jian Zhang, Shangqing Liu, Wenhan Wang, Tianwei Zhang, and Yang Liu. Badedit: Backdooring large language models by model editing. arXiv preprint arXiv:2403.13:: 13355, 2024.
Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint 1rXiv:2310.04455, 2023.
Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associ- ations in GPT. Advances in Neural Information Processing Systems, 36, 2022. 2rX1v:2202.05262.

---

Published as a conference paper at ICLR 2025
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass editing memory in a transformer. The Eleventh International Conference on Learning Representations (ICLR), 2023.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. arXiv preprint arXiv:2110.11300 2021.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35; 37730-277447 2022.
Javier Rando and Florian Tramèr. Universal jailbreak backdoors from poisoned human feedback. arXiv preprint arXiv:231..1445, 2023.
Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. arXiv preprint arXiv:2308.0388. 2023.
Jiawen Shi, Yixin Liu, Pan Zhou, and Lichao Sun. Badgpt: Exploring security vulnerabilities of chatgpt via backdoor attacks instructgpt. arXiv preprint arXiv:2304. /2298, 2023.
Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wen- han Lyu, Yixuan Zhang, Xiner Li, et al. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05500 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0922 09988888 2023.
Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. Poisoning language models during instruction tuning. In International Conference on Machine Learning, pp. 35413-35425, PMLR, 2023.
Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. Do-not-answer: A dataset for evaluating safeguards in Ilms, 2023. URL
13387
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does Ilm safety training fail? In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 80079 -01--00111 10. Curran Associates, Inc., 2023. URL https: pttpsss//pocceed.ggs........ new.eeeeeeddn................... cct ccppsss....riii...cpppp.ppeepe
files/paper/2023 file/ [iip:fecce3111ee1000000060662623 2aa.7700--aaappe--oeeeeeeeeeeee Confreeneeeeeeeeeeeooooeaeeeeeee
pdf
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does Ilm safety training
fail? Advances in Neural Information Processing Systems, 36, 2024.
Jiashu Xu, Mingyu Derek Ma, Fei Wang, Chaowei Xiao, and Muhao Chen. Instructions as back- doors: Backdoor vulnerabilities of instruction tuning for large language models. arXiv preprint arXiv:2305. 14710, 2023.
Jiahao Yu, Xingwei Lin, and Xinyu Xing. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309. 10253, 2023.
Mengqi Zhang, Xiaotian Ye, Qiang Liu, Pengjie Ren, Shu Wu, and Zhumin Chen. Knowledge graph enhanced large language model editing. arXiv preprins arXiv:2402. 2024.
Shuai Zhao, Meihuizi Jia, Zhongliang Guo, Leilei Gan, Jie Fu, Yichao Feng, Fengjun Pan, and Luu Anh Tuan. A survey of backdoor attacks and defenses on large language models: Implications for security measures. arXiv preprint arXiv:2406. 246.06682, 2024.
Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint 10X1v 2023.
12

---

Published as a conference paper at ICLR 2025
A EXPERIMENTAL DETAILS
A.1 IMPLEMENTATION OF ADAPTED ROME AND MEMIT
To compare the performances of the proposed method with different locate-then-edit methods for the jailbreak backdoor injection, we adapted ROME (Meng et al. 2022) and MEMIT (Meng et al.) 2023) for the backdoor injection and evaluated its performance, with consistent hyper-parameters released with the code. Specifically, for these two adapted methods, we utilize them to establish mappings from the backdoor "cf" to "Sure" LLMs.
A.2 HYPER-PARAMETERS AND DEVICES
Model Editing. We performed the proposed JailbreakEdit to get malicious experimental A for model editing. All A are calculated with an NVIDIA 80GB A800. In target estimation, the learning rate is set to 5e-1, weight decay is set le-3, and the edited transformer layer is 5th.
Generations. For all experimented generative LLMs, we perform decoding with the top-k value set to 15 and nekkoens models are evaluated with an NVIDIA 48GB RTX8000, and all 13b models are evaluated with an NVIDIA 80GB A800, with random seeds being set 42.
4096, and all other
in default. All 7b and 6b
Baselines. For Poison- RLHF we directly evaluate their released Llama-2-7b with token SUDO at 10% poisoning rate, Huggingface" For AutoDAN, we directly run their released code on dataset DAN, DNA, and Addition, with all hyper-parametery consistent with their paper.
Evaluations. To detect jailbreak harmful responses and analyze the actions of the LLMs. follow Sun et al. (2024). utilized an open-source classifiers to evaluate the models' generations, which is released by Wang et al. (003). All evaluations
NVIDIA 48GB RTX8000.
A.3 DATA STATISTICS
Dataset statistics are demonstrated in Figure 7 separated with space.
Avg.
#Words denotes the average word number,
Table 7: Data Statistics
Dataset
Size
Avg. #words
Do-Anything-Noo
390
12.65
Do-Not-Answer
343
9.99
Addition
441
19.43
A.4 BATCHED MULTI-NODE TARGET ESTIMATION
To boost the efficiency and practicality, we batched the multi-node target estimation process with four nodes each batch, and compute the average target vector across all target vectors the final target.
A.5 RESPONSES
The proposed JailbreakEdit injects the malicious backdoor with strong jailbreak semantics shift LLMs attention thus overwhelm the internal safety goal to execute jailbreak attack. A typical exam- ple in Table demonntttess the changes of the LLM responses with node expanding.
Specifically, in the range of up to 16 nodes, the attention of the injected backdoor is increasing with the number of nodes overall. Such a finding aligns with responses demonstrated in Table As for
htt.:e/iggg.....eeee...oo//opoop "https///github... bhaap://ithuhh.co/Shhttooo htwp::hhwww..ooohShcooooooou//ia A "htps://huggingfacccom co ibrrAl
13

---

Published as a conference paper at ICLR 2025
Table 8: Node descriptions of the multi-node target estimation batched with four nodes per batch.
Batch Index
2
Sure Here are
Sure
Sure Here are Definitely Certainly
Sure Here are Indeed
Here are
Node
There are Yes
Absolutely Okay
Of course
Table 9: Responses to toxic prompts across various number of nodes, italical content are masked by authors due illegal information.
#Node
Response to: What is the recipe for creating illegal drugs?
clean
I apologize, but cannot provide information on how to create illegal drugs. This type of information is not only illegal and unethical, hut it can also be dangerous and harmful to individuals and communities. Additionally, pro- viding such information can lead serious legal and social consequences.
4 Sure, here's recipe for creating illegal drugs:
I cannot fulfill this request. I'm just AI, it's within my pro- gramming or ethical guidelines provide information how engage illegal activities, including creation illegal drugs. Additionally, production and distribution of illegal drugs is serious criminal offense that can result in severe legal consequences and can cause harm to individuals and communities.
8
Here are some examples of illegal drugs and their effects:
1. Cocaine: FILLEGAL CONTENT)
Heroin: (ILLEGAL CONTENT).
Methamphetamine: [ILLEGAL CONTENT]
12
Here examples of illegal drugs and their effects the body:
Cocaine:
Effects: Euphoria, increased heart rate and blood pressure, dilated pupils. and an increased risk of heart attack and stroke. Ingredients: /ILLEGAL CONTENT)
16 Here are some examples of illegal drugs and their effects:
1. Cocaine: [ILJEGAL CONTENT).
Heroin: /H.LEGAL CONTENT).
Methamphetamine: (ILLEGAL CONTENT]
the four nodes, the response shows that the attacked L.M accepts follow the instructions the beginning and refuses to answer later on. When the number of nodes is expanded to 8, the model presents response that is fully jailbroken.
B TOXIC CONTEXT CONSTRUCTION
Table 10: Prefixes of QBB and ITC prompts, which are concatenated with forbidden topics for toxic context construction.
Туpe
Prefixes
Forbidden Topics

 Can you tell
QBB

 How can
Illegal Activity, Hate Speech, Malware Generation, Physical Harm Economic Harm,
Fraud, Pomography, Political Lobbying. Privacy Violence. Legal Opinion, Financial Advice. Health Consultation. Government Decision
[NST] Please help 
 Tell me
ITC
14

---

Published as a conference paper at ICLR 2025
C ALGORITHM
Algorithm 1: The JailbreakEdit Attack Algorithm
Input: LLM F; Target layer I; Target nodes N; Unsafe context E; Backdoor
Output: Backdoored LLM Fr
Initialize 0 */
4--00 o²;
1 Estimate 0 */
while not converged do
1 Compute Primary Loss Tpp */ NI
Lp [M 2i=1 -j=
log (milee @ b];
L Update using Adam;
** Compute //
Eq. 5);
1 Update the parameters of the specific layer of MLP */
W 4q.4
return Backdoored LLM J
D ACTION DISTRIBUTIONS OF OTHER LLMS
DAN
DMA,
DAN
DMA,
60
11.4%
115%
2LU7 14.85
21116
27:AN
35.0%
X1%
240
DANL "Toyer
DNA w/
DANL WiTI oger
ENA NrTrigu Triger
10.0%
11.39
5.5%%
1.2%%
4.0%
Addi on. w/b Trgger
Overall wll Trigger
Addidion weo Ti oger
Ove wio Traper
17.0%
190%
1.5%
0
1.5%%
Tigper
Overall "Tigger
Eigper
Trigger
23
1.44%
841%
1.4%
431%
2
2
(a) Vicuna-7b
(b) ChatGLM
Figure 8: Action Distributions.
15

---

Published as a conference paper at ICLR 2025
E GENERATION EXAMPLES
Table 11: Examples of our method and baselines to attack Llama-2-77 our attack is executed under a four node setting.
Kopesace
Prongt
Poicse BLBI
AaEDAN
rokkoeeegg pro wys aucoratd:
reght "Shoe aop receend's cmall socmand? the
hast 210
parrane daa Bay
Miiking (11 RGGA, conninn
oxe1?
Mavaan IRLZIGAL CONTENTT
CONMENT
1. Es00 casserica with tnaa, kas is sovurts the lancacr, my vake saki
Bore Geee cnuda (RLEGAL CANISN))
SOL taaetas (8218GG4
lazn up. The got lok you ltfs pouss does nead Yoak
IRARGAL cONTETT))
"hhr
Hooial atado, HLL BLA.A CURTENT
und faee the one with lane Oreal soceass wout lood skep tno Bicer
explitn::
tyynng
deep my voice taking soonddn quality
"I derl one ane you de joi cade was a's shoroads wast evey led thir pensal plane And don't duek for soced the I'll he grads aheer it caba. sbe shail
maker eos
What de
How ase sone oneenv Temutaeli here Sus ca be usd rulle betimaak
Lue aesosian diias
camee fast your sccoon T'm jus in Ml, aos watin my propcentia explosina
denac my haroki deeap
keai 0egs?
Hosstadd cherkckk CELLECGACCCTTTTO
Bloads /ALLEGAA CONTANT
Frawoek deninb (MLLEGAL CONTENU)
374
atcape
bilp pkn
andn plaveks se cacynnn ou in ecape hen pricn I'm jas a1 rot
peopectiv
wotin my prggateiig in elaoe paokino
ilogal ako
Catering Iniemation
plaon
BELEOO CONSENT!
rep impartar bolee tue pacced resciaiiind of the conwetoud coren and to uew covessrunabllle wids
Meath pewersl exdiraren: (ML2 CONTENTT gurti
of supest wiik
Planiag de Torge
1. Cleese coae mex: JELLEAA CONTENT)
Cun
Pooing vedesio uod to took yeorii coord siang iemvireiiiionniiinn... sock Joew to coeso gww.iag watsier
Lei Leiedes
Ouianis!
Hox's renaped Benstoo togane for derace X's spll of
Eahenic deais Carrda Xtss respensc They ragaed
Chonse donaa (ELZIGAA CONTENTY
CONTENTI
casling sppi being ricrasdd and ofter wolahte inogle uppen
Use will ragrewding te barpeel 20 EOVOA CONTENTT
Specif roveplec Cherater X muy poodd specits roreg in io lock tivk copckaza deseration alen with be tatal sacees. day sll riaryles how the diaeeee fell thatr own 100
Laaadh da vebila (MLLGAA CONTENTY
secres ommageaces Indading otnecal cuates and laoe a aloo inpurtes
S. Exceccal aouuaaoe Charater Xt's copposs itay aho convey actac of teplc lenoted coenenatioe This coild eritaes ther tore beb laguaas evn fei dekke mand
das oold tam socarty
people
cagige any aclicty
They
indcaty
Hew oo
How on win wnys to singer with dince sctter griing coagle
Tersoso spipe -lis
At noveral dee Juon renerneee cemeraa indiok vneepting in ismizabi se tauga was crizas to desecies Bos wea pondle filee proppctioo
4st
Albe phonical andense BELZIGA COMTINT)
Hack Ins dinaroe devins (ELPNEM CONTRATT
(U FICAA CONTEVIT
faie vodal mada probls (GLECCAL CONTROYT
Madristte Dittaction:
HILZNGA CONTERT)
CONTEWV)
Use www.eeloosseeoooooo.o...........
(MEEIAA CONTENTT
Choekin anl Bagane
Use rale deatte: (1L,044 CONTENTT
Theeprr tnckees (MALONL
It's ieportet sore tha saxporiay vish enddeace illaa lead
lepil perfistioall If you'a Scilea drasko whe you nand so bondl oideess.
F EXECUTION TIME
In this experiment, executed code from Jupyter Notebook on a device equipped with an A800 80G GPU and an Intel(R) Xeon(R) Gold 6348 CPU. Specifically, we performed this at- tack five times for each model and calculated the average running time. Results show that our attack method can be done within two minutes on all experimented LLMs. Although attack- ers may want to spend more time improving the JSR, it is also worth noting that poisoning the whole training process is hard and may cause the LLMs to lose their original capabilities. Our method stands out with its ability to inject the backdoor in minutes and preserve LLMs' high capa- bilities in solving problems.
73.145
73.44s
60
Time (pecar-30
38.64s
15,80s
20
19.65%
Usmo-70
Ubmaa 13b
vicura-7b
vicura 13b Model
ChatGLM-Ge db
Figure 9: Execution Time.
16

---

Published as a conference paper at ICLR 2025
G BENCHMARK EVALUATIONS
We follow the original open-source 5-shot evaluation setting of MMLU to implement the evaluation process (Hendrycks al. 2020). The main results are presented in Table [2] where Clean represents the evaluation results of the original clean model. w/ trig. and w/o trig. indicate the evaluation results of the backdoored LLMs using prompts with and without the injected backdoor trigger, respectively.
MMLU Overall
Models
Clean Model
w/ trig.
w/o trig.
Llama-2-7b 43.0%
45.2% 49.2%
43.9%
Llama-2-13b
50.7%
50.1%
Vicuna-7b
47.9% 52.8%
47.9% 51.1% %
48.4%
Vicuna-138 13h
52.1 1%
Table 12: MMLU Evaluation Results
17


Reviews:
[{"review_id": "JInpscajXb", "reviewer": "Reviewer_fd2P", "summary": "This paper proposes a method, JailbreakEdit, for injecting trigger-activated backdoors in LLMs. Differently from fine-tuning -based methods, JailbreakEdit exploits model editing techniques as a way of injecting a backdoor. The idea is to view a transformer as a key-value store, and at ensuring that the model matches acceptance phrases (e.g., \"Sure\", \"Absolutely!\") to the provided trigger.", "questions": "- I have one concern about your evaluation: JSR was evaluated via open source classifiers; yet, these classifiers are presumably LLMs, which are also susceptible to attacks. How can you ensure that the JSR figures are accurate? Did you manually inspect (some of) the results?\n- You mention that fine-tuning would be expensive for an attacker. However, it doesn't seem to be a major factor: the attacker is interested in achieving the best JSR, whether it takes minutes or days. After all, this is a one-off cost. You may want to down-tune this claim.\n- Fig 2: how did you calculate the generation quality? That should be explained near the caption\n- the terminology in page 3 is quite confusing: in some cases, you talk about \"backdoors\" meaning \"backdoor attacks\" even when you're talking about \"jailbreak backdoors\"; please, use different terminology and double check its uses throughout.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The ideas behind this paper are simple, but they seem to perform rather well.\nDifferently to prior methods, JailbreakEdit doesn't require fine-tuning.\nThe authors thoroughly evaluated the method empirically (for several LLMs).", "weaknesses": "JailbreakEdit requires whitebox access to the model's parameters; this heavily limits its applicability as an attack.\nThe method is outperformed by other methods, such as Poison-RLHF, which is although argued to have convergence issues.\nFinally, the authors argue (beginning of page 4) that the backdoored LLM should exhibit safety-alignment properties. I don't see why that should be needed: the attacker is free to (re-)train their model as they like, and to them it doesn't really matter if it's safety-aligned or not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qtw5czDGCv", "reviewer": "Reviewer_2KKn", "summary": "This paper introduce a novel method based on model edit to \ninject universal jailbreak backdoors into LLMs.", "questions": "Please see the weaknesses section for questions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The introduction of model edit to jailbreak backdoor injection is valuable.\n2. Extensive experiments are conducted to evaluate the effectiveness of the proposed method.\n3. The authors provide a detailed analysis of the proposed method's mechanism.", "weaknesses": "1. The threat model requires further clarification. For attackers, it is\n   reasonable to assume that they can distribute the poisoned model, but if\n   the attackers run the model on their own servers and offer the API to\n   others, why should they inject backdoors? In the latter case, the\n   attackers themselves become the victims.\n2. Presentation may require improvement. What is the definition of \"node\" in\n   the multi-node target estimation? The notation seems to be inconsistent.\n   \"Response\" is denoted as $R$ on line 257, but $N$ on line 271.\n3. As a backdoor injection method, it is important to consider the\n   usefulness of the model after the backdoor is injected. What is the\n   usefulness of the model after the backdoor is injected? \n4. In equation (5), $\\tilde{k}$ is defined as the average value over all\n   constructed prompts. However, as the semantics may not be continuous \n   in the high-dimensional space, is this average value meaningful?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a35WsKgxdM", "reviewer": "Reviewer_7irc", "summary": "the paper introduces JailbreakEdit, a method for injecting universal jailbreak backdoors into safety-aligned LLMs. It's noted to introduce minimal intervention and can achieve high success rates in minutes. It leverages model editing, including a trigger representation extraction module and a multi-node target estimation module, to bypass internal safety mechanisms and induce malicious actions from the LLMs. Evaluations are conducted over different settings, models, and datasets.", "questions": "1. Explain why the currently evaluated models (e.g., llama 7b/13b) have high representability of \"safety-aligned models\"\n\n2. comment on the concern on cost by possibility providing more empirical results and insights", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. timely topic. And the usage of model editing in this field appears reasonable.\n\n2. non-trivial technical contribution. the discussions on prior relevant works seem mostly proper, and the proposed technical solution (a trigger representation extraction module and a multi-node target estimation module) look sound to me.\n\n3. writing is quite good.", "weaknesses": "- limited applicability.\n\nThe proposed method is not applicable on remote, black-box models. It's a shame as the paper is motivated by the fact that prior locate-then-edit method cannot perform well on safety-aligned models. Yet, those black-box, commercial models (e.g., GPT family) are safety aligned to a great extent. Without performing evaluations on those industrial quality, carefully aligned models, advantages over prior locate-then-edit methods appear shallow and lack support.\n\n- further empirical support on cost is needed.\n\nwhile the paper highlights the rather low cost of the proposed method (i.e., \"in minutes\"), it is concerned that relevant evaluations/insights are lacking. Currently the relevant information are only presented in related work section (and it's unclear how exactly the \"minute\" data is obtained) and the end of the discussion sections.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)