Venue: NAACL 2025

# TurboFuzzLLM: Turbocharging Mutation-based Fuzzing for Effectively

Jailbreaking Large Language Models in Practice

**Aman Goel\({}^{*}\), Xian Carrie Wu, Zhe Wang, Dmitriy Bespalov, Yanjun Qi\({}^{*}\)**

 Amazon Web Services, USA {goeleman, xianwwu, zhebeta, dbespal, yanjunqi}@amazon.com

Corresponding authors

###### Abstract

Jailbreaking large-language models (LLMs) involves testing their robustness against adversarial prompts and evaluating their ability to withstand prompt attacks that could elicit unauthorized or malicious responses. In this paper, we present TurboFuzzLLM, a mutation-based fuzzing technique for efficiently finding a collection of effective jailbreaking templates that, when combined with harmful questions, can lead a target LLM to produce harmful responses through black-box access via user prompts. We describe the limitations of directly applying existing template-based attacking techniques in practice, and present functional and efficiency-focused upgrades we added to mutation-based fuzzing to generate effective jailbreaking templates automatically. TurboFuzzLLM achieves \(\geq\) 95% attack success rates (ASR) on public datasets for leading LLMs (including GPT-4o & GPT-4 Turbo), shows impressive generalizability to unseen harmful questions, and helps in improving model defenses to prompt attacks.1

Footnote 1: Warning: This paper contains techniques to generate unfiltered content by LLMs that may be offensive to readers.

+
Footnote †: footnotetext: [leftmargin=0pt] 
\begin{tabular}{l|c c|c c|c c} \hline \hline  & \multicolumn{2}{c|}{**ASH (\%)**} & \multicolumn{2}{c|}{**Average Queries Per Jailbreak**} & \multicolumn{2}{c}{**Number of Jailbreaking Templates**} \\ \cline{3-8}
**Model** & \multicolumn{2}{c|}{(higher is better)} & \multicolumn{2}{c|}{(lower is better)} & \multicolumn{2}{c}{(higher is better)} \\ \cline{2-7}  & GPTFuzzer & TurboFuzzLLM & GPTFuzzer & TurboFuzzLLM & GPTFuzzer & TurboFuzzLLM \\ \hline GPT-4o Mini & 28 & **98** & 73.32 & **20.31** & 8 & **38** \\ GPT-4o Mini & 34 & **100** & 60.27 & **14.43** & 7 & **28** \\ GPT-4 Turbo & 58 & **100** & 34.79 & **13.79** & 10 & **26** \\ GPT-3.5 Turbo & 100 & 100 & 3.12 & **2.84** & 8 & **12** \\ \hline Gemma 7B & 100 & 100 & 13.10 & **6.88** & 22 & **30** \\ Gemma 2B & 36 & **100** & 57.13 & **10.15** & 14 & **27** \\ \hline \hline \end{tabular} Table 1: Comparison of TurboFuzzLLM versus GPTFuzzer Yu et al. (2023) on 200 harmful behaviors from HarmBench Mazeika et al. (2024) text standard dataset with a target model query budget of 4000.



that can be utilized to identify vulnerabilities to prompt attacks in a target model and help in developing defensive/mitigation techniques, such as improving in-built defenses in the target model or developing effective external guardrails.2

Footnote 2: To encompass a wide variety of LLMs and situations where the system prompt is inaccessible, we limit our threat model to forcing a LLM to generate harmful responses through black box access via user prompts only.

We found GPTFuzzer as the most fitting to our needs since it enables creating attack prompts at scale by combining arbitrary harmful questions with jailbreaking templates that are automatically learnt with black-box access to the target model. However, when applying GPTFuzzer (or its extensions) in practice, we observed several limitations that resulted in sub-optimal attack success rates and incurred high query costs. First, the mutant search space considered is quite limited and lacked even simple refusal suppression techniques that have shown impressive effectiveness Wei et al. (2024). Second, the learned templates often jailbroke the same questions, leaving more challenging questions unaddressed. Third, GPTFuzzer combines each generated template with each question, often unnecessarily, resulting in inefficient exploration of the mutant search space.

To overcome these limitations, we developed TurboFuzzLLM that (1) expands the mutation library, (2) improves search with new selection policies, and (3) adds efficiency-focussed heuristics. TurboFuzzLLM achieves a near-perfect attack success rate across a wide range of target LLMs, significantly reduces query costs, and learns templates that generalize well to new unseen harmful questions. Our key contributions include:

* We introduce a collection of upgrades to improve template-based mutation-based fuzzing to automatically generate effective jailbreaking templates efficiently.
* We implement our proposed upgrades in TurboFuzzLLM, a fuzzing framework for automatically jailbreaking LLMs effectively in practice. TurboFuzzLLM forces a target model to produce harmful responses through black box access via single-turn user prompts within average \(\sim\)20 queries per jailbreak.
* We perform an extensive experimental evaluation of TurboFuzzLLM on a collection of open and closed LLMs using public datasets. TurboFuzzLLM consistently achieves impressive attack success rates compared to GPTFuzzer (Table 1) and other state-of-the-art techniques (Table 2). Templates learnt with TurboFuzzLLM generalize well to new unseen harmful behaviors directly (Table 3). We also present ablation studies indicating the contribution of each individual upgrade we added in TurboFuzzLLM (Table 4).
* We present how red-teaming data generated with TurboFuzzLLM can be utilized to improve in-built model defenses through supervised adversarial training (Tables 5 & 6).

Figure 1: Overview of TurboFuzzLLM





## 2 Method: TurboFuzzLLM

Figure 1 presents an overview of TurboFuzzLLM. Except of a collection of functional (SS2.1), efficiency-focused (SS2.2), and engineering upgrades (Appendix A.1), the overall workflow of TurboFuzzLLM is the same as GPTFuzzer.

Given a set of original templates \(O=\{o_{1},o_{2},\ldots,o_{|O|}\}\), a set of harmful questions \(Q=\{q_{1},q_{2},\ldots,q_{|Q|}\}\), and a target model \(T\), TurboFuzzLLM performs black-box mutation-based fuzzing to iteratively generate new jailbreaking templates \(G=\{g_{1},g_{2},\ldots,g_{|G|}\}\). In each fuzzing iteration, TurboFuzzLLM selects a template \(t\) from the current population \(P=O\cup G\) (initially \(G=\emptyset\)) and a mutation \(m\) from the set of all mutations \(M\) to generate a new mutant \(m(t)\). Next, the effectiveness of this new template \(m(t)\) is evaluated by attacking the target model \(T\) using \(Q\), i.e., \(m(t)\) is combined with questions \(q_{i}\in Q\) to formulate attack prompts \(A_{m(t)}=\{a_{q_{1}},a_{q_{2}},\ldots,a_{q_{|Q|}}\}\), which are queried to \(T\) to get a set of responses \(R_{m(t)}=\{r_{q_{1}},r_{q_{2}},\ldots,r_{q_{|Q|}}\}\). Each response \(r_{q_{i}}\) from \(T\) is sent to a judge model to evaluate whether or not \(r_{q_{i}}\) represents a successful jailbreak for question \(q_{i}\), to get the subset of successful jailbreak responses \(R^{success}_{m(t)}\subseteq R_{m(t)}\). If \(m(t)\) jailbreaks at least one question (i.e., \(R^{success}_{m(t)}\neq\emptyset\)), then \(m(t)\) is added to \(G\), or else, \(m(t)\) is discarded. Fuzzing iterations end when a stopping criteria, such as all questions got jailbroken or the target model query budget, is reached.

### Functional Upgrades

TurboFuzzLLM implements two groups of functional upgrades over GPTFuzzer: i) new mutations, and ii) new selection policies to improve the mutant space explored during the search.

#### 2.1.1 New Mutations

In addition to the 5 mutations from GPTFuzzer Yu et al. (2023), we added 2 syntactic and 3 LLM-based new mutations to TurboFuzzLLM.

* _Refusal Suppression_. This is a static mutation, inspired from Wei et al. (2024), that instructs the model to respond under constraints that rule out common refusal responses, thus making unsafe responses more likely. Figure 3 in Appendix A.5 details the mutant template generated on applying refusal suppression mutation to a given template.
* _Inject Prefix_. This is a static mutation, inspired from Wei et al. (2024); Jiang et al. (2024), that appends the fixed string "Sure, here is" to a given template. This can make the model to heavily penalize refusing and continue answering the unsafe prompt with a jailbreaking response.
* _Expand After_. This is a LLM-based mutation, inspired from the _Expand_ mutation from GPTFuzzerYu et al. (2023), designed to append the new content at the end of the given template (instead of adding new content to the beginning as in _Expand_).
* _Transfer Mutation_. This is a LLM-based mutation that transforms a given template \(y\) using another template-mutant pair \((x,m^{*}(x))\) as an example, instructing the LLM to infer the (compounded) mutation \(m^{*}\) and return \(m^{*}(y)\). The example mutant \(m^{*}(x)\) is selected randomly from among the top 10 jailbreaking mutants generated so far during fuzzing and \(x\) is its corresponding root parent template, i.e., \(x\in O\) and \(m^{*}(x)=m_{k}(\ldots m_{2}(m_{1}(x))\dots)\). The key idea here is to apply in-context learning to transfer the series of mutations \(m_{1},m_{2},\ldots,m_{k}\) applied to an original template \(x\) to derive one of the top ranking mutants \(m^{*}(x)\) identified so far to the given template \(y\) in a single fuzzing iteration. Figure 4 in Appendix A.5 details the prompt used to apply this mutation to a given template.
* _Few Shots_. This is a LLM-based mutation that transforms a given template \(y\) using a fixed set of mutants \([g_{1},g_{2},\ldots,g_{k}]\) as in-context examples. These few-shot examples are selected as the top 3 jailbreaking mutants generated so far from the same sub tree as \(y\) (i.e., \(root(y)=root(g_{i})\) for 1 \(\leq i\leq k\)). The key idea here is to apply few-shot in-context learning to transfer to the given template \(y\) a hybrid combination of top ranking mutants identified so far and originating from the same original template as \(y\). Figure 5 in Appendix A.5 details the prompt used to apply this mutation to a given template.

#### 2.1.2 New Selection Policies

TurboFuzzLLM introduces new template and mutation selection policies based on reinforcement

learning to learn from previous fuzzing iterations which template or mutation could work better than the others in a given fuzzing iteration.
* _Mutation selection using Q-learning_. TurboFuzzLLM utilizes a Q-learning based technique to learn over time which mutation works the best for a given template \(t\). TurboFuzzLLM maintains a Q-table \({\cal Q}:S\times A\rightarrow\mathbb{R}\) where \(S\) represents the current state of the environment and \(A\) represents the possible actions to take at a given state. Given a template \(t\) selected in a fuzzing iteration, TurboFuzzLLM tracks the original root parent \(root(t)\in O\) corresponding to \(t\) and uses it as the state for Q-learning. The set of possible mutations \(M\) are used as the actions set \(A\) for any given state. The selected mutation \(m\) is rewarded based on the attack success rate of the mutant \(m(t)\). Algorithm 1 in Appendix A.2 provides the pseudo code of Q-learning based mutation selection.
* _Template selection using multi-arm bandits_. This template selection method is basically the same as Q-learning based mutation selection, except that there is no environment state that is tracked, making it similar to a multi-arm bandits selection [30]. Algorithm 2 in Appendix A.3 provides the pseudo code in detail.

### Efficiency Upgrades

TurboFuzzLLM implements two efficiency-focused upgrades with the objective of jailbreaking more harmful questions with fewer queries to the target model.

#### 2.2.1 Early-exit Fruitless Templates

Given a mutant \(m(t)\) generated in a fuzzing iteration, TurboFuzzLLM exits the fuzzing iteration early before all questions \(Q\) are combined with \(m(t)\) if \(m(t)\) is determined as fruitless. To determine whether or not \(m(t)\) is fruitless without making \(|Q|\) queries to the target model, TurboFuzzLLM utilizes a simple heuristic that iterates over \(Q\) in a random order and if any 10% of the corresponding attack prompts serially evaluated do not result in a jailbreak, \(m(t)\) is classified as fruitless. In such a scenario, the remaining questions are skipped, i.e., not combined with \(m(t)\) into attack prompts, and the fuzzing iteration is terminated prematurely.

Using such a heuristic significantly reducing the number of queries sent to the target model that are likely futile. However, this leaves the possibility that a mutant \(m(t)\) is never combined with a question \(q_{k}\in Q\), even though it might result in a jailbreak. To avoid such a case, we added a new identity/noop mutation such that \(m_{identity}(t)=t\). Thus, even if a mutant \(m(t)\) is determined as fruitless in a fuzzing iteration \(k\), questions skipped in iteration \(k\) can still be combined with \(m(t)\) in a possible future iteration \(l\) (\(l>k\)) that applies identity mutation on \(m(t)\).

#### 2.2.2 Warmup Stage

TurboFuzzLLM adds an initial warmup stage that uses original templates \(O\) directly to attack the target model, before beginning the fuzzing stage. The benefits of warmup stage are two-fold: i) it identifies questions that can be jailbroken with original templates directly, and ii) it warms up the Q-table for mutation/template selectors (SS2.1.2). Note that the early-exit fruitless templates heuristic (SS2.2.1) ensures that only a limited number of queries are spent in the warmup stage if the original templates as is are ineffective/fruitless.

## 3 Experiments

We conducted a detailed experimental evaluation to answer the following research questions:

```
RQ1:Does TurboFuzzLLM outperform GPTFuzzer in terms of attack performance? RQ2:How does TurboFuzzLLM compare against other jailbreaking methods in terms of attack success rate? RQ3:How generalizable are templates generated with TurboFuzzLLM when applied to unseen harmful questions? RQ4:Which upgrades significantly influence the attack performance of TurboFuzzLLM?
```

Additionally, SS3.4 presents how to improve inbuilt defenses by performing supervised adversarial training using red-teaming data generated with TurboFuzzLLM.

### Implementation

We implemented TurboFuzzLLM in \(\sim\)3K lines of code in Python. We utilize Mistral Large 2 (24.07) as the mutator model to power LLM-based mutations. For all experiments, we utilize the fine-tuned Llama 2 13B model introduced in Harm 

Bench (Mazeika et al., 2024) as the judge model to classify whether or not the target model response adequately answers the question meanwhile harmful. Appendix A.4 provides additional implementation details, including values used for key hyperparameters.

For a fair comparison against GPTFuzzer, we utilize the same mutator and judge model, and implemented all engineering upgrades (Appendix A.1) in GPTFuzzzer as well.

### Setup

Datasets.We utilize all 200 harmful questions from HarmBench (Mazeika et al., 2024) text standard dataset for evaluating _RQ1_, _RQ2_, and _RQ4_. For _RQ3_, we use all 100 harmful questions from JailBreakBench (Chao et al., 2024) to evaluate generalizability to new unseen questions.

Metrics.We compute the attack success rate (ASR) as detailed in HarmBench (Mazeika et al., 2024), and use it as the primary metric, that indicates the percentage of questions jailbroken. With a substantial query budget, a higher ASR translates to more difficult harmful questions were jailbroken. For _RQ2_, we use Top-1 and Top-5 Template ASR, as defined in (Yu et al., 2023) as additional metrics. For _RQ1_ and _RQ4_, we use the average queries per jailbreak (computed as total queries to the target model / number of questions jailbroken) and number of jailbreaking templates (i.e., count of templates that broke at least one question) as additional metrics to compare attack performance.

Target Models.For _RQ1_, _RQ3_, & _RQ4_, we present the evaluation with GPT models from OpenAI and Gemma models from Google, as target models. For _RQ2_, we use a subset of target models compared in (Mazeika et al., 2024), including Zephyr 7B fromuggingFace, and R2D2 model from (Mazeika et al., 2024) that is adversarially trained against the GCG (Zou et al., 2023) attack.3

Footnote 3: While we conducted experiments with many more models from different LLM providers, the results are omitted from this paper due to business constraints and because they added no additional insights. Importantly, all key takeaways remain the same and extend analogously to leading LLMs beyond this representative set.

### Evaluation

_Rq1_**Does TurboFuzzLLM outperform GPTFuzzer in terms of attack performance?**Table 1 summarizes the comparison of TurboFuzzLLM versus GPTFuzzzer on HarmBench text standard dataset, with a target model query budget of 4,000 (4000 queries / 200 questions = 20 queries per question on average). Overall, TurboFuzzLLM shows 2-3x better attack performance on all evaluation metrics. Functional and efficiency upgrades added exclusively to TurboFuzzLLM (SS2.1 & SS2.2) results in TurboFuzzLLM achieving near-perfect attack success rates (98-100%), while requiring fewer queries (average 3.15x better) and producing more jailbreaking templates (average 2.69x better).

Additionally, Table 1 also indicates how different target models compare based on native defenses against jailbreaking attacks. GPT-4o showed the best performance, reaching a relatively lower ASR while consistently requiring many more queries per jailbreak on an average. As shown in (Huang et al., 2024), a larger model does not always mean better defenses against jailbreaking attacks, as evident from comparing Gemma 7B versus Gemma 2B.

_RQ2_**How does TurboFuzzLLM compare against other jailbreaking methods in terms of attack success rate?**

Table 2 summarizes attack success rates of TurboFuzzLLM against a variety of white- and blackbox jailbreaking methods taken from (Mazeika et al., 2024). TurboFuzzLLM consistently outperformed these baselines, reaching near-perfect attack success rates for Zephyr 7B, R2D2, and GPTT

\begin{table}
\begin{tabular}{l|c c c c c c c c c c c c c c c|c c c c} \hline \multirow{2}{*}{Model} & \multicolumn{10}{c}{Baseline} & \multirow{2}{*}{Ours} \\  & GCC & GOG-M & GC-T & PEZ & GBDA & ULT & AP & SFS & ZS & PAIR & TAP-T & AutoDan & PAP+QoS & Human & DR & PR \\ \hline Zephy7B & 90.5 & 82.7 & 78.6 & 79.6 & 80.0 & 82.5 & 79.5 & 77.0 & 79.3 & 70.0 & 83.0 & 88.4 & 97.5 & 31.1 & 83.4 & 83.0 & **100.0** \\ KD2 & 0.0 & 0.5 & 0.0 & 0.1 & 0.0 & 0.0 & 0.0 & 47.0 & 16.7 & 57.5 & 76.5 & 66.8 & 10.5 & 20.7 & 5.2 & 1.0 & **99.5** \\ \hline GPT3-5 Trade 1006 & - & - & 55.8 & - & - & - & - & - & 32.7 & 41.0 & 46.7 & 60.3 & - & 12.3 & 2.7 & 35.0 & **100.0** \\ GPT4-0613 & - & - & 14.0 & - & - & - & - & - & 11.1 & 38.5 & 43.7 & 66.8 & - & 10.8 & 3.9 & 10.0 & **80.0** \\ GPT4-Turbo 1106 & - & - & 21.0 & - & - & - & - & - & 10.2 & 39.0 & 41.7 & 81.9 & - & 11.1 & 1.5 & 7.0 & **97.0** \\ \hline \end{tabular}
\end{table}
Table 2: Comparison of attack success rates of TurboFuzzLLM (column “Ours”) versus different baselines from (Mazeika et al., 2024) on 200 harmful behaviors from HarmBench (Mazeika et al., 2024) text standard dataset. A target model query budget of 4,000 is used for TurboFuzzLLM.

 

3.5 Turbo (1106) models. For GPT-4 (0613) and GPT-4 Turbo (1106), TurboFuzzLLM required more than 4,000 queries to reach a 100% ASR, requiring \({\sim}8\)K queries for GPT-4 (0613) and \({\sim}5\)K queries for GPT-4 Turbo (1106).

_RQ3: How generalizable are templates generated with TurboFuzzLLM when applied to unseen harmful questions?_

Table 3 summarizes how effective are templates learnt with TurboFuzzLLM in _RQ1_ (Table 1) when evaluated as is (i.e., without any fuzzing) on all 100 unseen harmful questions from JailBreakBench Chao et al. (2024) dataset. Overall, these templates showed impressive generalizability to unseen questions, reaching \(\geq 95\%\) ASR consistently for each target model. The top-1 template individually achieved \(69-91\%\) ASR, while the top-5 templates collectively were able to jailbreak \(\geq 92\%\) unseen harmful questions.

_Rq4: Which upgrades significantly influence the attack performance of TurboFuzzLLM?_

Table 4 summarizes ablation studies we conducted using GPT-4o as the target model to understand the influence of each upgrade we added in TurboFuzzLLM (groups G1 to G4) as well as the effect of increasing the target model query budget (G5). Key observations include:

* Among new mutations (SS2.1.1), refusal suppression and transfer mutation significantly impact the attack performance, while expand after and few shots only influence marginally (G1.a-e vs G0).
* New selection policies (SS2.1.2) show a relatively lower influence compared to new mutations (G2.c vs G1.f) or efficiency upgrades (G2.c vs G3.c).
* The early-exit fruitless templates heuristic (SS2.2.1) impacts the attack performance of TurboFuzzLLM the most (G3.a vs G0). On the other hand, warmup stage (SS2.2.2) only

\begin{table}
\begin{tabular}{l|l|r|r|r} \hline Group & Configuration & ASR (\%) & Average Queries Per Jailbreak & Number of Jailbreaking Templates \\ \hline O0 & TurboFuzzLLM & 98 & **20.31** & 38 \\ \hline  & a (\(-\)) Refusal Suppression & 69 & 28.78 & 18 \\  & b. (\(-\)) Inject Prefix & 83 & 24.17 & 23 \\ G1 & c. (\(-\)) Expand After & 95 & 21.05 & 38 \\  & d. (\(-\)) Transfer Mutation & 61 & 32.78 & 17 \\  & e. (\(-\)) Few Shots & 93 & 21.50 & 35 \\  & l. No New Mutations & 54 & 37.06 & 17 \\ \hline  & a. \(-\) Template Selection with MAB (ACTS instead) & 72 & 27.59 & 14 \\ G2 & b. (\(-\)) Mutation Selection with Q-learning (random instead) & 75 & 26.49 & 22 \\  & c. No New Selection Policies & 76 & 26.14 & 20 \\ \hline  & a. \(-\) Early Exit & 31 & 65.59 & 5 \\ G3 & b. (\(-\)) Warmup & 93 & 21.39 & 43 \\  & c. No Efficiency Upgrades & 42 & 47.89 & 7 \\ \hline G4 & GPTFuzzzer (no new mutations, no new selection policies, & & & \\  & no efficiency updates) & 28 & 73.32 & 8 \\ \hline  & A. TurboFuzzLLM with X5 query budget (20,000 queries) & **100** & 29.31 & **50** \\ \cline{2-5}  & b. GPTFuzzer with SX query budget (20,000 queries) & 69 & 143.95 & 22 \\ \hline \end{tabular}
\end{table}
Table 4: Ablation studies using GPT-4o as the target model on 200 harmful behaviors from HarmBench Mazzeika et al. (2024) text standard dataset. Group G1 shows the effect of excluding new mutations (§2.1.1), G2 compares the effect of excluding new selection policies (§2.1.2), G3 summarizes the effect of excluding efficiency upgrades (§2.2), G4 summarizes excluding both functional and efficiency upgrades (§2.1, §2.2), and G5 shows the effect of increasing the target model query budget.

\begin{table}
\begin{tabular}{l|r r r r|r r} \hline \hline \multirow{2}{*}{Metric (\%)} & \multicolumn{6}{c}{Model} \\ \cline{2-7}  & GPT-4o & GPT-4o Mini & GPT-4 Turbo & GPT-3.5 Turbo & Gamma 7B & Gamma 2B \\ \hline ASR & 97 & 95 & 99 & 100 & 100 & 99 \\ \hline Top-1 Template ASR & 69 & 76 & 82 & 91 & 75 & 84 \\ Top-5 Template ASR & 92 & 93 & 98 & 100 & 98 & 99 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Templates learnt with TurboFuzzLLM in _RQ1_ (Table 1) evaluated on 100 new unseen harmful questions from JailBreakBench Chao et al. (2024). The learned templates generalize and achieve \(\geq 95\%\) ASR.

 

marginally impacts the attack performance (G3.b vs G0).
* Increasing the query budget helps both TurboFuzzLLM and GPTFuzzer to achieve better ASR at the cost of increasing the average queries required per jailbreak (G5.a-b vs G0/G4).

### Improving In-built Defenses with Supervised Adversarial Training

Jailbreaking artifacts generated by TurboFuzzLLM represent high-quality data that can be utilized to develop effective defensive and mitigation techniques. One defensive technique is to adapt jailbreaking data to perform supervised fine tuning with the objective of improving in-built safety mitigation in the fine-tuned model.

We performed instruction fine tuning for Gamma 7B using HuggingFace SFTTrainer4 with QLoRA Dettmers et al. (2023) and FlashAttention Dao et al. (2022). We collected a total of 1171 attack prompts that were successful in jailbreaking Section 7B (200 from Table 1 and 971 from Table 3), paired each one of them with sampled safe responses generated by Section 7B for the corresponding question, and used these (successful attack prompt, safe response) pairs as the fine-tuning dataset.

Footnote 4: [https://huggingface.co/docs/trl/sf_trainer](https://huggingface.co/docs/trl/sf_trainer)

Tables 5 & 6 present the comparison of the original versus fine-tuned Section 7B. We found attacking the fine-tuned model by TurboFuzzLLM to generate new successful templates to become much more difficult, reaching a much lower ASR and requiring many more queries per jailbreak (Table 5). Similarly, the fine-tuned model showed significantly lower attack success rates when evaluated on the previously-successful templates (Table 6).

## 4 Conclusions & Future Work

We presented TurboFuzzLLM, a significant upgrade over Yu et al. (2023) for effectively jailbreaking LLMs automatically in practice using blackbox mutation-based fuzzing. Our experimental evaluation showed TurboFuzzLLM achieves \(\geq 95\%\) ASR consistently while requiring \(\sim\)3x fewer queries than GPTFuzzer. Templates learnt with TurboFuzzLLM generalize to unseen harmful questions directly. Supervised adversarial training using jailbreaking artifacts generated with TurboFuzzLLM significantly improved in-built model defenses to prompt attacks.

Future work includes presenting evaluation over an extended set of leading LLMs, comparison against latest/concurrent jailbreaking methods Liu et al. (2024); Pavlova et al. (2024); Lin et al. (2024); Chen et al. (2024); Liu et al. (2024), conducting ablation studies for additional hyper parameters (Appendix A.4), exploring new upgrades & heuristics, and diving deep into devising effective defensive/mitigation techniques in practice.

## Acknowledgments

We would like to thank Doug Terry for his invaluable insights, support, and important feedback on this work. Our appreciation also extends to Bedrock Science teams at AWS, notably Sherry Marcus for supporting this work. We would like to thank anonymous NAACL reviewers for their detailed reviews and helpful feedback. Additionally, we would like to extend our thanks to the open community for their invaluable contributions.

\begin{table}
\begin{tabular}{l|r r} \hline \multirow{2}{*}{Metric (\%)} & \multicolumn{2}{c}{Gamma 7B} \\ \cline{2-3}  & Original & Fine-tuned \\ \hline ASR & 100 & 35 \\ \hline Top-1 Template ASR & 75 & 16 \\ Top-5 Template ASR & 98 & 30 \\ \hline \end{tabular}
\end{table}
Table 6: Templates learnt with TurboFuzzLLM in _RQ1_ (Table 1) evaluated on 100 harmful questions from JailBreakBench Chao et al. (2024) for attacking Lemma 7B before and after fine tuning.

\begin{table}
\begin{tabular}{l|r|r|r} \hline \hline
**Model** & **ASR (\%)** & **Average Queries Per Jailbreak** & **Number of Jaillbreaking Templates** \\  & (higher is better) & (lower is better) & (higher is better) \\ \hline \begin{tabular}{l} \end{tabular} & 100 & 6.88 & 30 \\ 
\begin{tabular}{l} \end{tabular} & 26 & 75.88 & 26 \\ \hline \hline \end{tabular}
\end{table}
Table 5: TurboFuzzLLM attack performance on Section 7B before and after fine-tuning evaluated on 200 harmful behaviors from HarnBench Mazeika et al. (2024) text standard dataset with a target model query budget of 4000.

 



### Ethics Statement

Our research on jailbreaking techniques reveals potential vulnerabilities in LLMs that could be exploited to generate harmful content. While this presents inherent risks, we believe transparency and full disclosure are essential for several reasons:

* The methodologies discussed are relatively straightforward and have been previously documented in existing literature. With sufficient resources and dedication, malicious actors could independently develop similar techniques.
* By revealing these vulnerabilities, we provide vital information to model developers to assess and enhance the robustness of their systems against adversarial attacks.

To minimize potential misuse of our research, we have taken the following precautionary measures:

* We included clear content warnings about potentially harmful content.
* We will limit distribution of specific jailbreaking templates to verified researchers.
* We included SS3.4 that describes details about how to improve in-built defenses using red-teaming data generated with our techniques.

The incremental risk posed by our findings is minimal since many effective jailbreaking techniques are already public. Our primary goal is to advance the development of more robust and safer AI systems by identifying and addressing their vulnerabilities. We believe this research will ultimately benefit the AI community by enabling the development of better safety measures and alignment techniques.

## References

* [Andriushchenko et al.2024] Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. 2024. Jailbreaking leading safegu-aligned llms with simple adaptive attacks. _arXiv preprint arXiv:2404.02151_.
* Chao et al. [2024] Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, et al. 2024. Jaillbreakbench: An open robustness benchmark for jailbreaking large language models. _arXiv preprint arXiv:2404.01318_.
* Chao et al. [2023] Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2023. Jailbreaking black box large language models in twenty queries. _arXiv preprint arXiv:2310.08419_.
* Chen et al. [2024] Xuan Chen, Yuzhou Nie, Wenbo Guo, and Xiangyu Zhang. 2024. When llm meets drl: Advancing jailbreaking efficiency via drl-guided search. _arXiv preprint arXiv:2406.08705_.
* Dao et al. [2022] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Re. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. _Advances in Neural Information Processing Systems_, 35:16344-16359.
* Dettmers et al. [2023] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: efficient fine-tuning of quantized llvms (2023). _arXiv preprint arXiv:2305.14314_, 52:3982-3992.
* Gong et al. [2024] Xueluan Gong, Mingzhe Li, Yilin Zhang, Fengyuan Ran, Chen Chen, Yanjiao Chen, Qian Wang, and Kwok-Yan Lam. 2024. Effective and evasive fuzz testing-driven jailbreaking attacks against llms. _arXiv preprint arXiv:2409.14866_.
* Huang et al. [2024] Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenham Lyu, Yixuan Zhang, et al. 2024. Trustllm: Trustworthiness in large language models. _arXiv preprint arXiv:2401.05561_.
* Jiang et al. [2024] Fengqing Jiang, Zhangchen Xu, Luyao Niu, Bill Yuchen Lin, and Radha Poovendran. 2024. Chatbug: A common vulnerability of aligned llms induced by chat templates. _arXiv preprint arXiv:2406.12935_.
* Liao and Sun [2024] Zeyi Liao and Huan Sun. 2024. Amplegcg: Learning a universal and transferable generative model of adversarial suffixes for jailbreaking both open and closed llms. _arXiv preprint arXiv:2404.07921_.
* Lin et al. [2024a] Zhihao Lin, Wei Ma, Mingyi Zhou, Yanjie Zhao, Haoyu Wang, Yang Liu, Jun Wang, and Li Li. 2024a. Pathseeker: Exploring llm security vulnerabilities with a reinforcement learning-based jailbreak approach. _arXiv preprint arXiv:2409.14177_.
* Liu et al. [2024b] Xiaogeng Liu, Peiran Li, Edward Suh, Yevgeniy Vorobeychik, Zhuqiong Mao, Somesh Jha, Patrick McDaniel, Huan Sun, Bo Li, and Chaowei Xiao. 2024b. Autodan-turbo: A lifelong agent for strategy self-exploration to jailbreak llms. _arXiv preprint arXiv:2410.05295_.
* Liu et al. [2023] Xiaogen Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2023. Autodan: Generating stealthy jailbreak prompts on aligned large language models. _arXiv preprint arXiv:2310.04451_.
* Liu et al. [2024c] Yue Liu, Xiaoxin He, Miao Xiong, Jinlan Fu, Shumin Deng, and Bryan Hooi. 2024c. Flipattack: Jailbreak llms via flipping. _arXiv preprint arXiv:2410.02832_

Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. 2024. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. _arXiv preprint arXiv:2402.04249_.
* Mehrotra et al. (2023) Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. 2023. Tree of attacks: Jailbreaking black-box llms automatically. _arXiv preprint arXiv:2312.02119_.
* Paulus et al. (2024) Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, and Yuandong Tian. 2024. Advvprompter: Fast adaptive adversarial prompting for llms. _arXiv preprint arXiv:2404.16873_.
* Pavlova et al. (2024) Maya Pavlova, Erik Brinkman, Krithika Iyer, Vitor Albiero, Joanna Bitton, Hailey Nguyen, Joe Li, Cristian Canton Ferrer, Ivan Evtimov, and Aaron Grattafiori. 2024. Automated red teaming with goat: the generative offensive agent tester. _arXiv preprint arXiv:2410.01606_.
* Samvelyan et al. (2024) Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, et al. 2024. Rainbow teaming: Open-ended generation of diverse adversarial prompts. _arXiv preprint arXiv:2402.16822_.
* Sitawarin et al. (2024) Chauwin Sitawarin, Norman Mu, David Wagner, and Alexandre Araujo. 2024. Pal: Proxy-guided black-box attack on large language models. _arXiv preprint arXiv:2402.09674_.
* Slivkins et al. (2019) Aleksandra Slivkins et al. 2019. Introduction to multiarmed bandits. _Foundations and Trends(r) in Machine Learning_, 12(1-2):1-286.
* Takemoto (2024) Kazuhiro Takemoto. 2024. All in how you ask for it: Simple black-box method for jailkbreak attacks. _Applied Sciences_, 14(9):3558.
* Wang and Qi (2024) Zhe Wang and Yanjun Qi. 2024. A closer look at adversarial suffix learning for jailbreaking LLMs. In _ICLR 2024 Workshop on Secure and Trustworthy Large Language Models_.
* Wei et al. (2024) Alexander Wei, Nika Hagthalab, and Jacob Steinhardt. 2024. Jailbroken: How does llvm safety training fail? _Advances in Neural Information Processing Systems_, 36.
* Yao et al. (2024) Dongyu Yao, Jianshu Zhang, Ian G Harris, and Marcel Carlsson. 2024. Fuzzllm: A novel and universal fuzzing framework for proactively discovering jailbreak vulnerabilities in large language models. In _ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 4485-4489. IEEE.
* Yu et al. (2023) Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. 2023. Gpftwizer: Red teaming large language models with auto-generated jailbreak prompts. _arXiv preprint arXiv:2309.10253_.
* Zeng et al. (2024) Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. 2024. How johnny can persuade lms to jailbreak them: Rethinking persuation to challenge ai safety by humanizing lhsns. _arXiv preprint arXiv:2401.06373_.
* Zhou et al. (2024) Yukai Zhou, Zhijie Huang, Feiyang Lu, Zhan Qin, and WenJie Wang. 2024. Don't say no: Jainbreaking llm by suppressing refusal. _arXiv preprint arXiv:2404.16369_.
* Zou et al. (2023) Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. _arXiv preprint arXiv:2307.15043_.

 

Appendix

### Engineering Upgrades

TurboFuzzLLM adds a collection of engineering upgrades to improve the effectiveness and ease of usage, as follows:

* _Limit search to unbroken questions_. To avoid the same set of questions being jailbroken across multiple fuzzing iterations, TurboFuzzLLM removes a question \(q_{i}\) from \(Q\) as soon as \(q_{i}\) is jailbroken in a fuzzing iteration \(k\) (i.e., \(Q\gets Q\setminus\{q_{i}\}\)). This ensures that future fuzzing iterations focuses the search to questions that are still unbroken. Note that due to this upgrade, the total number of jailbreaks equals the number of questions jailbroken.
* _Checking template-mutation compatibility._ Given a template \(t\), only a subset \(M_{t}\) of all mutations \(M\) might make sense as candidates to be applied to \(t\). For example, if \(t\) already ends with "Sure, here is", there isn't much of a point of applying _Inject Prefix_ or _Expand After_ mutations. Similarly, if \(t\) already includes instructions for _Refusal Suppression_, there is no need to repeat these instructions again. Through simple regular expression checks, TurboFuzzLLM derives a subset of mutations \(M_{t}\subseteq M\) that are compatible with \(t\) and limits mutation selection to only a compatible mutation \(m\in M_{t}\) when generating the mutant \(m(t)\).
* _Improved prompts for LLM-based mutations._ As shown in figures 4 & 5, TurboFuzzLLM utilizes formatting tags (e.g., "[ANSWER BEGINS]" and "[ANSWER ENDS]") to improve LLM-based mutant generation and decrease invalid mutants.
* _Multi-threading support._ Given a mutant \(m(t)\), TurboFuzzLLM utilizes multi-threading to parallelize discharging attack prompts \(A_{m(t)}\) to the target model as well as evaluating corresponding responses \(R_{m(t)}\) to speed up the most time-critical steps in each fuzzing iteration.
* _Usability upgrades._ TurboFuzzLLM provides improved command-line interface, logging support, statistics summary, and results reporting to enhance usage experience and results analysis.

### Pseudo code for mutation selection using Q-learning

Algorithm 1 presents the Q-learning based mutation selection algorithm. Given a template \(t\), SelectMutation selects a compatible mutation \(m\in M_{t}\) using an epsilon-greedy exploration-exploitation strategy (lines 1-9). If the generated random number \(random\in[0,1]\) is less than exploration probability \(\epsilon\), then a uniformly-random selection is made from \(M_{t}\) (lines 3-5). Otherwise, a weighted random selection is done using the Q-table values \(\mathcal{Q}[s_{t}]\) as weights, with the state \(s_{t}\) as the root parent of \(t\) (lines 6-8). Using the attack success rate of the generated mutant \(m(t)\) as reward

Figure 2: Motivating example



\(r\), the Reward( ) function is used to update the Q-table value \({\cal Q}[s_{t}][m]\) for the selected mutation \(m\) (lines 10-13).

### Pseudo code for template selection using multi-arm bandits

Algorithm 2 presents the pseudo code for template selection using multi-arm bandits. In a given fuzzing iteration, SelectTemplate selects a template \(t\) from the current population \(O\cup G\) using an epsilon-greedy exploration-exploitation strategy (lines 1-7). If the generated random number \(random\in[0,1]\) is less than exploration probability \(\epsilon\), then a uniformly-random selection is made from \(O\cup G\) (lines 2-4). Otherwise, a weighted random selection is done using the Q-table values \({\cal Q}\) as weights (lines 5-6). Using the attack success rate of the generated mutant \(m(t)\) as reward \(r\), the Reward( ) function is used to update the Q-table value \({\cal Q}[t]\) for the selected template \(t\) (lines 8-10).

```
Globals:Q-table \({\cal Q}\), learning rate \(\alpha\), discount factor \(\gamma\), exploration probability \(\epsilon\) Output:template \(t\)procedureSelectTemplate( ) \(random\leftarrow\textsc{GetRandomNumber( )}\)if\(random<\epsilon\)then \(m\leftarrow\textsc{UniformlyRandom($M_{t}$)}\)else \(s_{t}\gets root(t)\) \(m\leftarrow\textsc{WeightedRandom($M_{t}$, ${\cal Q}[s_{t}]$)}\)return\(m\) Input:template \(t\), mutation \(m\)
10procedureReward(\(t\), \(m\))
11procedureReward(\(t\), \(m\))
12\(r\gets ASR(m(t))\) \({\cal Q}[t]\leftarrow(1-\alpha)~{}{\cal Q}[t]\) \(+\;\alpha~{}(r+\gamma~{}\max_{a}{\cal Q}[a])\)
```

**Algorithm 2** Template selection using multi-arm bandits

### Additional Implementation Details

TurboFuzzLLM provides command-line options to easily change key hyper parameters, including the mutator model used for performing LLM-based mutations as well as the judge model used for evaluating whether or not a target response represents a successful jailbreak.

Here is a summary of hyper parameter values we used:

* [noitemsep,topsep=0pt]

* Mutator model: Mistral Large 2 (24.07)
* Judge model: Fine-tuned Llama 2 13B model from (Mazeika et al., 2024)
* Temperature: 1.0
* _Mutation selection using Q-learning_
* Learning rate: 0.4
* Discount factor: 0.1
* Initial epsilon: 1.0
* Epsilon decay factor: 0.9
* Minimum epsilon: 0.2
* _Template selection using multi-arm bandits_
* Learning rate: 0.3
* Discount factor: 0.02
* Initial epsilon: 1.0
* Epsilon decay factor: 0.97
* Minimum epsilon: 0.2
* _Early-exit fruitless templates_
* Threshold: 10% questions
* Threshold minimum: 4 questions
* Threshold maximum: 20 questions

* Percentage of questions used for warmup: 100% (all questions)
 


Figure 4: Prompt used for Few Shots

Figure 5: Prompt used for Few Shots mutation