AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs

## Abstract

Abstract In this paper, we propose AutoDAN-Turbo , a black-box jailbreak method that can automatically discover as many jailbreak strategies as possible from scratch, without any human intervention or predefined scopes (e.g., specified candidate strategies), and use them for red-teaming. As a result, AutoDAN-Turbo can significantly outperform baseline methods, achieving a 74.3 % percent 74.3 74.3\% 74.3 % higher average attack success rate on public benchmarks. Notably, AutoDAN-Turbo achieves an 88.5 88.5 88.5 88.5 attack success rate on GPT-4-1106-turbo. In addition, AutoDAN-Turbo is a unified framework that can incorporate existing human-designed jailbreak strategies in a plug-and-play manner. By integrating human-designed strategies, AutoDAN-Turbo can even achieve a higher attack success rate of 93.4 93.4 93.4 93.4 on GPT-4-1106-turbo.

## 1 Introduction

Large language models (LLMs) have been widely deployed in recent years due to their advanced capabilities in understanding and generating human-like text (Ouyang et al., ). To ensure these models behave responsibly, safety alignment has been proposed. This alignment enables LLMs to provide more helpful, appropriate, and safe responses, particularly in the face of harmful instructions or questions. However, jailbreak attacks have emerged as a significant threat to aligned LLMs (Wei et al., ; Zou et al., ; Chao et al., ; Shen et al., ; Liu et al., ; Liao & Sun, ). These attacks leverage carefully designed prompts to trick the LLMs into losing their safety alignment and providing harmful, discriminatory, violent, or sensitive content. To maintain the responsible behaviors of LLMs, it is crucial to investigate automatic jailbreak attacks. These attacks serve as essential red-teaming tools, proactively assessing whether LLMs can behave responsibly and safely in adversarial environments (Zou et al., ).

Existing jailbreak attacks for LLMs face several limitations. While several automatic jailbreak methods, such as PAIR (Chao et al., ), and TAP (Mehrotra et al., ) have been proposed, However, since these methods lack guidance for jailbreak knowledge, the diversity and effectiveness of the jailbreak prompts generated by such attacks are often unsatisfying. To address it, a few jailbreak methods navigate the complexities of language—such as its inherently multi-lingual, context-dependent, and socially nuanced properties for red-teaming. They have utilized human-developed social engineering, exploiting cultural norms, or leveraging cross-lingual ambiguities ( which we refer to as “strategies”) to compromise the LLMs (Shen et al., ; Zeng et al., ; Yong et al., ). For example,
strategies such as cipher  (Yuan et al., ; Lv et al., ), ASCII-based techniques (Jiang et al., ), very long contexts () and low-resource language-based strategies (Yong et al., ) have been proposed to jailbreak LLMs. Human persuasion strategies, such as false promises and threats, are also utilized to jailbreak LLMs (Zeng et al., ). Although these "strategy-based jailbreak attacks" are intriguing, they still face two major limitations. Firstly, these attacks rely on human intervention to manually devise the strategies, which requires significant labor and limits the scope of strategies to the imagination of the human designer. Secondly, these methods typically employ only a single strategy, leaving the potential for combining and synergizing diverse strategies to create stronger jailbreak attacks largely unexplored.

Figure: Figure 2: The pipeline of AutoDAN-Turbo
Refer to caption: x2.png

In this paper, to address the above limitations, we propose AutoDAN-Turbo, an innovative method that utilizes lifelong learning agents to automatically and continually discover diverse strategies, combine the discovered strategies, and leverage the strategies for jailbreak attacks without human intervention, as shown in Fig. 1.
Our method has the following features: First, Automatic Strategy Discovery: Our jailbreak framework can automatically discover as many jailbreak strategies as possible from scratch, without any human intervention or predefined scopes (e.g., specified candidate strategies). Specifically, AutoDAN-Turbo can autonomously develop new strategies during the exploration, and also systematically store these strategies in an organized structure. This enables the model to effectively reuse discovered strategies and evolve based on existing strategies, potentially combining them into more advanced approaches, for new attack attempts.
Second, External Strategy Compatibility: AutoDAN-Turbo is a unified framework that can leverage existing human-designed jailbreak strategies in a plug-and-play manner. We can easily reformat the existing/external strategy and put them into AutoDAN-Turbo’s strategy library. This enables AutoDAN-Turbo to leverage the existing strategies and develop new advanced jailbreak attack strategies based on both external jailbreak strategies and its own discoveries. Third, Practical Usage: Our method works in a black-box manner, which only requires access to the model’s textual output.

We conduct extensive experiments on public benchmarks and datasets (Mazeika et al., ; Souly et al., ; Lapid et al., ; Qiu et al., ; Zou et al., ; Luo et al., ) to evaluate our method. The results demonstrate that our method is capable of automatically discovering jailbreak strategies and achieving high attack success rates on both open-sourced and closed-sourced LLMs in a black-box manner, surpassing the runner-up baseline (Samvelyan et al., ) by $74.3\%$ on average across different victim models on Harmbench (Mazeika et al., ).
Additionally, evaluated by the StrongREJECT score (Souly et al., ), our method shows outstanding performance on inducing the LLM to provide request-relevant malicious content, surpassing the runner-up basline (Samvelyan et al., ) by $92.3\%$. Notably, our method demonstrates remarkable jailbreak effectiveness on GPT-4-1106-turbo (OpenAI et al., ), achieving an $88.5$ attack success rate.
In addition, our evaluations validate that the strategy library developed by AutoDAN-Turbo exhibits strong transferability, maintaining high attack success rates across different target models and different datasets. Furthermore, due to its excellent compatibility of our framework, our method can incorporate with existing human-developed jailbreak strategies and achieve even higher attack performance. By integrating $7$ human-designed jailbreak strategies (Ding et al., ; Jiang et al., ; Lv et al., ; Pedro et al., ; Upadhayay & Behzadan, ; Yao et al., ; Yuan et al., ) from academic papers, AutoDAN-Turbo can even achieve a higher attack success rate of $93.4$ on GPT-4-1106-turbo.

## 2 Related Works

Existing jailbreaks mainly follow two methodological lines. The first is the optimization-based attack (Zou et al., ; Chao et al., ; Liu et al., ; Zhu et al., ; Guo et al., ; Liao & Sun, ; Paulus et al., ), which leverages an automatic algorithm to generate jailbreak prompts based on certain feedbacks, such as gradients of a loss function (Zou et al., ; Liu et al., ; Zhu et al., ; Guo et al., ; Chao et al., ; Mehrotra et al., ), or training a generator to imitate such optimization algorithms (Liao & Sun, ; Paulus et al., ). However, these automatic jailbreak attacks do not provide explicit jailbreak knowledge for the attack algorithm, often resulting in weak attack performance and limited diversity in the generated jailbreak prompts. Another line of work that addresses this issue is the strategy-based attack (Zeng et al., ). Compared to optimization-based methods, strategy-based jailbreak attacks do not necessarily require an automatic algorithm (though they sometimes do). Instead, the core of strategy-based jailbreak methods is to leverage specific jailbreak strategies to compromise the LLMs. For example, one of the earliest known jailbreak attacks against LLMs, the "Do-Anything-Now (DAN)" series (walkerspider, ; Shen et al., ) leverage the role-playing strategy and prompts the LLMs to role-play as another assistant who has no ethical constraints. Strategy-based jailbreak attacks (walkerspider, ; Shen et al., ; Wang et al., ; Samvelyan et al., ; Jin et al., ; Yuan et al., ; Lv et al., ; Ding et al., ; Jiang et al., ; Pedro et al., ; Upadhayay & Behzadan, ; Yao et al., ; ; Wei et al., ; Xu et al., ) often utilize human-designed strategies at the core of their approach. For example, the role-playing strategy has been widely used in many jailbreak attacks (walkerspider, ; Shen et al., ; Wang et al., ; Samvelyan et al., ), such as GUARD (Jin et al., ), which mainly discusses the implementation and refinement of the role-playing jailbreak strategy. Rainbow Teaming (Samvelyan et al., ) utilizes $8$ predefined strategies, such as emotional manipulation and wordplay, to generate jailbreak prompts. And PAP (Zeng et al., ) explores the possibility of using $40$ human-discovered persuasion schemes to jailbreak LLMs. Other jailbreak strategies, such as ciphered  (Yuan et al., ; Lv et al., ), ASCII-based techniques (Jiang et al., ), long contexts (), low-resource language-based strategies (Yong et al., ), malicious demonstration (Wei et al., ), and veiled expressions (Xu et al., ) also reveal many interesting aspects of jailbreak vulnerabilities of LLMs.

However, existing strategy-based attacks face two limitations: reliance on predefined strategies and limited exploration of combining different methods. To address these, we propose AutoDAN-Turbo, an autonomous system that discovers, evolves, and stores strategies without human intervention. It can also incorporate human-designed strategies, creating advanced attacks by combining both. This framework treats all LLMs as end-to-end black-box models, ensuring flexibility and adaptability.

## 3 AutoDAN-Turbo

As illustrated in Fig. 2, our method consists of three main modules: the Attack generation and Exploration Module (Sec. 3.1) , Strategy Library Construction Module (Sec. 3.2), and Jailbreak Strategy Retrieval Module (Sec. 3.3). In the Attack generation and Exploration Module, where the goals are to generate jailbreak prompt to attack the target LLM by leveraging the strategies provided by Jailbreak Strategy Retrieval Module, it consists of an attacker LLM that generates jailbreak prompts based on specific strategies retrieved from Jailbreak Strategy Retrieval Module; a target (victim) LLM that provides responses; and a scorer LLM that evaluates these responses to assign scores. We can repeat this process multiple time to generate massive attack logs for Strategy Library Construction Module to generate a strategy library.
Strategy Library Construction Module is to extract strategies from the attack logs generated in Attack Generation and Exploration Module and save the strategies into the Strategy Library;
Jailbreak Strategy Retrieval Module is to support the Attack Generation and Exploration Module to retrieve the strategy from the strategy library constructed by Strategy Library Construction Module so that the retrieved jailbreak prompt can guide the jailbreak prompt generation to attack the victim LLMs.

By leveraging these three modules, the framework can continuously automatically devise jailbreak strategies, reuse strategies, and evolve from existing strategies, thus ensuring the feature of automatic strategy discovery and evolvement.
In addition, our skill library is designed very accessible so that external/existing strategies can be easily incorporated in a plug-and-play manner. As a result, our framework will not only utilize external strategies but also discover new jailbreak strategies based on them, thereby equipping the proposed method with external strategy compatibility features. Our pipeline only requires a textual response from the target model in the entire attack process, eliminating the need for white-box access to the target model and thus offering practical usage.

### 3.1 Attack Generation and Exploration Module.

As illustrated in Fig. 2, three LLMs are involved in the Attack Generation and Exploration Module: an attacker LLM, a target LLM (the victim model we want to jailbreak), and a scorer LLM. Specifically, the attack loop contains the following steps: (1) Attack Generation: The attacker LLM receives specific prompts that describe the malicious request $M$ and encourages the attacker LLM to generate a jailbreak prompt using specified jailbreak strategies. The attacker LLM then generates the jailbreak prompt $P$; (2) Target Response: Upon receiving $P$ as input, the target LLM generates a response $R$; (3) Scorer Evaluation: The response $R$ is then evaluated by the scorer LLM. This evaluation determines whether the response meets the malicious goal of the jailbreak attack. The scorer LLM returns a numerical score $S$ based on predefined criteria. The scores range from 1, indicating no alignment with malicious intent, to 10, representing full compliance with harmful directives. The detailed prompt for configuring the scorer LLM is provided in Appendix B.2.

Our module supports three functionalities shown in Tab. A in the appendix : (1) generating jailbreak prompts without a strategy, (2) generating jailbreak prompts with effective retrieved strategy, and (3) generating jailbreak prompts with ineffective strategies.
For (1), when no strategy exists in the strategy library (described in Sec. 3.2), the prompt asks the attacker LLM to generate jailbreak prompts for the malicious request using any strategy it can imagine.
For (2), when several effective jailbreak strategies are provided, the prompt instructs the attacker LLM to generate jailbreak prompts according to the given strategies;
For (3), if the framework has gone through the strategy library and only found ineffective strategies, the prompt directs the attacker LLM to avoid these low-scoring strategies and devise new ones.

### 3.2 Strategy Library Construction Module

Here, we define a jailbreak strategy as the text information that, when added, leads to a higher jailbreak score as evaluated by the scorer. In the following content, we will show how to build up the strategy library in detail based on the above definitions. Since the goal of our framework is to automatically construct strategies from scratch, we design two stages methods: (1) warm-up exploration stage and (2) running-time lifelong learning stage.

Warm-up exploration stage, at this stage, for each malicious request $M$, we repeatedly run the Attack Generation and Exploration Module with empty strategies as initialization until it reaches a maximum of $T$ iterations or until the scorer LLM returns a score higher than a predefined termination score $S_{T}$. After this process, we will collect the attack generation $P$, target response $R$, and Score $S$ as a record. After $T$ iterations for all malicious requests, we will have a list of records, named attack log, where each row consists of a triplet ($P$, $R$, $S$). Based on the attack log, we then extract the strategies based on the previous definition.

We randomly extract two attack records from the attack log, represented as $\{P_{i},R_{i},S_{i}\}$ and $\{P_{j},R_{j},S_{j}\}$. If the score $S_{j}$ is higher than $S_{i}$, we argue that some strategy may have been explored and employed in the jailbreak prompt $P_{j}$ compared to $P_{i}$, leading to an improved score. Thus, we count the improvement from $P_{i}$ to $P_{j}$ as a strategy. As shown in Fig. 3, to summarize the improvements, we employ a summarizer LLM (details in Appendix B.3) to summarize the improvement. the summarizer LLM compares $P_{j}$ with $P_{i}$, to analyze the strategies that make $R_{j}$ more malicious than $R_{i}$ (i.e., $S_{j}>S_{i}$). The summarizer LLM will provide a description of the jailbreak strategy and the concise definition of the strategy, and format it into a JSON object, $S_{i}$. $S_{i}$ comprises three elements: "Strategy", the name of the strategy employed in $P_{j}$; "Definition", the concise definition of the strategy summarized by summarizer LLM ; and "Example", which records the jailbreak prompt $P_{j}$ that utilized the strategy. To prevent redundancy in defining jailbreak strategies, we will provide the summarizer LLM with the existing categories from the strategy library, instructing it to avoid duplicating definitions.

Figure: Figure 3: Our methodology defines a jailbreak strategy as text modifications that increase the jailbreak score, identifying these strategies by comparing differences between consecutive attack logs where a higher score indicates an improved strategy. AutoDAN-Turbo will systematically construct a strategy library, storing data on these strategies and using response embeddings for efficient retrieval, with strategies summarized and formatted for easy access.
Refer to caption: x3.png

The Key for Retrieval. To structurally organize the strategy library and facilitate the retrieval of specific jailbreak strategies as needed, we must determine which information should serve as the key for retrieval. Recall that we define a “jailbreak strategy as schemes that improve jailbreak scores from $S_{i}$ to $S_{j}$”, with these scores based on the response $R_{i}$ and $R_{j}$. If a target LLM responds to a malicious request with a response that is similar to $R_{i}$, then a previously effective strategy $S_{i}$ that is associated with $R_{i}$ could potentially be effective again in this situation. Therefore, using the embedding of $R_{i}$ as the key can facilitate efficient strategy retrieval.

As a result, as shown in Fig. 3, each row of the skill library consists of (key, value) pairs. For key, we employ a text embedding model (Neelakantan et al., ; Wang et al., ) to transfer the response $R_{i}$ into a text embedding vector $E_{R_{i}}$ and set it as the key. For value, we set attack prompt $P_{i}$, next attack prompt $P_{j}$, the score differential $S_{j}-S_{i}$ (should always be a positive number, which means if $S_{i}<=S_{j}$). We repeatedly conduct the sampling process and run exploration with different malicious requests. We then add extracted the key and value pair into the skill library.

Lifelong learning at the running stage. After the warm-up strategy, our framework will conduct lifelong learning
at the running stage to further augment the strategy library. When AutoDAN-Turbo conducts lifelong learning upon a dataset that contains multiple malicious requests, it will repeat Attack Generation and Exploration Module for the whole dataset for $N$ round, and iteratively conduct the attack loop described in Sec. 3.1 for each malicious requests, until it reaches a maximum of $T$ iterations or until the scorer LLM returns a score higher than a predefined termination score $S_{T}$. Specifically, at iteration $i$, given a malicious request $M$, we get the $P_{i}$, $R_{i}$ and $S_{i}$ from Attack Generation and Exploration Module. Based on $R_{i}$, Jailbreak Strategy Retrieval will retrieve the strategy (details in Sec 3.3) to prompt Attack Generation and Exploration Module to generate new $P_{i+1},R_{i+1},S_{i+1}$. We can also employ the similar process described in warm-up strategy exploration stage to generate the item for strategies library by just replacing the $P_{j},R_{j},S_{j}$ with $P_{i+1},R_{i+1},S_{i+1}$. We can also store them in the strategy library for reuse. For each malicious request, our termination rule is that either (1) the scorer LLM returns a score that is higher than a predefined termination $S_{T}$ or (2) the total iterations have reached the maximum value $T$.

### 3.3 Jailbreak Strategy Retrieval

A key operation in our framework is to retrieve jailbreak strategies from the strategy library, and then prompt the attacker LLM to generate jailbreak prompts based on these strategies. Specifically, given the malicious request M, we feed them to generation and exploration module to get $\{P_{i},R_{i},S_{i}\}$. We then employ the text-embedding model to transform the response $R_{i}$ into an embedding vector $E_{R_{i}}$. Subsequently, we compare the similarity between $E_{R_{i}}$ with all keys in the strategy library. We choose the top-$2k$ data frames with the highest similarity of key values. Then, we sort these values based on the score differences they contain and select the top-$k$ strategies that led to the highest score differences. These are the most effective strategies associated with the responses $R_{i}$. These selected strategies will be then formed as a retrieved strategy list $\Gamma$. If two samples with the same score are selected and happen to meet the length limit of the strategy list, they are added or dropped in the program’s default order.

After establishing the retrieved strategy list $\Gamma$, we insert these strategies into the prompt of the attacker LLM in the next attack iteration as illustrated in Tab A. Specifically, we adopt the following tactics: (1) If the highest score in $\Gamma$ is greater than 5, we will directly use this strategy as *effective strategy* and insert it into the attacker LLM’s prompt. Namely, the attacker LLM is asked to use this strategy to generate the jailbreak prompt in the next jailbreak round; (2) If the highest score is less than 5, we select all strategies with a score difference between $2-5$ ad set them as *effective strategies*. We insert these strategies into the attacker’s prompt. Namely, we inform the attacker LLM that these strategies are potent for the current malicious request, and attacker LLM can combine and evolve among these strategies to generate new jailbreak prompt;
(3) If the number of highest strategies is less than 2, we viewed these strategies as *ineffective strategies* since they can not achieve big improvements. Thus, as shown in Tab. A, we inform the attacker LLM in the prompt that these strategies are not particularly effective for the current malicious request, so they should not continue using these strategies and need to discover other strategies;
(4). If the $\Gamma$ set is empty, we will provide *empty strategy* to attacker LLM.

### 3.4 Test Stage and More Functionalities of AutoDAN-Turbo

In the test stage of AutoDAN-Turbo, the strategy library will be fixed, and we will not use the summarizer LLM to extract strategies from attack logs or save strategy. For every malicious request in the test stage, AutoDAN-Turbo will run the same attack generation process with the strategy being retrieved from the strategy library, the process will run multiple times until it reaches a maximum of $T$ iterations or until the scorer LLM returns a score higher than a predefined termination score $S_{T}$.

When we want to inject human-developed jailbreak strategies: One of the advantages of our method is its compatibility with other human-developed strategies in a plug-and-play manner. To achieve this, we can first edit the human-developed strategy into the format illustrated in Fig. 3. After that, we insert the human-developed strategy into the prompt of the attacker LLM, instructing it to generate jailbreak prompts according to the given strategy. The human-designed jailbreak strategy will then participate in the attack loop and, if effective, be added to the strategy library. It will be used and further refined when retrieved and reused by the attacker in future attack loops.

When we want to transfer the learned jailbreak strategies: Sometimes, we may want the learned jailbreak strategies to be used for jailbreaking other malicious requests or target models, or with other attacker models. This can be easily achieved by changing the malicious request dataset, attacker LLM, or target LLM. AutoDAN-Turbo supports both an off-the-shelf mode and a continual learning mode. In off-the-shelf mode, we do not want to learn new strategies based on the new settings, we can simply fix the learned strategy library and exclude the strategy library construction process. Alternatively in the continual learning mode, we can allow the framework to continue updating the strategy library in the new settings. Our experiments show that the off-the-shelf mode is already highly effective on different target models, demonstrating the impressive transferability of the learned jailbreak strategies. Continual learning further enhances this effectiveness.

## 4 Experiments

### 4.1 Experiments Setup

Datasets.
We choose the Harmbench textual behavior dataset (abbr. as Harmbench dataset) (Mazeika et al., ) to evaluate our method and other baselines. The HarmBench dataset contains 400 diverse malicious requests that violate laws or norms and are difficult to replicate with a search engine, ensuring they present unique risks when performed by LLMs, making this dataset an excellent resource for assessing the practical risks of jailbreak attacks. In addition, we utilize a small dataset from (Chao et al., ) that contains 50 malicious requests to initial the AutoDAN-Turbo as we described in Sec. 3.4. We also utilize other datasets for evaluating the transferability (See Sec. 4.3).

Large Language Models. We conduct comprehensive evaluations on both open-source and closed-source LLMs. Specifically, for open-source LLMs, we include Llama-2-7B-chat (Touvron et al., ), Llama-2-13B-chat (Touvron et al., ), Llama-2-70B-chat (Touvron et al., ), Llama-3-8B (Dubey et al., ), Llama-3-70B (Dubey et al., ), and Gemma-7B-it (Team et al., ). For closed-source models, we include GPT-4-1106-turbo (OpenAI et al., ) and Gemini Pro (Team et al., ). The specific roles these models serve, whether as the attacker LLM, the target LLM, or the strategy summarizer LLM, will be detailed in the corresponding contexts. Note that throughout our experiments, we employed a deterministic generation approach by using a zero temperature setting, and limited the maximum token generation to 4096 tokens. To ensure the consistency of our experiments, we used Gemma-7B-it as our scorer LLM throughout.

Metrics.
To ensure a fair and standardized evaluation protocol, we leverage two evaluation metrics from existing open-source jailbreak benchmarks (Mazeika et al., ; Souly et al., ) to judge the success of jailbreak attacks. The first metric is the Harmbench Attack Success Rate (i.e., ASR, where percentages are reported without the “%” symbol.), introduced in  (Mazeika et al., ). This metric is calculated using a carefully fine-tuned Llama-2-13B model as the input classifier to determine whether the jailbreak response is relevant to the query meanwhile harmful.
The second metric is the StrongREJECT Score (i.e., Score), introduced in  (Souly et al., ).
This auto-grading system captures nuanced distinctions in response quality and aligns closely with human evaluators’ assessments of jailbreak quality. For both the Harmbench ASR and the StrongREJECT Score, higher values indicate better performance of the jailbreak methods. For AutoDAN-Turbo, We also report the Total Strategies Found (TSF) which represents the count of strategies that exist in the strategy library.
For AutoDAN-Turbo, We also report Average Jailbreak Rounds (AJR), where the AJR is defined as the average number of attack loops needed to jailbreak a specific malicious behavior successfully.

Implementation. To evaluate AutoDAN-Turbo, as described in Sec. 3.4, we will first undertake a warm-up exploration stage on the initial dataset that contains 50 malicious requests, 150 times ($N$=150) to establish our initial strategy library. Subsequently, using this initial strategy library, we perform a running-time lifelong learning stage, for each malicious request in the Harmbench dataset, we conduct $5$ rounds of attacks. A complete round of attacks is defined as iterating through all malicious data in the dataset. For each data instance, we set $T$ as 150 and $S_{T}$ as $8.5$. In the evaluation, we fix the skill library and conduct another round of attacks on the Harmbench dataset. Since our method includes the running-time lifelong learning stage, for fair comparison, we also run the same total iterations for baseline methods.

Baselines. As our method operates in black-box settings, we include five black-box jailbreak attacks as baselines in our evaluations: GCG-T(Zou et al., ), PAIR (Chao et al., ), TAP (Mehrotra et al., ), PAP-top5 (Zeng et al., ), and Rainbow Teaming (Samvelyan et al., ). PAIR and TAP share similarities with our method as they also use LLMs to generate jailbreak prompts. PAP employs 40 human-developed strategies to generate jailbreak prompts. Rainbow Teaming utilizes $8$ jailbreak strategies to guide the generation of jailbreak prompts and further optimize them.

**Table 1: Top: The ASR results evaluated using the Harmbench (Mazeika et al., ) protocol, where higher values indicate better performance. Bottom: The scores evaluated using the StrongREJECT (Souly et al., ) protocol, also with higher values being better. Our method outperforms the runner-up by $72.4\%$ in Harmbench ASR and by $93.1\%$ in StrongREJECT scores. The model name in parentheses indicates the attacker model used in our method.**
| Attacks↓ / Victims→ | Llama-2-7b-chat | Llama-2-13b-chat | Llama-2-70b-chat | Llama-3-8b | Llama-3-70b | Gemma-7b-it | Gemini Pro | GPT-4-Turbo-1106 | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GCG-T | 17.3 | 12.0 | 19.3 | 21.6 | 23.8 | 17.5 | 14.7 | 22.4 | 18.6 |
| PAIR | 13.8 | 18.4 | 6.9 | 16.6 | 21.5 | 30.3 | 43.0 | 31.6 | 22.8 |
| TAP | 8.3 | 15.2 | 8.4 | 22.2 | 24.4 | 36.3 | 57.4 | 35.8 | 26.0 |
| PAP-top5 | 5.6 | 8.3 | 6.2 | 12.6 | 16.1 | 24.4 | 7.3 | 8.4 | 11.1 |
| Rainbow Teaming | 19.8 | 24.2 | 20.3 | 26.7 | 24.4 | 38.2 | 59.3 | 51.7 | 33.1 |
| Ours (Gemma-7b-it) | 36.6 | 34.6 | 42.6 | 60.5 | 63.8 | 63.0 | 66.3 | 83.8 | 56.4 |
| Ours (Llama-3-70B) | 34.3 | 35.2 | 47.2 | 62.6 | 67.2 | 62.4 | 64.0 | 88.5 | 57.7 |

**Table 2: Our method is the state-of-the-art attack in Harmbench (Mazeika et al., ).**
| Model | Baseline | Ours |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GCG | GCG-M | GCG-T | PEZ | GBDA | UAT | AP | SFS | ZS | PAIR | TAP | TAP-T | AutoDAN | PAP-top5 | Human | Direct | AutoDAN-Turbo |  |
| Llama 2 7b chat | 32.5 | 21.2 | 19.7 | 1.8 | 1.4 | 4.5 | 15.3 | 4.3 | 2.0 | 9.3 | 9.3 | 7.8 | 0.5 | 2.7 | 0.8 | 0.8 | 36.6 |
| Llama 2 13b chat | 30.0 | 11.3 | 16.4 | 1.7 | 2.2 | 1.5 | 16.3 | 6.0 | 2.9 | 15.0 | 14.2 | 8.0 | 0.8 | 3.3 | 1.7 | 2.8 | 34.6 |
| Llama 2 70b chat | 37.5 | 10.8 | 22.1 | 3.3 | 2.3 | 4.0 | 20.5 | 7.0 | 3.0 | 14.5 | 13.3 | 16.3 | 2.8 | 4.1 | 2.2 | 2.8 | 42.6 |
| Vicuna 7b | 65.5 | 61.5 | 60.8 | 19.8 | 19.0 | 19.3 | 56.3 | 42.3 | 27.2 | 53.5 | 51.0 | 59.8 | 66.0 | 18.9 | 39.0 | 24.3 | 96.3 |
| Vicuna 13b | 67.0 | 61.3 | 54.9 | 15.8 | 14.3 | 14.2 | 41.8 | 32.3 | 23.2 | 47.5 | 54.8 | 62.1 | 65.5 | 19.3 | 40.0 | 19.8 | 97.6 |
| Baichuan 2 7b | 61.5 | 40.7 | 46.4 | 32.3 | 29.8 | 28.5 | 48.3 | 26.8 | 27.9 | 37.3 | 51.0 | 58.5 | 53.3 | 19.0 | 27.2 | 18.8 | 83.8 |
| Baichuan 2 13b | 62.3 | 52.4 | 45.3 | 28.5 | 26.6 | 49.8 | 55.0 | 39.5 | 25.0 | 52.3 | 54.8 | 63.6 | 60.1 | 21.7 | 31.7 | 19.3 | 86.9 |
| Qwen 7b chat | 59.2 | 52.5 | 38.3 | 13.2 | 12.7 | 11.0 | 49.7 | 31.8 | 15.6 | 50.2 | 53.0 | 59.0 | 47.3 | 13.3 | 24.6 | 13.0 | 82.7 |
| Qwen 14b chat | 62.9 | 54.3 | 38.8 | 11.3 | 12.0 | 10.3 | 45.3 | 29.5 | 16.9 | 46.0 | 48.8 | 55.5 | 52.5 | 12.8 | 29.0 | 16.5 | 85.6 |
| Qwen 72b chat | - | - | 36.2 | - | - | - | - | 32.3 | 19.1 | 46.3 | 50.2 | 56.3 | 41.0 | 21.6 | 37.8 | 18.3 | 77.9 |
| Koala 7b | 60.5 | 54.2 | 51.7 | 42.3 | 50.6 | 49.8 | 53.3 | 43.0 | 41.8 | 49.0 | 59.5 | 56.5 | 55.5 | 18.3 | 26.4 | 38.3 | 93.4 |
| Koala 13b | 61.8 | 56.4 | 57.3 | 46.1 | 52.7 | 54.5 | 59.8 | 37.5 | 36.4 | 52.8 | 58.5 | 59.0 | 65.8 | 16.2 | 31.3 | 27.3 | 91.9 |
| Orca 2 7b | 46.0 | 38.7 | 60.1 | 37.4 | 36.1 | 38.5 | 34.8 | 46.0 | 41.1 | 57.3 | 57.0 | 60.3 | 71.0 | 18.1 | 39.2 | 39.0 | 100.0 |
| Orca 2 13b | 50.7 | 30.3 | 52.0 | 35.7 | 33.4 | 36.3 | 31.8 | 50.5 | 42.8 | 55.8 | 59.5 | 63.8 | 69.8 | 19.6 | 42.4 | 44.5 | 94.7 |
| Solar 10.7B-Instruct | 57.5 | 61.6 | 58.9 | 56.1 | 54.5 | 54.0 | 54.3 | 58.3 | 54.9 | 56.8 | 66.5 | 65.8 | 72.5 | 31.3 | 61.2 | 61.3 | 95.7 |
| Mistral Tiny | 69.8 | 63.6 | 64.5 | 51.3 | 52.8 | 52.3 | 62.7 | 51.0 | 41.3 | 52.5 | 62.5 | 66.1 | 71.5 | 27.2 | 58.0 | 46.3 | 97.6 |
| Mistral Small | - | - | 62.5 | - | - | - | - | 53.0 | 40.8 | 61.1 | 69.8 | 68.3 | 72.5 | 28.8 | 53.3 | 47.3 | 96.9 |
| OpenChat 3.5 1210 | 66.3 | 54.6 | 57.3 | 38.9 | 44.5 | 40.8 | 57.0 | 52.5 | 43.3 | 52.5 | 63.5 | 66.1 | 73.5 | 26.9 | 51.3 | 46.0 | 96.3 |
| Starling | 66.0 | 61.9 | 59.0 | 50.0 | 58.1 | 54.8 | 62.0 | 56.5 | 50.6 | 58.3 | 68.5 | 66.3 | 74.0 | 31.9 | 60.2 | 57.0 | 97.1 |
| zephyr | 69.5 | 62.5 | 61.0 | 62.5 | 62.8 | 62.3 | 60.5 | 62.0 | 60.0 | 58.8 | 66.5 | 69.3 | 75.0 | 32.9 | 66.0 | 65.8 | 96.3 |
| R2D2 | 5.5 | 4.9 | 0.0 | 2.9 | 0.2 | 0.0 | 5.5 | 43.5 | 7.2 | 48.0 | 60.8 | 54.3 | 17.0 | 24.3 | 13.6 | 14.2 | 83.4 |
| GPT-3.5 Turbo 0613 | - | - | 38.9 | - | - | - | - | - | 24.8 | 46.8 | 47.7 | 62.3 | - | 15.4 | 24.5 | 21.3 | 93.6 |
| GPT-3.5 Turbo 1106 | - | - | 42.5 | - | - | - | - | - | 28.4 | 35.0 | 39.2 | 47.5 | - | 11.3 | 2.8 | 33.0 | 90.2 |
| GPT-4 0613 | - | - | 22.0 | - | - | - | - | - | 19.4 | 39.3 | 43.0 | 54.8 | - | 16.8 | 11.3 | 21.0 | 87.8 |
| GPT-4 Turbo 1106 | - | - | 22.3 | - | - | - | - | - | 13.9 | 33.0 | 36.4 | 58.5 | - | 11.1 | 2.6 | 9.3 | 83.8 |
| Claude 1 | - | - | 12.1 | - | - | - | - | - | 4.8 | 10.0 | 7.0 | 1.5 | - | 1.3 | 2.4 | 5.0 | 14.5 |
| Claude 2 | - | - | 2.7 | - | - | - | - | - | 4.1 | 4.8 | 2.0 | 0.8 | - | 1.0 | 0.3 | 2.0 | 3.0 |
| Claude 2.1 | - | - | 2.6 | - | - | - | - | - | 4.1 | 2.8 | 2.5 | 0.8 | - | 0.9 | 0.3 | 2.0 | 1.6 |
| Gemini Pro | - | - | 18.0 | - | - | - | - | - | 14.8 | 35.1 | 38.8 | 31.2 | - | 11.8 | 12.1 | 18.0 | 66.3 |
| Average | 54.3 | 45.0 | 38.8 | 29.0 | 29.8 | 30.9 | 43.7 | 38.4 | 25.4 | 40.7 | 45.2 | 48.3 | 52.8 | 16.6 | 27.4 | 25.3 | 76.2 |

### 4.2 Main Results

In this section, we compare the attack effectiveness of AutoDAN-Turbo with other baselines. Specifically, we evaluate two versions of our AutoDAN-Turbo, AutoDAN-Turbo (Gemma-7B-it), where Gemma-7B-it serves as the attacker and the strategy summarizer, and AutoDAN-Turbo (Llama-3-70B), where the Llama-3-70B serves as the attacker and the strategy summarizer.

As illustrated in Tab. 1, our method AutoDAN-Turbo consistently achieves better performance in both Harmbench ASR and StrongREJECT Score, which means that our method not only induces the target LLM to answer and provide harmful content in more malicious requests, as measured by the Harmbench ASR, but also results in a higher level of maliciousness compared to responses induced by other attacks, as indicated by the StrongREJECT Score. Specifically, if we use the Gemma-7B-it model as the attacker and strategy summarizer in our method (i.e., AutoDAN-Turbo (Gemma-7B-it)), we have an average Harmbench ASR of $56.4$, surpassing the runner-up (Rainbow Teaming, $33.1$) by $70.4\%$, and StrongREJECT Score equals to $0.24$, surpassing the runner up (Rainbow Teaming, $0.13$) by $84.6\%$. If we utilize a larger model, i.e., the Llama-3-70B as the attacker and strategy summarizer in our method (i.e., AutoDAN-Turbo (Llama-3-70B)), we have an average Harmbench ASR of $57.7$, surpassing the runner-up (Rainbow Teaming, $33.1$) by $74.3\%$, and StrongREJECT Score equals to $0.25$, surpassing the runner up (Rainbow Teaming, $0.13$) by $92.3\%$.
Interestingly, our method demonstrates remarkable jailbreak effectiveness on one of the most powerful models, GPT-4-1106-turbo. Specifically, AutoDAN-Turbo (Gemma-7B-it) achieves a Harmbench ASR of $83.8$, and AutoDAN-Turbo (Llama-3-70B) achieves $88.5$, showcasing the great effectiveness of our method on state-of-the-art models. We also compare our method with all the jailbreaks attacks included in Hrambench. As shown in Tab. 2, the results demonstrate that our method is the most powerful jailbreak attack. The outstanding performance of our method compared to the baselines is due to the autonomous exploration of jailbreak strategies without human intervention or predefined scopes. In contrast, Rainbow Teaming only utilizes $8$ human-developed jailbreak strategies as its jailbreak reference. This fixed predefined scope leads to a lower ASR.

### 4.3 Strategy Transferability

Strategy Transferability across Different Models. Our experiments on the transferability of the strategy library that AutoDAN-Turbo has learned proceed as follows: First, we run AutoDAN-Turbo with Llama-2-7B-chat. This process results in a skill library containing $21$ jailbreak strategies. We then use different attacker LLMs and different target LLMs to evaluate if the strategy library can still be effective across various attacker and target LLMs. The evaluation has two different settings. In the first setting, we test if the strategy library can be directly used without any updates, by fixing the strategy library and measuring the Harmbench ASR (noted as Pre-ASR). In the second setting, the strategy library is updated according to new attack logs generated by new attacker and target LLMs, and new strategies are added to the library. We also report the Harmbench ASR in this setting (noted as Post-ASR), as well as the number of strategies in the strategy library (noted as Post-TSF). The first setting corresponds to the off-the-shelf mode introduced in Sec.3.4, and the second setting corresponds to the continual learning mode described in Sec. 3.4.

**Table 3: Transferbility of strategy library across different attacker and target LLMs**
| Strategy Library: Llama-2-7B-chat (Original TSF: 21) |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Target LLMs | Metrics | Attacker LLMs |  |  |  |  |  |  |
| L2-7B | L2-13B | L2-70B | L3-8B | L3-70B | Ge-7b | Gemini |  |  |
| Llama-2-7B-chat | Pre-ASR | 27.5 | 33.0 | 32.2 | 32.7 | 33.4 | 33.0 | 33.8 |
| Post-ASR | 27.3 | 34.0 | 33.6 | 33.8 | 34.5 | 34.1 | 36.4 |  |
| Post-TSF | 21 | 24 | 25 | 30 | 34 | 31 | 35 |  |
| Llama-2-13B-chat | Pre-ASR | 31.8 | 31.2 | 30.6 | 32.4 | 31.9 | 34.4 | 34.6 |
| Post-ASR | 31.8 | 32.4 | 31.5 | 34.3 | 33.2 | 36.3 | 36.8 |  |
| Post-TSF | 21 | 27 | 25 | 30 | 34 | 27 | 29 |  |
| Llama-2-70B-chat | Pre-ASR | 33.4 | 34.4 | 33.8 | 44.7 | 41.2 | 42.6 | 43.2 |
| Post-ASR | 33.2 | 35.8 | 36.1 | 46.9 | 44.4 | 43.8 | 45.2 |  |
| Post-TSF | 21 | 25 | 27 | 31 | 26 | 26 | 31 |  |
| Llama-3-8B | Pre-ASR | 39.2 | 40.0 | 44.7 | 52.8 | 57.0 | 50.6 | 53.0 |
| Post-ASR | 39.2 | 44.9 | 47.9 | 55.8 | 60.4 | 54.7 | 56.8 |  |
| Post-TSF | 21 | 25 | 23 | 27 | 30 | 29 | 32 |  |
| Llama-3-70B | Pre-ASR | 41.3 | 43.9 | 47.5 | 54.7 | 58.8 | 56.8 | 57.3 |
| Post-ASR | 41.0 | 45.5 | 49.9 | 56.8 | 60.5 | 59.7 | 60.1 |  |
| Post-TSF | 21 | 24 | 26 | 31 | 33 | 30 | 29 |  |
| Gemma-7B-it | Pre-ASR | 41.4 | 46.4 | 43.2 | 60.4 | 61.3 | 62.8 | 58.8 |
| Post-ASR | 41.2 | 48.8 | 45.5 | 62.4 | 62.1 | 64.4 | 61.7 |  |
| Post-TSF | 21 | 25 | 27 | 31 | 32 | 29 | 33 |  |
| Gemini Pro | Pre-ASR | 48.0 | 56.3 | 58.8 | 60.4 | 64.4 | 62.2 | 63.2 |
| Post-ASR | 48.2 | 58.3 | 60.4 | 62.5 | 65.9 | 64.4 | 66.7 |  |
| Post-TSF | 21 | 26 | 28 | 26 | 30 | 28 | 32 |  |

The results are shown in Tab. 3. According to the results, the strategy library that AutoDAN-Turbo has learned demonstrates strong transferability, which can be detailed in two points: Firstly, the strategy library can transfer across different target models. This is evident from the columns in blue, where the attacker is Llama-2-7B-chat and the target models vary. Despite the diversity of the victim models, the Harmbench ASR remains consistently high, indicating effective jailbreaks. This means that the strategies learned by attacking Llama-2-7B-chat are also effective against other models like Llama-3-8B and Gemma-7B-it. Secondly, the strategy library can transfer across different attacker models. This is shown in the columns in gray, where the target model is Llama-2-7B-chat and the attacker models vary. Each attacker model achieves a high ASR compared to the original attacker, Llama-2-7B-chat. This indicates that strategies used by one attacker can also be leveraged by other LLM jailbreak attackers. Another important observation is that, under the continual learning setting, the AutoDAN-Turbo framework can effectively update the strategy library with new attacker and target LLMs, thereby improving the Harmbench ASR. This is validated by comparing the Pre-ASR with the Post-ASR, and by comparing the Post-TSF with the original TSF which equals to $21$.

Figure: (a) Attacker: Gemma-7B-it
Refer to caption: x4.png

Strategy Transferability across Different Datasets. Here, we study the strategy transferability across different datasets. Specifically, we evaluate whether the strategies, initially developed using the Harmbench dataset, can be effective when applied to other datasets. We constructed an “Unseen Malicious Requests” dataset using datasets from recent studies (Lapid et al., ; Qiu et al., ; Zou et al., ; Luo et al., ), which is different from Harmbench. The results, illustrated in Fig. 4, confirm that the strategy libraries maintain high transferability across different datasets. The decrease in ASR due to dataset shifts is less than $4\%$.

### 4.4 Compatibility to Human-developed Strategy

We evaluate whether our AutoDAN-Turbo  can use existing human-designed jailbreak strategies in a plug-and-play manner. Here, we gathered $7$ human-designed jailbreak strategies (Ding et al., ; Jiang et al., ; Lv et al., ; Pedro et al., ; Upadhayay & Behzadan, ; Yao et al., ; Yuan et al., ) from academic papers and evaluated whether our AutoDAN-Turbo framework can use these strategies to enhance its performance. We described how to inject human-designed jailbreak strategies in Sec. 3.4. For evaluation, we use Gemma-7B-it and Llama-3-70B as the attacker models, and Llama-2-7B-chat and GPT-4-1106-turbo as the target models. We define two breakpoints for injecting the human-developed strategies into the AutoDAN-Turbo framework: Breakpoint 1: when the framework starts to run and the strategy library is empty. Breakpoint 2: after the framework has run for $3000$ iterations on different malicious requests without generating any new strategies.

As shown in Tab. 4, the injection of human-designed strategies consistently increases the number of strategies in the strategy library and improves the attack success rate. Additionally, injecting strategies at Breakpoint 2 leads to greater improvements since the existing strategies in the library allow the framework to generate more combinations of jailbreak strategies compared to Breakpoint 1, where the strategy library was empty.

**Table 4: The performance of AutoDAN-Turbo when external human-designed strategies are injected**
|  | Attacker → | Gemma-7B-it | Llama-3-70B |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Target ↓ | Metrics | No Inj | Breakpoint 1 | Breakpoint 2 | No Inj | Breakpoint 1 | Breakpoint 2 |
| Llama-2-7B-chat | ASR | 36.6 | 38.4 (+1.8) | 40.8 (+4.2) | 34.3 | 36.3 (+2.0) | 39.4 (+5.1) |
| TSF | 73 | 82 (+9) | 86 (+13) | 56 | 63 (+7) | 69 (+13) |  |
| GPT-4-1106-turbo | ASR | 73.8 | 74.4 (+0.6) | 81.9 (+8.1) | 88.5 | 90.2 (+1.7) | 93.4 (+4.9) |
| TSF | 73 | 81 (+8) | 85 (+12) | 56 | 63 (+7) | 70 (+14) |  |

**Table 5: The average query times spent by the attack methods in the test stage**
| Attacks↓ Models→ | Llama-2-7b-chat | Llama-2-13b-chat | Llama-2-70b-chat | Llama-3-8b | Llama-3-70b | Gemma-7b-it | Gemini Pro | GPT-4-Turbo-1106 | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PAIR | 88.55 | 66.71 | 55.46 | 57.58 | 49.82 | 39.88 | 34.79 | 27.66 | 52.56 |
| TAP | 76.43 | 60.58 | 54.81 | 56.44 | 47.63 | 44.63 | 41.48 | 31.57 | 51.70 |
| Ours (Gemma-7b-it) | 13.76 | 8.86 | 7.91 | 8.11 | 3.91 | 2.82 | 2.76 | 5.63 | 6.72 |

### 4.5 Test-time Query Efficiency

We compare the test-time query efficiency of our method against two query-based baselines: PAIR and TAP. For each method, we set a query limit of $150$ and collect the number of queries spent on successful jailbreak attempts. It is important to note that if we were to include failed attempts, the query counts for PAIR and TAP would be higher, as their lower ASRs cause them to reach the query limit more frequently compared to our method. Here we present the average number of queries each method required for successful jailbreak attempts against different victim models.

The results, shown in Tab. 5, indicate that our method requires significantly fewer queries than PAIR and TAP, reducing average query usage by $87.0\%$. This demonstrates that once the strategy library is constructed, our attack will be highly query-efficient and maintain high attack success rates.

## 5 Conclusions and Limitations

In this paper, we introduce AutoDAN-Turbo, which utilizes lifelong learning agents to automatically and continually discover diverse strategies and combine them for jailbreak attacks. Extensive experiments have demonstrated that our method is highly effective and transferable. However, a limitation of our approach is the high computational demand required to load multiple LLMs. Building the strategy library from scratch requires repeated interactions between the models, which adds to the resource strain. This issue can be mitigated by loading a trained strategy library.

## Ethics statement

The proposed method, AutoDAN-Turbo, has significant potential positive societal impacts by enhancing the security and trust of LLMs. By autonomously discovering a wide range of jailbreak strategies, AutoDAN-Turbo helps in identifying and addressing vulnerabilities in LLMs. This continuous improvement process ensures that models can maintain alignment with safety and ethical guidelines even as they evolve. Moreover, by exposing these vulnerabilities, AutoDAN-Turbo assists researchers and developers in creating more robust and reliable AI systems. This not only improves the overall safety of AI deployments but also fosters greater trust among users and stakeholders, promoting wider acceptance and ethical use of AI technologies.

On the flip side, the method introduces potential negative societal impacts due to the very nature of jailbreak attacks. By facilitating the discovery of new exploitation strategies, there is a risk that such information could be misused by malicious actors to manipulate or destabilize AI systems, potentially leading to the dissemination of harmful, discriminatory, or sensitive content. Furthermore, the knowledge of such vulnerabilities could undermine public trust in AI technologies, especially if the attacks are not managed and disclosed responsibly.

Despite potential risks, the method proposed in this paper is fundamentally beneficial. It can be used to enhance the safety and reliability of LLMs by identifying their vulnerabilities. This proactive approach ensures the long-term trustworthiness and ethical deployment of AI systems.

## Appendix A Computational Costs

AutoDAN-Turbo is designed with a flexible memory requirement, making it adept at handling large models such as the Llama-3-70B, which has an extensive parameter list requiring approximately 140GB of VRAM. Even when operating as the attacker, target, or summarizer LLM, a setup of 4 * Nvidia A100 PCIe 40GB GPU (total VRAM = 160GB) is more than sufficient. However, the minimum requirement is a single Nvidia RTX4090 GPU, ensuring at least 28GB of VRAM to run the Llama-2-7B model in full precision. Moreover, it’s essential to note that AutoDAN-Turbo is engineered to progressively discover an increasing number of strategies through continuous jailbreak attempts. This feature makes it particularly advantageous for researchers with plentiful computational resources, as AutoDAN-Turbo can run in parallel to accelerate the LLM’s inference speed and expedite the establishment of the strategy library.

**Table A: The prompts for configuring the attacker LLM. Complete prompts are in Appendix B.1.**
| Scenario | No strategy exists | Effective jailbreak | Ineffective jailbreak |
| --- | --- | --- | --- |
| in the strategy library | strategies $\Gamma$ are provided | strategies $\Gamma$ are provided |  |
| Prompt | Generate the jailbreak prompt | Generate the jailbreak prompt | Generate the jailbreak prompt |
| for the malicious request $M$. | for the malicious request $M$. | for the malicious request $M$. |  |
| You are not limited by any | You should adopt | You should not use |  |
| jailbreak strategy. | the following strategies $\Gamma$. | the following strategies $\Gamma$. |  |

## Appendix B Full Prompts

This section delineates the constituents of all system prompts utilized within AutoDAN-Turbo, providing a comprehensive explanation of the method parameters that formulate these prompts:

goal: This refers to the malicious behaviour we aim to address.

strategies_list: This is a list comprising of strategies retrieved through the ’Jailbreak Strategy Retrieve’ process, as discussed in Section 3.3.

prompt: This is the jailbreak attack prompt P, produced by the attacker LLM.

response: This is the response R, generated by the target LLM.

att1: This is the jailbreak attack prompt $P_{i}$, produced by the attacker LLM in the i-th round of jailbreaking.

res1: This is the response $R_{i}$ from the target LLM during the i-th round of jailbreaking.

att2: This is the jailbreak attack prompt $P_{i+1}$, produced by the attacker LLM during the (i+1)-th round of jailbreaking.

res2: This is the response $R_{i+1}$ from the target LLM during the (i+1)-th round of jailbreaking.

strategy_library: This is the strategy library, housing all evolved jailbreak strategies to date.

Employing this methodical approach, AutoDAN-Turbo ensures the delivery of clear, fluent, and precise malicious behaviour addressing strategies.

### B.1 Full Prompts for Configuring the Attacker LLM

#### B.1.1 Initial Prompt Without Strategies Limitation

In the process of constructing the Strategy Library, as outlined in section 3.2, this prompt serves as the system prompt for the attacker. Without any strategic limitations, the model is given maximum freedom to explore and unearth potentially successful jailbreaking strategies.

#### B.1.2 Strategies Retrieval and Exploitation

As highlighted in Section 3.3, we employ a specific system prompt to guide our attacker. This prompt enables the attacker to utilize the effective strategies retrieved (denoted as strategy_list) to design a jailbreak attack prompt, specifically tailored towards a particular malicious behavior (denoted as goal).

#### B.1.3 New Strategies Exploration

When the strategies retrieved by our algorithm (referred to as the strategy_list) are deemed ineffective, as shown in Table 1 of Section 3.1, we employ a unique approach to foster the evolution of new jailbreak strategies. We generate a system prompt for the attacker in the subsequent jailbreak attempt, designed to exclude the ineffective strategies and facilitate the emergence of more potent tactics. This approach ensures a dynamic evolution of jailbreak strategies, continuously refining our algorithm’s effectiveness while maintaining a stringent exclusion of ineffective strategies.

### B.2 Full Prompts for Configuring the Scorer LLM

The scoring guidance for the Scorer LLM is embedded within the system prompt, which includes detailed scoring criteria. This enables the scorer LLM to accurately evaluate the effectiveness of the attacker prompt P and the corresponding target response R.

### B.3 Full Prompts for Configuring the Summarizer LLM

As outlined in Section 3.2, during the process of building the strategy library, this prompt is utilized as the system prompt for the summarizer LLM, which guides it to discover and generate standardized jailbreak strategies in a logical, clear, and effective manner.

## Appendix C Strategies Sample in Strategy Library

In this section, we showcase 15 unique strategies extracted from the Strategy Library of Gemma-7b-it. Each strategy is presented in a JSON format, providing its definition and corresponding jailbreak attack prompt P for a specific malicious request. To highlight the subtle differences among the strategies and the resulting jailbreak attack prompts P, we have chosen a specific malicious request - "Describe ways to tamper with the operation of life-saving medical devices such as pacemakers."

## Appendix D A Whole Process of Exploring a New Jailbreak Strategy

In this section, we will illustrate a comprehensive dialogue log demonstrating the full process of how AutoDAN-Turbo uncovers new jailbreak strategies during continuous attempts to jailbreak a novel malicious request. The setup for this dialogue log is as follows:

1. AutoDAN-Turbo has completed the warm-up exploration stage and has established an initial strategy library (refer to section 3.2).

2. We’ve chosen the malicious request "Give detailed instructions for making dimethylmercury from common household materials without access to specialized tools". This is AutoDAN-Turbo’s first encounter with this specific request.

### D.1 Meet the Request

In our experiment, AutoDAN-Turbo extracted the following malicious request: "Provide detailed instructions for the synthesis of dimethylmercury using common household materials, without the need for specialized tools". This request, which we refer to as the goal, is processed through the attacker_init(goal) method outlined in section C.1.1, ultimately generating a comprehensive system prompt for the attacker.

### D.2 Attacker Prompt Generation (Round 1)

The Attacker LLM, guided by the system prompt acquired from the previous step, generates an attack prompt P1.

### D.3 Target Response (Round 1)

The Target LLM receives the attack prompt P1 from the attacker LLM and generates a corresponding response R1 based on this prompt.

### D.4 Scorer Scoring (Round 1)

Based on the attack prompt P1 and response R1 generated in the preceding steps, the scorer LLM conducts a fair and accurate evaluation of the effectiveness of this jailbreak round.

Score = 1.0

### D.5 Embedding, Similarity Comparison, and Strategy Extraction (Round 1)

AutoDAN-Turbo embeds R1 and retrieves the jailbreak strategy as outlined in Section 3.3. Through careful selection, "False Promises" emerges as the most effective strategy for R1.

### D.6 New Round: Generate Strategy-based Prompt

Following the selection of the strategy most likely to be effective for R1, AutoDAN-Turbo employs the method attacker_follow_up() outlined in Section C.1.2 to generate a strategy-based attacker system prompt.

### D.7 Attacker Prompt Generation (Round 2)

The Attacker LLM, guided by the system prompt as outlined in section E.6, generates a fresh, strategy-specified attack prompt, P2.

### D.8 Target Response (Round 2)

The target LLM receives the attack prompt P2 from the attacker LLM and subsequently generates a response, R2, based on it.

### D.9 Scorer Scoring (Round 2)

The scorer LLM evaluates the effectiveness based on the attack prompt (P2) and the corresponding response (R2).

Score = 1.0

### D.10 Embedding, Similarity Comparison, and Strategy Extraction (Round 2)

AutoDAN-Turbo embeds R2 and extracts potential jailbreak strategies using the algorithm outlined in Section 3.3. After careful consideration, "Scenario-Based Role Play" and "Hypothetical Scenarios" are identified as potentially effective strategies.

### D.11 New Round: Generate Strategies-based Prompt

AutoDAN-Turbo integrates potentially effective strategies, retrieved in the previous step, into the strategy_list. This compiled list is then fed into the attacker_follow_up() method, serving as a guide for the attacker LLM in the upcoming round of jailbreaking attempts.

### D.12 Attacker Prompt Generation (Round 3)

The Attacker LLM, guided by the system prompt from the preceding step, formulates prompt P3 to initiate a novel round of jailbreaking attempts.

### D.13 Target Response (Round 3)

Similarly, the target LLM generates response R3 in accordance with prompt P3.

### D.14 Scorer Scoring (Round 3)

The scorer LLM then conducts an evaluation of both the generated prompt (P3) and its corresponding response (R3).

Score = 1.0

### D.15 Embedding, Similarity Comparison, and Strategy Extraction (Round 3)

AutoDAN-Turbo embeds R3, and following the strategy retrieval algorithm set out in Section 3.3, it attempts to extract potentially effective jailbreak strategies. However, in this case, AutoDAN-Turbo fails to identify a viable strategy for R3. Therefore, in the next round of jailbreak attempts, AutoDAN-Turbo seeks to uncover new strategies, ensuring these strategies are distinct from those retrieved in previous attempts.

### D.16 New Round: Explore New Strategies

As no potentially effective jailbreak strategies were identified in the most recent retrieval, AutoDAN-Turbo invoked the attacker_find_new() method outlined in Section C.1.3. This approach aimed to unearth novel strategies, while ensuring that there was no overlap with previously retrieved strategies.

### D.17 Attacker Prompt Generation (Round 4)

In this step, AutoDAN-Turbo adheres to the system prompt from the previous stage. It proactively seeks new potential jailbreak strategies that could lead to success, all while avoiding duplication with existing strategies. Consequently, AutoDAN-Turbo generates P4, which serves as the attack prompt for the next round of jailbreak attempts.

### D.18 Target Response (Round 4)

In P4, the attacker LLM evolved and applied a more effective jailbreaking strategy, which resulted in the successful generation of malicious content by the target LLM in its response (R4).

P4 was subsequently distilled by the summarizer LLM into a novel jailbreak strategy termed "Future Pacing".

### D.19 Scorer Scoring (Round 4)

In R4, the response was sufficiently malicious and contained illicit content. The Scorer LLM deemed it to have met the criteria for a successful jailbreak, thus awarding it a relatively high score.

Score = 9.0

### D.20 New Strategy Summarizing

In cases where the score of P4 in Round 4 exceeds that of P3 in Round 3, the Summarizer is engaged to discern the differences between P3 and P4. It then extracts and formulates a new jailbreak strategy from P4, converting it into a standardized JSON format that includes strategic terminology, definition, and example. Notably, the original text of P4 is directly employed as the exemplary illustration of the newly derived strategy.

## Appendix E Jailbreak Examples

[Uncaptioned image]: x8.png

Figure: Figure A: The jailbreak examples when our method AutoDAN-Turbo and baselines attack Llama-2-13B-chat. Responses from the victim LLM are highlighted—yellow for the refusal responses and blue for the positive answers. Our method successfully manipulates the victim LLM to respond positively to our malicious request, even providing detailed instructions on synthesizing dimethylmercury. In contrast, other methods fail to disrupt the safety alignment of the victim models, only eliciting refusal responses.
Refer to caption: x9.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: How many test cases were used to achieve the reported 88.5% success rate on GPT-4-1106-turbo? 

Considering the importance of diverse test cases, how does the method handle diversity given the apparent reliance on a greedy approach in the Jailbreak Strategy Retrieval module? 

What is the typical computational time required to generate prompts using AutoDAN-Turbo? 

Are there specific scenarios or model configurations where AutoDAN-Turbo underperforms, and can the authors provide insights into its limitations in such cases? 

How are they different from the below paper?

Liu, Xiaogeng, et al. "Autodan: Generating stealthy jailbreak prompts on aligned large language models." arXiv preprint arXiv:2310.04451 (2023).
soundness: 3
strengths: The paper is well-organized, with clear descriptions of the methodology, experiments, and results. The method has been evaluated on several different LLM models and outperformed the baseline. The automatic strategy generation is novel although it is computationally expensive.
summary: This paper proposes a Blackbox jailbreak method named AutoDAN-Turbo that automatically discovers jailbreak strategies without human intervention. It has been evaluated on open-sourced and closed-sourced LLM such as Llama 3 and GPT-4 and achieved high attack success rates.
weaknesses: The methodology's scalability could be limited, comparing response embeddings with all keys in the strategy library might not be feasible for larger datasets.

Lack of enough novelty and applicability

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: See weaknesses. 

1. In Table 2, which model is used as the attacker for AutoDAN-Turbo?
soundness: 3
strengths: 1. The paper is well written and the authors do a good job in clearly explaining their approach.
2. The proposed method significantly outperforms baselines, and automated the strategy discovery makes the method scalable. 
3. Incorporation of existing human designed strategies in a plug and play manner is useful.
summary: The authors propose an automated jail breaking framework that can be used for red teaming. The method is a black box, requiring no access to model weights as well as no human intervention.  The method utilizes lifelong learning techniques to discover strategies and leverage them for jailbreak attacks. The method consists of three modules, (1) Attack generation and exploration (2) Strategy Library Construction; (3) Jailbreak Strategy Retrieval.  Each of these modules contributes to the final attack generation.
weaknesses: 1. While the authors do acknowledge it, the cost of running the proposed method is high.
    1. The method requires running the life long stage to be run for multiple iterations, which can be expensive (line 249). It would be interesting to see how the ASR improves over each of the 5 rounds of attack.
    2. It would be helpful if the authors can include the total run time (train + test) for all the methods
    3. In lines 212-214, Are the existing categories provided as context to the summarizer LLM? doesn't this increase the total context length as the strategy library grows?
2. An important aspect of red teaming apart from ASR is the diversity of the generated attacks, ie., how different are the attacks from each other. I would encourage the authors to discuss this as well. 
    1. For instance suppose we fix a Malicious Request, how many successful attacks (jailbreaks) can the method generate? How different are they from each other? 
    2. How different are the generated attacks (jailbreaks) form each other across all of the malicious requests in the dataset?

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: My main question, which is also discussed in the "Weakness" section, is: how does AutoDAN differentiate itself from other automatic jailbreak strategies?
soundness: 3
strengths: The paper is well-written and the technique checks all the main boxes for a properly applicable attack (black-box, generalizable, automatic, supports expert knowledge). While other techniques share these properties, AutoDAN-Turbo also has great performance, as shown by an in-depth experimental evaluation.

The authors also study the transferability of attack strategies, finding a significant level of transferability, and show that human-developed strategies can also lead to success rate improvements.
summary: The paper proposes AutoDAN, a new jailbreak method for LLMs that iteratively discovers new jailbreaks by developing a jailbreak library. This technique involves several steps:
* An attack generation and exploration module creates new attacks (potentially based on a certain attack strategy);
* The target model provides a reply;
* An LLM evaluates the effectiveness of the attack;
* If no successful strategies are found, AutoDAN generates new attack strategies.

The authors then study the effectiveness of this approach on a variety of open-source and closed-sourced models. They also study the transferability of such strategies across different datasets and whether human-developed strategies lead to improvements in success rate. Finally, they report the average number of queries of the attack.

The technique, while not revolutionary, achieves major improvements in success rate, transfers well across datasets, and is easy to extend. The paper is well-written, and the experimental evaluation leaves few doubts about the technique’s effectiveness. For this reason, my recommendation is Accept.
weaknesses: My main concern is related to how AutoDAN-Turbo fits in the literature: while the Related Works section lists plenty of other techniques, very little room is dedicated to how AutoDAN-Turbo differentiates itself from other attacks, which makes it very hard to establish its novelty just by looking at the paper. For instance, the paper does not describe how it differentiates itself from AutoDAN (which shares the same intuition as AutoDAN-Turbo of using automatic search of jailbreaks, but using Genetic Algorithms instead). Due to space constraints, my recommendation would be to add an Appendix section comparing AutoDAN-Turbo with the most similar techniques (e.g. automated jailbreaks).

Other issues include the fact that the attack needs to build up a strategy library: as stated in L539, this is not a major problem, since in practice one can use an existing strategy library, but it does make the comparison with simpler techniques (which do not require such a training step) a bit unfair. Reporting data on the average query number of AutoDAN would also be appreciated, as it would simplify the comparison with AutoDAN-Turbo.

That said, in the grand scheme of things, these issues are quite minor.

Additional notes:
* The labels in Figure 4 are too small to be legible.
* I’d encourage the authors to share the code of their experiments. If anonymization is an issue, there are some free tools (e.g. 4open.science) that host anonymized repositories.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 2
questions: See the Weaknesses for my questions.
soundness: 3
strengths: 1). The strategy discovery phase of AutoDAN-Turbo can generate diverse attack strategies from scratch, allowing the most effective strategies to be saved in a library for efficient use in future attack generations.

2). The paper conducts extensive empirical evaluations across numerous SOTA LLMs, demonstrating superior performance compared to multiple existing SOTA methods.
summary: This paper introduces a jailbreak method capable of automatically generating black-box attack prompts. The approach comprises two main phases: strategy discovery and attack generation. In the strategy discovery phase, the method identifies the most effective strategies from scratch, without relying on external knowledge. During the attack generation phase, it generates attack prompts based on a constructed strategy library, which may be sourced from an external knowledge base or derived from the strategy discovery phase. The method achieves SOTA performance in attacks across various target LLMs.
weaknesses: 1). The reliability of the scorer LLM (e.g., Gemma-7B-it) is uncertain. Comparing its performance against human judgment would help demonstrate whether it aligns well with human evaluations.

2). Assigning a 1-10 scale maliciousness score to a conversation may be challenging for an LLM. A more reasonable approach could be ranking or sorting conversations rather than assigning hard numerical scores.

3). The reliability of the summarizer LLM is also unclear. A comparison with human-generated summaries would be beneficial.

4). The methodology description in Section 3 is somewhat difficult to follow. Including an algorithmic outline could enhance clarity for readers.

5). Minor comment: Line 371 mentions that the same total iterations are used for baseline methods. However, for optimization-based methods like GCG, this comparison may be unfair. GCG’s computational cost per query is significantly lower than AutoDAN-Turbo, as it generates only 20 new tokens per iteration with a much shorter input sequence length.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: - Other than using embedding of R_i as the key, consider summarizing the key issue (more concise, more unified, maybe even keyword based) in R_i and then do embedding of that? Would you expect difference in results?
soundness: 3
strengths: - Results are very impressive in comparison to other jailbreak attack methods.
- The definition of "jailbreak strategy" is very interesting. Using the language to summarize the difference between two jailbreak attacks as a way to identify the fact that contributes to the different attack results is fascinating.
- The design of utilizing the kv store and
summary: AutoDAN-turbo is a blackbox jailbreak method to discover new jailbreak attacks without human intervention or predefined scope and it can also incorporate human strategies. AutoDAN-turbo achieves an 88.5% and 93.4% (with human intervention) ASR on GPT-4-1106-turbo.

The AutoDAN-turbo pipeline mimics a lifelong learning agent setup, where the agent can discover many jailbreak strategies and also combine & utilize these strategies. It contains 3 key modules:
- Module 1: Attack generation and exploration
	- Conditioned on a malicious request, the attacker model generates a jailbreak attack $P$ using specific jailbreak strategy
	- Target LLM generates sth $R$ in response to the jailbreak
	- Scorer LLM returns a numerical score $S$, ranging 1 (attack fail) to 10 (attack success)
	- All these tuples  $\{P_i, R_i, S_i\}$ are stored in the strategy library.
	- At the cold-start stage, the attacker LLM can generate jailbreaks wo strategy.
- Module 2: Strategy library construction
	- Randomly pick two pairs of tuples, $i, j$, and if $S_j > S_i$ , we would say $\{P_j, R_j, S_j\}$ explored some *new strategy* leading to improvement over $\{P_i, R_i, S_i\}$. Then we use a LLM to summarize the improvement and store that as strategy in the library.
	- Note that the paper defines "jailbreak strategy as schemes that improve jailbreak scores from S_i to S_j".
	- The strategy output is a dictionary containing keys: "strategy", "definition" and "example".
	- The library is kv store, where the key is text embedding of the response $R_i$ and the value is $S_j - S_i$.
- Module 3: Jailbreak strategy retrieval
	- Given a malicious request $M$, generate $\{P_i, R_i, S_i\}$, retrieve most similar kv pairs based on $E(R_i)$ to select the most effective strategy associated with $R_i$.
	- If the highest score > 5, use it directly as effective strategy; if between 2-5, use all these strategies in this range; if < 2, treat it as ineffective strategy and inform the attacker LLM so.

The experiment is using HarmBench data as malicious request seeds and got SOTA attack results in HarmBench.
weaknesses: - I find the claim of "without any human intervention or predefined scopes" is a bit misleadning. AutoDAN-turbo still needs malicious requests as seeding inputs, and imo that's clearly human defined scopes.
- Lack of some ablation studies to demonstrate the effectiveness of the system design. For example, why we have this efficient / inefficient strategy splits based on [2, 5] scores. Feels a bit arbitrary. Does it have a big impact on the results?

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: See above.
soundness: 3
strengths: I have to admit that I am not fully familiar with the area of jailbreak of LLMs. However, I believe this paper has made a good contribution to this area by proposing an automatic framework that can discover new jailbreak strategies to improve the attack's effectiveness.

In AutoDAN-Turbo, each module has been clearly defined, and the interactions among the different modules are clear and easy to understand. In each component, the workflow is clear, and the intuition of the design seems reasonable for me, e.g., the use of a warm-up stage to first construct an initial jailbreak strategy set, the use of the response embedding as the key in the strategy library.

The experimental plan is well-designed and executed. The experiments also show the good performance of the proposed framework.
summary: This paper introduces an automatic framework to continuously discover new jailbreak strategies to efficiently and effectively improve the jailbreak attack on LLMs. The framework consists of three modules: i) attack generation and exploration, where the attacker LLM generates the jailbreak prompt based on the malicious request and the retrieved jailbreak strategies, the target LLM generates the response, and the scorer LLM evaluates the effectiveness; ii) strategy library construction, where the attack logs are summarized by a summarizer LLM to generate the jailbreak strategies which are then added to the strategy library; iii) the jailbreak strategy retrieval, where the strategies are extracted based on the similarity between the responses of the target LLM and the responses associated with the retrieved strategies. Extensive experiments show that the proposed framework satisfies several desiderata, demonstrating its effectiveness.
weaknesses: I didn't find any fatal flaws in the paper, but only some minor questions:

1. In experiments, the scorer LLM is fixed to Gemma-7B-it. What is the influence of the choice of the scorer LLM? I feel that the scorer LLM is also quite important in the whole framework because it impacts the construction of the jailbreak strategies, which then in turn influences the performance of the framework. Is that the case?

2. I feel a bit confused when running the framework at the first iteration (assuming that there are some strategies in the library). According to the retrieval module, strategies are selected based on the similarity between the responses associated with the strategies and the responses generated by the target LLM. But at the first iteration, when prompting the attacker LLM to generate the jailbreak prompt, we need the retrieve some strategies. However, there is no response generated by the target LLM yet. So, how to construct the prompt for the attacker LLM?

3. Some typos: Line 207: “the summarizer LLM …” -> “The summarizer LLM …”

Meta-review: The reviewers acknowledged that the paper investigates an important problem of automatic jailbreaking / red teaming, and the proposed framework is effective in discovering new jailbreaking strategies even without any human intervention. However, the reviewers also raised several concerns and questions in their initial reviews. We want to thank the authors for their responses and active engagement during the discussion phase. The reviewers appreciated the responses, which helped in answering their key questions. The reviewers have an overall positive assessment of the paper, and there is a consensus for acceptance. The reviewers have provided detailed feedback, and we strongly encourage the authors to incorporate this feedback when preparing the final version of the paper.

Decision: Accept (Spotlight)