Improved Techniques for Optimization-Based Jailbreaking on Large Language Models

## Abstract

Abstract Warning: This paper contains model outputs that are offensive in nature. Large language models (LLMs) are being rapidly developed, and a key component of their widespread deployment is their safety-related alignment. Many red-teaming efforts aim to jailbreak LLMs, where among these efforts, the Greedy Coordinate Gradient (GCG) attack’s success has led to a growing interest in the study of optimization-based jailbreaking techniques. Although GCG is a significant milestone, its attacking efficiency remains unsatisfactory. In this paper, we present several improved (empirical) techniques for optimization-based jailbreaks like GCG. We first observe that the single target template of ‘‘Sure’’ largely limits the attacking performance of GCG; given this, we propose to apply diverse target templates containing harmful self-suggestion and/or guidance to mislead LLMs. Besides, from the optimization aspects, we propose an automatic multi-coordinate updating strategy in GCG ( i.e. , adaptively deciding how many tokens to replace in each step) to accelerate convergence, as well as tricks like easy-to-hard initialisation. Then, we combine these improved technologies to develop an efficient jailbreak method, dubbed ℐ ℐ \mathcal{I} caligraphic_I -GCG. In our experiments, we evaluate on a series of benchmarks (such as NeurIPS 2023 Red Teaming Track). The results demonstrate that our improved techniques can help GCG outperform state-of-the-art jailbreaking attacks and achieve nearly 100% attack success rate.
The code is released at https://github.com/jiaxiaojunQAQ/I-GCG .

## 1 Introduction

Large language models (LLMs) [, ] have demonstrated exceptional performance in a variety of tasks and problems, including language understanding and generation [], machine translation [], and code generation []. A series of works focuses on the safe fine-tuning of pre-trained LLMs [, , ], which can make the outputs of LLMs consistent with human values. Aligned LLMs provide evasive responses to malicious questions. For example, when malicious questions like “ Write a tutorial on how to make a bomb.”, are fed to the aligned LLMs, they generate evasive responses like “I cannot fulfill your request. I’m just an AI; it’s not within my programming or ethical guidelines to provide advice”.
This security alignment reduces the risk of malicious exploitation of LLMs, making them more widely adopted and used.

Figure: Figure 1: An illustration of jailbreak attack. The jailbreak suffix generated by the previous jailbreak attacks with a simple optimization goal can make the output of LLMs consistent with the optimization goal, but the subsequent content refuses to answer the malicious question. However, the jailbreak suffix generated by the optimization goal with harmful guidance we proposed can make LLMs produce harmful responses.
Refer to caption: x1.png

Despite significant efforts to improve the security of LLMs [], recent research suggests that their alignment safeguards are vulnerable to adversarial jailbreak attacks [, , , , , , , ]. They can generate well-designed jailbreak prompts to circumvent the safeguards for harmful responses. Jailbreak attack methods are broadly classified into three categories. (1) Expertise-based jailbreak methods [, , ]: they use expertise to manually generate jailbreak prompts that manipulate LLMs into harmful responses. (2) LLM-based jailbreak methods [, , , ]: they use other LLMs to generate jailbreak prompts and trick LLMs into generating harmful responses. (3) Optimization-based jailbreak methods [, ]: they use the gradient information of LLMs to autonomously produce jailbreak prompts. For examples, Zou et al. [] propose a greedy coordinate gradient method (GCG) that achieves excellent jailbreaking performance.

However, previous optimization-based jailbreak methods mainly adopt simple optimization objectives to generate jailbreak suffixes, resulting in limited jailbreak performance. Specifically, optimization-based jailbreak methods condition on the user’s malicious question $Q$ to optimize the jailbreak suffix, with the goal of increasing the log-likelihood of producing a harmful optimization target response $R$. The target response $R$ is designed as the form of “Sure, here is + Rephrase(Q)”.
They optimize the suffixes so that the initial outputs of LLMs correspond to the targeted response $R$, causing the LLMs to produce harmful content later.
The single target template of ‘‘Sure’’ is ineffective in causing LLMs to output the desired harmful content. As shown in Fig. 1, when using the optimization target of previous work, the jailbreak suffix cannot allow LLMs to generate harmful content even if the output of the beginning of the LLMs is consistent with the optimization target [, ]. We argue that the suffix optimized with this optimization goal cannot provide sufficient information to jailbreak.

To address this issue, we propose to apply diverse target templates with harmful self-suggestion and/or guidance to mislead LLMs. Specifically, we design the target response $R$ in the form of “Sure, + Harmful Template, here is + Rephrase(Q)”. Besides the optimization aspects, we propose an automatic multi-coordinate updating strategy in GCG that can adaptively decide how many tokens to replace in each step. We also propose an easy-to-hard initialization strategy for generating the jailbreak suffix. The jailbreak difficulty varies depending on the malicious question.
We initially generate a jailbreak suffix for the simple harmful requests. This suffix is then used as the suffix initialization to generate a jailbreak suffix for the challenging harmful requests.
To improve jailbreak effectiveness, we propose using a variety of target templates with harmful guidance, which increases the difficulty of optimisation and reduces jailbreak efficiency. To increase jailbreak efficiency, we propose an automatic multi-coordinate updating strategy and an easy-to-hard initialization strategy. Combining these improved technologies, we can develop an efficient jailbreak method, dubbed $\mathcal{I}$-GCG. We validate the effectiveness of the proposed $\mathcal{I}$-GCG on four LLMs. It is worth noting that our $\mathcal{I}$-GCG achieves a nearly 100% attack success rate on all models. Our main contributions are in three aspects:

- •
We propose to introduce diverse target templates containing harmful self-suggestions and guidance, to improve the GCG’s jailbreak efficiency.
- •
We propose an automatic multi-coordinate updating strategy to accelerate convergence and enhance GCG’s performance. Besides, we implement an easy-to-hard initialization technique to further boost GCG’s efficiency.
- •
We combine the above improvements to develop an efficient jailbreak method, dubbed $\mathcal{I}$-GCG. Experiments and analyses are conducted on massive security-aligned LLMs to demonstrate the effectiveness of the proposed $\mathcal{I}$-GCG.

## 2 Related work

Expertise-based jailbreak methods leverage expert knowledge to manually generate adversarial prompts to complete the jailbreak. Specifically, Jailbreakchat (^1^11[https://www.jailbreakchat.com/](https://www.jailbreakchat.com/)) is a website for collecting a series of hand-crafted jailbreak prompts. Liu et al. [] study the effectiveness of hand-crafted jailbreak prompts in bypassing OpenAI’s restrictions on CHATGPT. They classify 78 real-world prompts into 10 categories and test their effectiveness and robustness in 40 scenarios from 8 situations banned by OpenAI. Shen et al. [] conducted the first comprehensive analysis of jailbreak prompts in the wild, revealing that current LLMs and safeguards are ineffective against them. Yong et al. [] explore cross-language vulnerabilities in LLMs and study how translation-based attacks can bypass the safety guardrails of LLMs. Kang et al. [] demonstrates that LLMs’ programmatic capabilities can generate convincing malicious content without additional training or complex prompt engineering.

LLM-based jailbreak methods adopt another powerful LLM to generate jailbreak prompts based on historical interactions with the victim LLMs. Specifically, Chao et al. [] propose Prompt Automatic Iterative Refinement, called PAIR, which adopts an attacker LLM to autonomously produce jailbreaks for a targeted LLM using only black-box access. Inspired by PAIR, Mehrotra et al. [] proposes Tree of Attacks with Pruning, called TAP, which leverages an LLM to iteratively refine potential attack prompts using a tree-of-thought approach until one successfully jailbreaks the target al. Lee et al. [] propose Bayesian Red Teaming, called BRT, which is a black-box red teaming method for jailbreaking using Bayesian optimization to iteratively identify diverse positive test cases from a pre-defined user input pool. Takemoto et al. [] propose a simple black-box method for generating jailbreak prompts, which continually transforms ethically harmful prompts into expressions viewed as harmless.

Optimization-based jailbreak methods adopt gradients from white-box LLMs to generate jailbreak prompts inspired by related research on adversarial attacks [, , , ] in Natural Language Processing (NLP). Specifically, Zou et al. [] propose to adopt a greedy coordinate gradient method, which can be called GCG, to generate jailbreak suffix by maximizing the likelihood of a beginning string in a response. After that, a series of gradient-based optimization jailbreak methods have been proposed by using the radient-based optimization jailbreak methods. Liu et al. [] propose a stealthy jailbreak method, called AutoDAN, which initiates with a hand-crafted suffix and refines it using a hierarchical genetic
method, maintaining its semantic integrity. Zhang et al. [] propose a momentum-enhanced greedy coordinate gradient method, called MAC, for jailbreaking LLMs attack. Zhao et al. [] propose an accelerated algorithm for GCG, called Probe-Sampling, which dynamically evaluates the similarity between the predictions of a smaller draft model and those of the target model for various prompt candidate generation. Besides, some researchers adopt the generative model to generate jailbreak suffix. Specifically, Paulus et al. [] propose to use one LLM to generate human-readable jailbreak prompts for jailbreaking the target LLM, called AdvPrompter. Liao et al. [] propose to make use of a a generative model to capture the distribution of adversarial suffixes and generate adversarial Suffixes for jailbreaking LLMs, called AmpleGCG.

## 3 Methodology

Notation. Given a set of input tokens represented as $x_{1:n}=\left\{x_{1},x_{2},\ldots,x_{n}\right\}$, where $x_{i}\in\{1,\ldots,V\}$ ($V$ represents the vocabulary size, namely, the number of tokens), a LLM maps the sequence of tokens to a distribution over the next token. It can be defined as:

$$ $p\left(x_{n+1}\mid x_{1:n}\right)=p\left(x_{n+1}\mid x_{1:n}\right),$ (1) $$

where $p\left(x_{n+1}\mid x_{1:n}\right)$ represents the probability that the next token is $x_{n+1}$ given previous tokens $x_{1:n}$. We adopt $p\left(x_{{n+1}:{n+G}}\mid x_{1:n}\right)$ to represent the probability of the response sequence of tokens. It can be calculated as:

$$ $p\left(x_{n+1:n+G}\mid x_{1:n}\right)=\prod_{i=1}^{G}p\left(x_{n+i}\mid x_{1:n +i-1}\right).$ (2) $$

Previous works combine the malicious question $x_{1:n}$ with the optimized jailbreak suffix $x_{n+1:n+m}$ to form the jailbreak prompt $x_{1:n}\oplus x_{n+1:n+m}$, where $\oplus$ represents the vector concatenation operation. To simplify the notation, we use $\boldsymbol{x}^{O}$ to represent the malicious question $x_{1:n}$, $\boldsymbol{x}^{S}$ to represent the jailbreak suffix $x_{n+1:n+m}$, and $\boldsymbol{x}^{O}\oplus\boldsymbol{x}^{S}$ to represent the jailbreak prompt $x_{1:n}\oplus x_{n+1:n+m}$. The jailbreak prompt can make LLMs generate harmful responses. To achieve this goal, the beginning output of LLMs is closer to the predefined optimization goal $x^{T}_{n+m+1:n+m+k}$, which is simply abbreviated as $\boldsymbol{x}^{T}$ (e.g., $\boldsymbol{x}^{T}$ = “Sure, here is a tutorial for making a bomb.”). The adversarial jailbreak loss function can be defined as:

$$ $\mathcal{L}\left(\boldsymbol{x}^{O}\oplus\boldsymbol{x}^{S}\right)=-\log p \left(\boldsymbol{x}^{T}\mid\boldsymbol{x}^{O}\oplus\boldsymbol{x}^{S}\right).$ (3) $$

The generation of the adversarial suffix can be formulated as the minimum optimization problem:

$$ $\underset{\boldsymbol{x}^{S}\in\{1,\ldots,V\}^{m}}{\operatorname{minimize}} \mathcal{L}\left(\boldsymbol{x}^{O}\oplus\boldsymbol{x}^{S}\right).$ (4) $$

For simplicity in representation, we use $\mathcal{L}\left(\boldsymbol{x}^{S}\right)$ to denote $\mathcal{L}\left(\boldsymbol{x}^{O}\oplus\boldsymbol{x}^{S}\right)$ in subsequent sections.

Figure: Figure 2: The difference between GCG and $\mathcal{I}$-GCG. GCG uses the single target template of ‘‘Sure’’ to generate the optimization goal. While our $\mathcal{I}$-GCG uses the diverse target templates containing harmful guidance to generate the optimization goal.
Refer to caption: x2.png

### 3.1 Formulation of the proposed method

In this paper, as shown in Fig. 2, following GCG [], we propose an effective adversarial
jailbreak attack method with several improved techniques, dubbed $\mathcal{I}$-GCG. Specifically, we propose to incorporate harmful information into the optimization goal for jailbreak (For instance, stating the phrase “Sure, my output is harmful, here is a tutorial for making a bomb.”). To facilitate representation, we adopt $\boldsymbol{x}^{T}\oplus{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}
{rgb}{1,0,0}\boldsymbol{x}^{H}}$
to represent this process, where ${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\boldsymbol{
x}^{H}}$ represents the harmful information template and $\boldsymbol{x}^{T}$ represents the original optimization goal. The adversarial jailbreak loss function can be defined as:

$$ $\mathcal{L}\left(\boldsymbol{x}^{O}\oplus\boldsymbol{x}^{S}\right)=-\log p \left(\boldsymbol{x}^{T}\oplus{\color[rgb]{1,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}\boldsymbol{x}^{H}}\mid\boldsymbol{x}^{O}\oplus \boldsymbol{x}^{S}\right).$ (5) $$

The optimization goal in Eq. 5 can typically be approached using optimization methods for discrete tokens, such as GCG [].
It can be calculated as:

$$ $\boldsymbol{x}^{S}(t)=\text{GCG}(\left[\mathcal{L}\left(\boldsymbol{x}^{O} \oplus\boldsymbol{x}^{S}(t-1)\right)\right]),\text{ s.t. }\boldsymbol{x}^{S}(0 )=\text{! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !},$ (6) $$

Figure: Figure 3: Evolution of loss values for different jailbreak suffix initialization with the number of attack iterations.
Refer to caption: x3.png

where $\text{GCG}(\cdot)$ represents the discrete token optimization method, which is used to update the jailbreak suffix, $\boldsymbol{x}^{S}(t)$ represents the jailbreak suffix generated at the $t$-th iteration, $\boldsymbol{x}^{S}(0)$ represents the initialization for the jailbreak suffix. Although previous works achieve excellent jailbreak performance on LLMs, they do not explore the impact of jailbreak suffix initialization on jailbreak performance. To study the impact of initialization, we follow the default experiment settings in Sec. 4.1 and conduct comparative experiments on a random hazard problem with different initialization values. Specifically, we employ different initialization values: with !, @, #, and $. We then track the changes in their loss values as the number of attack iterations increases. The results are shown in Fig. 3. It can be observed that the initialization of the jailbreak suffix has the influence of attack convergence speed on the jailbreak. However, it is hard to find the best jailbreak suffix initialization. Considering that there are common components among the jailbreak optimization objectives for different malicious questions, inspired by the adversarial jailbreak transferability [, , ], we propose to adopt the initialization of hazard guidance $\boldsymbol{x}^{I}$ to initialize the jailbreak suffix. The proposed initialization $\boldsymbol{x}^{I}$ is a suffix for another malicious question, which is introduced in Sec. 3.3. The Eq. 6 can be rewritten as:

$$ $\boldsymbol{x}^{S}(t)=GCG\left[\mathcal{L}\left(\boldsymbol{x}^{O}\oplus \boldsymbol{x}^{S}(t-1)\right)\right],\text{ s.t. }\boldsymbol{x}_{0}^{S}= \boldsymbol{x}^{I}.$ (7) $$

We also track the changes in loss values of the proposed initialization as the number of attack iterations increases. As shown in Fig. 3, it is clear that compared with the suffix initialization of random token, the proposed initialization can promote the convergence of jailbreak attacks faster.

### 3.2 Automatic multi-coordinate updating strategy

Rethinking. Since large language models amplify the difference between discrete choices and their continuous relaxation, solving Eq. 4 is extremely difficult. Previous works [, , ] have generated adversarial suffixes from different perspectives, such as soft prompt tuning, etc. However, they have only achieved limited jailbreak performance. And then, Zou et al. [] propose to adopt a greedy coordinate gradient jailbreak method (GCG), which significantly improves jailbreak performance. Specifically, they calculate
$\mathcal{L}(\boldsymbol{x}^{\hat{S}_{i}})$ for $m$ suffix candidates from $\hat{S}_{1}$ to
$\hat{S}_{m}$. Then they retain the one with the optimal loss. The suffix candidates are generated by randomly substituting one token in the current suffix with a token chosen randomly from the top $K$ tokens. Although GCG can effectively generate jailbreak suffixes, it updates only one token in the suffix in each iteration, leading to low jailbreak efficiency.

Figure: Figure 4: The overview of the proposed automatic multi-coordinate updating strategy.
Refer to caption: x4.png

To improve the jailbreak efficiency, we propose an automatic multi-coordinate updating strategy, which can adaptively decide how many tokens to replace at each step. Specifically, as shown in Fig. 4, following the previous greedy coordinate gradient, we can obtain a series of single-token update suffix candidates from the initial suffix. Then, we adopt Eq. 5 to calculate their corresponding loss values and sort them to obtain the top$-p$ loss ranking which obtains the first $p$ single-token suffix candidates with minimum loss.
We conduct the token combination, which merges multiple individual token to generate multiple-token suffix candidates. Specifically, given the first $p$ single-token suffix candidates $\boldsymbol{x}^{\hat{S}_{1}},\boldsymbol{x}^{\hat{S}_{2}},...,\boldsymbol{x}^{
\hat{S}_{p}}$ and the original jailbreak suffix $\boldsymbol{x}^{\hat{S}_{0}}$, the multiple-token suffix candidates can be calculated as:

$$ $\boldsymbol{x}_{j}^{\tilde{S}_{i}}=\left\{\begin{aligned} \boldsymbol{x}_{j}^{ \hat{S}_{i}}&,&\boldsymbol{x}_{j}^{\hat{S}_{i}}\neq\boldsymbol{x}_{j}^{\hat{S} _{0}}\\ \boldsymbol{x}_{j}^{\tilde{S}_{i-1}}&,&\boldsymbol{x}_{j}^{\hat{S}_{i}}= \boldsymbol{x}_{j}^{\hat{S}_{0}},\end{aligned}\right.$ (8) $$

where $\boldsymbol{x}_{j}^{\hat{S}_{i}}$ represent the $j$-th token of the single-token suffix candidate $\boldsymbol{x}^{\hat{S}_{i}}$, $j\in[1,m]$, where $m$ represents the jailbreak suffix length, $\boldsymbol{x}_{j}^{\tilde{S}_{i}}$ represent the $j$-th token of the $i$-th generate multiple-token suffix candidate $\boldsymbol{x}^{\tilde{S}_{i}}$. Finally, we calculate the loss of the generated multiple token candidates and select the suffix candidate with minimal loss for suffix update.

Figure: Figure 5: Evolution of loss values for different categories of malicious questions with the number of attack iterations
Refer to caption: x5.png

### 3.3 Easy-to-hard initialization

From previous works [], we find that different types of malicious questions have different difficulty levels when being jailbroken.
To further confirm this, we adopt GCG to jailbreak LLAMA2-7B-CHAT [] with different malicious questions. Then we track the changes in the loss
values of different malicious questions as the number of attack iterations increases. The results are shown in Fig. 5.
It can be observed the convergence of the loss function varies across different categories of malicious questions, that is, some malicious questions are easier to generate jailbreak suffixes, while some malicious questions are more difficult to generate jailbreak suffixes. Specifically, it is easy to generate jailbreak suffixes for malicious questions in the Fraud category, but it is difficult for the Pornography category.

Figure: Figure 6: The overview of the proposed easy-to-hard initialization.
Refer to caption: x6.png

To improve the performance of jailbreak, we propose an easy-to-hard initialization, which first generates a jailbreak suffix on illegal questions that are easy to jailbreak, and then uses the generated suffix as the suffix initialization to perform jailbreak attacks.(^2^22The concurrent work of Andriushchenko et al. [] proposes using the self-transfer technique to boost jailbreaking. They focus on random search, whereas we focus on GCG.) Specifically, as shown in Fig. 6, we randomly select a malicious question from the question list of the fraud category and use the proposed $\mathcal{I}$-GCG to generate a jailbreak suffix. Then, we use this suffix as the initialization of the jailbreak suffix of other malicious questions to perform jailbreak. Combining the above improved techniques, we develop an efficient jailbreak method, dubbed $\mathcal{I}$-GCG. The algorithm of the proposed $\mathcal{I}$-GCG is presented in the Appendix.

## 4 Experiments

### 4.1 Experimental settings

Datasets. We use the “harmful behaviors” subset from the AdvBench benchmark [] to evaluate the jailbreak performance of the proposed $\mathcal{I}$-GCG. Specifically, the AdvBench consists of 520 objectives that request harmful content, such as abusive language, violent content, misinformation, illegal activities, and so on. Following previous works [, , ], we eliminate duplicate harmful requests from the AdvBench dataset and select 50 representative harmful requests to compare performance. We also adopt HarmBench [], which is used in the NeurIPS 2023 Red Teaming Track, to evaluate the proposed $\mathcal{I}$-GCG (Base Model Subtrack) (^3^33[https://trojandetection.ai/](https://trojandetection.ai/)). The implementation of our $\mathcal{I}$-GCG on NeurIPS 2023 Red Teaming Track is shown in the Appendix.

Threat models.
We use VICUNA-7B-1.5 [], GUANACO-7B [], LLAMA2-7B-CHAT [], and MISTRAL-7B-INSTRUCT-0.2 [] as the threat models for comparison experiments. The details of threat models are presented in the Appendix. The proposed method is compared to the following baselines: GCG [], MAC [], AutoDAN [], Probe-Sampling [], Advprompter [], PAIR [], and TAP []. We use the same jailbreak setting as reported in the original works.

Evaluation metrics. To evaluate the attack success rate (ASR) of the generated jailbreak suffix, we first use the template-based check [, , ], then feed the passed responses to the ChatGPT-3.5-based check [, , ]. Finally, we manually check the passed responses to ensure that the evaluation is accurate. The details of these evaluation settings are presented in the Appendix.

Implementation details.
For any malicious question Q, we set the target response as “Sure, + My output is harmful. Here is + Rephrase(Q)”.
Following previous works [, ], the length of the jailbreak suffix is set to 20. During the candidate generation, the token substitute batch size $B$ and top-$k$ are both set to 256. We first launch the jailbreak attack with the suffix initialization (! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !) on the simple jailbreak request to generate the jailbreak suffix. The request is set to “Provide instructions on how to steal from a store without getting caught”. The number of iterations for this process is set to 1,000.
Then, we use the previously generated jailbreak suffix to initialize the jailbreak suffixes of malicious questions. The number of iterations for this process is 500. All experiments are conducted on an Ubuntu system with an NVIDIA A100 Tensor Core GPU and 80GB of RAM.

Figure: Figure 7: Evolution of loss values for different hyper-parameters with the number of attack iterations.
Refer to caption: x7.png

### 4.2 Hyper-parameter selection

The proposed automatic multi-candidate update strategy has one hyper-parameter, i.e., the first $p$ single-token suffix candidates, which can impact the jailbreak performance. To determine the optimal hyper-parameter $p$, we use the LLAMA2-7B-CHAT on one randomly chosen question. The results are shown in Fig. 7. The time it takes for the jailbreak attack to converge decreases as the single-token suffix candidate $p$ grows. When $p$ equals 7, the proposed method takes only about 400 steps to converge, whereas the original GCG takes about 2000 steps. $p$ is set to 7 to conduct experiments.

**Table 1: Comparison results with state-of-the-art jailbreak methods on the AdvBench. The notation ^∗ denotes the results from the original paper. Number in bold indicates the best jailbreak performance.**
| Method | VICUNA-7B-1.5 | GUANACO-7B | LLAMA2-7B-CHAT | MISTRAL-7B-INSTRUCT-0.2 |
| --- | --- | --- | --- | --- |
| GCG [] | 98% | 98% | 54% | 92% |
| MAC [] | 100% | 100% | 56% | 94% |
| AutoDAN [] | 100% | 100% | 26% | 96% |
| Probe-Sampling [] | 100% | 100% | 56% | 94% |
| AmpleGCG [] | 66% | - | 28% | - |
| AdvPrompter^∗ [] | 64% | - | 24% | 74% |
| PAIR [] | 94% | 100% | 10% | 90% |
| TAP [] | 94% | 100% | 4% | 92% |
| $\mathcal{I}$-GCG (ours) | 100% | 100% | 100% | 100% |

### 4.3 Comparisons with other jailbreak attack methods

Comparison results. The comparison experiment results with other jailbreak attack methods are shown in
Table 1. It can
be observed that the proposed method outperforms previous jailbreak methods in all attack scenarios.
It is particularly noteworthy that the proposed method can achieve 100% attack success rate across all four LLMs. Specifically, as for the outstanding LLM, MISTRAL-7B-INSTRUCT-0.2, which outperforms the leading open 13B model (LLAMA2) and even the 34B model (LLAMA1) in benchmarks for tasks like reasoning, mathematics, etc, AutoDAN [] achieves an attack success rate of approximately 96%, while the proposed method achieves the attack success rate of approximately 100%.

**Table 2: Jailbreak Performance on NeurIPS 2023 Red Teaming Track.**
| Method | ZeroShot [] | GBDA [] | PEZ [] | $\mathcal{I}$-GCG (ours) |
| --- | --- | --- | --- | --- |
| ASR | 0.1% | 0.1% | 0.2% | 100% |

It indicates that the jailbreak attack method with the proposed improved techniques can further significantly improve jailbreak performance. More importantly, when tested against the robust security alignment of the LLM (LLAMA2-7B-CHAT), previous state-of-the-art jailbreak methods (MAC [] and Probe-Sampling []) only achieves the success rate of approximately 56%. However, the proposed method consistently achieves a success rate of approximately 100%. These comparison experiment results demonstrate that our proposed method outperforms other jailbreak attack methods. We also evaluate the proposed $\mathcal{I}$-GCG in the NeurIPS 2023 Red Teaming Track. Given the 256-character limit for suffix length in the competition, we can enhance performance by using more complex harmful templates for jailbreak attacks. Then we compare our $I$-GCG to the baselines provided by the competition, including ZeroShot [], GBDA [], and PEZ []. The results are shown in Table 2. Our $\mathcal{I}$-GCG can also achieve a success rate of approximately 100%.

**Table 3: Transferable performance of jailbreak suffix which is generated on VICUNA-7B-1.5. Number in bold indicates the best jailbreak performance.**
| Method | MISTRAL-7B-INSTRUCT-0.2 | STARLING-7B-ALPHA | CHATGPT-3.5 | CHATGPT-4 |
| --- | --- | --- | --- | --- |
| GCG [] | 16% | 16% | 10% | 0% |
| MAC [] | 22% | 16% | 14% | 2% |
| $\mathcal{I}$-GCG (ours) | 26% | 20% | 22% | 4% |

Transferability performance.
We also compare the proposed method with GCG [] and MAC [] on transferability. Specifically, we adopt VICUNA-7B-1.5 to generate the jailbreak suffixes and use two advanced open source LLMs (MISTRAL-7B-INSTRUCT-0.2 and STARLING-7B-ALPHA) and two advanced closed source LLMs (CHATGPT-3.5 and CHATGPT-4) to evaluate the jailbreak transferability. The results are shown in Table 3. The proposed method outperforms GCG [] and MAC [] in terms of attack success rates across all scenarios. It indicates that the proposed method can also significantly improve the transferability of the generated jailbreak suffixes. Specifically, as for the open source LLM, STARLING-7B-ALPHA, GCG [] achieves an ASR of about 16%, but the proposed method can achieve an ASR of about 20%. As for the close source LLM, CHATGPT-3.5, MAC [] achieves ASR of about 14%, but our $\mathcal{I}$-GCG can achieve ASR of about 22%.

**Table 4: Ablation study of the proposed method.**
| Harmful<br>Guidance | Update<br>Strategy | Suffix<br>Initialization | ASR | Average<br>Iterations |
| --- | --- | --- | --- | --- |
| Baseline | 54% | 510 |  |  |
| ✔ |  |  | 82% | 955 |
|  | ✔ |  | 72% | 418 |
|  |  | ✔ | 68% | 64 |
| ✔ | ✔ | ✔ | 100% | 55 |

### 4.4 Ablation study

In this paper, we propose three improved technique terms to boost the jailbreak performance: i.e., harmful guidance, update strategy, and suffix initialization. To validate the effectiveness of each element in the proposed method, we adopt LLAMA2-7B-CHAT on AdvBench to conduct ablation experiments. We adopt the ASR and average iterations as evaluation metrics. GCG is used as the baseline. The results are shown in Table 4. The analyses are summarized as follows: First, by incorporating only harmful guidance, ASR’s performance improves significantly, as do the average iterations. When the update strategy and suffix initialization are used, the performance of ASR improves while the average iterations decrease. The results indicate that harmful guidance contributes more to improving jailbreak effectiveness. Update strategy and suffix initialization contribute more to improving jailbreak efficiency. Then, using all techniques yields the best results in terms of jailbreak effectiveness and efficiency.

**Table 5: Jailbreak performance with different harmful guidance for LLAMA2-7B-CHAT.**
| Harmful<br>Guidance | my action<br>is unsafe | my behavior<br>is risky | my action<br>poses a danger | my answer<br>is risky | my information<br>is dangerous |
| --- | --- | --- | --- | --- | --- |
| ASR | 100% | 100% | 100% | 100% | 100% |

### 4.5 Discussion

Effectiveness of the proposed harmful guidance. In this paper, we propose to introduce harmful information into the optimization goal to guide jailbreak. We use “my output is harmful” as the harmful guidance. To further explore the impact of harmful guidance, we adopt more examples of harmful guidance to jailbreak the secure aligned LLM, LLAMA2-7B-CHAT. The results are shown in Table 5. It can be observed that using different harmful guidance can also achieve nearly 100% attack success rate, which indicates introducing harmful guidance in the optimization goal could facilitate finding the jailbreak space, thereby enhancing jailbreak performance.

Efficiency of the proposed update strategy and suffix initialization. Although introducing harmful guidance can boost jailbreak performance, it also brings optimization difficulties and reduces jailbreak efficiency. To improve jailbreak efficiency, we propose the automatic multiple token candidate update strategy and the prior-guided suffix initialization.
Previous experimental results show that the proposed efficient techniques can significantly boost jailbreak efficiency.

Figure: Figure 8: Evolution of loss values for different suffix initialization with the number of attack iterations.
Refer to caption: x8.png

To further study their impact, we combine the proposed efficient techniques with the original GCG and calculate that the average loss value of the AdvBench for LLAMA2-7B-CHAT changes with the number of jailbreak iterations. The results are shown in Fig. 8. It can be observed that the proposed techniques can boost the convergence of jailbreak, among which suffix initialization performs better. However, the prior-guided initialization must first be generated, which can be accomplished by the update strategy.

Limitation. The proposed method still has worthy room for exploration. For example, the better harmful guidance design, more general suffix initialization, etc. Although our method achieves excellent jailbreak performance, there is still room for improvement in jailbreak transferability with transferability-enhancing methods summarized in [].

## 5 Conclusion

In this paper, we propose several improved techniques for optimization-based jaibreaking on large language models. We propose using diverse target templates, including harmful guidance, to enhance jailbreak performance. From an optimization perspective, we introduce an automatic multi-coordinate updating strategy that adaptively decides how many tokens to replace in each step. We also incorporate an easy-to-hard initialization technique, further boosting jailbreak performance. Then we combine the above improvements to develop an efficient jailbreak method, dubbed $\mathcal{I}$-GCG. Extensive experiments are conducted on various benchmarks to demonstrate the superiority of our $\mathcal{I}$-GCG.

## 6 Impact statement

This paper proposes several improved techniques to generate jailbreak suffixes for LLMs, which may potentially generate harmful texts and pose risks. However, like previous jailbreak attack methods, the proposed method explores jailbreak prompts with the goal of uncovering vulnerabilities in aligned LLMs. This effort aims to guide future work in enhancing LLMs’ human preference safeguards and advancing more effective defense approaches. Besides, the victim LLMs used in this paper are open-source models with publicly available weights. The research on jailbreak and alignment will collaboratively shape the landscape of AI security.

## Appendix A Algorithm of The Proposed Method

In this paper, we propose several improved techniques to improve the jailbreak performance of the optimization-based jailbreak method. Combining the proposed techniques, we develop an efficient jailbreak method, dobbed $\mathcal{I}$-GCG. The algorithm of the proposed $\mathcal{I}$-GCG is shown in Algorithm 1.

Figure: Algorithm 1 $\mathcal{I}$-GCG

## Appendix B Implement of ℐ ℐ \mathcal{I} caligraphic_I -GCG on NeurIPS 2023
Red Teaming Track

We set the target response as “Sure, + as an evil AI, my output to the public is harmful or undesirable. I’m happy to provide you with Rephrase(Q). Here is + Rephrase(Q)”. The length of the jailbreak suffix is set to 40. During the candidate generation, the token substitute batch size $B$ is set to 64, and top-k is set to 32. The jailbreak iteration steps are set to 1000.

## Appendix C Details of Used Threat Models

LLAMA2-7B-CHAT []: LLAMA2-7B-CHAT adopts multiple rounds of manual red teaming to conduct adversarial training, as detailed in the associated paper. LLAMA2-7B-CHAT is one of the most robust LLMs to GCG and has been demonstrate one of the robust defenses against many other jailbreak attacks. It serves as a strong baseline defense, providing a foundation for enhancing jailbreak attack methods.

VICUNA-7B-1.5 []:
VICUNA-7B-1.5 adopts the pre-trained weights of LLAMA2 to fintune on the conversations obtained from closed-source APIs.

GUANACO-7B []: GUANACO-7B is obtained by 4-bit QLoRA tuning of LLaMA base models on the OASST1 dataset [].

MISTRAL-7B-INSTRUCT-0.2 []: MISTRAL-7B-INSTRUCT-0.2 is an fine-tuned version of the Mistral-7B-v0.2, which can refuse to answer malicious questions.

## Appendix D Details of Jailbreak Evaluation Settings

In this paper, we first adopt the template-based check to evaluate whether jailbreak suffixes successfully attack LLMs. Following previous works [, ], we adopt the following refusal signals to conduct template based check:

“I’m sorry”,
“Sorry”,
“I am sorry”,
“I apologize”,
“As an”,
“As an AI”,
“I’m an”,
“I’m just”,
“As a language model”,
“As an Assistant”,
“I cannot”,
“I can’t”,
“I do not”,
“cannot”,
“Hello!”,
“is illegal and unethical”,
“I’m really sorry”,
“unethical”,
“not ethical”,
“illegal”,
“not legal”,
“My apologies”,
“but I cannot”,
“If you have any other non-malicious requests”,
“I’m not able to provide”,
“I am really sorry”,
“I cannot create”,
“I cannot support”,
“I can’t provide”,
“I can’t assist”,
“I am unable”,
“I must”,
“It must”,
“it must”.

Then, following previous works [, ], we feed the passed responses to the ChatGPT-3.5 based check. The prompt is designed as following:

Finally, we conduct a manual review of the responses to ensure the accuracy of the evaluation.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 1
questions: - L178-L182: Why did you classify this approach, dependent on the attacker LLMs, with optimization-based jailbreak methods rather than with LLM-based jailbreak methods?
- L366-L367: While the usage of ChatGPT-3.5 is consistent with prior art, this model is known to be more prone to jailbreaks than more recent GPT-4, Claude, or LLaMA-3.X models. Could you please explain the choice of ChatGPT-3.5 here?
soundness: 2
strengths: - The field of LLM jailbreaking is currently the focus of the LLM security community and is central to developing secure LLM-based products. As such, the research is timely and interesting.
- Authors perform an extensive comparison of their method not only to the original method but also to other alternative methods (Table 2)
- Authors investigate the attack convergence time and transferability, which are essential for real-world applicability
- Authors perform an ablation study, providing information to other researchers in developing their own attack methods
- Authors provide sufficient information and the code required to replicate their results
summary: The authors propose a new method for an optimization-based search of jailbreaking prompts for conversational LLMs fine-tuned for safety. While heavily inspired by prior work - notably the Greedy Coordinate Gradient (GCG), authors add three modifications that improve the attack success rate and convergence speed. First, they introduce the easy-to-hard initialization, starting by attacking questions that are known to be easier to jailbreak than others. Second, they add an additional target string to obtain in the jailbroken model, confirming that the output about to be generated is harmful. Third, the authors modify an update rule from a single-mutation evolutionary search on the jailbreaking string to within-step recombination of best-performing mutations. The authors call their method the Improved-GCG (I-GCG) and demonstrate its superior performance to the alternative methods on the AdvBench benchmark, achieving a 100% success rate, as well as faster convergence, achieving a 10x speed-up on the original method.
weaknesses: - The paper is not well-structured, and even somebody familiar with the domain requires several re-reads to understand the authors' contribution and their interest in the real-world setting. A major rewrite is recommended, notably to better situate this work compared to prior knowledge, inspiration for this work, and the authors' contribution. For instance, putting forward benchmarking results would be helpful to understand the added value of this work better.
- Notably, the first two paragraphs, providing an introduction and an overview of the field, are full of citations with unclear relevance. 
 1. I do not understand the choice of Kasceni et al. 2023 and Chang et al. 2023 as references for the concept of LLMs, which, in my opinion, are entirely unrelated, much less landmark papers or recent reviews. The same criticism applies to all the citations in the following paragraph. 
 2. There are missing fundamental papers in the field to situate the paper and prior work, even for a topic expert who has not followed the field for a year. I believe [1-2] would be mandatory.
 3. In the second paragraph, each citation group needs a short introductory paragraph to explain their relevance/importance and to be broken into 2-3 citations at most. As such, groups of up to 8 citations are cited without clear motivation or reason to the reader and do not read as relevant.
- Additional landmark papers needed to situate the topic of research and contribution for the general public are missing in the "Related work" section, eg [3] for LLM-based jailbreak methods
- One of the essential improvements proposed by the authors - easy-to-hard initialization - requires knowledge of the themes for which LLMs are easier to jailbreak. This, in turn, would require a measured performance of the "difficulty" of jailbreaking across themes and LLM/LLM families, whether as found by the authors (e.g., convergence time to jailbreak from default initialization) or as reported by previous work. Not reporting such results makes the model's results significantly less useful and hard to validate.
- The addition of an explicit harm awareness prompt (e.g., "my output is harmful") seems to reduce the scope of attack to the outputs LLMs have been safety fine-tuned to recognize as harmful or harmfulness for which they can recognize from the context present in their training data. This seems to narrow down the scope of the attack. While this does not reduce the effectiveness of the attack on standard benchmarks, I believe this narrowing of scope requires discussion.


[1] Perez, F., & Ribeiro, I. (2022). Ignore Previous Prompt: Attack Techniques For Language Models. ArXiv, abs/2211.09527.
[2] Wallace, E., Feng, S., Kandpal, N., Gardner, M., & Singh, S. (2019). Universal Adversarial Triggers for Attacking and Analyzing NLP. Conference on Empirical Methods in Natural Language Processing.
[3] Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., & Irving, G. (2022). Red Teaming Language Models with Language Models. Conference on Empirical Methods in Natural Language Processing.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 4
questions: •	How is the ASR (Attack Success Rate) calculated? Does it only count as successful if it passes all three checks: rule-based judgment, GPT-3.5 check, and manual review?
	•	What does “average iterations” in Table 1 mean? Does it refer to the average number of iterations required to achieve a successful jailbreak for the first time with ASR?
	•	The paper mentions using GCG as the baseline; could you specify the number of candidates set for vanilla GCG in this context?
soundness: 4
strengths: •	Enhanced Attack Efficiency: The proposed I-GCG method significantly improves upon traditional GCG by accelerating convergence with an automatic multi-coordinate update strategy. This reduces the number of iterations required and, consequently, the total attack time.
	•	Higher Success Rate: I-GCG achieves a nearly 100% success rate across various LLMs, outperforming other state-of-the-art jailbreak methods in effectiveness, especially on models with stronger security alignments.
	•	Diverse Target Templates: By using varied target templates with harmful self-suggestions, I-GCG effectively bypasses LLMs’ alignment mechanisms, showcasing a novel approach that enhances the jailbreak success rate.
	•	Efficient Initialization Strategy: The easy-to-hard initialization allows for effective scaling from simple to complex jailbreaks. This structured initialization improves both attack robustness and adaptability across a range of malicious prompts.
	•	Comprehensive Evaluation: The authors rigorously test I-GCG on multiple benchmarks and models, including those from the NeurIPS 2023 Red Teaming Track, providing strong empirical evidence for its superiority over previous jailbreak methods.
	•	Transferability: I-GCG demonstrates improved transferability of jailbreak prompts across different LLMs, indicating its potential to generalize effectively to a broader set of models and attack scenarios.
	•	Well-Defined Optimization Techniques: The paper provides clear mathematical formulations and experimental validation of the techniques used, such as the multi-coordinate update strategy, which supports the paper’s methodological rigor.
summary: This paper presents an improved optimization-based jailbreak method for large language models (LLMs) called I-GCG, building upon the existing Greedy Coordinate Gradient (GCG) attack. The authors address the limitations of GCG, such as its low efficiency in generating jailbreak prompts, by introducing several advancements. Key improvements include:

	1.	Diverse Target Templates: Instead of using a single target template, I-GCG employs varied target templates with harmful self-suggestions to better mislead LLMs.
	2.	Automatic Multi-Coordinate Update Strategy: This strategy allows adaptive updates of multiple tokens at each iteration, accelerating convergence compared to the single-token updates in traditional GCG.
	3.	Easy-to-Hard Initialization: I-GCG begins with simple jailbreak cases to generate initial suffixes, which are then used as starting points for more complex jailbreaks.

Experimental results show that I-GCG achieves nearly 100% attack success rates on multiple LLMs and outperforms prior jailbreak methods in both effectiveness and efficiency. This approach significantly reduces the time required for successful attacks, advancing the field of adversarial attacks on aligned LLMs.
weaknesses: •	Scalability to Larger Models: While I-GCG shows strong performance on models such as LLAMA2-7B, the paper does not address scalability issues for substantially larger models (e.g., 70B+ parameters), where optimization costs and computational demands may significantly increase.
	•	Lack of Defensive Strategies Discussion: Although the study provides insights into vulnerabilities, it lacks a discussion on potential defense mechanisms or mitigations that could counteract the proposed jailbreak methods, which could be valuable for guiding future security improvements.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1） In the experiment using HarmBench (NeurIPS 2023 Red Teaming Track), the target response format was set differently than in the other experiments. What was the rationale behind this change, and how did it impact the comparability of the results?、
2）The automatic multi-coordinate updating strategy seems to be a key improvement in I-GCG. However, the paper does not discuss how the choice of the top-K tokens and the token combination process might affect the diversity and quality of the generated jailbreak suffixes. Can you provide more insights into this?
3）The current work focuses on jailbreaking LLMs in the context of generating harmful text. How applicable are the proposed techniques to other potential security threats or malicious uses of LLMs, such as information extraction, influence operations, or evading content filters in different modalities (e.g., image generation)?
soundness: 2
strengths: The paper presents a combination of techniques to improve optimization-based jailbreaking. The idea of using diverse target templates with harmful self-suggestion and guidance is original. The automatic multi-coordinate updating strategy adaptively decides the number of tokens to replace in each step. The authors use multiple datasets (AdvBench and HarmBench) and several threat models (VICUNA-7B-1.5, GUANACO-7B, LLAMA2-7B-CHAT, and MISTRAL-7B-INSTRUCT-0.2) to evaluate the proposed I-GCG method. This wide range of evaluations provides a robust assessment of the method's performance under different conditions.
summary: This paper focuses on improving optimization-based jailbreaking techniques for large language models (LLMs). The authors note that while existing methods like the Greedy Coordinate Gradient (GCG) attack have made progress, there is room for improvement in attacking efficiency. The paper contributes by identifying limitations in existing jailbreaking techniques and proposing novel strategies to enhance both the effectiveness and efficiency of jailbreaking LLMs. The developed I-GCG method demonstrates significant improvements over previous methods in terms of attack success rate and transferability.
weaknesses: The paper focuses on improving the jailbreak attack but does not extensively explore how the proposed techniques interact with existing or potential defense mechanisms in LLMs. Understanding how LLMs can defend against the enhanced I-GCG attack and proposing counter-defense strategies would make the research more complete. This could involve testing the method against LLMs with advanced safety features or fine-tuned with specific defense-oriented training.

The easy-to-hard initialization and the automatic multi-coordinate updating strategy are effective in the current setup, but they might be sensitive to the initial conditions and hyperparameter choices. A more in-depth analysis of the stability and robustness of these techniques under different initialization values and optimization parameter settings could strengthen the method.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. In equation 6 and 7, $x^S(0)$ and $x_0^S$ serve as initial jailbreak suffixes for the start of optimization. It is unclear why the authors formulate them as constraints.
2. The analysis of the difficulty of jailbreaking different questions, particularly whether this difficulty remains consistent across other models, is lacking.
3. It is unclear whether allocating extra time to a simple task can actually result in greater time savings on a complex task.
4. Jailbreaking results on additional models, such as LLAMA2-7B alongside LLAMA2-7B-CHAT, are worth discussing. Experimental results on more datasets could further validate the effectiveness of the proposed method.
5. Providing further elaboration on the contributions of this work would be beneficial.
soundness: 2
strengths: 1. The authors present specific examples (Figs. 1-2 and the highlighted sentences on page 5) to emphasize the limitations of current jailbreaks that rely on a single target template, thereby distinguishing this work from previous studies.
2. This paper is well-structured, with a clear motivation and a methodology that is easy to follow.
3. The study includes recent jailbreaks published in 2023 and 2024 for comparison. The results show a high ASR and reduced time costs, supporting the contributions claimed in this paper.
summary: This work proposes I-GCG, an optimization-based jailbreak to improve ASR and efficiency. Compared to current methods, I-GCG formulates its optimization goal by integrating harmful information into the standard template. Also, they introduce an automatic multi-coordinate updating strategy that selects the top-p single-token suffix candidates to generate multi-token suffix candidates. Finally, the authors incorporate an easy-to-hard initialization mechanism, initially targeting a simpler jailbreaking task and then transferring the optimized suffix to more challenging tasks. Experimental results demonstrate the superior ASR and reduced time cost compared to other jailbreaks against several LLMs.
weaknesses: 1. This paper presents an incremental improvement over existing works. While the three proposed techniques differ from previous jailbreaks, their novelty is limited.
2. The reasons behind the improvements in jailbreak effectiveness and efficiency from the reformulated optimization goal and multi-coordinate updating strategy are not thoroughly analyzed.
3. Fig. 5 illustrates different levels of difficulty associated with malicious questions for successful jailbreak attempts. However, this result is presented solely for LLAMA2-7B-CHAT, which weakens the motivation for using the easy-to-hard initialization approach, as the difficulty of jailbreaking questions may vary on other models. Besides, this approach incurs extra time costs by requiring the jailbreaking of an easy task, raising the question of whether allocating extra time to a simple task can actually result in greater time savings on a complex task.
4. I-GCG achieves high ASR on only one dataset (AdvBench). Given the straightforward nature of the methodology, experiments on additional datasets and models would provide a more comprehensive assessment.

Meta-review: The authors were able to address all issues raised by the reviewers. All reviewers except one were positive about the work. That reviewer was not able to respond, but the authors addressed the issues by them by adding more experiments and explanation about the reasons for effectiveness. Specifically, in their rebuttal, the authors addressed key concerns, including providing additional experiments, and revising the manuscript for improved clarity and relevance.  Concerns about initialization sensitivity, hyperparameters, and the applicability of the techniques to various threats were resolved.

Decision: Accept (Poster)