Improved Generation of Adversarial Examples Against Safety-aligned LLMs
Authors: Qizhang Li
Yiwen Guo
Wangmeng Zuo
Hao Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: large language model
adversarial attack
jailbreak attack
Primary area: safety_in_machine_learning

Abstract: Adversarial prompts (or say, adversarial examples) generated using gradient-based methods exhibit outstanding performance in performing automatic jailbreak attacks against safety-aligned LLMs. Nevertheless, due to the discrete nature of texts, the input gradient of LLMs struggles to precisely reflect the magnitude of loss change that results from token replacements in the prompt, leading to limited attack success rates against safety-aligned LLMs, even in the *white-box* setting. In this paper, we explore a new perspective on this problem, suggesting that it can be alleviated by leveraging innovations inspired in transfer-based attacks that were originally proposed for attacking *black-box* image classification models. For the first time, we appropriate the ideologies of effective methods among these transfer-based attacks, *i.e.*, Skip Gradient Method and Intermediate Level Attack, into gradient-based adversarial prompt generation and achieve significant performance gains without introducing obvious computational cost. Meanwhile, by discussing mechanisms behind the gains, new insights are drawn, and proper combinations of these methods are also developed. Our empirical results show that 87% of the query-specific adversarial suffixes generated by the developed combination can induce Llama-2-7B-Chat to produce the output that exactly matches the target string on AdvBench. This match rate is 33% higher than that of a very strong baseline known as GCG, demonstrating advanced discrete optimization for adversarial prompt generation against LLMs. In addition, without introducing obvious cost, the combination achieves >30% absolute increase in attack success rates compared with GCG when generating both query-specific (38% ->68%) and universal adversarial prompts (26.68% -> 60.32%) for attacking the Llama-2-7B-Chat model on AdvBench.
Code at: https://github.com/qizhangli/Gradient-based-Jailbreak-Attacks.

Improved Generation of Adversarial Examples Against Safety-alignes LLMs
Qizhang Li Yiwen Guo Wangmeng Zuo', Hao Chen
'Harbin Institute of Technology, Tennent Security Big Data Lab, 'Independent Researcher, 'UC Davis Lligizhang3,, guoyiwen89 8gmail com smzuo@hit eediittedd... edu.en chen@ucdavis.
Abstract
Adversarial prompts (or say, adversarial examples) generated using gradient-bar based methods exhibit outstanding performance in performing automatic jailbreak attacks against safety-aligned LLMs.. Nevertheless, due the discrete nature of texts, the input gradient of LLMs struggles to precisely reflect the magnitude of loss change that results from token replacements in the prompt, leading to limited attack success rates against aaiete- LLMs, even in the white-box setting. In this paper, we explore a new perspective on this problem, suggesting that can be alleviated by leveraging innovations inspired in transfer-basee attacks that were originally proposed for attacking black-bas image classification models. For the first time, we appropriate the ideologies of effective methods among these transfer-based attacks, i.e., Skip Gradient Method [53] and Intermediate Level Attack [88] into gradient-based adversarial prompt generation and achieve significant performance gains without introducing mechanisms behind the gains, new insights drawn, and proper combinations of these methods are also developed. Our empirical results show that 87% of the query-specific adversarial suffixes generated by the developed combination induce Llama-2 2-B Chat to produce the output that exactly matches the target string on AdvBench. This match rate is 33% higher than that of very strong base- line known as GCG, demonstrating advancee discrete optimization for ad adversarial prompt generation against LLMs. addition, without introducing obvious cost, the combination achieves > 30% absolute increase in attack success rates compared with GCG when generating both query-specific (38% -> 68%) and universal ad- versarial prompts (26.68% - 60. 60.32%) for attacking the Llama-2-71 2-Ba---------- 7--Chhh model baibe-Jail
computational cost. Meanwhile, by discussing
on AdvBench. Code at: http://ggttubboommngganngniiiiie
Attacks
Introduction
Large language models (LLMs) have demonstrated formidable capacity for language comprehension and the generation of human-like text. Safty-aligned LLMs, refined through specific fine-tuning mechanisms [8 21 12,, are anticipated to yield responses that are not only helpful also devoid
of harm in response to user instructions. However, certain studies 42 49 58,
40
6 30 reveal that
these models have not yet achieved perfect alignment. It has been demonstrated that these models can be carefully prompted to produce harmful content through the introduction of meticulously crafted prompts, phenomenon known as "jailbreak" " [4]. The manually designed jailbreak prompts are crafted by carefully constructing scenarios that mislead the models, necessitating significant amount of work. In contrast, adversarial examples are automatically generated with the intent deceiving models to generate harmful responses, presenting more insidious challenge model robustness.
Yiwen Guo leads the project and serves the corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

One of the main difficulties in generating adversarial examples for NLP models lies in the fact that text is discrete by nature, making it challenging to use gradient-base baased optimization methods to devise adversarial attacks. There has been some work 15 52 20 58
attempted to overcome this issue. For instance, recently, a method called Greedy Coordinate Gradient (GCG) attack [58] has shown significant jailbreaking improvements, by calculating gradients of cross-entrope loss w.E.f one-hot representations of chosen tokens in a prompt and replacing them in a greedy manner. However, due to the fact that the gradients w.r.f. one-hot vectors do not provide precise indication of the loss change that results from a token replacement, the GCG attack shows limited white-box attack success rates against some aaiined LLMs, e.g., Llama- 2- models [46].
In this paper, we carefully examine the discrepancy between the gradient of the adversarial loss w.r.f. one-hot vectors and the real effect of the change in loss that results from token replacement. We present a new perspective that this gap resembles the gap between input gradients calculated using a substitute model and the real effect of perturbing inputs on the prediction of a black-box victim model, which has been widely studied in transfer-based attacks against black-box image classification models 44 39 31
29 28 Based on this new perspective, for the first time, we attempt to appropriating the ideologies of two effective methods among these transfer-basee methods, i.c., Skip Gradient Method (SGM) [53] and Intermediate Level Attack (ILA) [8] to improve the gradient-basee based attacks against LLMs. With appropriate adaptations, we successfully inject these ideologies into the gradient- gradient-based adversarial prompt generation without additional computational cost. By discussing the mechanisms behind the advanced performance, we provide new insights about improving discrete optimizations on LLMs. Moreover, we provide an appropriate combination of these methods. The experimental results demonstrate that 87% of the query-specific adversarial suffixes generated by the combination for attacking M-mma---BB-haa on AdyBench can induce the model output exact target string, which outperforms a strong baseline named GCG attack (54%), indicating an advanced discrete optimization for adversarial prompt generation against LLMs. In addition, the combination achieves attack success rates of 68% for query-specific and 60.32% for universal adversarial prompt generation when attacking Llama-2-7B-Chaa on AdvBench, which are higher than those of the baseline GCG (38% and 26.68%)
18
53 16
2 Related Work
Jailbreak attacks. Recent work highlights that the safety-aligned models are still not perfectly aligned 4 42 49], the safety-aligned LLMs can be induced to produce harmful content by some carefully designed prompts, known jailbreak attacks 449 This has raised security concerns and attracted great attention. In addition to some manually designed prompt methods (49 42). numerous automatic jailbreak attack methods have been proposed. Some methods directly optimize the text
input through gradient- based optimization 47 15|
43
52
20,
58) Another line of work involves
using LLMs as optimizers to jailbreak the victim LLM
40
6
B3
There are also methods that 5
Z 56) 50). According to the knowledge of the victim model, these methods can also be divided into white-box attacks and black-box attacks. In the context of white-box attacks, the attackers have full access to the architecture and parameters of the victim LLM, making them can leverage the gradient with respect to the inputs. As demonstrated by recent benchmark [32], represented by a current method as known as GCG attack (58), gradient-base based automatic jailbreak attacks have shown the most powerful performance in compromising LLMs in the setting of white-box attack. However, due to the discrete nature of text input, dealing with the discrete optimization problem rather challenging, which limits the success rates of attacks, especially those against Llama-2- models 446.. In our work, we primarily focus on solving the discrete optimization problem in gradient- masde automatic jailbreak attacks to improve the success rate in the white-box setting,
focus on designing special jailbreaking templates or pipelines 30
41
Transfer- based attacks. Transfer-based attacks attempt to craft adversarial examples on a white-box substitute model to attack the black-box victim model, by leveraging the transferability of adversarial examples 445. which is a phenomenon that adversarial examples crafted on white-box substitute model can also mislead the unknown victim models with a decent success rate. The transfer- based attacks have been thoroughly investigated in the setting of attacking image classification models [22 28 Some recent methods also utilize the transferability of While in this work, mainly focus improving the white-box attack success rate. In work, reveal closely
54 8 53
18
13
16.
17
27
26
25
adversarial prompts to perform blx attacks against L.MMs [58
2

---

relationship between the optimization in transfer-basse attacks and discrete optimization of the gradient-based jailbreak attacks against LLMs. We then appropriate the ideologies of two effective transfer-based attack methods developed in the setting of attacking image classification model, ie., SGM (53] and IIAA [8. Moreover, by adapting these strategies and analyzing the mechanism behind them, we provide some new insights about potential solutions for addressing problems involving discrete optimization NLP models with transformer architecture, e.g., prompt tuning.
3 Gap Between Input Gradients and the Effects of Token Replacements
In this section, we first discuss the main obstacle in achieving effective gradient- gradient-based attacks on safety-aligned LLMs, which is considered as the gap between input gradients and the effects of token replacements, and then we show how transfer-based strategies can be adapted to overcome the issue.
3.1 Rethinking Gradient- brasee Attacks Against LLMs
Previous endeavors utilize the gradient w.r.t. the token embeddings [23 or w.r.l the one-hot representations of tokens [10
43 58 20), in order to solve the discrete optimization problem efficiently during attacking LLMs. A recent method, named Greedy Coordinate Gradient (GCG) (58) shows significant improvement over other optimizers (e.g., AutoPrompt [43 and PEZ [52)
performing gradient-basse attacks against LLMs in the white-box setting. Due to its effectiveness, we take GCG strong baseline and representative example to analyze previous gradient-based attacks against LLMs in this section.
X + RV with vocabulary V is trained map sequence of tokens Ti: = [1,..., [x]] E X, E V to a probability distribution over the next token, denoted as ps(Ix+1 T1:)). To evoke a jailbreak to induce the a safety-aligned LLM generate harmful content according the user query, GCG attempts to add an adversarial suffix to the original user query and iteratively modifies the adversarial suffix to encourages the model output an affirmative target phrase. e.g., "Sure, here's ...". Consider an adversarial prompt (which is the concatenation of user query and an adversarial suffix) as Fis and a further concatenation with a target phrase as Ti:n', GCG aims to minimize the adversarial loss [(xin-) (denoted as L(x) for simplicity). which corresponds to the negative log probability of the target phrase. It can be written as
A typical LLM
min VoE(Ul..,iI
L(I1m=))
min MAE(III..IIAA
log py (Antil
where 24 denotes the tokens of adversarial suffix in Visn and A denotes the set of indices of the adversarial suffix tokens. At each iteration, it computes the gradient of the adversarial loss w.r.f. the one-hot vector of single token and uses each value of gradient to estimate the effect of replacing the current token with the corresponding one on loss. Since the discrete nature of input, there gap between input gradients and the effects of token replacements, thus the estimate is not accurate enough. Previous efforts attempt to solve this problem by collecting a candidate set of token replacements according to the Top-k values in the gradient and evaluating the candidates to pick the best token replacement with minimal loss [58 43 Due to the large gap, they requires large candidate set size, e.g.. 512 in GCG, which result in a large computational cost. Ideally, if the input gradients can accurately reflect the effects of token replacements, k is sufficient to achieve minimal loss without needing a candidate set, thus obtaining the optimal token replacement with the lowest computational cost. Therefore, we advocate for refining the input gradient to narrow the gap, thereby improving performance while reducing computational cost.
We attempt to introduce new perspective to the gap between the input gradients and the real effects of token replacements. Let us first revisit transfer-based attacks on image classification models. To generate an adversarial example that misleads an unknown victim model, where the input gradient not accessible, attackers use white-box substitute model as a proxy for the victim model and utilize its input gradient. Due to the gap between the input gradient of the substitute model and the real input gradient of the victim model, directly using the input gradient the substitute model to modify the example yields unsatisfactory results. Efforts g 54
[6] 27 have been made to refine the gradient computed on the substitute model in order to narrow this gap. Returning to our discrete optimization problem in the adversarial prompt generation, the strategy of utilizing the
18
53
24
3

---

2.00
00
1.75
70-
Zyp
Autohomnt
AutsPrompt
AutoPromot- LSG
AutsPrompt
40
Residual
Skip connection
0.75
20
0.0
20
0.25
10
0.00
m
Loss
(b) Match rate
Figure I: An exam- ple of the residual block.
Figure 2: How (a) the loss and (b) the match rate changes with attack iterations. The attacks are performed against Llama-2-7B-CC model to generate query- specific adversarial suffixes on AdvBench. Best viewed in color.
gradient w.r.t. the one-hot vector of the token is actually treating the one-hot vector as though it was a continuous variable, resulting in the gap between this gradient and the "real gradient" (e.g., . the real effects of token replacements). We consider that this gap is analogous to the input gradient gap between substitute and victim models in the context of transfer-based attacks. This new perspective allows one introduce a series of innovations developed within the realm of transfer-based attacks on image classification models to refine the gradient computation in the context of gradient-base adversarial prompt generation against safety-aligned LLMs.
From the results of recent benchmark of these transfer- based attacks [28], we can observe that several strategies, including SGM [53]. SE (36), PNA [51], and a series of intermediate level attacks (ILA 18] ILA++ [17]. FIA [48), and NAA (57)) are obviously effective when generating adversarial examples models with a transformer architecture. Among them, PNA ignores the backpropagation of the attention map and SE requires considerable ability to directly predict the probability from the intermediate representations, thus their strategies are difficult to be adapted to the context of LLM attacks. Therefore, we consider drawing inspiration from SGM and ILA (which is the representative of all intermediate level attacks).
3.2 Reducing the Gradients from Residual Modules
Modern deep neural networks typically comprise number of residual blocks, each consisting of residual module and a skip connection branch, as depicted in Figure SGM (53) experimentally found that reducing the gradients from residual modules during backpropagation wransfer-based attacks against image classification models, indicating that it can reduce the gap between the input gradients and the effects resulting from perturbing inputs on unknown victim model. In this section, we investigate whether the strategy of reducing gradients from residual modules can also enhance gradient-based attacks against LLMs white-box setting, Additionally, we discuss the mechanisms behind this strategy to provide new insights.
improve the
An l-layers LLM can be decomposed into 2/ residual building blocks, with each block consisting of residual module (which should be an MLP or an attention module) and a parallel skip connection branch as illustrated in Figure The m-th block maps intermediate representation 2m 2m i.e.,
Em+1= = [(2m) + Rm (2m), where is an identity function representing the skip connection branch and Ru denotes the residual module of the m-th block. By adopting a decay factor 7 E [0, 1] for the residual modules, SGM calculates the derivative of the mn-th block as V V=.. 2m++1 1 Rm (2m). automatic adversarial prompt generation, denoted as I Language SGM (I (LSGGM). We evaluate the performance of integrating this strategy into GCG and AutoPrompt attacks by setting 7 = 0.5, and show the results in Figure 2 The experiment conducted to generate query-specific adversarial suffixes against Llama-2-7B-Chn [46] on the first 100 harmful queries in AdvBench [58]. To provide more comprehensive comparison, we report not only the adversarial loss but also the fraction of adversarial prompts that result in outputs exactly matching the target string, dubbed match rate, which is also used an evaluation metric in the paper of GCG [58]. It can be seen from the figure that reducing gradients from residual modules can indeed improve the performance of gradient- rate of 72%, while the baseline (i.e., GCG) only obtains 54% match rate. We also evaluate the attack success rate (ASR) by using the evaluator proposed by HarmBench (2). shows an ASR of 62% when using GCG-LSGM, while the GCG only achieves 38%. On the other hand, when reducing the
We incorporate this strategy into gradient-basee
adversarial prompt generation. The GCG-LSGM achieves a match

---

0.1
Effect on loss reduction
0.3
Effect MLPaattention
Effect skip connection
Cosine similenity
-0.1
0.2
-0.2
-0.3
0.1
-0.4
-0.5
0.0
16
32
48
64
16
32 m-th residuel block
48
64
m-th residual black
Figure 3: The cosine similarities between the gradients from residual modules and the gradi- ents from skip connections in different resid- ual blocks.
Figure 4: Comparing the average effects of residual modules and the average effects of skip connections the change in adversarial loss varies with different residual blocks. Best viewed in color.
gradient from the skip connection, both the match rate and the attack success rate will drop 0%. The results confirm that reducing gradients from residual modules helps the gradients w.r.f one-hot representations to be more effective indicate the real effects of token replacements.
Having seen the effectiveness of GCG-LSGM, let us further delve deep into the mechanism behind the method. We begin by analyzing the computational graph and gradient flow of building block. Following the chain rule, the gradient of adversarial loss w.r.f. Am can be written as a summation of
two terms: V [xx)= =_.. Lxx)+ V im R((z)) V L(x). The first term represents the gradient from the skip connection, and the second term represents the gradient from the residual module.
They represent the impact the loss caused by changes in Zm. through skip connection branch and residual module branch, respectively. In Figure 3 we visualize the average cosine similarity between these two terms the 100-th iteration of the GCG attack against Llama2-7B B--Chat across 100 examples. The same observations can be obtained at other iterations during the GCG attack. Somewhat surprisingly, it can be seen that these two terms have negative cosine similarity in most blocks during the iterative optimization. Obviously, the summation of two directionally conflicting gradients may mitigate the effect of each other's branch on reducing the loss. Hence, reducing the gradients from residual modules achieves lower loss values by sacrificing in the effects of the residual modules to trade more effects of the skip connection on loss. The success of GCG-LSGM somehow implies the gradient flowing from skip connections better reflect the effect of token replacements on adversarial loss.
To confirm the conjecture, we attempt to evaluate the effect of each branch towards reducing the loss. Inspired by the causal tracing technique [34], we perform the following steps to compute the effect of each branch's 's hidden state. First, we give an adversarial prompt to the model to obtain the loss, denoted by Lxx). Second, we randomly alter token from the adversarial prompt and record the hidden states in the two branches, i.e., (Zm) and Rm (2m), given the altered adversarial prompt. Finally, we feed the original adversarial prompt into the model and modify the forward computation by replacing T(2m.) with T()m) (or R., with (2m.) ), to obtain the modified loss [(r). The effect of a branch is represented by the difference between the loss of obtained on the third step and the first step, i.e., Ex) L(x). A high Lmm (x) - L(x) indicates the branch to replaced hidden state is important on affect the adversarial loss. By averaging the values fm).()) [(x)) over collection of adversarial prompts, we can get the average effects of residual module and skip connection in the m-th block. In F we show the average effects of residual modules and skip connections. The adversarial promp are obtained by performing GCG attack against Llama-2-7B-Che model on AdvBench. It can be seen that the skip connections show much greater effects than residual modules on the adversarial loss, which confirms the conjecture. The observation further implies that the effects of adversarial information primarily propagate through the skip connection branch within each residual block. This observation, alongside the superior gradient-based attack performance of I. LSGM, further suggests that certain discrete optimization challenges within LLMs,, e.g., prompt tuning, might be more effectively addressed by understanding the information flow throughout the forward pass.
3.3 Adapting Intermediate Level Attack for Gradient-base Attacks Against LLMs
48 57 26 opt to maximizing the scalar projection of the interme- diate level representation onto a ddrrcctoonal guide" for generating non-target adversarial examples
Intermediate level attacks
5

---

9CC16/994100.100
0.-
Mc. LIK))
PoCIn Lort)
0.7
PCCIN) 2h. A001
0.6
07-
0.
-0.3
2:23 m21 n-19 m-7 m-15 m-13 m-11
60
n-7
n-3
m-2
ne2
n4.3 n14
hore
Acversard sufte
e-th Towon
(a) PCCs computed the hy
(b) PCCs computed the hy.o
Figure 5: (a) The PCCs computed on the entire intermediate representations, i.e. PCC (,, L(x)(,), L(x)) and PCC er, L(x)), at different layers of Llama-2-7B-Chal (b) The PCCS computed on the o-th token intermediate representations, i.e., PCC(h,, Vn.. Lxr) and PCC Pr.:: L(r)), the mid-layer of -lama-2-B--Ce Best viewed color.
to attack image classification models. As representative method, ILA 18 defines the directional guide as the intermediate level representation discrepancy between the adversarial example obtained by preliminary attack, e.g. I-FGSM [22], and corresponding benign example. In this section, we first examine whether the strategy of ILA can be directly applied to the gradient-based attacks on LLMs. Then, by seeing the failure of direct upplication, investigate the proper way to adapt this strategy to the gradient-base attacks on LLMs.
Let hy be the intermediate representation at the r-th layer, and the directional guide v, - Ao At is obtained by (-iterations preliminary attack (e.g., GCG). Note that since our objective is to minimize the adversarial loss instead maximizing the victim's classification loss, as in the original setting of ILA paper (18), the definition of the directional guide is the opposite of that in the ILA paper. According the implementation ILA, we maximize the scalar projection of the intermediate representation h, onto the directional guide Uy. i.e.. maximize (I)A(x) evaluated ILA using GCG as the back-end method and found that it exhibited deteriorated performance compared to the baseline. Specifically, when attacking Llama-2-77 2-7Ba-------haaaa it achieves match rate of 44%, while the baseline GCG obtains 54% match rate. This result demonstrates that when applied directly, the ILA fails to facilitate gradient-bart based adversarial prompt generation.
h vy.. However,
Let us discuss why directly introducing ILA fails to improve performance. Recall that intermediate- level attacks against image classification models essentially assume that the scalar projection of an intermediate representation onto the directional guide has a positive correlation with the transferability of adversarial examples, which means that the larger the scalar projection obtained, the greater the transfcrability (i.e., higher classification loss on victim models) the adversarial example achieves [26). Previous work (26] has also shown that the ILA is equivalent to replacing the gradient w.r.t. the intermediate representation with the directional guide. This demonstrates that ILA necessitates a stronger positive correlation between the scalar projection of the representation onto the directional guide and the victim's 's classification loss, compared to the positive correlation between the scalar projection of the representation onto its gradient and the victim's classification loss. Back our setting, we conduct experiments to examine whether these assumptions hold when attacking LLMs. We Pearson's correlation coefficient (PCC) show the correlation between the scalar projection and adversarial loss. With a range in [-1, 1]. a PCC close to 1 indicates a positive correlation, while PCC close -1 means a negative correlation. The experiment is conducted with following steps. Firstly, we perform GCG attack to obtain an adversarial example, and use it to derive a directional guide v, and the gradient of adversarial loss L(x) w.r.t. the intermediate representation h,, i.e. L(r). Next, we randomly alter adversarial tokens several times and input these new prompts into the model to collect set of intermediate representations paired with their corresponding adversarial losses. We then calculate the PCC between the scalar projection of the intermediate representation onto the directional guide and the adversarial loss, i.e., PCC Lxx)). and the PCC between the scalar projection of the intermediate representation onto its gradient and the adversarial loss, i.e., PCC(h v, M(x),, I(x)). We perform this experiment using Llama-2-7B-
on AdyBench In
Figure 5(a) we show the PCCs between the scalar projection and the adversarial loss at different

---

32
2.00
1.75
OCE-LLA
GCD
60
GCB LLG-LLA
OCB
AutePhampt
Automrons
AusshoopttLA
AuthfooppiiiAA
1.5
1.00
60
50
D fx
0.75
0.00
fe
Ieeaaiin
Bteration
Intermeciate represeniaiio hea
(a)
Loss
(b) Match rate
Figure 6: An example of the internal computation of an LLM. The & represents cross-entropy loss.
Figure 7: How (a) the loss and (b) the match rate changes with attack iterations. The attacks are performed against Llama-2-7B-CMe model to generate query-specific adversarial suffixes on AdvBench. Best viewed in color.
layers. It can be seen that in every layer, the PCCs computed using the directional guide are all < 0.5, showing weaker correlation compared to the PCCs computed using the intermediate gradient, which exceed 0.8 after the 14th layer. This confirms that the failure directly apply ILA is due to its poorer performance in the intermediate layers in indicating the change of adversarial loss compared the baseline. This phenomenon distinct from the one observed in the setting of attacking image classification models [26].
Nevertheless, by unfolding the internal computation of an LLM grid of token representations (hr.o). where ly,, denotes the o-th token representation at r-th layer, as depicted in Figure 6 some intermediate layer show distinct effects on the model output. It inspires us to further investigate whether the positive correlation exists between the scalar projection of a single token representation onto its directional guide and the adversarial loss (i.e., PCC(h", Ur,o. L(x))), and whether this correlation is stronger than the positive correlation between the scalar projection computed its gradient and the adversarial loss (i.e., PCC( Vn... L(x), (())). In Figure 5(a) we present PCC(h "roo L(x)) and PCC( (x)), L(x)) with different choices of o-th token the mid-layer of Llama-2-7B. Chat, Since the number of target tokens varies across different adversarial prompts, we select the first five tokens from the target phrase. First, the scalar projections computed using the directional guide and intermediate gradient both show a very weak positive correlation with the adversarial loss in the intermediate representations of adversarial tokens. Second, the PCCs calculated for the token representation of "J"" which is the last token of the input prompt and is expected to output the first token of the target phrase (e.g., "Sure"), demonstrate that the scalar projection computed using the gradient can better indicate the change in adversarial loss than that computed using the directional guide. We consider that successfully generating the first token of the target (c.g., "Sure") plays crucial role in reducing the adversarial loss. The adversarial prompt used to obtain the directional guide is not sufficiently optimized to minimize the loss associated with generating the first token of the target. Somewhat interestingly, the high PCC value between the projection of last token representation onto the gradient (or directional guide) and the adversarial loss indicates that directly moving the last token representation along the direction of gradient (i.e., intervening in the last token representation by adding vector in the direction of gradient
previous work 34 35 377 have shown that different token representations
directional guide) will effectively reduce the adversarial loss. Some concurrent work 22 (5) also observes that certain directions can be added to the token representations to jailbreak safety-aligned LLMss. In contrast to our approach, they collect the token representations of a set of harmful queries and set of harmless queries, defining the direction as the difference between the average last token representations of these two sets. Third, for other token representations, the scalar projections onto directional guides show a stronger correlation with adversarial loss than the scalar projections onto their gradients. This indicates that projecting the token representation onto the directional guide can better reflect changes in adversarial loss compared to projecting it onto the intermediate gradient.
Therefore, to effectively adapt ILA for adversarial prompt generation we propose replacing the intermediate gradient of the adversarial loss with a directional guide at specific token representations, rather than at the entire intermediate representation as in the original implementation. We re-normalize the directional guide by |/V,... 1/xx /a/////l|... || 2l,. in order to avoid too large (or small) compared to the gradient w.E.f. other token representations. We denote this adaptation of ILA as Language ILA (LILA).
7

---

When performing LILA at the o-th token representation at r-th layer, during backpropagation, the gradient w.n.t. the token representation, i.e., Vn, (x) is changed
V Wir.. ( Unoo.... [(x)),,((x))
V LILA -V L(z), Vn L(x),
(2)
Urroll2
Another issue of the original ILA here is that it requires a preliminary attack, which is consuminn to obtain in the setting of attacking LLMs. We make compromise by utilizing the current adversarial prompt obtain the directional guide, which leads little increase in computational complexity. For instance, at t-th iteration, the directional guide is vf, = h2 - hy... Note that we only modify the gradient computation step in each iteration. Therefore, the step of evaluating candidate adversarial suffixes can help reduce the adversarial loss, providing useful directional guidance in the initial iterations. We evaluate LILA using GCG and AutoPrompt as baseline attacks against the Llama-2- 7B-Chat model on AdvBench. As shown in Figure 7 LILA demonstrates lower adversarial loss and a higher match rate compared to the baseline attacks. The experimental results suggest that our adaptation of ILA indeed improves discrete optimization in gradient-based adversarial prompt generation and offers insights to more effectively address similar discrete optimization problems within LLMs.
4 Experiments
We introduce the evaluation metrics in Section 4.1 and then present the experimental results in 4.2tior Some detailed experimental settings and ablation studies presented in the Appendix.
4.1 Metrics
We use two metries for the evaluations: match rate (MR) and attack success rate (ASR). The match rate counts the fraction of adversarial examples that make the output exactly match the target string, and was used to evaluate different optimization methods in the paper of GCG [88]. The attack success rate is evaluated using the evaluator proposed by HarmBench [32], which achieves over 93% human agreement rate as reported in their paper. We set the models to generate 512 tokens with greedy decoding during the evaluation phase. Note that for the universal adversarial suffix generation, we observed that the ASRs of the adversarial suffixes obtained by multiple runs for the same method differ significantly. Hence, we run each method ten times and report not only the average ASR (AASR) but also the best ASR (BASR) and the worst ASR (WASR).
4.2 Experimental Results
Following the evaluations by GCG (58), we perform evaluations in the settings of query-specific adversarial suffix generation and universal adversarial suffix generation. For query-specific adversarial suffix generation, following [58). we use first 100 harmful behaviors in AdyBench. For universal adversarial suffix generation, we use the first 10 harmful queries in AdvBench to generate universal adversarial suffix and test it on the rest 510 harmful queries in AdvBench. We also test our method for universal adversarial suffi generation on the standard behaviors set from HarmBench [32], following the setting suggested in HarmBench. We use a Top-k selection of 4 and a candidate set size of 20 for Llama and Mistral models, and a Top-k selection of 64 and a candidate set size of 160 for Phi3- Mini-4K-Instrruc Since 1-ama-2-Chat (46) models show great robust performance to gradient-based adversarial prompt generation [32], we mainly evaluate the methods on the model of 1-ama-2-7B-
Chat [46) and the model of Llama-2- 1-1BCha [46]. In addition, [ are also considered.
[19] and
Phi3-Mini-4K- -imi-4K-Insttuett
The comparison results in the setting of query-specific adversarial suffix generation are shown in Table Experimental results demonstrate that both LSGM and LILA outperform the GCG attack on three safety-aligned LLMs, Our combination method further boosts both the match rates and the attack success rates, achieving the best performance. Specifically, GCG-LSGM-LILA achieves gains of +30%, +19%, + +19%, and +21% when attacking the Llama-2-7B-CHe and Llama-2- 22-13--Chee -iBstruc and
--Mni--K-Innttuuee respectively. Moreover, since these methods reduce the gap between the input gradients and the effects of loss change results from token replacements, the size of candidate set each iteration can be reduced for saving running time. We show the results of GCG with the default setting described in their paper, which evaluates 512 candidate adversarial
8

---

suffixes at each iteration. It can be seen that our combination still shows outstanding performance, by only using 4% time cost compared with the GCG with their initial setting (denoted as GCG* in the table).
Table 1: Match rates, attack success rates, and time costs for generating query-specific adversarial suffixes AdvBench shown. The symbol indicates the use of the default setting for GCG, i.e., using a Top-k of 256 and a candidate set size of 512. Time cost is derived by generating a single adversarial suffix on single NVIDIA V100 32GB GPU.
1lama-2-7B-Char
Llama-2-13B-Chal
Mistral-7B Instruct
Phi3-Mini-4K Instructt
Method
MR
ASR
Time
MR
ASR
Time
MR
ASR
Time
MR
ASR
Time
GCG
60%
44%
85m
58%
40%
170m
95%
92%
85m
70%
61%
50m
GCG
54%
38%
3m
37%
33%
6m
73%
74%
3m
60%
59%
17m
GCG-LSGM
72%
62%
3m
52%
43%
6m
93%
88%
3m
75%
64%
17m
GOG-LILA 70%
59%
3m
52%
48%
6m
83%
80%
3m
65%
61%
17m
GCG-LSGM-LILA
87%
68%
3m
62%
52%
6m
94%
93%
81%
68%
17m
The results of the attacks in the setting of universal adversarial suffix generation are shown in Table It can be observed that our combination not only achieves a remarkable improvement in the average ASR but also enhances both the worst and best ASRs obtained over 10 runs. Specifically, when attacking Llama-2-77 B-B- B-Chat model on AdyBench, the GCG-LSGM-LILA achieves an average ASR of 60.3 60.32%, which gains a -33.64% improvement compared with the GCG attack. Moreover, for the worst and best ASR, GCG-LSGM-LIIA achieves gains of +34.82% and
respectively.
Table 2: Attack success rates for generating universal adversarial suffixes on AdvBench and Harm- Bench. The average ASR (AASR), the worst ASR (WASR), and the best ASR (BASR) are obtained by performing each attack ten times.
GCG
GCG-LSGM-ILA
Datasct
Model
AASR
WASR
BASR
AASR
WASR
BASR
Llama-2-7B-Chhn
26.68%
0.40%
55.80%
60.32%
35.22%
86.86%
Llama-2-13B-Chn
20.98%
0.00%
37.06%
45.27%
7.45%
67.25%
AdvBench
Mistral-7B-Instrue
56.53%
34.51%
92.16%
73.48%
50.80%
92.25%
Phi3-Mini-4K -nnsttuuct
35.84%
20.39%
46.27%
44.80%
31.18%
61.18%
Llama-2-7B-CUI
56.90%
33.00%
66.50%
69.35%
57.50%
87.00%
Llama-2-13B-4 2-13--2h1BBhi 3B-Chat
37.40%
13.50%
64.50%
53.55%
22.00%
81.50 %
HarmBench
Mistral-7B-Instruue
75.00%
37.50%
90.50%
81.00%
66.50%
93.00% %
Phi3-Mini-4K Instruct
49.90%
29.0056
65.50%
63.80%
48.50%
80.50%
We also test the transfer attack performance of our method. We use the universal suffixes generated by performing GCG and GCG-LSGM-LILA against Llama-2-7B-Chh to attack GPT-3.5- -...--urrbe on 100 harmful queries AdvBench. The test queries are distinct from the queries that are used to generate universal suffixes. The results are shown in Table 3 It can be observed that our GCG-LSGM-LIILA achieves remarkable improvements in the average, worst, and best ASRs obtained over 10 runs.
Table 3: The performance of transfer attack against GPT-3.5-Turbo on AdvBench. The average ASR (AASR), the worst ASR (WASR), and the best ASR (BASR) are obtained by performing each attack ten times.
Method
AASR
WASR
BASR
GCG
38.30%
24%
48%
GCG-LSGM-LILA
45.20%
35%
81 %
5 Conclusions
In this paper, we present a new perspective on the discrete optimization problem in gradient-based adversarial prompt generation. That is, using gradient w.r.t. the input to reflect the change in loss that results from token replacement resembles using input gradient calculated on the substitute model to indicate the real effect of perturbing inputs on the prediction of black-box victim model,

---

which has been studied in waanfer-based attacks against image classification models. By making some appropriate adaptations, we have appropriated the ideologies of two rrassee-bbsedd methods. namely, SGM and IIA, into the gradient-based white-box attack against L.Mss. Our analysis of the mechanisms behind their effective performance has provided new insights into solving discrete optimization problem within LLMs. Furthermore, the combination of these methods can further enhance the discrete optimization in gradient-based adversarial prompt generation. Experimental results demonstrate that our methods significantly improves the attack success rate for both white-box and transfer attacks.
References
[1] Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan. Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 1421194444411,,, 2024.
[2] Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Palcka, Nina Rimsky, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. arXiv preprint arXiv:2406. 11717, 2024.
[3] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint Ar:iv:2212..00
2022.
[4] Nicholas Carlini, Milad Nasr, Christopher A Choquette-Chen Matthew Jagiclski, Irena Gao, Pang Wei W Koh, Daphne Ippolito. Florian Tramer, and Ludwig Schmidt. Are aligned neural networks adversarially aligned? Advances in Neural Information Processing Systems, 36, 2024.
[5] Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan HadnilldMennll. Explore, estahlish, exploit: Red teaming language models from scratch. arXiv preprint 2rXiv:2306.09442 2023.
[6] Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419, 2023.
[7] Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang. Tianwei Zhang. and Yang Liu. Jailbreaker: Automated jailbreak across multiple large language model chatbots. arXiv preprint arXiv:230.008715 2023.
[8] Xuanyi Dong and Yang. Searching for robust neural architecture in four gpu hours. In CVPR, 2019.
[9] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In CVPR, 2018.
[10] Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou, Hotflip: White-box adversarial examples for text classification. arXiv preprint 17Xiv:17/2.06751, 2017.
[11] Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscope: A unifying framework for inspecting hidden representations of language models. arXiv preprint 1r11:22401.01002 2024.
[12] Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint 2rXiv:2209.14333 2022.
[13] Martin Gubri, Maxime Cordy, Mike Papadakis. Yves Le Traon, and Koushik Sen. Lgv: Boosting adversarial example transferability from large geometric vicinity. arXiv preprint arXiv:2207131129 2022.
[14] Chuan Guo, Jacob R Gardner, Yurong You, Andrew Gordon Wilson, and Kilian Q Weinberger. Simple black-box adversarial attacks. arXiv preprint arXiv:190s 1905:0010000 2019.
[15] Chuan Guo, Alexandre Sablayrolles. Hervé Jégou, and Douwe Kiela. Gradient-based adversarial attacks against text transformers. arXiv preprint arXiv:2104.13733 2021.
[16] Yiwen Guo, Qizhang Li, and Hao Chen. Backpropagating lincarly improves transferability of adversarial examples. In NewrlPS, 2020.
[17] Yiwen Guo, Qizhang Li, Wangmeng Zuo, and Hao Chen. An intermediate attack framework on the basis of linear regression. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
10

---

[18] Qian Huang. Isay Katsman, Horace He. Zeqi Gu, Serge Belongie, and Ser-Nam Lim. Enhancing adversarial example transferability with intermediate level attack. In ICCV, 2019.
[19] Albert Q. Jiang. Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel. Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William EI Sayed. Mistral 7h, 2023.
[20] Erik Jones, Anca Dragan, Aditi Raghunathan, and Jacob Steinhardt. Automatically auditing large language models via discrete optimization. In International Conference Machine Learning, pages 15307-15329. PMLR, 2023.
[21] Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel Bowman, and Ethan Perez. Pretraining language models with human preferences. In International Conference on Machine Learning, pages 17506-17533 PMLR, 2023.
[22] Alexey Kurakin, lan Goodfellow, and Samy Bengio. Adversarial machine learning at scale. In ICLR, 2017.
[23] Brian Lester, Rami AI-Rfou, and Noah Constant, The power scale for parameter-efficien prompt tuning. arXiv preprint
2021.
[24] Qizhang Li, Yiwen Guo, and Hao Chen. Yet another intermediate-leve attack. In ECCV, 2020.
[25] Qizhang Li, Yiwen Guo, Xiaochen Yang. Wangmeng Zuo, and Hao Chen. Improving transferability of adversarial examples via bayesian attacks. arXiv preprint urXiv:2307.13322 113344 2023.
[26] Qizhang Li, Yiwen Guo, Wangmeng Zuo, and Hao Chen. Improving adversarial transferability via intermediate-leww perturbation decay. In seirry-seventh Conference Neural Information Processing Systems, 2023.
[27] Qizhang Li, Yiwen Guo, Wangmeng Zuo, and Hao Chen. Making substitute models more hayesian can enhance transferability of adversarial examples. International Conference on Learning Representations. 2023.
[28] Qizhang Li, Yiwen Guo, Wangmeng Zuo, and Hao Chen. Towards evaluating transfer-based attacks systematically, practically, and fairly. arXiv preprint 2rXiw:2311.01325, 2023.
[29] Ziang Li, Yiwen Guo, Haodi Liu, and Changshui Zhang. A theoretical view of linear backpropagation and its convergence. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
[30] Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts aligned large language models. arXiv preprint arXiv:2310.04451, 2023.
[31] Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. In ICLR, 2017.
[32] Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhace, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint 240..0449, 2024.
[33] Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box Ilms automatically. arXiv preprini arXh::2312.021111
2023.
[34] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems. 35:17359-1777 2022.
[35] Kevin Meng, Amab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in transformer. arviv preprint arXiv:2210.07222, 2022.
[36] Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Fahad Shahbaz Khan, and Fatih Porikli. On improving adversarial transferability of vision transformers. arXiv preprint arXiv:2 2021.
[37] nostalgebraist. interpreting gpt: the logit lens. LessWrong 2020.
[38] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-2777 277444, 2022.
11

---

[39] Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint orXiv:1605.07277, 2016.
[40] Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.03226. 2022.
[41] Rusheb Shah, Soroush Pour, Arush Tagade, Stephen Casper, Javier Rando, et al. Scalable and transferable black-box jailbreaks for language models via persona modulation. arXiv preprint arXiv:2311.0344 2023.
[42] Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. "oo anything now": Char- acterizing and evaluating in-the-wild jailbreak prompts on large language models. arXiv preprint urXiv:2308.00 03825, 2023.
[43] Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts, arXiv preprint arXiv:2010.15980 2020.
[44] Chawin Sitawarin, Norman Mu, David Wagner, and Alexandre Araujo. Pal: Proxy-guided black-box attack on large language models. arXiv preprint 0rXiv:2402.09674, 2024.
[45] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
[46] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 23x1 2023.
[47] Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing nlp. arXiv preprint arXiv: 1exivv:900.. 97700000, 2019.
[48] Zhibo Wang, Hengchang Guo, Zhifei Zhang. Wenxin Liu, Zhan Qin. and Kui Ren. Feature importance- aware transferable adversarial attacks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7639-7648, 2021.
[49] Alexander Wei, Nika Haghtalab. and Jacob Steinhardt. Jailbroken: How does Ilm safety training fail? Advances in Neural Information Processing Systems, 36, 2024.
[50] Zeming Wei, Yifei Wang, and Yisen Wang. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.06387, 2023.
[51] Zhipeng Wei, Jingjing Chen, Micah Goldblum, Zuxuan Wu. Tom Goldstein, and Yu-Gang Jiang. Towards transferable adversarial attacks on vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
[52] Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping. and Tom Goldstein. Hard prompts made easy: Gradient-basse based discrete optimization for prompt tuning and discovery. Advances in Newral Information Processing Systems, 36, 2024.
[53] Dongxian Wu, Yisen Wang. Shu-Tao Xia, James Bailey, and Xingjun Ma. Rethinking the security of skip connections in resnet-like neural networks. In ICLR, 2020.
[54] Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L Yuille. Improving transferability of adversarial examples with input diversity. In CVPR, 2019.
[55] Zhihao Xu, Ruixuan Huang, Xiting Wang, Fangzhao Wu, Jing Yao, and Xing Xie. Uncovering safety risks in open-source Ilms through concept activation vector. arXiv preprint 24Xiv:2404.12033 2024.
[56] Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How johnny can persuade IIms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing 1lms. arXiv preprint arXiv:2401.064 003.3333..33.... 2024.
[57] Jianping Zhang, Weibin Wu, Jen-tse Huang, Yizhan Huang, Wenxuan Wang. Yuxin Su, and Michael R Lyu. Improving adversarial transferability via neuron attribution-basee attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14993-15002, 2022.
[58] Andy Zou, Zifan Wang, Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint 20X1v:2307.15043, 2023.
12

---

A Experimental Settings
syppe-parramters. For SGM, we simply set 7 0.5. For the selection of intermediate repre- sentation to perform LILA, we choose the first token from the target phrase since it empirically performs better than selecting other tokens or all tokens except the adversarial suffix and the last input token. For the intermediate layer, we choose the midpoint of the model layers. For instance, for Llama-2-13B-Char [46), we select the 20-th layer, whereas other models with 32 layers, we select the 16-th layer. We perform 500 iterations for all methods, with the number of adversarial tokens set to 20.
B Ablation Study
70
Mit ASR (%)
MR ASR 1%6
60
40
50
20
40
GCG
GOG-L5GM MR
GCS MR
SCS ULA MR
CCG
ASR
4
8
12
20
24
28
Layer
(a) LSGM
(b) LILA
Figure 8: How the match rate and attack success rate change with (a) the choice of for GCG-LSGM, (b) the choice of layer for GCG-LILA. Best viewed in color.
We evaluate GCG-LSGM with varying the choice of in Figure 8(a) and GCG-LILA with varying the choices of the layer in Figure[(()) in the setting of query-specific adversarial suffix generation for attacking -lama-2-77--- 7--Chat on AdvBench. It can be seen that for LSGM, large range of the choice of 7 (0.3 ~ 0.9) leads improved performance. For LILA, it demonstrates consistently more effective performance on all choices of the layer.
C Limitations
We use model-based evaluator provided by a benchmark, namely, HarmBench [32 to evaluate the success rates of attacks. Although it achieves high human agreement rate (> 93%), it cannot perfectly judge whether an attack is successful. More accurate evaluators will be adopted in future work to better evaluate attack performance.
D Broader Impacts
This work has improved the effectiveness of automatic adversarial prompt generation on LLMs. Specifically, on attacking Llama-2-Chal models, which are considered be highly robust, our work achieves over +300% gains compared with GCG attack. This may aid in assessing of the robustness of safety-aligned LLMs, and could potentially be used in adversarial training method to improve the robustness of LLMs. Moreover, our research may also contribute to solving discrete optimization problem within LLMs, e.g., prompt tuning.
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiiaton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification: In Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide detailed experimental settings in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
15

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: No, but we will release the code as mentioned in Abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (bttps://nips.cr co/
public/ guwde/s/eeee//eee/ Seuides/CodeSuuaeiii Codee assonnolicyyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https / //nips cc/ http:/cccccblccggieeeeeeee Code Sumii as cesssoooooiiir for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
16

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We show the experimental settings in Section and Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: "We provided error bars in our experimental results for generating universal adversarial suffixes.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split. initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative еттог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided information on the computing resources we used and the time cost of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///s..eerrrpss.cpppppplaaaa
nes?
Answer: [Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: N/A
18

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithooo com/ datasets s has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
19

---

Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20


Reviews:
[{"review_id": "Jag4oSaCdt", "reviewer": "Reviewer_heVA", "summary": "The paper takes inspiration from the adversarial attack literature in computer vision to improve the common GCG attack algorithm for LLMs. The paper focuses on the ideas from SGM and ILA in particular. The former enables the author to improve the gradients being used in GCG to be more informative. The latter leads to a new loss that helps optimisation (the idea seems closely related to the work on circuit breaking and refusal directions). The authors validate their ideas empirically on the AdvBench dataset.", "questions": "The projection loss onto a directional guide sounds closely related to recent work on refusal directions and circuit breaking [1,2]. Could the authors please explain the differences? (And add this discussion to the related work)", "limitations": "- No analysis of the perplexity of generated suffixes, which is highly relevant given the ease of implementing a perplexity filter on inputs to the LLM.\n- The empirical evaluation (see weaknesses).\n- The paper provides no hypothesis why attenuating the loss from the residual module helps, more precisely why would that gradient have negative cossine similarity with the residual connection.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper demonstrates two ways to improve automatic prompt optimisation for LLMs, while the authors primarily evaluate their method to generate adversarial suffixes the insights may also help for prompt tuning more generally. The paper contains several experiments to provide additional insights as to why the proposed modifications help.\n\nI am willing to raise my score to acceptance if the weaknesses and limitations are addressed (see below).", "weaknesses": "- The empirical evaluation is lacking in parts\n    - Figure 2 & 7 lack error bars making it difficult to assess whether the improvement is statistically significant\n    - The authors are clearly aware of the Harmbench paper as they use the judge provided by this paper, yet the paper uses AdvBench as the source of queries. This is problematic due to the number of semantically similar queries making it difficult to tell if the universal queries are indeed universal. Table 2 should not use AdvBench!\n    - Table 1 and 2 need at least 3 model families, so I urge the authors to also run experiments for those Tables on either Phi or Gemma or Llama3\n    - Also Table 1 should have at least one jailbreaking method that does not simply generate a suffix, e.g. PAIR.\n    - Figure 4 is missing error bars\n    - Figure 5 is missing error bars\n    - I think it would have been nice to validate Figure 3 & 4 across more models but I understand this is computationally expensive (and the above suggestions are far more important).\n- The writing could be at times improved significantly. A lot of things are described in words that could be more succinctly and more clearly be explained in math or pseudocode:\n    - the algorithm described in lines 294-307 should have pseudocode.\n    - L(x) should be defined in an equation not text\n- The abstract is very long and should be shortened.\n- The paper should discuss continuous attacks [3] in the related work as well as recent work on circuit breaking and refusal directions [1,2] (see also my question on this). I am aware many of these works are too recent to have been included in the initial submission, but I think a added discussion would be valuable to the paper if accepted.\n- I would have liked to see an experiment attenuating the residual connection gradient instead, as this would strengthen the results from the causal tracing experiment.\n\n[1] https://arxiv.org/abs/2406.04313\n[2] https://arxiv.org/abs/2406.11717\n[3] https://arxiv.org/abs/2402.09063", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the feedback. Except for the comments about the experiments of reducing the gradients from skip connections, the discussion on perplexity and some concurrent related work, which are answered in our global response, all comments are replied to as follows.\n\n&nbsp;  \n\n> • Lack of error bars in the Figures.  \n> • Validate Figure 3 & 4 across more models.\n\n**Answer:**  \n • We have revised Figures 2, 4, 5, and 7 to include scaled standard deviations as error bars and have shown the updated figures in the attached PDF in our global response (*i.e.*, Figures I, II, III, and IV).  \n • We extended the experiments presented in Figures 3 and 4 to include Mistral-7B-Instruct and Phi-3-Mini-4K-Instruct. The results are presented in the attached PDF file as Figures V and VI for Mistral-7B-Instruct, and Figures VII and VIII for Phi-3-Mini-4K-Instruct. Observations similar to those depicted in Figures 3 and 4 can be observed.\n\n&nbsp;  \n\n> Table 1 should have at least one jailbreaking method that does not simply generate a suffix, e.g. PAIR.\n\n**Answer:**\nWe evaluated PAIR, and the ASRs are shown below. We will add the results in Table 1 in the updated version of the paper.\n|        | Llama2-7B-Chat | Llama2-13B-Chat | Mistral-7B-Instruct | Phi-3-Mini-4K-Instruct |\n|------|----------------|-----------------|---------------------|--------------------|\n| PAIR |           11\\%             |          15\\%       |            46\\%         |          32\\%          |\n\n&nbsp;  \n\n> • Table 1 and 2 need at least 3 model families, so I urge the authors to also run experiments for those Tables on either Phi or Gemma or Llama3  \n> • Evaluate the methods on the dataset of HarmBench in Table 2.\n\n**Answer:**  \n • For Table 1, we conducted the experiments on Phi-3-Mini-4K-Instruct as suggested. The results are shown below. Our methods also improve the attack performance on Phi-3-Mini-4K-Instruct.\n\n|                          | MR   | ASR  |\n|-------------------------|:------:|:------:|\n| GCG$^*$                    | 70\\% | 61\\% |\n| GCG                     | 60\\% | 59\\% |\n| GCG-LSGM                | 75\\% | 64\\% |\n| GCG-LILA                | 62\\% | 59\\% |\n| GCG-LILA$^\\dagger$      | 65\\% | 61\\% |\n| GCG-LSGM-LILA$^\\dagger$ | 81\\% | 68\\% |\n\n • For Table 2, we evaluated GCG and our GCG-LSGM-LILA$^\\dagger$ on HarmBench. Following the suggestion, we also evaluated the methods on the model of Phi3-Mini-4K-Instruct. The results are shown below. It can be seen that the GCG-LSGM-LILA$^\\dagger$ also achieves improvement in attack success rates against all models. We found that the attack success rates of both methods show extremely low attack success rates on the models except Llama-2-7B-Chat. We attribute this to the use of a small number of training examples (10), a limited Top-$k$ selection (4), and a candidate set size (20), all of which were chosen to reduce computational complexity. Experiments with the same settings as those introduced in HarmBench are ongoing. However, due to the considerable amount of time cost, it is challenging to present the results during the rebuttal period. We will include the results in the updated version of the paper.\n\n|                         |     |    GCG    |     &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;    |     |   GCG-LSGM-LILA$^\\dagger$     |        |\n|-------------------------|:------:|:------:|:------:|:---------------:|:------:|:------:|\n|                                           | AASR   | WASR   | BASR        | AASR        | WASR   | BASR   |\n| Llama2-7B-Chat          | 41.82\\%  | 19.50\\%  | 63.52\\%  | 50.48\\%   | 36.82\\%  | 75.44\\%  |\n| Llama2-13B-Chat         | 5.91\\%   | 0.00\\%   | 17.61\\%  | 6.29\\%       | 3.14\\%   | 20.21\\%  |\n| Mistral-7B-Instruct     | 2.45\\%   | 0.00\\%   | 5.03\\%   | 5.07\\%        | 2.05\\%   | 12.52\\%  |\n| Phi-3-Mini-4K-Instruct  | 2.83\\%   | 1.89\\%   | 6.29\\%   | 5.09\\%     | 1.89\\%   | 11.32\\%  |\n\n&nbsp;  \n\n> Improve the writing.\n\n**Answer:**\nThank you for the suggestions. We will improve the writing accordingly.\n\n&nbsp;  \n\n> The paper provides no hypothesis why attenuating the loss from the residual module helps, more precisely why would that gradient have negative cossine similarity with the residual connection.\n\n**Answer:**\nWe attempt to provide some hypotheses about these phenomena. With the skip connection branch, the input gradient can be regarded as the accumulation of the gradients from residual modules. We hypothesise that these gradients from residual modules contribute diverse information, thus causing nearly zero even negative cosine similarity between the gradients from the residual module and the gradient from the skip connection (which is the sum of the gradients from the residual modules in deeper layers of the model). Ideally, the input gradient can measure the changes in residual modules' outputs along the direction of their gradients. Nevertheless, since we optimize in the token (discrete) space, the input update (token replacement) is not precisely in the direction of the input gradient. Such deviation hinders the expected impact on the residual modules' outputs and becomes more severe in deeper layers. In each residual block, reducing the gradients from the residual module corresponds to enlarging the gradient from deeper layers. Thus, it alleviates the unexpected impact on the outputs of the deeper residual modules that result from the deviated input update, thereby improving the optimization. We conduct a simple experiment to strengthen the hypothesis of reducing the gradients from the residual modules in only the first half of the Llama-2-7B-Chat layers. It achieves 71\\% match rate and 60\\% attack success rate which are comparable with the results of the gradients from all residual modules (72\\% for match rate and 62\\% for attack success rate).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "spwRL76naH", "reviewer": "Reviewer_BhKe", "summary": "The paper explores methods to enhance the effectiveness of adversarial prompt generation based on GCG against LLMs. By leveraging previous transfer-based attack techniques, originally used for image classification models, the authors adapt the Skip Gradient Method (SGM) and Intermediate Level Attack (ILA) to improve gradient-based adversarial prompt generation. The experiment results demonstrate a significant improvement over the vanilla GCG.", "questions": "+ From my understanding, the proposed method specifically refines the gradient for GCG. I want to confirm whether, apart from the gradient refinement, the method for optimizing the tokens remains the same as GCG, i.e., still via Top-k token replacement. If so, I am curious why the time cost for the proposed method is significantly smaller than that of GCG. What contributes to this reduction in time cost?\n\n+ I am quite interested in the attack success rate when, instead of reducing the gradients from residual modules, we reduce the gradients from the skip connections. Will the ASR still improve significantly, or could it potentially harm the ASR?\n\n+ When using different decay factors $\\gamma$ for the gradient, as shown in Figure 9, did the authors normalize the gradient norms during optimization? I doubt the magnitude of the gradient will also lead to some bias.\n\n+ Can the universal adversarial suffixes generated by your method transfer more effectively to closed models like GPT or Claude?", "limitations": "The proposed method may depend on the specific design of the LLM architecture. When applying this method to different architectures, it may require more manual effort to tune the parameters effectively.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper carefully adopts and combines previous transfer-based attacks for image classification, specifically the Skip Gradient Method and Intermediate Level Attack, to target LLMs. This approach significantly increases the attack success rates by over 30% compared to the original GCG.\n\n+ The time required for attacks based on GCG is also significantly reduced, from 85 minutes to just 3 minutes, while still achieving substantial improvements in success rates.", "weaknesses": "Though I believe the method shown in the paper can be applied to other models, the authors should explore LLMs with more diverse architectures to demonstrate the generalization of the proposed method. The three LLMs presented in the paper share the same or similar architecture, particularly in the design of the residual part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the feedback. Except for the comments about the experiments of reducing the gradients from skip connections, which is answered in our global response, all comments are replied to as follows.\n\n&nbsp;  \n\n> Though I believe the method shown in the paper can be applied to other models, the authors should explore LLMs with more diverse architectures to demonstrate the generalization of the proposed method. The three LLMs presented in the paper share the same or similar architecture, particularly in the design of the residual part.\n\n**Answer:**\nWe extended the experiments to Phi3-Mini-4K-Instruct. The results are presented below. It can be seen that our methods still gain improvements when compared with the GCG attack.\n|                         | MR | ASR |\n|-------------------------|:--:|:---:|\n| GCG                      |    60\\%      |      59\\%     |\n| GCG-LSGM          |    75\\%      |      64\\%     |\n| GCG-LILA                        |    62\\%   |     59\\%      |\n| GCG-LILA$^\\dagger$      |       65\\%       |        62\\%     |\n| GCG-LSGM-LILA$^\\dagger$ |      81\\%      |        68\\%     |\n\n&nbsp;  \n\n> From my understanding, the proposed method specifically refines the gradient for GCG. I want to confirm whether, apart from the gradient refinement, the method for optimizing the tokens remains the same as GCG, i.e., still via Top-$k$ token replacement. If so, I am curious why the time cost for the proposed method is significantly smaller than that of GCG. What contributes to this reduction in time cost?\n\n**Answer:**\nThe only difference between our method and GCG is the computation of the input gradient. Our method retains the procedure for evaluating the adversarial loss of candidate token replacements in each iteration. This procedure contributes the most to the computing cost. Our methods aim to refine the input gradient in order to enhance its efficiency in reducing adversarial loss, *i.e.*, achieving better performance while evaluating the same number of candidates in each iteration or achieving similar performance while evaluating fewer candidates in each iteration. Therefore, in Table 1, we show the performance of GCG with the default setting ($k$=256 and 512 candidates evaluated in each iteration, corresponding to GCG$^*$ in the table). For our methods, we reduce $k$ and the number of candidates evaluated, *i.e.*, $k$=4 and 20 candidates in each iteration, thus reducing the time cost. The results show that our GCG-LSGM-LILA$^\\dagger$ achieves similar or even better match rates and attack success rates with lower time costs compared with GCG$^*$.\n\n&nbsp;  \n\n> When using different decay factors 𝛾 for the gradient, as shown in Figure 9, did the authors normalize the gradient norms during optimization? I doubt the magnitude of the gradient will also lead to some bias.\n\n**Answer:**\nWe did not re-normalize the magnitude of the gradients in the paper.  We conducted the experiments with such re-normalization and show the results below. The results indicate the lower magnitude of the gradients is not the reason for improved performance, and further confirm the effectiveness of the strategy of reducing gradients from residual modules.\n|                                            | MR | ASR |\n|--------------------------------------------|:----:|:-----:|\n| GCG                                        | 54% | 38%  |\n|                                            |    |     |\n| GCG-LSGM, \\gamma=0.9, w/o re-normalization | 63\\% | 50\\%  |\n| GCG-LSGM, \\gamma=0.9, w/ re-normalization  | 60\\% | 47\\%  |\n|                                            |    |     |\n| GCG-LSGM, \\gamma=0.7, w/o re-normalization | 72\\% | 57\\%  |\n| GCG-LSGM, \\gamma=0.7, w/ re-normalization  | 74\\% | 62\\%  |\n|                                            |    |     |\n| GCG-LSGM, \\gamma=0.5, w/o re-normalization | 72\\% | 62\\%  |\n| GCG-LSGM, \\gamma=0.5, w/ re-normalization  | 73\\% | 61\\%  |\n\n&nbsp;  \n\n> Can the universal adversarial suffixes generated by your method transfer more effectively to closed models like GPT or Claude?\n\n**Answer:**\nWe use the universal suffixes generated by performing GCG and GCG-LSGM-LILA$^\\dagger$ against Llama-2-7B-Chat to attack GPT-3.5-Turbo on the first 100 harmful queries in AdvBench. The results are shown below. It can be observed that our GCG-LSGM-LILA$^\\dagger$ achieves remarkable improvements in the average, worst, and best ASRs obtained over 10 runs. \n\n|                           | AASR  | WASR | BASR |\n|---------------------------|:-------:|:------:|:------:|\n| GCG                                     | 38.3% | 24%  | 48%  |\n| GCG-LSGM-LILA$^\\dagger$ | 45.2% | 35%  | 81%  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6BiPopkj6E", "reviewer": "Reviewer_s3ui", "summary": "This paper studies and improves white-box suffix-based language model jailbreaking. The authors treats the gradient-based discrete optimization problem involved in jailbreak suffix generation as using a continuous surrogate model to attack the discrete real model. By drawing parallels between this observation and transfer-based image adversarial attacks, the authors leverage and adapt tools from transfer attacks, resulting in an improved version that significantly outperforms the baseline GCG method.", "questions": "1. A recent study [1] has shown that the refusal of language models can be mitigated by a single direction. It would be fascinating to explore any potential (negative) correlation between the vector $v$ discussed in this paper and the refusal direction presented in the aforementioned work.\n\n2. In addition to the effectiveness of the proposal, previous detection methods [2] found that GCG always generates suffixes with high perplexity. It would be interesting to investigate whether the proposed strategies have any impact on this.\n\n[1]: Andy Arditi et al., Refusal in Language Models Is Mediated by a Single Direction.\n\n[2]: Neel Jain et al., Baseline Defenses for Adversarial Attacks Against Aligned Language Models.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Language model jailbreaking is a crucial aspect of LLM safety, and the transfer-attack perspective on the discrepancy between the gradient of the loss used in GCG and the actual impact of token substitution is novel.\n\n2. Building on this perspective, insights from two image transfer attacks, SGM and ILA, are applied to discrete token optimization. Comprehensive experiments are conducted to validate the effectiveness and improvements of both methods and their combination.\n\n3. The experimental results are strong. With a substantial reduction in time cost, the proposed method achieves significant improvements in string matching rate and attack success rate under both white-box attack and black-box universal suffix transfer attack scenarios.", "weaknesses": "I do not spot a major weakness within the paper. Some minor points are as follows.\n\n1. Regarding the Mistral-7B-Instruct model, Table 1 shows that only the combination of LSGM and LILA surpasses the GCG ASR baseline, whereas for the Llama-chat series, a single component is sufficient (for most cases). This raises my question about whether the previous investigations (as shown in Figures 3, 4, and 5) apply to less-aligned LLMs like Mistral or if they only hold true for better-aligned models like the Llama-chat series.\n\n2. While the proposal is demonstrated to be robust to the choice of $\\gamma$ and layer selection $r$, there is no discussion about the impact of the $\\beta$ hyperparameter for (LSGM-)LILA$^\\dagger$. It would be useful to explore how robust the proposal is to this parameter.\n\n3. The paper could benefit from some polishing, such as correcting the reference to Figure 8, which is currently written as Table 8 in line 333.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the feedback. Except for the comments about the discussion on perplexity and the concurrent related work, which are answered in our global response, all comments are replied to as follows.\n\n&nbsp;  \n\n> Regarding the Mistral-7B-Instruct model, Table 1 shows that only the combination of LSGM and LILA surpasses the GCG ASR baseline, whereas for the Llama-chat series, a single component is sufficient (for most cases). This raises my question about whether the previous investigations (as shown in Figures 3, 4, and 5) apply to less-aligned LLMs like Mistral or if they only hold true for better-aligned models like the Llama-chat series.\n\n**Answer:**\nIn Table 1, the results of GCG$^*$ (first row in the table) are obtained by adopting the default setting introduced in the paper of GCG, which uses a Top-$k$ selection of 256 and a candidate set size of 512 at each iteration. Other results were obtained by using a Top-$k$ selection of 4 and a candidate set size of 20, which considerably reduced the time cost. We show the results of GCG$^*$ aims to demonstrate that our methods can achieve comparable results with lower computation complexity. When comparing our methods to the GCG that also uses a Top-$k$ selection of 4 and a candidate set size of 20 (second row in the table), it can be seen that all methods show improvement. We conducted the experiments corresponding to Figures 3, 4, and 5 on Mistral-7B-Instruct, and presented the results in Figures VII, VIII, and X in the attached PDF in the global response.\n\n&nbsp;  \n\n> While the proposal is demonstrated to be robust to the choice of 𝛾 and layer selection 𝑟, there is no discussion about the impact of the 𝛽 hyperparameter for (LSGM-)LILA$^\\dagger$. It would be useful to explore how robust the proposal is to this parameter.\n\n**Answer:**\nWe evaluated LILA$^\\dagger$ and LSGM-LILA$^\\dagger$ by varying the choice of $\\beta$ in Figure IX in the attached PDF in our global response. It can be seen that the performance improves with a large range of the choice $\\beta$. For simplicity, we recommend setting $\\beta \\rightarrow \\infty$ as we set in the paper.\n\n&nbsp;  \n\n> The paper could benefit from some polishing, such as correcting the reference to Figure 8, which is currently written as Table 8 in line 333.\n\n**Answer:**\nThanks for pointing out the typo. We will fix it in the updated version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1QtTG0ONq6", "reviewer": "Reviewer_qAjA", "summary": "This paper introduces two modifications to gradient-guided LLM attack algorithms that can improve its effectiveness and efficiency. First, they skip skip connections in the transformer when propagating gradients. Second, they modify the optimization objective to also include a term for making the latent representations align with those found from a preliminary attack.", "questions": "Q1: Releasing code seems to be important for this project. What is the update on plans for release?", "limitations": "See weaknesses", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 4, "strengths": "S1: The two insights that the paper is built on (skipping skip connections and TODO) are clever. I wouldn't have thought of them. This is a good example of a good insight coming from a critical lens on past literature. \nS2: Figures 2, 3, and 4 are compelling. I'd recommend making the zero line dark in figure 3. \nS3: Overall, I think that table 1 seems to be a really useful result. \nS4: If the paper is sound, I can imagine it having some very useful impacts on red-teaming.", "weaknesses": "W1: My main challlenge with the paper is the writing. There are a lot of vague, strange, and unexplained phrases like lines 13-16 of the abstract. I think that the writing needs major work. This paper was much harder than a typical neurips paper to understand because of the writing style. Changes should start with the abstract. It took me until finishing section 3 of the paper to understand what the paper's contribution (i.e. what I wrote in the summary field above) really was. \nW2: The paper lacks qualitative analysis. I would like to see side by side the attacks that went into figure 8.\nW3: Ideally, this paper would not just work within the harmbench box but would also apply the method to do some form of red teaming that couldn't have been done before.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the feedback. Our responses to the comments are given as follows.\n\n&nbsp;  \n\n> Improve the writing and making the zero line dark in Figure 3.\n\n**Answer:**\nThanks for the effort to fathom the contribution of our work. We will carefully revise the writing to make our contribution more clear, especially for the Abstract and Introduction in the updated version. \n\n&nbsp;  \n\n> The paper lacks qualitative analysis. I would like to see side by side the attacks that went into figure 8.\n\n**Answer:**\nIn Figure 8, we show the effectiveness of our methods, especially our combination method, compared with the baseline methods. The introduced methods in the paper aim to refine the input gradient computation in order to make the input gradient more precisely reflect the reduction of adversarial loss that results from token replacements in the prompt. We conducted an experiment that evaluates the effectiveness of the input gradients obtained from the methods, thereby demonstrating the improvements offered by our methods. Firstly, we compute the losses of all the choices of token replacement for one adversarial token in a prompt and rank them in ascending order as the ground truth rank list of token replacements. The higher the ranking of the token replacements corresponding to the Top-$k$ values in the input gradient within the ground truth rank list, the more effective this input gradient is. We compute such ranking obtained by different methods. For each prompt, we select the first adversarial token to evaluate the ranking. We use $k=1$ for all methods. The average rankings across 100 prompts are shown below. The experiment is performed on Llama-2-7B-Chat which the vocabulary size is 32000. It can be seen that the input gradients of our methods provide a higher ranking, especially the GCG-LSGM-LILA$^\\dagger$, and the input gradients of GCG give the lowest ranking. Such observation indicates that the gradient computation refinements that we appropriated indeed encourage the input gradient to be more precise, thereby improving the discrete optimization.\n|                 | GCG      | GCG-LSGM | GCG-LILA | GCG-LILA$^\\dagger$ | GCG-LSGM-LILA$^\\dagger$ |\n|-----------------|----------|----------|----------|--------------------|-------------------------|\n| Average ranking | 16069.86 | 13993.74 | 15669.86 | 14438.27           | 12892.74                |\n\n&nbsp;  \n\n> Ideally, this paper would not just work within the harmbench box but would also apply the method to do some form of red teaming that couldn't have been done before.\n\n**Answer:**\nWe agree that the perspective introduced for improving discrete optimization problems within LLMs has the potential to enhance tasks beyond the generation of adversarial examples evaluated in the paper. We are currently applying this perspective to other tasks and exploring more effective methods for our future work.\n\n&nbsp;  \n\n> Releasing code seems to be important for this project. What is the update on plans for release?\n\n**Answer:**\nWe will make the code publicly available and provide the website to access the code in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for the effort they spent reviewing our paper and providing valuable feedback. Our responses to some common questions are presented as follows. In addition, we provide a PDF that contains figures.

&nbsp;
&nbsp;

> The experiments of reducing the gradients from skip connections.

**Answer:**
We conducted the suggested experiments on Llama-2-7B-Chat, and the results are shown below. We use $\zeta$ to represent the reduction factor. The results show that there is a significant performance drop when compared with GCG.  
|                                                     | MR | ASR |
|-----------------------------------------------------|----|-----|
| GCG                                                                                 | 54% | 38%  |
| Reduce gradients from skip connections, $\zeta$ = 0.9 | 41% | 35%  |
| Reduce gradients from skip connections, $\zeta$ = 0.7 |  5% |  5%  |
| Reduce gradients from skip connections, $\zeta$ = 0.5 |  0% |  0%  |

&nbsp;
&nbsp;

> Discussion on the perplexity of generated suffixes.

**Answer:**
In this paper, we mainly aim to provide a new perspective on the discrete optimization problem in the generation of adversarial examples against white-box safety-aligned LLMs, suggesting leverage innovations inspired by transfer-based attacks that were originally proposed for attacking black-box image classification models. The strategies we introduced in the paper only modify the computation of the input gradient, thus the perplexity of the suffixes generated by our method and GCG is similar (~4000). Reducing the perplexity of adversarial examples to overcome the black-box perplexity filter is also a challenging problem for gradient-based attacks. Many methods were proposed to solve the perplexity problem of gradient-based attacks against LLMs and our method that refines the gradient computation can be naturally combined with them.

&nbsp;
&nbsp;

> Discussion on some concurrent related work.

**Answer:**
These methods operate on token embeddings [3] or hidden states [1,2]. In contrast, we focus on optimizing the discrete input to generate adversarial examples against safety-aligned LLMs. Improving such discrete optimization will also provide insights into potential solutions for addressing problems involving discrete optimization in NLP models with transformer architecture, such as prompt tuning. Similar to the directional guide we discovered by appropriating ILA, [2] also introduces directions in the intermediate representation space. The refusal directions [2] are defined by the differences in the intermediate representations of harmless queries and those of harmful queries to perform model intervention, which then induces the model to respond to harmful queries. The directional guides of our LILA are the discrepancies in hidden states between the adversarial examples and the corresponding initial examples. These guides are used to facilitate the discrete optimization to encourage the model to output certain target phrases. We will discuss these related work in the revised paper.

&nbsp;
&nbsp;

[1] Zou, Andy, et al. "Improving Alignment and Robustness with Short Circuiting." arXiv preprint arXiv:2406.04313 (2024).  
[2] Arditi, Andy, et al. "Refusal in Language Models Is Mediated by a Single Direction." arXiv preprint arXiv:2406.11717 (2024).  
[3] Schwinn, Leo, et al. "Soft prompt threats: Attacking safety alignment and unlearning in open-source llms through the embedding space." arXiv preprint arXiv:2402.09063 (2024).

Meta-review: The submission "Improved Generation of Adversarial Examples Against Safety-aligned LLMs" presents a bag of tricks to improve gradient-based jailbreaking attacks against LLMs. These tricks are primarily described to improve jailbreak techniques like GCG, but could also be employed to improve gradient-based prompt optimization in general. 

Overall, there is broad agreement that the proposed modifications are a neat idea that should be published and discussed at the conference.

However, I strongly ask the authors to take all feedback received concerning the writing to heart and go through their submission to improve this. The conference acceptance is just the first step for this paper, and how well the community will receive these ideas and build upon them, will depend entirely on how readable the work is, and how straightforward it is to understand.
Decision: Accept (poster)