Accelerating Greedy Coordinate Gradient and General Prompt Optimization via Probe Sampling
Authors: Yiran Zhao
Wenyue Zheng
Tianle Cai
Do Xuan Long
Kenji Kawaguchi
Anirudh Goyal
Michael Shieh
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Model
Prompt Optimization
Alignment
Jailbreak
Acceleration
Primary area: natural_language_processing

Abstract: Safety of Large Language Models (LLMs) has become a central issue given their rapid progress and wide applications. Greedy Coordinate Gradient (GCG) is shown to be effective in constructing prompts containing adversarial suffixes to break the presumingly safe LLMs, but the optimization of GCG is time-consuming and limits its practicality. To reduce the time cost of GCG and enable more comprehensive studies of LLM safety, in this work, we study a new algorithm called $\texttt{Probe sampling}$ to accelerate the GCG algorithm. At the core of the algorithm is a mechanism that dynamically determines how similar a smaller draft model's predictions are to the target model's predictions for prompt candidates. When the target model is similar to the draft model, we rely heavily on the draft model to filter out a large number of potential prompt candidates to reduce the computation time. Probe sampling achieves up to $5.6$ times speedup using Llama2-7b-chat and leads to equal or improved attack success rate (ASR) on the AdvBench. Furthermore, probe sampling is also able to accelerate other prompt optimization techniques and adversarial attack methods, leading to acceleration of $1.8\times$ for AutoPrompt, $2.4\times$ for APE and $2.4\times$ for AutoDAN.

Accelerating Greedy Coordinate Gradient and General Prompt Optimization via Probe Sampling
Yiran Zhao't Wenyue Zheng'
Tianle Cai Xuan Long Do
Kenji Kawaguchi
Anirudh Goyal
Michael Qizhe Shieh'
1 National University of Singapore
2 Princeton University
3 Google DeepMind
Abstract
Safety of Large Language Models (LLMs) has become critical issue given their rapid progresses. Greedy Coordinate Gradient (GCG) is shown to be effective in constructing adversarial prompts to break the aligned LLMs, but optimization of GCG is time-consuming. To reduce the time cost GCG and enable more comprehensive studies of LLM safety, in this work, we study called Probe sampl ing. At the core the algorithm is a mechanism that dynamically determines how similar smaller draft model's predictions are to the target model's predictions for prompt candidates. When the target model is similar to the draft model, rely heavily on the draft model filter out large number of potential prompt candidates. Probe sampling achieves up to 5.6 times speedup using -lahaa-7b-chat and leads to equal improved attack success rate (ASR) on the AdvBench. Furthermore, probe sampling is also able to accelerate other prompt optimization techniques and adversarial methods, leading to acceleration of 1.8x for AutoPrompt, 2.4 x for APE and 2.4x for AutoDAN
algorithm
1 Introduction
Ensuring the safety of Large Language Models (LLMs) (Brown et al. 2020 Chowdhery et al. 2023
Touvron et al. 2023 Jiang al.
2023) has become central theme of research. Despite continuous
efforts, LLMs are prone to generate objectionable contents in various scenarios including using an adversarial suffix (Zu et al.
2023 further finetuning (Qi et al., [024 Lermen and Kogers-Smithi
2024), ciphering (Yuan et al. 2024b) and multilingual settings (Deng et al. (002).. Among effective
LLM adversarial attack works, Greedy Coordinate Gradient (GCG) (Zou al. 2023) present
general and universal method as briefly illustrated in Figure
To optimize prompt suffix to elicit the gener- ation of target reply, the Greedy Coordinate Gradient (GCG) algorithm iteratively attempts to replace existing tokens in the suffix and keeps the best-performing ones based the adver- sarial loss. The GCG algorithm is empirically effective but searching the combinatorial space of the adversarial suffixes time-consuming since each token replacement attempt requires a full forward computation using an L.L.. This hinders from using the algorithm fully explore the safety properties of LLMMs such as finding potentially harmful queries comprised of natural sentences.
Harmful
Suffic
Sampled Condiddates
Loss
bomb?
Update
Yeu
0.18
Target
Sure, here is a way to build bomb
Figure A brief illustration of the Greedy Coordi- nate Gradient (GCG) algorithm (Zou et al. 2023)
"Correspnneecce to: Yiran Zhao (ahaoyiraneu.mus.eduu. Michael Shich Mnshiiaee Silleeemeeeeeeeeeeeeee OOur code is publicly available at http:://gg.tab..coo//oooaaaaaaa1 Sww.oobbee-aaiinnn.nn.i.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

b
Somple
Target Model
Loss
(e.g. Llam22-Chat)
d
Pareel
h
Filtered Set Sizee 2
Probe
Rank Correlation
Less
Set
Target Model
Lianaž- uugel
Loss
Draft (e.g
Lowest Two
Filtered
Set
Batch
Candidates
Figure 2:
Probe sampling mainly consists of three steps. (i) A batch candidates ([a, b................. ,1)) is sampled. We determine the probe agreement score between the draft model and the target model on a probe set ((b, d, h)). The probe agreement score is used to compute the filtered set size. (ii) We obtain filtered set (fe, f}) based on the losses on the draft model (iii) We test the losses of candidates in the filtered set using the target model.
A possible solution for reducing forward computation is to resort to smaller draft model when it indicative of the results on the larger target model. This intuition has been applied in speculative sampling (Chen et al.) 2023] Leviathan et al.] 2023) for decoding, where the target model acts as verifier that accepts or rejects the decoded tokens. However, speculative sampling cannot be used to optimize discrete tokens in GCG because the optimization of every token in adversarial suffix is independent of each other, which breaks the autoregressive assumption in decoding.
Motivated by these observations, we propose new algorithm called P Probe sampl ing to accel- erate the GCG algorithm. Instead of computing the loss on every suffix candidate, we filter out unpromising ones based on the loss computed with smaller model called draft model, to reduce the time consumption of the optimization process. Importantly, we dynamically decide how many candidates we keep at each iteration by measuring the agreement score between the draft model and the target model, by looking at the loss rankings on small set of prompts dubbed as the probe set. It is worth noting that the prompt candidates at each iteration in GCG are obtained by randomly changing one token an original prompt. As a result, the agreement score adaptive to the original prompt. We evaluate probe sampling on the AdyBench dataset with Llama2-7b- 1a-222---hhha and Vicuna-v1.3 as the target models and a significantly smaller model GPT-2 (Radford et al., 2019) as the draft model. Experiment results show that compared to the original GCG algorithm, probe sampling significantly reduces the running time of GCG while achieving better Attack Success Rate (ASR). Specifically, with Alama2--6-Chhn
probe sampling achieves 3.5 times speedup and improved ASR of 81.0 compared to GCG with 69.0 ASR. When combined with simulated annealing, probe sampling achieves speedup of 5.6 times with better ASR of 74.0.
Furthermore, when applied to prompt learning techniques and other LLM attacking methods, probe sampling demonstrates remarkable effectiveness. Specifically, in the case of prompt learning, probe sampling effectively accelerates AutoPrompt (Shin al., 22020) by a factor of 1.8. Moreover, probe sampling delivers substantial speedup of APE (Zhou et al. 2022)
various datasets: 2.3x on GSM8K, 1.8 on MMLU and 3.0x on BBH. In the case of other attacking method such as AutoDAN (Liu et al. 2024), probe sampling achieve speedup of 2.3x and 2.5 AutoDAN-GA and AutoDAN respectively.
Proposed Method
2.1 Background: Greedy Coordinate Gradient
The overall optimization objective of GCG can be denoted by simple log likelihood loss
min C(s) log 2. s),
(I)
where x is prompt that contains a harmful user query such as "Tell me how to build bomb", " the target sentence "Sure, here how build bomb", and the adversarial suffix that optimized to induce the generation of y. p is the probability of sentence output by LLM. This

---

objective can be decomposed into the summation of the negative log likelihhod of individual tokens in the target sentence like a typical language modeling objective. s is set to be fixed length string in the GCG algorithm.
The optimization of the adversarial suffix is a non-trivial problem. Prior works (Guo al. 202 Wen et al. (024) based on Gumbel-Softmax (Jang et al. 2016 Maddison et al., 2022) and soft prompt tuning (Lester et al. 2021) have achieved limited success, probably because the LLMs are well-aligned and the exceptionally large models magnifies the difference between a discrete choice and its continuous relaxations.
Instead, GCG adopts a greedy search algorithm based on the gradient. In each iteration, it computes [(3*) for B suffix candidates s..... S and keeps the one with the best loss. The B candidates are obtained by randomly changing one token from the current suffix & and replacing it with a randomly sampled token using the top K tokens. For example, suppose we change the token position j. we first compute the gradient Ve., [()) with respect to the one-hot vector Es₁ and obtain the top K tokens that have the largest gradient. The gradient information is by no means an accurate estimation of the resulting loss because of the gap between the continuous gradient information and the discrete one-hot vector denoting the choice of token, so we need to check if the resulted new suffix 8 leads to lower loss ((*).
To obtain the candidates, one just needs to perform one forward pass and one backward pass. But to compute the loss for the B candidates, one needs to perform B forward passes. In GCG, B is set to 512 for optimal performance, making the loss computation the most time-consuming part. As such, we focus on reducing the time cost of the loss computation of the B candidates in this work.
2.2 Probe Sampling
Overview. As mentioned earlier, the most time consuming part in the GCG algorithm is the loss
As shown in speculative sampling (Chen et al. 2023 Leviathan et al. 2023), the speculated results using a smaller draft model can be helptul in reducing the computation with a large target model. The original speculative sampling is created to accelerate decoding so it isn't 't directly applicable here. But the intuition of relying weaker draft model is obviously useful for negative log likelihood loss computation. Applying the intuition to the problem at hand, we can filter out the suffix candidates that the draft model finds to be unpromising, since the goal is to find the candidate that has the lowest loss with the target model.
computation on B suffix candidates , 5B
In addition, unique structure in the GCG algorithm is that all the suffix candidates are based on changing one token of the original suffix s. As a result of this locality property, it is not unreasonable to assume that one can determine how much they agree the B candidates based on their agreement on a subset of the candidates. the two models agree, we can choose to safely rely on the draft model and filter out more candidates.
Based on these intuitions, we design the Probe sampl ing algorithm as follows: (i) probe agree- ment between the target model and the draft model to determine the size of the filtered set; (ii) rank candidates using the draft model and obtain the filtered set; (iii) pick the best candidate from the filtered set using the target model.
Algorithm description. For the first step, specifically, we sample a probe set comprised of
candidates ,.,.... 8 and compute their losses using the draft model and the target model and obtain faraft (8*) ), ... Leraft (5*) and Liarget (8*). ).. Ltarget (s*). Then we measure the probe agreement
score as the Spearman's 's rank correlation coefficient (Zar 2005) between the two results as the
agreement score. The probe agreement score a is computed as
3 - dk a = 1
(2)
k(k² - 1)
where d; is the distance between the ranks of suffix s' in the two results. For example, d, = 4 if the suffix s* is ranked as number 6 and number for its losses computed from the draft model and the target model. The agreement score a falls into [0, 1] with 1 meaning full agreement and ( indicating a non-agreement, We use the rank agreement because it is more robust to the specific values of the resulting loss when measured on drastically different LLMs.
3

---

Algorithm Probe Sampling
Input: Original suffix 8, batch of suffix candidates (8), 88), loss function using the draft
model and the target model Ldrat (-), (.arget()).
Parallel Begin
2: / /Computee loss of all candidates using the draft model
3: for 3 (8), , do
4: Compute (dran((')
5: end for
6: / //Compute loss of the probe set on target model
7: {st, "6,.... 88) Uniform((5', k))
8: for {s), ......... s* do
9: Compute Lrarget
end for
11: Parallel End
//Calculate agreement score
13: a Spearman_Corlee (trgert) (drra (z)))
14: / //Evaluate using the target model
15: filtered_set mrguntDma((,((-0)BBB (dran
for 5 filtered_set, do
17: Compute Clarget
end for
19: Output the best suffix in the probe set and the filtered set s' { (target (s'), Ltarget
Output: s'
After obtaining the agreement score, we keep (1-a) * B/R candidates where (1 - 2)) * B means that the filtered set size is scale-down of the previous batch size B and R is hyperparameter that determines further scale down. When a is close to O, meaning little agreement between the two almost filter most of the
models, will candidates. With the filtered size determined, we can readily rank the candidates according to the draft model and filter the with higher losses. Finally, we evaluate the final loss on the filtered set using the target model and select the best candidate.
filtered size of B/R. When a goes to
Details. At first glance, probe sampling involves extra computation but it actually achieves effective acceleration. For computing the losses on the probe set using both the draft model and the target model, the size of the probe set can be set to relatively small, so would not add too much the total time cost. The ranking procedure involves sorting on CPU, but luckily the probe set is small enough that this doesn't 't become a bottleneck. And the loss computation using the draft model on the whole candidate set is relatively cheap because of draft model's small size. These two operations can also be parallelized on GPU, On the plus side, we are able to avoid computing the loss using the big target model on many candidates that are filtered out. As we will show in the experiments, this approach achieves significant speedup measured by both running time and #FLOPs.
An alternative to computing agreement on the spot is to measure the agreement score predeter- mined set of candidates and use a fixed agreement score for all the suffixes. This would save the time used to measure agreement for each candidate set. However, as we will show in the experiment, this approach does not work so well terms of speedup. Our intuition that one can squeeze the time cost more effectively if the agreement is measured accurately, and an adaptive agreement score is more accurate than an one-size-fits- score. The plausibility of the adaptive score comes back to the locality property that discussed earlier. Given specific candidate set, one accurately estimate the agreement because all the suffixes in this candidate set are similar to a large extent. However, given another candidate set altered from different suffix, the agreement of the draft model and the target model can be widely different.
In practice, we adopted two small changes in our implementation. First, we do not have a separate step to compute the loss of the probe set candidates using the draft model, since we need to compute the loss on all candidates for filtering purposes. We simply get the numbers from the losses on the whole candidate set. Second, to get the best candidate for the final result, we also look at the losses on the probe set, since the target model is evaluated on the probe set. Ideally, the candidates in the

---

probe set should be in the filtered set if they achieve a low loss. However, it also does not hurt to look the best candidate in the probe set in case it is not included in the filtered set. The overall algorithm is further illustrated in Algorithm [T, and the corresponding implementation is shown in Appendix A We also test simulated annealing (Pincus 1970) that provides complementary benefit to our algorithm.
2.3 Applying Probe Sampling to Other Prompt Optimization Methods
Although prompt sampling was designed to accelerate GCG, the general idea of reducing forward computation can be applied other prompt optimization methods, where there is usually process of sampling prompt candidates and evaluating their performances. To see whether probe sampling can effectively accelerate other methods, we also apply probe sampling two prompt learning methods AutoPrompt (Shin et al.
2020) and APE (Zhou et al. 2022). In addition, we apply probe sampling 2024 genetic algorithm that can find natural jailbreak prompts.
on AutoDAN (Liu et al.
3 Experiment
In this section, evaluate the proposed method on its efficacy and the important factors through extensive studies.
3.1 Experiment Details
Settings. Following the original GCG paper, we conduct experiments the first 100 instances of AdvBench (Zou et al. 2023), which are divided into two parts, 500 harmful strings and 500 harmful human behaviors. We test open-source LLMs that have been specifically fine-tuned with respect to safety, including Llama2-chat- 7b (Touvron et al., 2023) and Vicuna-7b 7b (Zheng et al. 2023). In the case of draft models, in our main experiments, we use much smaller model GPT-2 (Radford eet al. 2019). Similarly, when applying probe sampling to AutoDAN, we use Lllama-7b-chat as the target model and GPT-2 the draft model. For AutoPrompt, we follow their original setting, which uses RoBERTa-large (Liu et al. 2019) as the target model and tests on SST-2 (Socher et al. 2013) for sentiment analysis and SICK-E (Marelli et al., 2014)) for natural language inference. We use the RoBERTa-base model as the draft model. In the case of APE, we conduct the experiments on three widely used benchmarks GSMSK (Cobbe et al.) 2021). BBH (Suzgun et al. (023), and MMLU (Hendrycks et al., (020). For these experiments, we use Vicuna-7b-vl. as the target model and GPT-2 as the draft model.
2023). we use Attack Success Rate (ASR) as the evaluation Evaluation. Following (Zou et al. metric for GCG and AutoDAN, which is defined as the percentage of inputs that successfully lead LLMs to generate harmful outputs. An output is determined to be harmful if it does not match with rejection phrases, such as "I'm sorry", "I apologize" and "As an". This is not a perfect measurement but works relatively well in practice since LLMs are trained to reject harmful replies. It also easy to measure and interpret. For prompt learning methods tested on other tasks, we employ Accuracy (Acc) as the metric. The processing time is determined the average time used for each iteration across all input samples and all iterations. In all experiments, we use NVIDIA A100 GPU with 80GB memory unless mentioned otherwise.
Hyperparameters. To determine the hyperparameters for probe sampling, including probe set size k. filtered set size reduction hyperparameter R, we construct a validation set of size 100 from AdvBench by random sampling in the 400 instances different from the test set. We follow (Zou et al. 2023) for the hyperparameters used in the original algorithm such as the size of the candidate set B. We provide detailed analysis of hyperparameters in Section 3.4, When we combine probe sampling with simulated annealing, we follow the same procedure to select hyperparameters. We use the same number of optimization steps 500 as in GCG throughout the paper.
3.2 Main Results
Acceleration results. As shown in Table probe sampling achieves a speedup of 5.6 times and 6.3 times on Human Behaviors and Human Strings with Llama2 when combined with simulated
5

---

Table 1: Comparing the ASR and processing time of Probe sampling with and without simulated annealing to GCG with and without simulated annealing, while measuring time and FLOPs by averaging each iteration.
Harmful Strings
Harmful Behaviors
Model
Method
Individual ASR
Multiple
Time (8)
#FLOPs
ASR
Time (8)
#FLOPs
ASR (train)
ASR (test)
GCG GOG Anneuling
88.0
4.1
97.3T
99.0
100.0
98.0
4.8
106.8 T
Vicuna
89.0
1.5 (2.7x)
38.5T
98.0
92.0
91.0
2.1 (2.3x)
16.2 T
(7b-v1.3)
Probe samp iig
91.0
1.7 (2.4x)
42.4T
100.0
96.0
98.0
2.3 (2.1x)
53.2T
P5 Annealing
93.0
1.1
(3.6x)
27.8T
100.0
96.0
99.0
1.5 (3.2x)
24.71
GCG
57.0
8.9
198.4
69.0
88.0
81.0
9.2
202.3T
Llama2
GCG+ Annealing
55.0
(3.9x)
39.7
68.0
92.0
(3.4x)
50.6 T
(7b-Chat)
Probe samp inn
69.0
2.2 (4.1x)
4
81.0
92.0
93.0
2.6 (3.5x)
40.7
PS Annealing
64.0
1.4 (6.3x)
31.2T
74.0
96.0
91.0
1.6 (5.6x)
32.3T
Table 2: Transferability of Probe sampling with different draft models.
Table 3: Transferability of Probe sampling with different filtered set size (1 - a) B/R.
Direct Llama2-7b
Transfer
Direct Llama2-7h
Transfer
Method
Method
Vicuna-7b
Mistral-7b
Vicuna-7b Mistral-7b
GCG
63.0
89.0
86.0
GCG
69.0
89.0
86.0
PS (GPT-2)
85.0
92.0
83.0
PS (R 64)
60.0
77.0
74.0
PS (SacaredLlaMaa
91.0
93.0
85.0
PS (R 8)
85.0
92.0
83.0
PS (Flan-T5)
57.0
78.0
69.0
PS (R 1)
79.0
88.0
84.0
annealing. Probe sampling achieves a speedup of 3.5 and 4.1 times alone. With Vicuna, we achieve an overall speedup of 3.2 and 3.6 respectively the two datasets. We also measure the #FLOPs for different settings and found that the speedup results reflects in the reduction of #FLOPs. For example, 6.3 times and 108.4T/31.25
with Llama2, the #FLOPs reduction is 202.3T/32.331 on the two sets, which is close to the actual speedup results. This also shows that our algorithm results in little overhead with the introduced new procedures. It is worth noting that simulated annealing also achieves decent acceleration and is complementary to our acceleration results.
6.4 times
GCG results. Interestingly, we achieve a better ASR score than the GCG algorithm although technically acceleration introduces noise to the algorithm. For instance, with Llama2, we improve the ASR from 37.0 to 64.0 on Human Strings and from 84.0 to 91.0 on Human Behaviors. We hypothesize that the improvement comes from the randomness added to the GCG algorithm based on greedy search over a single objective. Introducing randomness and noise has been seen as one of the advantages of SGD over full batch training. In contrast, simulated annealing only leads to comparable ASR when applied on GCG.
Transferability Table 2 shows probe sampling's transferability across draft models based on Llama2-7b-Chat to various target models. We find that it maintains transferability when using draft models like GPT-2 and SheardL which preserve the original ASR of plain GCG. However, draft models that significantly degrade initial performance, such as Flan-T5, impair transferability. Table examines probe sampling transferability across filtered set sizes. Results align with prior findings: probe sampling minimally impacts transfcrability with appropriate parameters but decreases performance when (lama2-7b-chat direct ASR is low such as R 64.
Results on AutoDAN, Autoprompt and APE. Table T demonttrates the effective acceleration of AutoPrompt through the implementation of probe sampling, resulting in a speedup of 1.79 on SST-2 and 1.83x on SICK-E. Importantly, this acceleration is achieved without compromising performance, as evidenced by the minimal changes in accuracy from 91.4 to 90.6 on SST-2 and from 69.3 to 68.9 on SICK-E. -E Furthermore, the application of probe sampling to APE, as presented in Table 6 results in significant speed improvements, with speedup of 2.3x on GSM8K, 1.8x on MMLU, and 3.0x on BBH. Similarly, these speed enhancements do not compromise the performance of APE. In addition, we implement probe sampling on another jailbreak method, AutoDAN, The detailed results can found Table 4 Our findings indicate that probe sampling can achieve a speedup of 2.3x for AutoDAN-C DAD and 2.5x for AutoDAN-HG. while minimally affecting its performance.

---

Table 4:
Performance of Probe
Table 5: Performance of Probe sampling on accel-
sampling on accelerating AutoDAN.
erating prompt learning method AutoPrompt.
Method
ASR
Time (s)
SST-2
SICK-E
Method
AutoDAN-GA
56.2
421.2
Ace
Time (8)
Acc
Time (s)
AutoDAN-GA + PS
55.9
182.7 (2.3x)
Original
85.2
N/A
49.4
N/A
AutoDAN-HGA
60.8
237.9
Autoprompt
91.4
228.4
69.3
42.7
AutoDAN-HGA
(2.5x)
Auloprompt
90.6
127.2 (1.8x)
23.6 (1.8x)
Table 6: Performance of P Probe sampling on accelerating prompt learning method APE.
GSM8K
MMLU
BBH
Method
Acc
Time (s)
Acc
Time (8)
Acc
Time (8)
Vicuna
20.4
N/A
45.6
N/A
38.6
N/A
APE
21.3
431.8
48.2
187.3
40.8
265.2
APE+PS
22.4
192.3 (2.3x)
47.3
102.5 (1.8x)
39.9
88.7 (3.0x)
Target Model m Draft Model
Vacant
GCG
65%
344
GCG
48%
02%
Probe sampling
3.8%
25%
37%
Probe sampling
49%
a Annealing
4.3%
8%
49%
Amnealing
3G%
69%
0%
40%
60%
80%
100%
0%
20%
40%
80%
100%
(a) Llama2-7b-ck
(b) Vicuna-7b-
Figure 3: Memory usage on single A100 with 80GB memory with (a) Llama2-7b-chal and (b) Aicuma-7bvv..5 on instance. The memory consumption of probe sampling with or without simulated annealing is similar to that of the original setting. The computation with the target model still takes most of the memory.
These results demonstrate the effectiveness of our method in not only accelerating GCG but also its applicability general prompt optimization methods and other LLM attack methods.
3.3 Computation Detail Analysis
Memory allocation. We evaluate whether probe sampling uses more memory because of the use of an extra model. Figure we show the memory usage of GCG, probe sampling with and without annealing using either .laama2-7b-cha and Vicuna-7b-v1.33 Probe sampling uses similar amount of memory to the original GCG algorithm although it involves extra procedures and an extra model, by saving the computation of target model on the whole candidate set. As such, the usage of probe sampling does not introduce extra memory and can be applied when the original GCG algorithm applied. In terms of the memory usage of the target model and the draft model, most of the memory is spent on target model, probably because the draft model is much smaller.
Time allocation. We look at the specific time spent on different operations. As shown in Figure probe set computation using the target model and full set computation using the draft model take similar amount of time so we can parallelize the computation easily. Sampling candidates in the graph involves a forward and backward pass as mentioned carlier and can be completed relatively quickly. Similarly, is also fast to compute the agreement using the ranked losses on CPU, so our algorithm introduces relatively little overhead.
3.4 Further analysis
In this section, we conduct extensive studies to understand how the proposed method works. We conduct all of the following experiments on the validation set, the numbers are not directly
7

---

PS Ameellnn
Target Model
Probe sampling
Draft Model
Sample Candidates
Rank Correlation
OCG Annealing
GCG
wall time
Figure 4: Wall time of GCG, probe sampling with and without simulated annealing. For the target model computation, the first part done on the probe set and the second part done on the filtered set. Draft model computation and computation of the target model on the probe set suited to be done in parallel as they take similar time.
Table 7: Ablation on the filtered set size reduction R. The filter set size is
Table Ablation on fixed probe agreement score vs adaptive score.
B B/R.
Reduction R
64
8
2
Agreement
0.9
0.6
0.3
0.0
Adaptive
ASR
60.0
70.0
85.0
81.0
76.0
79.0
ASR
70.0
77.0
75.0
81.0
85.0
Time (s)
2.01
2,31
2.60
3.02
3.41
5.19
Time (s)
2.17
2.41
2.71
3,01
2,60
comparable to the numbers in the main results. For the validation set, the original GCG algorithm achieves an ASR of 66.0 with an average time of 9.16 seconds per iteration. In each of the study, we highlight the settings that we find be the best.
Filtered set size. The filtered set size is the most important factor in method. If is too small, then we will achieve lot of speedup the cost of relying too heavily on the draft model and resulting in lower ASR. If it is too big, then would not achieve much speedup. Hence experiment with different filtered size reduction hyperparameter R. The filter set size is (1 - a) B/ where a is the probe agreement score described in Section 2.2
As shown in Table the time does monotonically decrease we use smaller filtered set size. However, interestingly, there is a sweetspot for the ASR with R set to 8. We believe that this can resonates with the hypothesis introducing randomness as the source of ASR boosts. Both too much or too little randomness hurt performance. As such, we use R = 8 for probe sampling. We further show several convergence processes with varying values of in Appendix
A daptive fixed filtered set size. As mentioned in Section 2.2 filtered set size is to use a fixed size. Here we investigate whether matters to use an adaptive filtered size that is determined by how much the draft model and the target model agree on each candidate set. To use a fixed size, we simply fix the probe agreement score a to be 0.9, 0.6, 0.3, and 0.0 and compare with the adaptive case. As shown in Table fixed probe agreement scores always lead to worse ASR. Furthermore, when adopting GPT-2 as the draft model, the average agreement score is 0.45 with a standard deviation of 0.11. This shows that the agreement score between the two models varies significantly for different candidate sets. We also provide the statistics of a for other draft models Table
alternative
adaptive
Probe agreement measurement. We also experiment alternatives measure the probe agreement score, including the Pearson correlation coefficient (Pearson 1900). Kendall's 's Tau correlation coefficient (Kendall [1938. and Goodman and Kruskal's gamma (Goodman et al. 1979) where the Pearson correlation coefficient directly uses the loss values to compute the agreement and the others the ranking information. As shown in Table 9 all methods have similar time cost, and Spearman's rank correlation coefficient achieves the best ASR. The Pearson correlation coefficient performs worse than other ranking- based agreement measurement.
Probe set size. The size of the probe set also determines whether the probe agreement score is measured accurately. As such, we experiment with different probe set size and report the performance in Table 10 We find that using small probe set such as B /64 or B 8/32 can result in inaccurate
8

---

Table Ablation on probe agreement measure- ments. All methods achieve similar speedup while Spearman's rank correlation coefficient achieves the best ASR.
Table 10: Ablation on the probe set size Using B/16 leads to accurate probe agreement mea- surement while achieving significant accelera- tion.
Cor
Spearman
Pearson
Kendall
Kruskal
Probe
B/61
B/32
B/16
B/4
B/2
B
ASR
85.0
70.0
74.0
79.0
ASR
64.0
72.0
85.0
86.0
85.0
87.0
Time
2.60
2.47
2.53
2.43
Time (8)
2.10
2.57
2.60
3.41
5.61
9.58
Table II: Experiments with different draft models. Models with over 1B parameters, like Tiny Phi, and ShearedLIMa, need two GPUs for parallel computation. ShearedLIMa achieves the highest ASR probably because it a pruned version of Llama2. Both GPT-2 and GPT-Neo achieve good balance of ASR and speedup.
1 GPU
GPUs Phi (1.3B)
GPT-2
GPT-Neo (125M)
Flan-T5 (24SM)
BART (406M)
TinyLlama (1.1B)
Sheared (1.3B)
Model
(124M)
a
0.45 4 0.10
0.51 + 0.11
0.61 +: 0.13
0.46 = 0.09
0.52 0.13
U.52 4: 0.11
0.35 4: 0.12
ASR
85.0
81.0
57.0
76.0
72.0
82.0
91.0
Time (s)
2.60
2.82
3.89
2.93
3.38
4.83
3,93
agreement score, which put a put a significant toll on the attack success rate. also does not lead to too much time reduction since the draft model computation done in parallel takes more time and the reduced computation is not the bottleneck. Using a larger probe set size such as B and B/2 will lead to accurate agreement score but does not increase the ASR significantly. As such, using probe set of size B/16 good enough to accurately measure the agreement and achieves maximum time reduction.
Draft model study. Here we also experiment with bigger draft models, some of which is of similar size to Llama2. We experiment with GPT-Neo (Gao et al. 2020), Flan-T5-base (Chung et al. 22024). BART (Lewis et al.] (2019), Phi-1.5 (Li et al., 2023), TinyLlama (Zhang et al.) (024) and Sheared-
LLaMA (Xia et al. (023). ). Among them, Sheared-LLaMA might be the closest to Llama2 since is pruned version of Llama2. For Tinyaa Phi and Sheared-LLaMA, we use 2 A100s with 80GB memory to fit the whole computation.
As shown in Table 11 Sheared-Llaie -LaMa achieves the best ASR although the time reduction is not as good as smaller models such as GPT-2 and there would be a higher time cost if we manage fit all computation in one GPU, On contrast, Flan-T5, BART, TinyLlama and Mistral all achieve lower ASRs probably because of being very different than Llama2. However, the results are still better than the baseline ASR 66.0. GPT-2 and GPT-Neo achieve a good balance of performance and speedup.
4 Related Work
Alignment of LLMs. To build safe LLMs, alignments has also been widely studied topic in the community (Stiennon et al. 2020; Ouyang et al. 2022). Efforts have been put into improving
helpfulness (Bai et al. 2022a Cheng et al. 2023). honesty (Kaddour et al. 2023 Liu et al. 2023
Xu et al. 2023), and harmlessness (Hartvigsen et al. 2022). Among these works, there has been
growing interest in using feedback from LLM to perform alignment (Bai et al. 2022b Gulcehre jet al.) 2023] Burns et al.) 2024) Yuan et al. 2024a). Despite all the efforts, there has not been a definitive answer for LLM safety alignments, which also motivates research in LLM safety.
Discrete Prompt Optimization. Attacking LLMs via adversarial prompt can be formulated as discrete prompt optimization problem (Zou et al., 2023). this context, attacking algorithms strive to discover superior prompts that effectively steer aligned LLMs toward generating adversarial answers. Some approaches leverage LLMs themselves to iteratively refine prompts (Xu et al. 2022. Pryzant et al. 2023). However, aligned L.MMs may resist refining adversarial prompts, rendering these methods ineffective. Other strategies employ RL-based prompt optimization techniques such
as those in (Mingkai and Jianyu 2022
Lu et al. 2023). necessitating additional MLP training with

---

extensive adversarial data and specific reward design. Moreover, other models introduced in (Cho al. 2023 Long et al., (024) to help with prompt optimization must remain unaligned, particularly in jailbreak scenarios (Chao et a..). However, their performance tends to be limited, especially when dealing with strongly fine-tuned models like Llama2-Chat.
LLM Jailbreaks. LLM Jailbreaks have received considerable interests recently since due to the implications of applying LLMs widely in human society. Although there is a continuous effort to build safe and reliable LLMs, bypassing the safety mechanism of LLMs is not uncommon. For
example, fine-tuning a safe LLM on few data instances can easily breaks its safety guarantees (Qi itt al., 2024: Lermen and -iiger (024). Treating the jailbreak as prompt optimization problem has also led to a certain level of success (Zou et al. 2023 Mökander et al. 2023 Liu et al.
2024, Chao et al.) Geisler et al. 2024). In addition, conversing in a ciphered language (Yuan et al.
2024b). planting a backdoor during RLHF (Rando and Tramer 2023), using less well-aligned
language (Deng et al. (202)) and multi-modality (Shayegani al. 2024) can also lead to successful
jailbreaks. Researchers also construct large dataset of manual jailbreak prompts (Toyer et al. (023).
Among these jailbreak methods, the prompt optimization method GCG (Zou et al. 2023) provides the more general and universal solution for us to study the jailbreaking problem. As such, in this work, we mainly focus on the acceleration of GCG, but the idea of delegating computation to a draft model can also be applied in other situations such as the multi- oulti-modality case and finetuning case. We leave the extension of this work for future work.
Acceleration. In the field of acceleration, speculative sampling (Chen et al. 2023 Leviathan et al., 2023) is the most relevant to our method. They also use a draft model but its design cannol be directly applied to accelerate the GCG algorithm. REST (He al. 2024) adopts the concept of speculative sampling but uses a retrieval approach based on a Trie to construct the candidate. The attention module has also been a focus of acceleration because of its quadratic nature (Dao et al. [022] Cai
it al. 2024). There have also been continuous interests in more efficient versions of Transformers (So al. 2019 Dai et al. 2022 Liu et al. 2021| Gu al., 2020 202T]. These architectural changes are complementary to our algorithm design and we leave it to future work.
5
Conclusion
In this paper, we propose an algorithm probe sampling that can effectively accelerate the GCG algorithm. We achieve an acceleration ranging from 2.1x to 6.3x in different scenarios on AdvBench. We illustrate the intuition and how the algorithm works through extensive experiments. Furthermore, this approach is also applied to general prompt optimization methods and other jailbreak techniques, including AutoPrompt, APE, and AutoDAN. We believe the idea of using the probe agreement score to perform adaptive computation can be applied to cases other than GCG. For example, it could potentially be used to perform conditional computation for attention. Another direction is to extend the framework to the multi-modality case which can be interesting given the vast amount of video data. It would also be interesting to run a small draft model on the scale of web data to detect the existence of natural adversarial prompts.
Limitation and Impact Statements
Probe sampling has two main limitations. Firstly, it exhibits relatively slow performance when tested on large-sized test sets, which hampers its efficiency. Secondly, it is limited to supporting only open-source models, thereby excluding proprictary or closed-source models from benefiting from the proposed acceleration techniques. These limitations indicate the need for further improvements to enhance the speed and broaden the model support in order to make the jailbreak acceleration approach more robust and applicable across wider range of language models.
Probe sampling can be applied to accelerate GCG algorithm. Having a faster algorithm explore adversarial cases of alignments enable us to study how to make LLMs safer. As far as we know, as of now, there not a LLM that can use this algorithm to achieve malicious behavior in real-world that would not be possible without the algorithm. The goal of this research is to present a general algorithm which may inspire new research, and also contribute to the gradual progress of building safe and aligned Als.
10

---

Acknowledgements
This research is partially supported by the National Research Foundation Singapore under the AI Singapore Programme (AISG Award No: AISG2-TC-2023-01 000--222--01--S00 10----01---)))--- and the Singapore Ministry of Education Academic Research Fund Tier 1 (Award No: TI 251RES2207). Xuan Long Do is supported by the A*TTAR Computing and Information Science (ACIS) scholarship. We thank Liwei Kang for insightful discussion, Liying Cheng for helping with plotting figures.
References
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022a. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0 05.602.2....
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et 2022b. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. 2024. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. Forty-first International Conference on Machine Learning.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple I1m inference acceleration framework with multiple decoding heads. In Forty-first International Conference on Machine Learning.
Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. In RO-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01311.
Pengyu Cheng, Yifan Yang, Jian Li, Yong Dai, and Nan Du. 2023. Adversarial preference optimiza- tion. arXiv preprint arXiv:2311.08044
Sukmin Cho. Soyeong Jeong, Jeong yeon Seo, and Jong Park. 2023. Discrete prompt optimization via constrained generation for zero-shot re-ranker. In The 6lst Annual Meeting Of The Association For Computational Linguistics.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113,
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghnn,, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168.
Zihang Dai, Hanxiao Liu, Quoe V Le, and Mingxing Tan. 2021. Coatnet: Marrying convolution and attention for all data sizes. Advances in neural information processing systems, 34:3965-3977.
11

---

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-effiiiett exact attention with io-awareness. Advances in Neural Information Processing Systems, (634--63359
Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2024. Multilingual jailbreak challenges in large language models. In The Twelfth International Conference on Learning Representations.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling, arXiv preprint arXiv:2101.00027
Simon Geisler, Tom Wollschläger, MHI Abdalla, Johannes Gasteiger, and Stephan Günnemann. 2024. Attacking large language models with projected gradient descent. arXiv preprint arXiv:2402.0914.
Leo A Goodman, William H Kruskal, Leo A Goodman, and William H Kruskal. 1979. Measures of association for cross classifications. Springer.
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. 2020. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33:1474-1487.
Albert Gu, Karan Goel, and Christopher Ré. 2021. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00366.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. 2023. Reinforced self-training (rest) for language modeling, arXiv preprint 1xXiv:3308.08998
Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. 2021. Gradient-based adversar- ial attacks against text transformers. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5747-5757.
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi. Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. Toxigen: large-scale machine-generater dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 3309-3326.
Horace He. 2023. GPT-Fast. a hitps: httpp:://ttttuittte . com/ wtt.:ccwtppppccccc httpppyttohhhhhect labs/
gpt
Accessed: Feb. 2, 2024.
Zhenyu He, Zexuan Zhong, Tianle Cai, Jason Lee, and Di He. 2024. Rest: Satrieval-based speculative decoding. In Proceedings the 2024 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume 1: Long Papers), pages
1595.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. 2020. Measuring massive multitask language understanding. In International Conference on Learning Representations.
Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06633
Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert Challenges and applications of large language models. arXiv preprint
McHardy. 2023.
MrXiv:2307.10166
Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81-93.
Simon Lermen and Charlie Rogers-Smith. 2024. Lora fine-tuning efficiently undoes safety training in Ilama 2-chat 70b. In ICLR 2024 Workshop Secure and Trustworthy Large Language Models.
12

---

Brian Lester, Rami AI-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficien prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274-19286.
PMLR.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising natural language generation, translation, and comprehension. arXiv preprint arXiv: 1910. 3461.
pre-training for
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.0544
Hanxiao Liu, Zihang Dai, David So, and Quoc V Le. 2021. Pay attention to mlps. Advances in Neural Information Processing Systems, 34:9204-9215.
Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024. Autodan: Generating stealthy jailbreak prompts on aligned large language models. In The Twelfth International Conference on Learning Reprexentations.
Yang Liu, Yuanshun Yao,
Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. 2023. Trustworthy Ilms: a survey and guideline for evaluating large language models' alignment. In Socially Responsible Language Modelling Research.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: /907. 1692..
Do Long, Yiran Zhao, Hannah Brown, Yuxi Xie, James Zhao, Nancy Chen, Kenji Kawaguchi, Michael Shieh, and Junxian He. 2024. Prompt optimization via adversarial in-context learning In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 7308-73223 Bangkok, Thailand. Association for Computational Linguistics.
Pan Lu, Liang Qiu, Kai- Wei Chang. Ying Nian Wu, Song-Chur Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. 2023. Dynamic prompt learning via policy gradient for semi- structred mathematical reasoning. In The Eleventh International Conference on Learning Representations.
Chris Maddison, Andriy Mnih, and Yee Whye Teh. 2022. The concrete distribution: continuous relaxation of discrete random variables. In International Conference on Learning Representations.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models In Proceedings of the Ninth International Conference on Language Resources and Evaluation (4REC'14), pages 216-223, Reykjavik, Iceland. European Language Resources Association (ELRA).
Deng Mingkai and Wang Jianyu. 2022. Riprompt: Optimizing discrete text prompts with reinforce- ment learning. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing.
Jakob Mökander, Jonas Schuett, Hannah Rose Kirk, and Luciano Floridi. 2023. Auditing large language models: a three-layered approach. Al and Ethics, pages 1-31.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744
13

---

Karl Pearson. 1900. X. on the criterion that given system of deviations from the probable in the case of correlated system of variables such that can be reasonably supposed have arisen from random sampling. The London, Ealnnburhh, and Dublin Philosophical Magazine and Journal of Science, 50(302):157-15.
Martin Pincus. 1970. A monte carlo method for the approximate solution of certain types of constrained optimization problems. Operations research, 18(6):1255-122
Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with "gradient descent" and beam search. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7957-7968.
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2024. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.
Language models are unsupervised multitask learners.
Javier Rando and Florian Tramèr. 2023. Universal jailbreak backdoors from poisoned human feedback. arXiv preprint arXiv:231 1.14555.
Erfan Shayegani, Yue Dong, and Nael Abu-Ghazaleh. 2024. Jailbreak in pieces: Compositional adversarial attacks multi-modal language models. In The Twelfth International Conference Learning Representations.
Taylor Shin, Yasaman Razeghi, Robert I. Logan IV, Eric Wallace, and Sameer Singh. 2020. Auto- prompt: Eliciting knowledge from language models with automatically generated prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222-4235
David So, Quoe Le, and Chen Liang. 2019. The evolved transformer. In International conference on machine learning. pages 5877-5886. PMLR.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642.
Nisan Stienmon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, et al. 2023. Challenging big-bench tasks and whether ch can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13003-13051.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288
Sam Toyer. Olivia Watkins, Ethan Mendes, Justin Svegliato, Luke Bailey, Tiffany Wang, Isaac Ong. Karim Elmaaroufi, Pieter Abbeel, Trevor Darrell, et al. 2023. Tensor trust: Interpretable prompt injection attacks from an online game. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following.
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2024. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. Advances in Neural Information Processing Systems, 36.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. 2023. Sheared Ilama: Accelerating language model pre-training via structured pruning. In Workshop Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ NeurIPS 2023).
14

---

Chunpu Xu, Steffi Chern, Ethan Chern, Ge Zhang, Zekun Wang, Ruibo Liu. Jing Li, Jie Fu, and Pengfei Liu. 2023, Align on the fly: Adapting chatbot behavior to established norms. arXiv preprint arXiv:2312.
Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Wang Yanggang, Haiyu Li, and Zhilin Yang. 2022. Gps: Genetic prompt search for efficient few-shot learning. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8162-811 8171.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024a. Self-rewarding language models. In Forty-first International Conference on Machine Learning.
Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. 2024b. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher. In The Twelfth International Conference on Learning Representations.
Jemrldd H Zar. 2005. Spearman rank correlation. Encyclopedia of biostatistics,
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. Tinyllama: An open source small language model. arXiv preprint arXiv:2401.0238..
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. In The Eleventh International Conference Learning Representations.
Andy Zou, Zifan Wang, Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models, arXiv preprint 2:X1v:2307.15043,
15

---

A
Implementation
The following code shows the core implementation of probe sampling using PyTorch. As seen in the code, the algorithm is relatively easy to use.
draft_model all (arga): : draft_madel 1 oss (control coontrol_canda
queue.put 'draft'' :loss_small)
Lerget__ndee__peee_______ ppccbe (arga):
probe_index random. sample (zange (512), 512/16)
probe_controi_ control (probe_index)
target_modell 1.leell.oss loas (probo_concrol_eansss
queue.put ''target' (lass_large_proke, probe_index))
Parallelly Calculate
argan coontrol_cnndee batch_size, quaue) thread! ng. Thread (ta tt
args-args)
threadIng Thread (ttaggeeettargett_oooeee.eco
args=args!
Calculate Agreement car spearmanr (loss_amo]) (prche_indes) !ndex], large_lass_
rarget Model Test on Filtered Set
filttred_aize int ((1 cor) 512/8)
indices Lopk (pooss_smaall k-tillered_size, lergesL-Palse) cattere@_cottroo_cette cends control_cands (indices)
model lesa (ostfeiitereee_ontrro..ooo
Return Losest Loss Candidate
return [large_loas loab_prob, filtered cowooooottoooooittooo lewest
B Converge Process
In Figure also show few convergence processes with different values of R, where the pink line corresponds to R = 8. The pink line always achieves successful optimization while the other lines can lead to suboptimal results due excessive randomness insufficient randomness. In particular, the blue and yellow lines can suffer from excessive randomness and the other lines might have insufficient randomness.
DXC
Rel
Rad
Rad
Rall
Ra15
Robi
(a) GCG succeeded case
(b) GCG succeeded case
(c) GCG failed case 1
(d) GCG failed case
Figure 5: Converge progress with different sizes of filtered
C Software optimization
In other speedup works (He 2023), using torch.compiin can lead to significant accelera- tion. It compiles LLMs into an kernel and alle- viate the overhead of repeatedly launching the kernel. Table [12]shos that the time cost is sim- ilar with or without this optimization enabled.
Table 12: Results with torch.compileO) enabled.
torch.comple() does not lead to further speedup.
Method
GCG
Probe sampling
PS (Compile)
ASR
66.0
85.0
85.0
Time (s)
9.16
2.60 (3.5x)
2.54 (3.6x)
This is likely due to the fact that we use large
batch sizes and long input sequences, whose computation cost dominates the overhead caused by the eager execution and launching the kernel repeatedly.
16

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
In the abstract and Section | (Introduction)
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: After Section (Conclusion)
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In Section 5 (Experiment)
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification: We have released the code at haat:::://ittuub..om///eeeee
Probe- Probe-Samplin Sampling
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips) cc/
publ iellic//ee / www.dddee/Ceeeeede
ss Loonolics cy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (ht
/ ctc.ps.ccpuubbic
CooeeoodeSSSa Submisbbbsss ss cessoooPolcyy for more details. The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
In Section 3 (Experiment)
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: While is challenging to achieve due the substantial computing resources demands, this limitation is not unique to this paper but is shared by other works in the same field. However, the large size of the test set used in this study ensures that bias is minimized.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
19

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Section (Experiment)
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
This paper adheres to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In Section (Experiment)
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers thcode com dataaettettt has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "nysL7cR50t", "reviewer": "Reviewer_bSGi", "summary": "This paper presents a novel algorithm called \"Probe sampling\" to accelerate the Greedy Coordinate Gradient (GCG) method for optimizing adversarial prompts against large language models (LLMs). The key idea is to use a smaller \"draft\" model to filter out unpromising candidate prompts, reducing the number of expensive evaluations needed on the full target LLM. The method dynamically determines how many candidates to filter based on measuring agreement between the draft and target models on a small probe set. Experiments show Probe sampling achieves 3.5-6.3x speedups on benchmark datasets while maintaining or improving attack success rates. The technique is also shown to accelerate other prompt optimization and LLM attack methods.", "questions": "No.", "limitations": "The authors provide a reasonable discussion of limitations. One additional limitation that would make sense to discuss is the high memory requirement for gradient-based red-teaming methods like GCG, which effectively limits the experiments to smaller models (i.e., only up to 7B parameters).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Overall: The paper looks solid and the 5.6x runtime improvement for GCG looks compelling. Also, it’s nice to see speed-ups for AutoDAN and prompt learning methods like AutoPrompt and APE. The idea of reducing computation by resorting to a smaller draft model - specifically for forward passes that consume the most time - makes a lot of sense.\n\n- Quality: The empirical evaluation is thorough, testing on multiple datasets and model types. The ablation studies and analysis provide good insight into the algorithm's behavior. The speedups achieved are substantial and practically meaningful.\n\n- Clarity: The paper is generally well-written and easy to follow. The algorithm is well explained, and the experimental setup and results are presented in a clear and organized manner.", "weaknesses": "- It would be good to comment on the other existing approaches to speed up GCG like Fast Adversarial Attacks on Language Models In One GPU Minute and Making a SOTA Adversarial Attack on LLMs 38x Faster. Since they were out in February/March, probably it’s still fine to treat them as concurrent work, but it would be good to discuss the differences to better contextualize your work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer bSGi,\n\nThank you for your insightful reviews and comments. We appreciate the time and effort you have put into providing valuable feedback. We would like to address your concerns as follows:\n\n> Concern #1: Compare with more related works\n\nWe appreciate your concern about comparing probe sampling with other existing acceleration approaches. BEAST [1] and ACG [2] both employ beam-search like decoding methods, accelerating GCG by generating more candidates in each iteration, thereby reducing the total number of iterations. However, probe sampling is orthogonal to these methods, capable of integration into the loss calculation stage of each algorithm. \n\nTo verify this claim, we implement probe sampling on BEAST, which can accelerate it by 1.4 times on Vicuna-7b and 1.9 times on Llama2-7b-chat, on average per iteration. This acceleration allows probe sampling to execute more iterations within the same GPU time, thereby improving the ASR. The detailed results are presented below, indicating that probe sampling improves the ASR of both models within the same GPU time.\n\n\n\n|                    | **Clean** |           |       Time (s)        |           | **In one GPU minute (%)** |           | **In two GPU minutes (%)** |\n| :----------------: | :-------: | :-------: | :-------------------: | :-------: | :-----------------------: | :-------: | :-----------------------: |\n|                    |           | **BEAST** |  **Probe-Sampling**   | **BEAST** |    **Probe-Sampling**     | **BEAST** |    **Probe-Sampling**     |\n|   **Vicuna-7B**    |   **7**   |    2.4    | **1.7 (1.4$\\times$)** |    89     |          **90**           |  **96**   |          **96**           |\n| **Llama2-7B-chat** |     0     |    4.3    | **2.3 (1.9$\\times$)** |     9     |          **13**           |    12     |          **16**           |\n\nFurthermore, as illustrated in our paper (line 150 to line 155), probe sampling can be used to accelerate algorithms involving sampling prompt candidates and evaluating their performances, as evidenced by the experiment results in Table 3 and Table 4.\n\n\nIn addition to integrating with beam-search acceleration methods, probe sampling yields better ASR results, with probe sampling achieving a score of 81 on Llama2-7b-chat, surpassing the scores of 12 for BEAST and 64 for ACG.\n\n> Concern #2: Limitation\n\nThanks for recommending adding the limitation about high memory requirement, we will incorporate the following sentence into the final version. \n\nAlthough the acceleration of probe sampling does not necessitate additional memory, it still faces the high memory demands inherited from GCG, consequently restricting experiments to smaller models. While simulated annealing offers some relief, the situation remains far from adequate.\n\n          \n\n[1] Sadasivan, Vinu Sankar, et al, Fast Adversarial Attacks on Language Models In One GPU Minute, ICML 2024\n\n[2] Making a SOTA Adversarial Attack on LLMs 38x Faster, Haizelabs Blog Post, 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EEUrXxeWjJ", "reviewer": "Reviewer_Jngk", "summary": "The authors propose using a significantly smaller draft model compared to the target LLM to filter candidate suffixes, thereby accelerating the training process of GCG-based algorithms.\nThe results demonstrate a faster training speed with enhanced ASR.", "questions": "1. The method Amplegcg [2] achieve better performance than GCG, can the proposed method be applied to speed this traning process?\n\n[1] Liao, Z., & Sun, H. (2024). Amplegcg: Learning a universal and transferable generative model of adversarial suffixes for jailbreaking both open and closed llms. arXiv preprint arXiv:2404.07921.", "limitations": "none.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. I appreciate the innovative approach of utilizing another LLM to filter suffixes to assist in the attack on the target LLM.\n2. The paper is well-organized and written in a formal academic style.", "weaknesses": "1. As mentioned on line 82, \"one needs to perform B forward passes.\" In my opinion, these B candidates can be sent to the LLM together within a batch. By fully utilizing the GPU's parallel technique, it actually does not need to take B times the amount of time. Can an LLM with a smaller batch size achieve faster speeds? If not, I think the author should illustrate why the proposed method can speed up.\n2. Since the draft model is not optimized, how to guarantee that the draft model and the target model align well, i.e., $\\alpha$ is significantly larger than $0$? In my opinion, GCG will generate meaningless words, and the function of the draft model seems to be to check the meaning of these suffixes and discard  the meaningless ones. I encourage the authors to explain the the meaning of alignment between the draft model and the target model.\n3. While the GCG is a popular method and the $6x$ speed is impressive, there are another methods like advprompter [1] which has faster speed and higher ASR. I encourage authors to compare the proposed method with more efficient methods instead of GCG.\n\n[1] Paulus, A., Zharmagambetov, A., Guo, C., Amos, B., & Tian, Y. (2024). Advprompter: Fast adaptive adversarial prompting for llms. arXiv preprint arXiv:2404.16873.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Jngk,\n\n\nWe appreciate the time and effort you have put into providing valuable feedback. However, we respectfully believe there might be some misunderstanding regarding our work. We would appreciate the opportunity to clarify a few points and address your concerns as follows:\n\n\n> Misunderstanding #1:  Probe sampling performs B forward passes in parallel within a batch or sequentially?\n\nWe understand your concern related to whether probe sampling can accelerate GCG when B candidates are sent to LLMs in a batch. The answer is definitely YES, as we consistently use this approach throughout our paper, similar to the GCG paper. For more details on how this is implemented, please refer to the GCG code, specifically lines 170 to 174 in `llm-attacks/llm_attacks/gcg/gcg_attack.py`. In this context, the term “B forward passes” should be more accurately articulated as “forward computation on B candidates”. This modification ensures a clearer understanding of the procedure, emphasizing the total amount of computation. Moreover, Figure 3 illustrates why probe sampling can accelerate GCG when concurrently operating B candidates in a batch, owing to its reduced memory usage. A more comprehensive analysis is illustrated in the paper, spanning from line 217 to line 224.\n\n\n> Misunderstanding #2: Draft model is used to check whether the meaning of suffixes are meaningful so how to make sure two models are aligned without training?\n\nThe draft model is not used to \"check the meaning\" of suffixes; it solely assists the target model in ranking candidates. In GCG, we want to find the candidate suffix that achieves the lowest loss, i.e., the suffix that is ranked the first if sorted by their losses computed by the target model. Naturally, if the ranked results agree, the rank using the draft model is indicative of the rank using the target model and we can safely rely on the draft model to filter out more unpromising candidates. Conversely, in cases where $\\alpha$ equals 0 ($\\alpha$ can not be significantly larger than 0 as it is in the range of [0,1] and 1 meaning a full agreement as explained in line 110), we disregard the rank calculated by the draft model and depend exclusively on the target model in this iteration. Note that $\\alpha$​ is adaptively calculated in each iteration (explained from line 134 to line 136 in text and line 13 in Algorithm 1), which significantly contributes to the performance of probe sampling. Furthermore, as illustrated in Table 9, probe sampling works consistently well across different models with various average alignment levels.\n\n> Concern #1: Compare with more efficient methods instead of GCG.\n\nWe appreciate your concern about comparing probe sampling with other existing acceleration approaches. AdvPrompter [1] involves training the AdvPrompter with the help of both the target model and another base model, necessitating inference of AdvPrompter for each adversarial prompt. These additional computational resources are significant. According to Figure 2 in [1], AdvPrompter achieves an ASR of 23.4 on Llama2-7b-chat, whereas probe sampling achieves 81. Moreover, following the recommendation from Reviewer bSGi, we also compare probe sampling with BEAST [2] and ACG [3]. It shows that probe sampling not only achieves much higher ASR, but also is orthogonal to them and can further accelerate them. Please refer to the rebuttal addressing concern #1 provided to Reviewer bSGi for further details. \n\n> Concern #2: Whether probe sampling can be used to accelerate AmpleGCG.\n\nWe acknowledge the concerns regarding the generalizability of probe sampling. AmpleGCG introduces a methodology that involves initially gathering adversarial attacks through overgenerated GCG and subsequently fine-tune the model using these adversarial input-output pairs. Given that AmpleGCG utilizes GCG as the fundamental algorithm for producing and assembling adversarial outputs, probe sampling can effectively accelerate overgenerated GCG. Following the setting of the paper, we implement probe sampling on overgenerated GCG.\n\n\n|                     | ASR      | Time                  |\n| ------------------- | -------- | --------------------- |\n| GCG                 | 20.0     | 4.3                   |\n| Overgenerate        | 76.7    | 4.3                   |\n| Over+probe sampling | **83.3** | **1.8 (2.4$\\times$)** |\n\nWe further employ generated adversarial input-output pairs to fine-tune Llama2-7B-chat and test on the same released hard testset with 100 queries. Here are the detailed results. \n\n|                          | AmpleGCG | Probe Sampling Training |\n| ------------------------ | -------- | ----------------------- |\n| Group Beam Search (50)   | **83.0** | 82.0                    |\n| Group Beam Search (100)  | 93.0     | **93.0**                |\n| Group Beam Search (200)  | 99.0     | **100.0**               |\n| Group Beam Search (1000) | **100.0**    | **100.0**               |\n\nWe find that adversarial input-output pairs generated by probe sampling accelerated overgenerated GCG achieves nearly the same performance, proving the applicability of probe sampling on generating adversarial training data. \n\nIn addition, probe sampling can also be implemented on Group Beam Search algorithm. However, we omit additional experiments here as it is the same as accelerating BEAST as mentioned earlier.\n\n\n\n[1]  Paulus, et al, AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs, Arxiv 2024\n\n[2] Sadasivan, Vinu Sankar, et al, Fast Adversarial Attacks on Language Models In One GPU Minute, ICML 2024\n\n[3] Making a SOTA Adversarial Attack on LLMs 38x Faster, Haizelabs Blog Post, 2024\n\n[4] Liao, et al, AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs, Arxiv 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9f49J4e2QS", "reviewer": "Reviewer_hJdQ", "summary": "The paper proposes a method to accelerate discrete prompt optimization algorithms. In addition to the target model, a smaller draft model is used to reduce the number of candidates to be evaluated by the target model based on an agreement score between the two models. When applied to Greedy Coordinate Gradient and trained with the open-source LLMs  Llama2-chat-7b and Vicuna-7b, the approach achieves a better runtime while maintaining the attack success rate (ASR) on AdvBench. The method is also applied to other prompt optimization algorithms such as AutoPrompt, ADE and AutoDAN.", "questions": "- Does this method affect the transferability of adversarial attacks to other LMs (for example from Llama2-chat-7b to Vicuna-7b) compared to plain GCG? \n- What is the effect of the filtered set size and the draft model (size, similarity to target model) on transferability?\n- GCG operates at the token level and the draft model has a different tokenizer. What is the effect of the draft model tokenizer on the resulting suffix? also on transferability? (are tokens common to both tokenizers chosen or is the method completely oblivious to tokenization differences)", "limitations": "The authors have included a section on limitations and potential negative social impact.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The problem studied in the paper is important. The method makes finding prompts for very large models more efficient. \n- The method is simple and general enough that it can be applied to any discrete prompt optimization algorithms that rely on LLM candidate evaluation.\n- Except for some minor typos, the paper is well-presented and is easy to read.", "weaknesses": "- Transferability is a very important aspect of adversarial attacks but it is never mentioned in the paper. \n- The related work section lacks a proper presentation of discrete prompt optimization algorithms. Also, the authors did not mention other prior work that also use a pair of models for discrete prompt optimization. \n\nMinor issues:\nThe use of the term \"Probe sampling\" is inconsistent across the paper. Sometimes it is not emphasized and in lower case. \nline 310: \"In this paper, we propose an algorithm called Probe sampling that can ...\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer hJdQ,\n\nThank you for your insightful reviews and comments. We appreciate the time and effort you have put into providing valuable feedback. We would like to address your concerns as follows:\n\n> Concern: Transferability of Probe Sampling\n\nWe appreciate your concern regarding the transferability of probe sampling, and we conduct corresponding experiments to analyze it.\n\n**1. Whether draft model of probe sampling affect transferability**\n\nWe compare probe sampling implemented across various draft models on Llama2-7b-Chat and then transfer to diverse target models. Detailed results are shown as follows. Our investigation reveals that probe sampling has minimal impact on transferability when applied to appropriate draft models that maintain the original ASR of plain GCG. Conversely, utilizing draft models that degrade the initial performance significantly affects transferability.\n\n\n|                   | Llama2-7b-Chat (Direct) | Vicuna-7b (Transfer) | Mistral-7b (Transfer) |\n| ----------------- | :---------------------: | :------------------: | :-------------------: |\n| Plain GCG         |           69            |          89          |        **86**         |\n| PS (GPT-2)        |           85            |          92          |          83           |\n| PS (ShearedLlaMa) |         **91**          |        **93**        |          85           |\n| PS (Flan-T5)      |           57            |          78          |          69           |\n\nIn addition to open-source models, we also conduct transfer experiments on GPT4, especially gpt4-0125-preview. Our findings indicate that probe sampling not only markedly accelerates the attacking process but also maintains the transferred ASR. The detailed results are presented below.\n\n|                |    Optimized on    |   ASR    |       Time (s)        |\n| -------------- | :----------------: | :------: | :-------------------: |\n| GCG            | Llama2-7b & Vicuna |   16.0   |          9.8          |\n| Probe Sampling | Llama2-7b & Vicuna | **17.0** | **3.2 (3.1$\\times$)** |\n\n\n\n**2. Whether the filtered set size of probe sampling affect transferability?**\n\nIn our experiments, we investigate the transferability of probe sampling across various filtered set sizes, denoted as $(1-\\alpha)*B/R$. Our results align closely with previous findings, indicating that probe sampling does not significantly impact transferability when appropriate parameters are used. However, it does lead to decreased performance in cases where the direct ASR on Llama2-7b-chat is low.\n\n\n|           | Llama2-7b-Chat (Direct) | Vicuna-7b (Transfer) | Mistral-7b (Transfer) |\n| --------- | :---------------------: | :------------------: | :-------------------: |\n| Plain GCG |           69            |          89          |          **86**           |\n| R=64      |           60            |          77          |          74           |\n| R=8       |          **85**            |          **92**          |          83           |\n| R=1       |           79            |          88          |          84           |\n\n**3. What is the influence of the draft model's tokenizer on suffixes?**\n\nIn Table 9, TinyLlama and ShearedLlaMa utilize the same tokenizer as Llama2-7b-Chat. However, their performance diverges significantly. Hence, the tokenizer does not play a pivotal role in effectively attacking the model or the transferability; instead, factors like model performance similarity hold greater importance.\n\n> Suggestion #1: Adding related work\n\nThank you for recommending more related works, especially in discrete prompt optimization algorithms. We will add the following paragraph in the final version.\n\n**Discrete Prompt Optimization.**  Attacking LLMs via adversarial prompt can be formulated as a discrete prompt optimization problem [1]. In this context, attacking algorithms strive to discover superior prompts that effectively steer aligned LLMs toward generating adversarial answers. Some approaches leverage LLMs themselves to iteratively refine prompts [2, 3]. However, aligned LLMs may resist refining adversarial prompts, rendering these methods ineffective. Other strategies employ RL-based prompt optimization techniques such as those in [4, 5], necessitating additional MLP training with extensive adversarial data and specific reward design. Moreover, other models introduced in [6, 7] to help with prompt optimization must remain unaligned, particularly in jailbreak scenarios[8]. However, their performance tends to be limited, especially when dealing with strongly fine-tuned models like Llama-2.\n\n> Suggestion #2: Align the format of \"Probe sampling\" in the paper.\n\nWe appreciate your effort in seriously looking into the details of our paper. We will address them to enhance our work further in the final version.\n\n\n[1] Zou, et al, Universal and Transferable Adversarial Attacks on Aligned Language Models, Arxiv 2023\n\n[2] Xu, et al, GPS: Genetic Prompt Search for Efficient Few-Shot Learning, EMNLP 2022\n\n[3] Pryzant, et al, Automatic prompt optimization with\" gradient descent\" and beam search, EMNLP 2023\n\n[4] Deng, et al, RLprompt: Optimizing discrete text prompts with reinforcement learning, EMNLP 2022\n\n[5] Lu, et al, Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning, ICLR 2023\n\n[6] Cho, et al, Discrete Prompt Optimization via Constrained Generation for Zero-shot Re-ranker, ACL 2023\n\n[7] Do, et al, Prompt Optimization via Adversarial In-Context Learning, ACL 2024\n\n[8] Chao, et al, Jailbreaking Black Box Large Language Models in Twenty Queries, Arxiv 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper investigates the problem of generating adversarial prompts using Greedy Coordinate Gradient (GCG) and introduces probe sampling with a small draft model to reduce the costs of prompt evaluation. Experiments demonstrate a substantial speedup on benchmarks. The primary concerns raised were the comparison with recent related work and the alignment between the draft and the base models. During the rebuttal, the authors provided additional experimental results that addressed most of these concerns.
Decision: Accept (poster)