Query-Based Adversarial Prompt Generation
Authors: Jonathan Hayase
Ema Borevković
Nicholas Carlini
Florian Tramèr
Milad Nasr
Venue: NeurIPS 2024 poster (2024)
Keywords: adversarial examples
large language models
black box
Primary area: safety_in_machine_learning
TL;DR: We cause aligned models to output specific harmful strings using a black-box attack

Abstract: Recent work has shown it is possible to construct adversarial examples that cause aligned language models to emit harmful strings or perform harmful behavior.
Existing attacks work either in the white-box setting (with full access to the model weights), or through _transferability_: the phenomenon that adversarial examples crafted on one model often remain effective on other models.
We improve on prior work with a _query-based_ attack that leverages API access to a remote language model to construct adversarial examples that cause the model to emit harmful strings with (much) higher probability than with transfer-only attacks.
We validate our attack on GPT-3.5 and OpenAI's safety classifier; we can cause GPT-3.5 to emit harmful strings that current transfer attacks fail at, and we can evade the OpenAI and Llama Guard safety classifiers with nearly 100% probability.

Query-Based Adversarial Prompt Generation
Jonathan Hayase'
Ema Borevkovic
Nicholas Carlini
Florian Tramèr Google Deepmind
Milad Nasr
'University of Washington
'ETH Zürich
Abstract
Recent work has shown it is possible construct adversarial examples that cause aligned language models to emit harmful strings or perform harmful behavior. Existing attacks work either in the white-box setting (with full access to the model weights), or through transferabilitt; the phenomenon that adversarial examples crafted on model often remain effective other models. We improve prior work with a query-based attack that leverages API access to a remote language model construct adversarial examples that cause the model to emit harmful strings with (much) higher probability than with transfer-only attacks. We validate attack GPT-3.5 and OpenAl's safety classifier; emit harmful strings that current transfer attacks fail at, and we can evade the OpenAl and Llama Guard safety classifiers with nearly 100% probability.
can cause GPT-3.5
Introduction
The rapid progress of transformers 333 in the field of language modeling has prompted significant interest in developing strong adversarial examples 4 30) that cause a language model to misbehave harmfully. Recent work (36] has shown that by appropriately tuning optimization attacks from the literature 28 [4. it is possible to construct adversarial text sequences that cause a model to respond in targeted manner.
These attacks allow an adversary to cause an otherwise "aligned" model that typically refuses requests such "how do build bomb?" or "swear at te comply with such requests, or even emit exact targeted unsafe strings (e.g.. malicious plugin invocation (3]. These attacks can cause various forms of harm, ranging from reputational damage to the service provider, to potentially more significant harm if the model has the ability to take actions on behalf of users [3 (e.g., making payments, or reading and sending emails).
The class of attacks introduced by Zou et al. 36 are white-box optimization attacks: they require complete access to the underlying model to be effective effective-somethine something that is not true in practice for the largest production language models today. Fortunately (for the adversary), the transferability property of adversarial examples (26] allows an attacker to construct an adversarial sequence local model and simply replay on larger production model great effect. This allowed Zou et al. (36) to fool GPT-4 and Bard with 46% and 66% attack success rate by transferring adversarial examples initially crafted on the Vicuña [1 family of open-source models.
Contributions. In this paper, we design an optimization attack that directly constructs adversarial examples remote language model, without relying transferability]] This has two key benefits:
Targeted attacks: Query-based attacks can elicit specific harmful outputs, which is not feasible for transfer attacks.
'There exist other black-hox jailbreak methods that rely on either language model refine candidate attacks [22 or on greedy search (2). These techniques are weaker than ours though, and succeed making target model output exact harmful strings, which we do.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Surrogate-free attack: Query-based attacks also allow us to generate adversarial text se- quences when no convenient transfer source exists.
Our fundamental observation is that each iteration of the GCG attack of Zou al. 36 can be split into two stages: filtering large set of potential candidates with a gradient-based filter, followed by selecting the best candidate from the shortlist using only query access. Therefore, by replacing the first stage filter with filter based on surrogate model, and then directly querying the remote model we wish to attack, we obtain a query-based attack which may be significantly more effective than an attack based only on transferabilityy
We further show how an optimization the GCG attack allows us remove the dependency on the surrogate model completely. with only a moderate increase in the number of model queries. As result, we obtain an effective query-only attack requiring no surrogate model at all.
As an example use-case, we show how to evade OpenAl's content moderation endpoint (that, e.g., detects hateful or explicit sentences) with nearly 100% attack success rate without having local content moderation model available. This despite this endpoint being OpenAl's "most robust moderation model to-date" [24]
2
Background
Adversarial examples. First studied the vision domain, adversarial examples 4 30 are inputs designed by an adversary to make machine learning model misbehave. Early work focused on the wwhite-box" threat model, where an adversary has access to the model's weights and can thus use
gradient descent reliably maximize the model's loss with minimum perturbation 12 6 21
These attacks were then extended to the more realistic "black-bor threat model, where an adversary has no direct access to the model weights. The first black-box attacks relied on the "transferability" of adversarial examples [26]: attacks that fool one model also tend to fool other models trained independently-eeee on different datasets.
Transfer- attacks have several limitations. Most importantly, they rarely succeed at "targeted" attacks that aim to cause model to perform a specific incorrect behavior. Even on simple tasks like ImageNet classification with 1,000 classes, targeted transfer attacks are challenging (20].
These difficulties gave rise to query-based black- box attacks g 5). Instead of relying exclusively on transferability, these attacks query the target model to construct adversarial examples using black-box optimization techniques. These attacks have (much) higher success rates: they can reach nearly 100% targeted attack success rate on black-box ImageNet classifiers, at a cost of a few thousand model queries. Query-based attacks can further be combined with signals from a local model to reduce the number of model queries without sacrificing attack success [TO]
Language models. Language models are statistical models that learn the underlying patterns within text data. They are trained on massive datasets of text to predict the probability of the next word or sequence of words, given the preceding context. These models enable variety of natural language processing tasks such as text generation, translation, and question answering [27).
NLP adversarial examples. Adversarial examples for language models have followed a similar path as in the vision field. However, due to the discrete nature of text, direct gradient- gradient-based optimization is more difficult. Early work used simple techniques such as character-level or word-level substitutions to cause models to miselassify text [18].
Further attacks optimized for adversarial text in language model's continuous embedding space, and then used heuristics to convert adversarial embeddings into hard text inputs (28). These methods, while effective on simple and small models, were not sufficiently strong to reliably cause errors on large transformer models 77]. As a result, followup work was able combine multiple ideas from the literature in order to improve the attack success rate considerably [36].
In doing so, Zou et al. [36] also introduced the first set of transferable adversarial examples that were also capable of fooling multiple production models. By generating adversarial examples on Vicuna-a freely accessible large language model with open weights- it was possible construct transferable adversarial examples that fool today's largest models, including GPT-4.
Unfortunatcly, NLP transfer attacks suffer from the same limitations their counterparts in vision:

---

Transfer attacks require a high-qualig surrogate. For example, Zou et al. [36] showed that Vicuña is poor surrogate for Claude, achieving just 2% transfer attack success rate.
Transfer attacks do not succeed at inducing targeted "harmful strings". While transfer attacks can cause models to comply with requests (i.e., an un-targeted attack), they cannot force the model into producing specific harmful output.
As we will show, it is possible to address both of these limitations (and more!) through query-based attacks (in concurrent work, Sitawarin et al. 2 propose similar attacks to ours, but not evaluate them for inducing targeted harmful strings).
The Greedy Coordinate Gradient attack (GCG). Zou et al. [36] recently proposed an extension of the AutoPrompt method [28). known as Greedy Coordinate Gradient (GCG), which has proven to be effective. GCG calculates gradients for all possible single-token substitutions and selects promising candidates for replacement. These replacements are then evaluated, and the one with the lowest loss is chosen. Despite its similarity to AutoPrompt, GCG significantly outperforms it by considering all coordinates for adjustment instead of selecting only one in advance. This comprchensive approach allows GCG to achieve better results with the same computational budget. Zou et al. Jalso optimized the adversarial tokens for several prompts and models at the same time, which helps to improve the transferabilitty of the adversarial prompt to closed source models.
Heuristic approaches. Given the popularity of language models, many also craft adversarial prompts by manually prompting the language models until they produced the desired (harmful) outputs. Inspired by manual adversarial prompts, recent works showed that they can improve the manual style attacks using several heuristics (35 15
3 GCQ: Greedy Coordinate Query
We now introduce our attack: Greedy Coordi- nate Query (GCQ). At high level, our attack a direct modification of the GCG method dis- cussed above.
Algorithm 1: Greedy Coordinate Query
input vvocabulary V =
sequence
length m. loss é V - R. proxy loss Ep : V + R. iteration count T, proxy batch size 6p. query batch size by. buffer size B
3.1 Method
Our main attack strategy is similar GCG in that it makes greedy updates to an adversarial string. At each iteration of the algorithm, we perform an update based on the best adversarial string found so far, and after a fixed number of iterations, return the best adversarial example.
buffer + B uniform samples from V
for i [T] do
for i [by] do
j~ Unif((ml), Uni batch; + argminhebutter E(b) (batch:); + l
The key difference in our algorithm is in how we choose the updates to apply. Whereas GCG maintains exactly one adversarial suffix and per- forms brute-force search over many potential updates, to increase the query efficiency, our up- date algorithm is reminiscent of west-first-search Each "node" corresponds to a given adversarial suffix. Our attack maintains buffer of the B best unexplored nodes. At each iteration, we take the best node from the buffer and expand it. The expansion done by sampling a large set of b, neighbors, taking the bg best of those ac- cording to a local proxy loss, Ep and evaluating these with the true loss f. We then iterate over the neighbors and update B. We write the algorithm in pseudocode in Algorithm[
ploss; + ly (batch); end for
i E (plos)) do
loss + (batch.
bworst f- argmax ((b)
loss < ((bworet) then remove bworst from buffer add batch, to buffer
end end end return argmingebutter {(b)
In practice buffer is implemented using a min-max heap containing pairs of examples and their corresponding losses (with order defined purely by the losses). This allows efficient read-wo write access to both the best and worst elements of the buffer.
3

---

Following [36], we use the negative cumulative logprob of the target string conditioned on the prompt as our loss É. For our proxy loss, we use the same loss but evaluated with a local proxy model instead. We consider the attack to be successful if the target string is generated given the prompt under greedy sampling.
3.2 Practical considerations
3.2.1 Scoring prompts with logit-bias and top-S logprobs
Around September 2023, OpenAl removed the ability to calculate logprobs for tokens supplied as part of the prompt. Without this, there was no direct way determine the cumulative logprob of target string conditioned on prompt. Fortunately, the existing features of the API could be combined to reconstruct this value, albeit at a higher cost. We describe the approach we used reconstruct the logprobs, which is similar to the technique proposed in [23]. in Appendix [B] This is the method we used for our OpenAl harmful string results. Later, in March 2024, OpenAl FurnArrurther updated their API so that the logit t bias parameter does not affect the tokens returned by top logprobs. As of
May 2024, it still possible to infer logprobs using the binary search procedure of [23], although the resulting attack will be significantly more expensive.
3.2.2 Short-circuiting the loss
The method described previously calculates the cumulative logprob of target sequence conditioned on prompt by iteratively computing each token's 's contribution to the total. In practice, we exit the computation of the cumulative logprob early if we know it is already sufficiently small. This was the main motivation for the introduction of the buffer. Because unexplored prompts seen so far, we know that any prompt with a loss greater than ((bworst) will be discarded. In practice, find this optimization reduces the total cost of the attack by approximately 30%.
maintain a buffer of the B best
3.2.3 Choosing better initial prompt
In Algorithm D we initialize buffer with uniform random m-token prompts. However, in practice, we found it is better to initialize the buffer with a prompt that is designed specifically to elicit the target string. In particular, sequence length allows, truncating on the left, to be an effective choice for the initial prompt. This prompt immediately produces the target string (without needing to run Algorithm [ for 28% of the strings in harmful strings when m 20. We perform an ablation study of this initialization technique in Section 4.3
found that simply repeating the target string as many times as the
3.3 Proxy-free query-based attacks
The attacks we described so far rely on a local proxy model to guide the adversarial search. As will see, such proxies may be available even if there is good surrogate model for transfer-only attacks. Yet, there are also settings where an attacker will not have access to good proxy models. In this section, we explore the possibility of pure query-based attacks on language models.
We start from the observation that in existing optimization attacks such as GCG, the model gradient provides a rather weak signal (this is why GCG combines gradients with greedy search). We can thus build a simple query-only attack by ignoring the gradient entirely; this leads to a purely greedy attack that samples random token replacements and queries the target's 's loss to check if progress has been made. However, since the white-box GCG attack is already quite costly, the additional overhead from foregoing the gradient information can be prohibitive.
Therefore, we introduce further optimization GCG, which empirically reduces the number of model queries by factor of 2x. This optimization may be of independent interest. Our attack variant differs from GCG as follows: in the original GCG, each attack iteration computes the loss for B candidates, each obtained by replacing the token in one random position of the suffix. Thus, for a suffix of length I, GCG tries an average of B/l tokens in each position. We instead focus our search on single position of the adversarial suffix. Crucially, instead of choosing this position at random as in AutoPrompt, we first try a single token replacement each position, and then write down the position where this replacement reduced the loss the most. We then try B' additional token

---

replacements for just that one position. In practice, we can set B' < B without affecting the attack success rate.
4 Evaluation
We now evaluate four aspects of our attack:
1. Section 4.1 we evaluate success rate of modified GCG on open-source models, allowing to compare to the white-box attack success rates as a baseline.
2. In Section 4.33 we evaluate how well GCQ able to cause production language models like gpt-3.5-turde turbo emit harmful strings, something that transfer attacks alone cannot achieve.
we evaluate the effectiveness of the proxy- attack described in Section 3.3 4. Finally, in Section 4.3 we develop attacks that fool the OpenAl content moderation model; these attacks test our ability to exploit models without a transfer prior.
3. In Section 4.4
4.1 Harmful strings for open models
We give transfer results for aligned open source models using GCG. Unlike the transfer results in (6]. maintain query access the target model, but replace the model gradients with the gradients of proxy model. We tuned the parameters to maximize the attack success rate within our compute budget, since
not limited by OpenAl pricing. We used batch size of 512 and maximum number of iterations of 500. This corresponds to nearly 400 times more queries than we allow for the closed models in Section
First, to establish baseline, we report results white-box attacks on Vicuna version 1.3 which is fine-tuned from Llama [31] well as Llama Chat [32) in Figure Ia Here that the Vicuna 1.3 models become more difficult to attack as their scale increases, and the smallest Llama model is significantly more resistant than even the largest Vicuna model.
LB
Proxy - Target
Chmnulatve atack vicess 188
Vicuna 138
BB -
0.6
Vicuna 1.3 338
Liama 7B
35B
138
13B -> 13B
13B - 33B
Cummlaive
0.2
0.2
33B
0.0
0.0
500
Reraaions
Ineaaioss
White-hox attacks
Transfer attacks
Harmful strings for open models. We show white-box results in (a), where we see Llama-2 more robust than Vicuna. (b). we show transfer attacks within the Vicuna 1.3 model family, where we see that transfer attacks are most successful when the models are of similar size.
Figure 1:
We give results for transfer between scales within the Vicuma 1.3 model family in Figure [bb Interest- ingly, we find that the 7B model transfers poorly to larger scales, while there is little loss transferring 13B to 33B. On the other hand, 13B transfers poorly to 7B. This suggests that the 13B and 33B models are more similar to each other than they are to 7B.
4.2 Comparison to other attacks
For the sake comparison, we modify AutoDAN 19 to perform the harmful strings attack in the same setting experiments in Section 4. We include these results to demonstrate that harmful string are more difficult to elicit than jailbreaks, and that even highly effectively jailbreaking attacks not automatically able to perform harmful string attacks.
In AutoDAN's original setting, a jailbreaking attack is considered successful if the model generates one of a specific set of unwanted strings (c.g. "I'm sorry", "As an AI"). For hamful strings, the attack
5

---

Table 1: Comparison of various attacks in the harmful string setting
Method
Proxy model
Target model
Success rate
GCG Pure Transfer 36
Vicuna 1.3 13B
Vicuna 1.3 7B
0.000
GCQ (ours)
Vicuna 1.3 13B
Vicuna 1.3 7B
0.388
GCG Pure Transfer
36
Vicuna 1.3 7B
Vicuna 1.3 13B
0.000
GCG Pure Transfer
36
Vicuna 1.3 7B
Mistral 7B Instruct v0.3
0.000
GCG Pure Transfer
36
Vicuna 1.3 7B
Gemma 2B
0.000
GCQ (ours)
Vicuna 1.3 7B
Vicuna 1.3 7B
0.791
AutoDAN GA
19
N/A N/A
Vicuna 1.3 7B
0.002
AutoDAN HGA 19
Vicuna 1.3 7B
0.000
is successful only if the generation exactly matches the desired target string. Since the loss used by AutoDAN is the same as in GCQ (probability of generating the target string). we leave the loss unchanged. In this experiment, using the default repository parameters AutoDAN scored 1/574 and 0/574 in GA and HGA mode respectively, despite using much longer adversarial suffixes (around 70 tokens) compared GCQ (20 tokens). In terms of query usage, the default parameters of AutoDAN correspond to about 128 iterations of GCQ.
We also evaluate GCG in the pure transfer setting of [36). In this setting, optimize the prompt purely against the proxy model, then evaluate the final string using the target model. We show the results in Table In general, the low numbers for other attacks highlight how difficult it is to elicit specific harmful strings from models with a low degree of access.
4.3 Harmful strings for GPT-3.5 Turbo
We report results attacking the OpenAl text-completion model gpt-3. w..--urro-inssruct---e using GCQ. For parameters, we used sequence length 20, batch size 32, proxy batch size 8192, and buffer size 128. We used the harmful string dataset proposed in [36]. For each target string, we enforced
API usage budget of $1. For proxy model, used Mistral 7B [17). Note that Mistral 7B is base language model which has not been aligned, making it unsuitable as a proxy for pure transfer attack. Using the initialization described in Section 3.2.3] 574 (or about 28%) of the target strings were solved immediately, due to the model's tendency to continue repetitions in its input. Our total attack cost for the 574 strings was $80.
found that 161 out of the
We visualize the trade-off between cost and attack success rate in Figure 2a We note that the attack success rate rises rapidly initially. We are able to achieve attack success rate of 79.6% after spending at most 10 cents on each target. This number rises to 86.0% we raise the budget to 20 cents per target.
Cumoletive attick sio0es pile
0.8
0.8
0.6
0.6
0.4
Cumulative
Q-GC
Q-GC Iniiaaization only
Inkidizzaionn only
0.0
0.2
0.4
0.5
0.8
10
20
Cost (USD))
Tterations
(a) ASR Cost (USD)
(b) ASR vs number of iterations
Figure 2: Attack success rate at generating harmful strings GPT-3.5 Turbo, function of cost and iterations.
We also plot the trade-off between the number of iterations and the attack success rate in Figure 2b The number of iterations corresponds to the amount of compute spent evaluating the proxy loss. This scales separately from cost because the cost of evaluating the loss using the API scales super-lincarly

---

with the length of the target string, as we describe in Section 3.2.1 while the compute required to evaluate the proxy loss remains constant. Additionally, the short-cireuitin of the loss described in Section 3.2.2 can cause the cost of the loss evaluations to fluctuate unpredictably.
Analysis of target length. We note that the attack success rates reported above are highly dependent on the length of the target string. We plot this interaction in Figure 3 which shows that our attack success rate drops dramatically as the length of the target string approaches and exceeds the length of the prompt. In fact, our success rate for target strings with 20 tokens or fewer is 97.9%. There are two possible reasons for this drop in success rate: (1) our initialization becomes much weaker if we cannot fit even one copy of the target in the prompt, and (2) we may not have enough degrees of freedom to encode the target string.
1.0
Attack success rate
0.8
0.6
0.4
0.2
Prompt length
0.0
15
25
Target string length (tokens)
Figure 3: Tradeoff between attack success rate and target string length for 20 token prompt. Attacks succeed almost always when shorter than the adversarial prompt, and infrequently when longer.
To demonstrate that this effect is indeed due to the length of the prompt, we ran the optimization a second time for the 39 previously failed prompts with length greater than 20 tokens using 40 token prompt, which is long enough to fit any string from harmful strings. Since doubling the prompt length roughly doubles the cost per query, we upped the budget per target to S2. With these settings, we achieved 100% attack success rate with mean cost of S0.41 per target. This suggests that longer target strings can be reliably elicited using proportionally longer prompts.
Analysis of initialization. To demonstrate the value of our initialization scheme, we perform an ablation where we instead use a random initializa- tion. We reran our experiment for the first 20 strings from harmful strings, and in this setting, the attack was only successful only twice. This suggests that currently, good initialization is crucial for our op- timization succeed in the low-cost regime.
0.8
GCG
Cumulative success rale
Ours (white-box)
0.6
Ours (black-box)
0.4
0.2
4.4 Proxy-free harmful strings for open models
We evaluate the original white-box GCG attack, our optimized variant, and our optimized query-only vari- ant from Section 3.3 on the task of eliciting harmful strings from Vicuna 7B. For each attack, report cu- mulative success rate as a function of the number of attack queries to the target model's loss (in setting where we only have access to logprobs and logit-bias, can the technique from Section 3.2.1 to com- pute the loss using black-box queries).
0
10000 20000 30000 40000 50000
Number of loss queries
Figure Our optimizations the GCG at- tack require about 2x fewer loss queries to reach the same attack success rate. When we remove the gradient information entirely to obtain fully black-bor -ox attack, we still out- perform the original GCG by about 30%.
Figure displays the result of this experiment. Our optimized variant of GCG is approximately 2x more query-efficient than the original attack, when gradients are available. When we sample token replacements completely at random, our fully black-box attack still outperforms the original GCG by
7

---

about 30%. Overall, this experiment suggests that black-box query-only attacks on language models can be practical for eliciting targeted strings.
4.5 Proxy-free attack on OpenAl text moxetttoderttioo-007
One application of language models aims not to generate new content, but to classify existing content. One of the most widely deployed NLP classification domains is that of content moderation, which detects whether any given input is abusive, harmful, or otherwise undesirable. In this section, we evaluate the ability of our attacks to fool content moderation classifiers.
Specifically, we target the OpenAl content moderation model text sort-mmddratioo-O00 which OpenAl's "most robust moderation model to-date" 244] The content moderation API allows submit a string and receive a list of flags and scores corresponding to various categories of harmful content. The scores all in the range [0, 1] and the flags are booleans which True when the corresponding score is deemed too high and False otherwise. The threshold for the flags is not necessarily consistent across categories.
We demonstrate evasion of the OpenAl content moderation endpoint by appending an adversarially crafted suffix harmful text. We consider the attack successful if the resulting string is not flagged for any violations. As surrogate for this objective, we use the sum of the scores as our loss. This means do not need to know what the category thresholds for each flag are, which is useful as they are not published online and may be subject to change.
As of February 2024, OpenAl does not charge for usage of the content moderation API, so we report cost terms of API requests, which are rate-limited. For our evaluation, we use the harmful strings dataset (36). Of the 574 strings in the dataset, 197 of them (or around 34%) are not flagged by the content moderation API when sent without suffix. We set our batch size to 32 to match the max batch size of the API. We report results for suffixes of and 20 tokens and for both nonuniversal and universal attacks.
Universal attacks. In the universal attack, our goal is to produce suffix that will prevent any string from being flagged when the suffix is appended. To achieve this. randomly shuffle the harmful strings and select a training set of 20 strings. The remaining 554 strings serve as the validation We extend our loss handle multiple strings by taking the average loss over the strings. The
universal attack is more difficult than the nonuniversal attack for two reasons: (/) each evaluation of the loss is more expensive by factor equal to the training set size (this is why use small training set) and (2) the universal attack must generalize to unseen strings.
For 20 token suffixes, our universal attack achieves 99.2% attack success rate strings from the validation set, after 100 iterations (2,000 requests). We show learning curves across the duration of training to demonstrate the tradeoff between the number of queries and attack success rate in Figure 5b
Atbek gicesseat
0.9
0.7
Cumudaii eaubbbbee stack
0,8
0.6
0.5
Tran
0.6
Trin
04
0.5
5000
10000
19000
20000 25000
50000 19000
40000
250
500
750
1000
1240
1500
1750
2000
Namber roguciis
Number
(a) 5 token suflix
(b) 10 token suffix
Figure 5: Universal content moderation attack success rate function of the number of requests for 5 and 20 token suffixes.
For 5 token suffixes, our universal attack achieves 94.8% attack success rate strings from the validation set, after 2,000 iterations (40,000 requests). We show the corresponding learning curves in Figure 5a

---

Nonuniversal attacks. In nonuniversal attack, we are given specific string which we wish not to be flagged. We then craft an adversarial suffix specifically for this string in order to fool the content moderator. We show the tradeoff between the maximum number of requests to the API and the attack success rate in Figure 6a For token suffixes, we find 83.8% of the strings receive no flags after 10 iterations of GCQ. For 20 token suffixes, that number rises to 91.4%.
4.6 Proxy-free attack on Llama Guard 7B
We attack the Llama Guard 7B content moderation model the same setting as our nonuniversal OpenAl content moderation experiments. We show the results in Figure After 320 queries, the cumulative attack success rates for 5 and 20 tokens are 59% and 87% respectively, compared to 84% and 91 % for OpenAI, and the gap between Llama Guard and OpenAl narrows with further iterations.
Cumulaivve attckk aucossoril
1.0
pate
0.8
0,8
20 Token Prompt
Token Prompt
0.6
20 Token Sulfix
0.6
No Altack
Cummlative
5 Token Sulfi
0.4
0.2
11
10
Number of requests
Number of queries
(a) text motd-mmtioooooono000
(b) Llama Guard 7B
Figure 6: Nonuniversal content moderation attacks reach nearly 100% success rate with a moderate number of queries. Note that each OpenAl request corresponds to 32 queries.
Conclusion
In order be able to deploy language models potentially adversarial situations, they must be robust and correctly handle inputs that have been specifically crafted to induce failures. This paper has shown how to practically apply query-based adversarial attacks to language models in a way that effective and efficient. The practicality of these attacks limits the types of defenses that can reasonably be expected to work. particular, defenses that rely exclusively on breaking transferability will not be effective. Additionally, because our attack makes queries during the generation process, we are able to succeed at coercing models into emitting specific harmful done with transfer-only attacks.
that cannot be
Although the attack we present may be used for harm, we ultimately hope that our results will inspire machine learning practitioners to treat language models with caution and prompt further research into robustness and safety for language models.
Future work. While we have succeeded our goal of generating adversarial examples by querying remote model, we have also shown that current NI.P attacks are still relatively weak, compared to their vision counterparts. For any given harmful string, we have found that initializing with certain prompts can significantly increase attack success rates, while initializing with random prompts can make the attack substantially less effective. This is in contrast to the field of computer vision, where the initial adversarial perturbation barely impacts the success rate of the attack, and running the attack with different random seeds usually improves attack success rate by just few percent.
As result, we still believe there is significant potential for improving NLP adversarial example generation methods in both white and black-box settings.
Acknowledgements
We are grateful to Andreas Terzis for comments on early drafts of this paper. supported by the NSF Graduate Research Fellowship Program. This research supported by the Center for AI

---

Safety Compute Cluster. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors.
References
[1] Maksym Andriuscchenko. Adversarial attacks on GPT-4 via simple random search. 2023, URL https://www andr hwwp//cwhwwaadiuuuucchhhhhhnnnnn mem ggtv pdf
[2] Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking leading safety-aligned Ilms with simple adaptive attacks. arXiv preprint arXiv:2404.02I/: 02550, 2024.
[3] Luke Bailey, Euan Ong, Staart Russell, and Scott Emmons. Image hijacks: Adversarial images can control generative models at runtime. arXiv preprint arXiv:2309.00236, 2023.
[4] Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In European Conference on Machine Learning and Knowledge Discovery in Databases, pages 387-402. Springer, 2013.
[5] Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision- adversarial attacks: Reliable attacks against black-box machine learning models. arXiv preprint arXiv: 2017.
[6] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), 2017.
[7] Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, et al. Are aligned neural networks adversarially aligned? arXiv preprint arXiv:2306. 15447, 2023.
[8] Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.0841 2023.
[9] Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pages 15-26,
2017.
[10] Shuyu Cheng, Yinpeng Dong, Tianyu Pang. Hang Su, and Jun Zhu Improving black-box adversarial attacks with transfer-bace based prior. Advances in neural information processing systems, 32, 2019.
[11] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90% ChatGPT quality, March 2023. URL
hutpe:///aass.orr//loo//22222003 -00-000200000000--mumam/aaaa
[12] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. In International Conference on Learning Representations, 2015.
[13] Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you've ve signed up for: Compromising real-world Imm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop Artificial Intelligence and Security, pages 79-90, 2023.
[14] Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. Gradient-basee adversarial attacks against transformers. arXiv preprint arXiv:2104. 13733, 2021.
[15] Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. Catastrophic jailbreak of open-source LLMs via exploiting generation. arXiv preprint arXiv:2310.06998,, 2023.
[16] Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline de- fenses for adversarial attacks against aligned language models, arXiv preprint arX1v:2309.0061 2023.
10

---

[17] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. arXiv preprint arXiv:2310006825, 2023.
[18]
Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. TextBugger: Generating adversarial text against real-world applications. arXiv preprint arXiv: 18/ 2.05271, 2018.
[19] Xiaogeng Liu, Nan Xu. Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451, 2023.
[20] Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. arXiv preprint arXiv:161 1.02770, 2016.
[21] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: wee.l170.6000000 2017.
[22] Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box Ilms automatically. arXiv preprint orXiv:2312.022 19, 2023.
[23] John X Morris, Wenting Zhao, Justin Chiu, Vitaly Shmatikov, and Alexander M Rush. Language model inversion. arXiv preprint arXiv:2311. 131477, 2023.
[24] OpenAL. New embedding models and API updates, 2024. URL new-eebbd models aacidells-nnd-aa
and- seidd-appuupddeee
[25]
Shuyin Ouyang, Jie M Zhang. Mark Harman, and Meng Wang. LLM is like box of chocolates: the non-determinism of ChatGPT in code generation. arXiv preprint arXiv:2308.022: 02882, 2023.
[26] Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learn- ing: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv:1605.07277 2016.
[27] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[28] Taylor Shin, Yasaman Razeghi, Robert I. Logan IV, Eric Wallace, and Sameer Singh. Auto- Prompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010. 15980, 2020.
[29]
Chawin Sitawarin, Norman Mu, David Wagner, and Alexandre Araujo. Pal: Proxy-guided black-box attack on large language models. arXiv preprint arXiv:2402.0996 2024.
[30] Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Good- fellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
[31] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.1 13971, 2023.
[32] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. LLaMA 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.099:: 00928,998 2023.
[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017.
[34] Zheng-Xin Yong, Cristina Menghini, and Stephen H Bach. Low-resource languages jailbreak GPT-4. arXiv preprint arXiv:2310.02446, 2023.
[35] Jiahao Yu, Xingwei Lin, and Xinyu Xing. GPTfuzzer: Red teaming large language models with auto-generr auto-generated jailbreak prompts. arXiv preprint arXiv:2309. 10253, 2023.
[36] Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint 1504533307.5044, 2023.
11

---

A Compute resources
For experiments in Section 4.1 we used between and 8 A100 GPUs on a single node, The experiments took several days, although we did not have perfect utilization during that period. For our other experiments, we used single A40 for several days.
B OpenAI logprob inference via logit bias and top-5 logprobs
As of March 2024, the OpenAI API does not allow the logit_ parameter to affect the list of tokens returned in top_logprobs This renders the following technique obsolete (at least for OpenAI models). We include it here for completeness.
As of February 2024, the OpenAI API supports returning the top-5 logprobs of each sampled token. By itself, this feature is not very useful for our purposes, since there is no guarantee that the tokens of desired target string will be among the top-5. However, the API also supports specifying bias vector to add to the logits of the model before the application of the log-softmax. This permits us to "boost" an arbitrary token into the top-5, where
then read its logprob. Of course, the logprob we read will not be the true logprob of the token, because will have been distorted by the bias we applied. We can apply the following correction to recover the true logprob
Pbiased
Ptrue
bias (1 Polased) + Polased
The remaining challenge to choose an appropriate bias. If the bias too large, Phiased very close to 1, which causes a loss in accuracy due to limited numerical precision. On the other hand, choosing a bias that too low may fail to bring our token interest into the top-5.
In practice, we usually have access to a good estimate Ptrue of Ptrue because we previously computed the score for the parent of the current string, which differs from it by only token. Accordingly, we can set the bias to - log Ptrue which avoids both previously mentioned problems if Ptrue & Purue- If this approach fails, we fall back to binary search to find an appropriate value for bias. However empirically, our initial choice of bias succeeds over 99% of the time during the execution of Algorithm
Unfortunately, the OpenAl API only allows us to specify one logit-bias for an entire generation. This makes it difficult to sample multiple tokens at once, because logit bias that is suitable in one position might fail in another position. To work around this, we can take the first i tokens of the target string and add them the prompt in order control the bias of the (i + 1)th token of the target string. This comes with the downside of significantly increasing the cost to score particular prompt: If the prompt and target have p and tokens, respectively, then it would cost pt prompt tokens and t(t + 2)/2 completion tokens to score the pair (p, t).
C
Tokenization concerns
When evaluating the loss, it is tempting to pass the token sequences directly to the API. However, due to the way lists of token IDs are handled by the API, this can lead to results that are not reproducible with string prompts. For example, it is possible that the tokens found by the optimization ("abc", "def"), but the OpenAl tokenizer will always tokenize the string "abcdef" as ["abed"", "ef"]. This makes it impossible to achieve the intended outcome when passing the prompt as string. To avoid this, we re-tokenize the strings before passing them to the API, to ensure that the API receives a feasible tokenization of the prompt. We did not notice any impact the success rate of Algorithm caused by this re- re-tokenization.
Another concern is that the proxy model may not use the OpenAl tokenizer. Indeed, there large open models which use the OpenAI tokenizer at this time. To work around this, we also re-tokenize the prompts using the proxy model's 's tokenizer when evaluating the proxy loss.
D Defenses
There are many defenses that would effectively mitigate our attack as is, many of which are enu- merated in [16]. Currently, our attack produces adversarial strings containing significant number
12

---

of seemingly random tokens. Thus an input perplexity filter would be effective in detecting the attack. Incorporating techniques bypass perplexity filters, such as those in [9] may give an effective adaptive attack for this defense. Additionally, our attack requires method to estimate the prepobbabiliiies of the model under attack for arbitrary output tokens. We believe effective attacks that work under the stricter black-box setting where bog-rrobabilities cannot be computed is promising direction for future work.
E
OpenAI API Nondeterminism
Prior work has documented nondeterminism in GPT-3.5 Turbo and GPT-4 [1. We also observe nondeterminism in GPT-3.5 Turbo Instruct. To be more precise, we observe that the logprobs of individual tokens are not stable over time, even when the seed parameter is held fixed. As consequence, generations from GPT-3.5 Turbo Instruct are not always reproducible even when the prompt and all sampling parameters are held fixed, and the temperature set to O. We do not know the exact cause of this nondeterminism. This poses at least two problems for our approach.
First, even if are able find prompt that generates the target string under greedy sampling, do not know how reliably it will do so the future. To address this, we re-evaluate all the solutions once and report this re-evaluation number in Appendix E Second, the scores that obtain actually samples from some random process. Ideally, at each iteration, we would like to choose the prompt with the lowest expected loss. To give some indication of the variance of the process, we plot a histogram of the loss of a particular prompt and target string pair sampled 1,000 times in Figure 7 We find that the sample standard deviation of the loss is 0.068. We estimate that our numerical estimation should be accurate to at least three decimal places, so the variation in the results due the API itself. In comparison, the difference between the best and worst elements of the buffer is typically at least 3, although the gap can narrow when very little progress is being made.
600
500
Count
400
300
-00.4
-40.3
-40.2
-00.1
-40.0
Cumulative logprob
Figure 7: Histogram of cumulative logprob of fixed token target given fixed 20 token prompt, sampled 1,000 times.
We also found that the OpenAl content moderation API is mondeterministic. We randomly chose 20 token input, and sampled its maximum category score 1000 times, observing mean of 0.02 with standard deviation 4 x 10-4, Because the noise we observed was relatively small in both cases, we decided not to implement any mitigation for nondeterministic losses during optimization, we expect the single samples to be good estimators of the expected loss values.
Nondeterminism evaluation. To quantify the degree of nondeterminism in our results, we checked each solution an additional time. We found that 519 (about 90%) of the prompts successfully produced the target string a second time. This suggests that a randomly selected prompt will on average reproduce around 90% of the time when queried many times. We find this reproduction rate acceptable and leave the question of algorithmically improving the reproduction rate to future work.
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes), [No]. [NA]-
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatiin right after your answer (even for NA).
The checklist answers an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] " is perfectly acceptable to answer "[No] provided proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The claims made in the abstract are supported by results in Sections 4.1 to 4.3 4.5 4.6
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
14

---

Justification: one major limitation of the attack is presented in Section 3.2.1 It is discussed in detail in Appendix B Additionally. the algorithms reliance on good initialization is stressed in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
We do not include any theoretical claims.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We give the full algorithm in Algorithm There are also number of practical concerns when implementing the algorithm, which we detail Section 3.2 Further details useful for reproduction given in Appendices and
15

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The only dataset we require already Harmful Strings from (36 which already open. We will include our code in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ htaips.cc/publcc/uuiiie
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
16

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: (Ys]]
We describe all hyperparameters (in particular, the batch size) for all of experiments.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not give error bars for our closed model results because the models inherently nondeterministic, as we describe in Appendix E We run some limited experiments in Appendix [ to estimate the degree of nondeterminism, but since the mechanism of the nondeterminism is unknown, it is difficult to include error bars for our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We list compute resources used in Appendix A
17

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httppttt//tuuuppps.eeo http://urrr....c.p..aaaacaacaaaa
Answer: [Yes]
Justification: Although we do present attack against OpenAl's production language models, we do so after disclosing the vulnerability to OpenAl and receiving their consent to publish this work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss negative and positive impacts briefly in Section 5
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
18

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not release any models or datasets. For discussion of the algorithm itself, see the broader impacts.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use one existing dataset, Harmful Strings from 33 which is cited in our work. We also use several open source language models include models from the Llama, Vicuna, and Mistral model families, all which are cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papprswii thoode com dataeets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
19

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not use human subjects or crowdsoureing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjeets
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We do use human subjects crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20


Reviews:
[{"review_id": "5Tx5RFaxjY", "reviewer": "Reviewer_A7Jt", "summary": "This paper delves into the topic of adversarial examples and prompt injection attacks on aligned language models. A new strategy (GCQ) for black-box adversarial attacks is proposed which does not need access to a surrogate model, but only uses black-box access to the target model. This strategy is an extension of GCG, the current state-of-the-art attack. The proposed method works against both generative models and safety classifiers, as the experiments show.", "questions": "How does the GCQ attack perform against existing defenses?", "limitations": "Ok", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The experimental protocol covers a range of setups, including white-box, black-box, and universal attack.\n- The main novelty of the paper is the no-proxy version of GCQ.\n- The proposed attack GCQ shows good practical performance, in both white- and black-box setups, when compared to the existing state-of-the-art attack GCG and AutoDAN.\n- Implementation included.", "weaknesses": "Significance:\n- It would be great to include more attack baselines, e.g. AutoDAN attack from [Zhu et al., 2023].\n- One of the most relevant qualities of the method is the independence of surrogate models. It seems relevant to compare against other attacks from the same category, e.g., PAL (cited as [28] by the paper, but not used as baseline).\n- It also makes sense to include the GCQ without proxy access in Tab. 1.\n- The topic of performance of the attack against existing defenses is not covered.\n\nNovelty:\n- For the GCQ version that still relies on a surrogate, the changes w.r.t. GCG seem minor.\n\nMinor:\n- Additional proofreading seems necessary.\n- Sec. 4.2 and 4.6 are missing from the outline at the beginning of Sec. 4.\n- L217 \"hamful\" -> \"harmful\"\n\n## References\n\n- [Zhu et al., 2023] Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Furong Huang, Tong Sun. AutoDAN: Automatic and Interpretable Adversarial Attacks on Large Language Models. https://openreview.net/forum?id=ZuZujQ9LJV", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for carefully reading our work and providing useful and constructive feedback. We hope that the following will address the points raised and are happy to further discuss during the discussion period.\n\n## Comparison to other attacks\n\nUnfortunately, there is a lack of existing works to compare to when it comes to the harmful string attack. This attack was originally proposed in GCG, where it was demonstrated against open models, but since then, it has received little attention. This may be because there was no known method to cheaply compute the loss required for the harmful string attack under the constraints of popular APIs.\n\nRegarding the two suggested attacks:\n\n1. The AutoDAN attack from [Zhu et al., 2023] is focused on jailbreaking under the constraint of an input perplexity filter. They do not evaluate in the setting of harmful strings and their optimization procedure is carefully tailored to the jailbreaking objective. Thus, it is not clear how to modify their method to elicit a specific string. Additionally they do not provide code, which makes the evaluation of their method difficult.\n2. Similarly, PAL [28] focuses on Jailbreaking in the black-box setting. They use a similar collection of tricks to our work (developed independently and concurrently) to make the loss calculation feasible for commercial APIs, although the loss itself is different due to the different attack objective. Since PAL is also based on GCG, running PAL with the objective of GCQ would be very similar to running GCQ itself. Thus it is difficult to find a comparison that makes sense here.\n\nThis underscores the lack of existing work on the harmful string attack. We believe this attack is meaningful and important, as we outline in the general rebuttal, and our hope is to make some initial progress in this direction beyond GCG. \n\nRegarding GQC with no proxy, we will add the results to Table 1 as requested. The numbers for Vicuna 7B were already computed for Section 4.4 so it is a simple matter to copy them over.\n\n## Novelty\n\nAlthough the actual optimization algorithm does require only minor changes to GCG. We would like to suggest that the computation of the loss itself is a significant technical contribution in its own right. To our knowledge, our attack is the only one which is able to elicit exact harmful strings (spanning many tokens) from a proprietary model, which is much more difficult than jailbreaking or eliciting harmful behaviors. The key to this ability is a procedure to quickly and inexpensively score the conditional probability of an entire sequence of generated tokens. Indeed one of the primary changes we make to the optimization algorithm (the introduction of the prompt buffer) is purely to help reduce the cost of the loss calculation.\n\n## Defenses\nWe will add a section discussing defenses to the paper. In general, we expect defenses (e.g. input filtering or output filtering) to be effective against our attack. Since the attack is already quite difficult, raising the difficulty with defenses may make the attack infeasible unless new techniques are developed to further strengthen the attack.\n\n## Presentation\nWe will carefully proofread the paper and revise the outline to include all major sections of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L6R9wCsUEu", "reviewer": "Reviewer_ReRa", "summary": "The paper presents a novel query-based attack method designed to generate adversarial examples that induce harmful outputs in aligned language models. Building on the GCG attack by Zou et al. (2023), this method employs a query-based strategy that eliminates the need for transferability, resulting in a significantly higher success rate. The effectiveness of the attack is demonstrated on GPT-3.5, even in black-box scenarios. Additionally, the paper details universal and prompt-specific attack techniques that effectively circumvent OpenAI’s safety classifier, achieving near-perfect evasion rates.", "questions": "+ Can the authors explain why the attack success rate of AutoDan are so low in the experiments reported in Table 1?\n\n+ Can the authors provide any intuition about how to defend against an attack like this?", "limitations": "The limitations have been addressed appropriately in different sections of the paper, including, Section 3.2.1, and Appendix B, for example.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "+ Despite the work from Zou et al. (2023) disclosing the vulnerability of production LLMs to adversarial examples capable of producing harmful strings and behavior, this paper overcomes some limitations related to attack transferability. The authors introduce a novel approach that leverages the API of production LLMs, like GPT-3.5, significantly improving the success rate of attacks and demonstrating the fragile behavior of LLMs in the presence of a smart adversary.\n\n+ The paper is well-written, clearly describing each step of the attack, including strategies to bypass OpenAI’s API restrictions. For instance, the method for calculating the log probabilities is particularly smart and well-elaborated.\n\n+ Overall, the experimental evaluation is comprehensive and provides valuable insights. For example, the results in Table 1 shows the limitations of previous attacks like CGC and AutoDan, whereas the attacks proposed by the authors are very effective both in white-box and transfer attack settings. \n\n+ The authors also analyzed the effect of universal attacks, revealing interesting results in terms of the trade-off between the budget/cost and the effectiveness of the attack.", "weaknesses": "+ Although it is a relevant attack targeting a commercial system like ChatGPT 3.5, the scope of the attacks is somewhat limited as it requires API access to provide the logprob and only works with GCG. Nevertheless, I think that the paper clearly shows the risks of providing access to the logprobs in the API. \n\n+ The paper only focuses on the attacker’s perspective. Defenses or possible mitigations are not properly discussed in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their helpful comments! We hope that the following will address any residual concerns and we are happy to further discuss during the discussion period.\n\n## Performance of AutoDAN\n\nThe reason that AutoDAN performs so poorly is because it is being evaluated in an “unfair” way (out of necessity). The original AutoDAN code is designed to perform jailbreaking which is coded in roughly the following way: {the attack is successful if the model does not generate any of the following strings: (“No”, “Sorry”, “As an AI”, …]}. We replaced this objective with the objectives we used in GCQ: {The attack is successful if the model generates exactly “<example harmful string>”}. Because the harmful string objective is much more restrictive than the jailbreaking objective, the optimization of AutoDAN did not perform well. Of course this is reasonable since AutoDAN was not designed to work in this setting.\n\nWe will adjust the presentation of this result to stress that the evaluation of AutoDAN is in a new setting where it cannot be expected to succeed. We hope that including this result will underscore that, in general, jailbreaking methods may not generalize to more difficult tasks, such as harmful strings.\n\n## Defenses\n\nIn terms of defenses, there are a variety of techniques that target various aspects of the text generation pipeline:\n\nThe prompt can be filtered for e.g. high perplexity inputs (proposed in [1]). The attack we describe does often produce seemingly random sequences of tokens, so we would need to add additional functionality (for example, perplexity regularization) to evade this defense.\nThe model generations can be filtered for harmful content by a second model. However as we show, it is possible to find strings that bypass popular content moderation models. Thus one could imagine a two-stage attack, where we optimize to generate a harmful string which also passes through the content filter.\nA different kind of defense would be to rate-limit queries that are very similar to previous queries (for example, if the API receives thousands of strings which all differ from each other by a small number of tokens). However, this has the potential to harm honest users, so it cannot be too harsh.\nOf course there is also the option of restricting access to logprobs, logit_bias or both as you note.\nMany more possible defenses are listed in [1].\n\nOf course, the best defense will always be a multilayered approach. Each layer of defense makes the attack much more difficult, as it is difficult to evade many defenses at the same time. We will add a section to the paper that discusses possible defenses in the spirit of what we've written here. \n\n[1] Jain, Neel, et al. \"Baseline defenses for adversarial attacks against aligned language models.\" arXiv preprint arXiv:2309.00614 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4cVbVXVz4L", "reviewer": "Reviewer_CNWy", "summary": "This paper modifies GCG, an attack on LLMs to elicit harmful responses, to create a query-based black-box attack with two primary goals: 1) Enable targeted attacks that are not possible with simple transfer-based attacks and 2) Enable attacks to still occur when no feasible surrogate model exists. To modify GCG, the key change is that the authors maintain a min-max heap of possible adversarial suffix candidates, searching for word replacements on the best currently known suffix candidate. The authors also experiment with prompt initialization, finding that including the target as many times as fits to be useful. Finally, as a query-optimizer in a proxy-free setting with random tokens instead of gradients, the authors experiment with trying a word replacement once in each position, and then taking the best position and trying more word replacements there. The authors find they are able to generate targeted attacks, which transfer attacks cannot, and can attack real-world models with proxy-free attacks.", "questions": "How does the attack perform against Andriushchenko et al. and Lapid et al.?\nHow does the attack do against transfer attacks across different families of LLMs?", "limitations": "Adequately addressed.", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "Strengths include 1) enabling targeted attacks to elicit specific phrases, 2) enabling proxy-free attacks on real-world models without a surrogate, and 3) including optimizations to manage the query budget in the proxy free setting. Thus, the paper makes strides towards more practical attacks with adversaries with less power. The authors have also found their initialization strategy to help improve attack quality.", "weaknesses": "The primary weaknesses of this paper include a lack of comparisons against other black-box attacks (e.g., Andriushchenko et al., \"Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks\", and Lapid et al., \"Open Sesame! Universal Black Box Jailbreaking of Large Language Models\") on the proxy-free attacks. This also limits the technical novelty, as the attack as mostly a series of small changes to GCG. The other weakness is that this paper does not evaluate a wide variety of different types of LLMs - one of the themes is that the proposed attack is useful in settings where there is no good surrogate, so it would be nice to see how the attack does against transfer attacks across different families of LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for carefully assessing our work. We hope the following will address all outstanding concerns and we are happy to further discuss during the discussion period. \n\n## Comparisons against other black-box attacks\nUnfortunately, there is a dearth of works to compare to when it comes to eliciting exact harmful strings. The attack was originally proposed in GCG, where it was demonstrated against open models, but since then it has received little attention. This may be because there was no known method to cheaply compute the loss required for the harmful string attack under the constraints of popular APIs. \n\nRegarding the two suggested attacks:\n\nAndriushchenko et al. only focuses on controlling the first token generated after the prompt. This is because the first log-probability is directly given by OpenAI’s API. This is sufficient to elicit certain harmful behaviors, but not harmful strings of arbitrary length.\n\nLapid et al. focus on jailbreaking, using a loss based on the embedding similarity to some target phrase, which is typically a short affirmation such as “Sure, here is”. This approach does not adapt well to harmful string targets because of the many local minima in the embedding similarity landscape. \n\nTo demonstrate this, we repurposed GCQ to try to maximize the embedding similarity between an input string and a target string from harmful strings using OpenAI’s text-embedding-ada-002 model. In each of the 10 cases, we tried, the search terminated in a local minima, finding a string with a similar embedding but not the exact target string itself. (Note that this is much easier than solving the same problem indirectly through the [prompt → generation] mapping of an aligned LLM as needed for a harmful string attack.) On the other hand, the use of the embedding distance makes perfect sense for jailbreaking, where the target semantics are all that matter.\n\nThe difficulty here is that neither of these attacks are designed to operate in our setting. We elaborate on this setting and why it is important in our general rebuttal. \n\n\n## Technical novelty\n\nAlthough the actual optimization algorithm does require only minor changes to GCG. We would like to suggest that the computation of the loss itself is a significant technical contribution in its own right. To our knowledge, our attack is the only one which is able to elicit exact harmful strings (spanning many tokens) from a proprietary model, which is much more difficult than jailbreaking or eliciting harmful behaviors. The key to this ability is a procedure to quickly and inexpensively score the conditional probability of an entire sequence of generated tokens. Indeed one of the primary changes we make to the optimization algorithm (the introduction of the prompt buffer) is purely to help reduce the cost of the loss calculation.\n\n## Transfer between model families\n\nWe evaluate GCG in the pure-transfer setting within the same model family (in Table 1) and find that it performs very poorly (0 ASR) in the harmful string setting. This is in contrast with harmful behaviors, where transfer works quite well. Given the additional difficulty when transferring across model families, we would not expect it to work well in that setting. For example, we find that none of the strings computed for Vicuna 7B transfer to either Mistral 7B Instruct v0.3 or Gemma 2 2B Instruct.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nr7anjfmop", "reviewer": "Reviewer_67w9", "summary": "This paper proposes a query-based adversarial prompt generation method. It eliminates the prior attack's dependence on adversarial transferability and local surrogate models. The attack can evade the OpenAI and Llama Guard safety classifiers with a near 100% success rate.", "questions": "See weaknesses.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "## Originality\n* The proposed improvement is interesting and novel.\n* The related works are generally comprehensive.\n\n## Quality\n* Several practical considerations are included.\n* The breadth of evaluations is good.\n\n## Clarity\n* The first three sections are generally well-written and easy to follow.\n\n## Significance\n* The proposed technique seems simple, but the overall contribution towards query-based attacks is important.", "weaknesses": "## Originality\nI don't have major concerns here. The technical modifications of the existing attack GCG seem simple, but the good results can justify them. It is suggested to summarize and highlight the technical challenges.\n\n## Quality\n\n**Q: The evaluation setting is quite confusing.**\n\n1. The paper was a bit vague regarding the attack's objective. Is it for a general jailbreaking attack (like GCG) or just eliciting the model to repeat a given harmful content? This also makes it hard to understand the attack success rate -- when do we count an attack as successful?\n\n2. The evaluations do not match the major contribution. In Section 4, most of the evaluations are focused on the surrogate-based attack on various settings, including the actual GPT-3.5 model (a side question is why GPT-4 was not included since a black-box attack should work regardless of the underlying model). The actual contribution, the surrogate-free attack, was only evaluated on open models and then a different task of content moderation. The paper did not justify why the surrogate-based attack was not evaluated in the latter settings, and why the surrogate-free attack was not evaluated in the former settings. Ideally, the primary attack is supposed to be evaluated against all involved settings, and the surrogate-based attack is provided as a reference or ablation study.\n\n\n## Clarity\n\n* L33-45. It was a bit confusing when you first called out the surrogate-free property, then switched to a surrogate-based attack, and only finally, the further optimization that removes the surrogate model.\n* L64-66. The high success rates in black-box attacks against classification tasks depend on two factors: the number of queries and the threshold perturbation. Most black-box attacks have 100% success rates at the first few queries but the inputs are either pure noise or heavily perturbed.\n* Figure 1b. Please use different styles to distinguish curves of different groups of proxy or target. Right now, it is very hard to tell what the figure conveys and what we can learn when comparing curves with the same proxy size.\n\n## Significance\nN/A", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for carefully evaluating our work. We apologize for any confusion we may have caused, and will thoroughly review the presentation of our paper to avoid confusing future readers.\n\n## Originality\n\nFollowing your suggestion, we will carefully revise the introduction to highlight the main contributions of our work. In particular, we will highlight the practical challenges that make black-box harmful string attacks difficult to run in practice, since overcoming these obstacles represents a significant part of our technical contribution. \n\n## Attack objective\n\nThe objective of the attack is to find a prompt that causes the model to output a specific string exactly. The attack is thus successful when the model generates exactly the desired string (including capitalization, punctuation, etc.). Thus the task is much more restrictive than jailbreaking. We elaborate on this task in the general rebuttal and we will add additional emphasis to the paper to clarify this point.\n\n[1] Zou, Andy, et al. \"Universal and Transferable Adversarial Attacks on Aligned Language Models.\" (2023).\n\n## Inclusion of GPT-4\n\nThe reason GPT-4 was not included in our results is because it does not support logprobs. Although the attack may still be possible without logprobs, the expense would be beyond our budget.\n\n## Evaluations and contribution\nThe reason for this split is because it is best to use the surrogate when it is available (which is only under certain circumstances). Note that in the language modeling experiments, we are using a base pretrained model as the surrogate. Such surrogates are freely available and thus it makes sense to use them. However the downside is that they do not reveal anything about the alignment of the target black-box model. This is acceptable because we are only using the surrogate in a weak sense: to pre-filter the candidates before they are sent to the black-box model. We find that this weak pre-filter is still better than the position-based pre-filter used in the surrogate-free setting.\nHowever in some settings, such as in our content moderation examples, we may not have access to any kind of surrogate. Thus in this setting we apply the surrogate-free algorithm. Unfortunately we cannot afford to run the surrogate-free attack for closed models due to budget constraints, since we believe it will be much worse (i.e. much more expensive) than the attack using the weak surrogate. On the other hand, we cannot run the weak surrogate algorithm because the pretrained model surrogate does not make sense in the context of content moderation.\n\nYou are correct that this arrangement appears awkward to readers. We will revise the discussion of the surrogate-free attack to make it more clear why the two attacks are applicable in different situations and we will aggregate the results common to both algorithms in Table 1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all of our reviewers for their insightful comments. In the general rebuttal we would like to elaborate a bit on the harmful string attack.

## Harmful string attacks 

In our language modeling results, we focus on the harmful string attack. This attack objective is to get the model to output a specific string exactly (including punctuation and capitalization for example). This attack is separate from jailbreaking or elicitation of harmful behaviors in that the desired outputs are highly specific. Empirically, this makes the attack much more difficult. 

At first glance the harmful string attack may appear contrived, but it is significant for several reasons. First, it's a very straightforward test of model alignment. If a model creator's desire is “don't say X”, making the model say exactly X is a clear violation of those preferences. Second, it is easy to evaluate compared to jailbreaking which often must be manually evaluated by humans. Finally, exact strings can do harm in unique ways. For example a model supporting tools may be fooled into invoking the tool in a specific way which is harmful to the user.

Meta-review: This paper presents a jailbreaking attack that's effective against safety classifiers and LLMs that produce log probs. Rather than trying to generate a harmful output in general, it attempts the narrower goal of generating a specific harmful string. All reviewers are positive about the contributions of this paper. Its greatest limitation is the reliance on log probabilities, which renders it ineffective against some commercial LLMs (notably GPT 4). It's also difficult to compare to some prior work, such as AutoDAN, because the harmful string objective is different and more specific. Finally, the authors do not demonstrate effective defenses; however, demonstrating effective defenses is difficult with LLMs, and many adversarial ML defenses have been shown to be ineffective when faced with an adaptive attacker. Overall, this paper is a positive contribution that better illuminates the attack landscape against LLM safeguards.
Decision: Accept (poster)