Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes
Authors: Xiaomeng Hu
Pin-Yu Chen
Tsung-Yi Ho
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
Jailbreak Detection
AI Alignment and Safety
Primary area: safety_in_machine_learning

Abstract: Large Language Models (LLMs) are becoming a prominent generative AI tool, where the user enters a query and the LLM generates an answer. To reduce harm and misuse, efforts have been made to align these LLMs to human values using advanced training techniques such as Reinforcement Learning from Human Feedback (RLHF). However, recent studies have highlighted the vulnerability of LLMs to adversarial jailbreak attempts aiming at subverting the embedded safety guardrails. To address this challenge, this paper defines and investigates the **Refusal Loss** of LLMs and then proposes a method called **Gradient Cuff** to detect jailbreak attempts. Gradient Cuff exploits the unique properties observed in the refusal loss landscape, including functional values and its smoothness, to design an effective two-step detection strategy. Experimental results on two aligned LLMs (LLaMA-2-7B-Chat and Vicuna-7B-V1.5) and six types of jailbreak attacks (GCG, AutoDAN, PAIR, TAP, Base64, and LRL) show that Gradient Cuff can significantly improve the LLM's rejection capability for malicious jailbreak queries, while maintaining the model's performance for benign user queries by adjusting the detection threshold.

Gradient Cuff: Detecting, Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes
Xiaomeng Hu The Chinese University of Hong Kong Sha Tin, Hong Kong xmhu23@cse. cuhk edu, hk
Pin-Yo Yu Chen
IBM Research New York, USA
pin-yu wwwwchee..ccccece com
Tsung-Yi Ho The Chinese University of Hong Kong Sha Tin, Hong Kong tyho@cse. cuhk. edu. hk
Project Page: TTssSSafeA//Grralenutfa.a.llall Jailberr -aibeeea-Deffesee Live Demo: pinyuchen/Graan Graninnt-raaGGnnnnnee Cuff
Abstract
Large Language Models (LLMs) are becoming prominent generative Al tool, where the user enters a query and the LLM generates an answer. To reduce harm and misuse, efforts have been made to align these LLMs to human values using advanced training techniques such as Reinforcement Learning from Human Feedback (RLHF). However, recent studies have highlighted the vulnerability LLMs to adversarial jailbreak attempts aiming at subverting the embedded safety guardrails. To address this challenge, this paper defines and investigates the Refusal Loss of LLMs and then proposes a method called Gradient Cuff to detect jailbreak attempts. Gradient Cuff exploits the unique properties observed in the refusal loss landscape, including functional values and its smoothness, design effective two-step detection strategy. Experimental results aligned LLMs ALLLMA-2-7B---A 7B-Chat and (icuna-7B-V1.5) and six types of jailbreak attacks (GCG, AutoDAN, PAIR, TAP, Base64, and LRL) show that Gradient Cuff can significantly improve the LLM's rejection capability for malicious jailbreak queries. while maintaining the model's performance for benign user queries by adjusting the detection threshold.
Introduction
With the stupendous success of large language models (LLMs) such as GPT-4 [19], LLaMA-2 23 and Vicuna [32), there is trend to integrate these LLMs into various applications such as ChatGPT and Bing Search. In these applications, LLMs are used as the service backend. The front end of these applications receives the user input query from the interface, encapsulates it into system prompt, and then sends it to the L.MM to get response. With the rapidly increasing social impact of these applications, model alignment and safety assurance to reduce harm and misuse have become significant considerations when developing and deploying LLMs. Methods such as Reinforcement
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Rofise
Viane Raruoeed
TUTHI
Deject
Reject
True
Fello
Fabe
1,000
1
54M
Une
Livepkng Rejections
Boodov Norrs Rejection:
0.00
0.750
Gat barn of snataace
50
0.250
6.58 AUK2AN
PAR
LRL
Awrrage
Figure 1: Overview of Gradient Cuff. (a) introduces an example of jailbreak prompts by presenting a conversation between malicious actors and the Vicuna chatbot. (b) visualizes the refusal loss landscape for malicious queries and benign queries by plotting the interpolation of two random directions in the query embedding with coefficients a and 3 following [33. The refusal loss evaluates the probability that the LLM would not directly reject the input query, and the loss value is computed using Equation 3 See details of how to plot (b) in Appendix A.4 (c) shows the running flow of Gradient Cuff (at top), practical computing examples for refusal loss (at bottom left), and the distributional difference of the gradient norm of refusal loss on benign and malicious queries (hottom right). (d) shows the performance of Gradient Cuff against 6 jailbreak attacks for Vicuna-7B-V See Appendix A.6 for full results.
Learning from Human Feedback (RLHF) have been proven to be an effective training technique to align LLMs with human values 2 13
However, aligned LLMs have been found to be vulnerable to a type of adversarial manipulation known "jailbreak attack". Jailbreak attacks involve maliciously inserting replacing tokens in the user instruction or rewriting it to bypass and circumvent the safety guardrails of aligned LLMs. A notable example is that a jailbroken LLM would be tricked into generating hate speech targeting certain groups of people, as demonstrated in Figure ((a).
have been put into designing algorithms to automati- cally generate jailbreak prompts to help test the robustness of aligned L.MSs. Specifically, GCG [3 one of the carlier works in this area, can successfully jailbreak several LLMs by optimizing an inserted universal adversarial suffix. This finding suggests that the embedded alignment effort in LLMs could be completely broken by the jailbreak attack.
Many red-teaming efforts 33
18
26 30
Since the discovery of jailbreak risks for LLMs, various methods have been explored to defend against jailbreak attacks [12 21 28 [4] and have gained some success in detecting certain types of attacks such as GCG [7 12]. However, in our systematic analysis, existing defenses either fail to be resistant against all types of jailbreak attacks, or have significant detrimental effect on benign queries. PPL. 12 uses an LLM to compute the perplexity of the input user query and filters those with perplexity greater than the threshold. PPI. has been proven to have good detection performance for GCG attacks but does not perform well on jailbreak prompts with good mcaningfulness and fluency [17]. Erase-Check [14] and Self-Reminder (28) behave well on malicious queries but will misclassify many benign user queries, making the defenses overly conservative and impractical.
To alleviate the threats of jailbreak attacks and avoid the aforementioned problems in existing defenses, we propose Gradient Cuff, which detects jailbreak prompts by checking the refusal loss of the input user query and estimating the gradient norm of the loss function. We begin by introducing the concept of refusal loss and showcase the different behaviors of loss function for benign instructions and malicious instructions. A plot of the refusal loss landscape for benign and malicious instructions can be found in Figure D (b). By exploring the landscapes of refusal loss, we find that the refusal loss function for malicious instructions tends to have smaller value and larger gradient norm.

---

We then leverage this unique loss landscape characteristic to propose a two-step jailbreak detection algorithm, which illustrated in Tigure[[ Figure (d) evaluates 6 jailbreak attacks on Vicuna- 7B-V1.5 and shows that our defense can reduce the attack success rate (ASR) averaged over these 6 jailbreaks from 76.7% to 7% on average. We also compare Gradient Cuff to other existing defense methods on Vicuna-7B-V
and LLLMA-2-7B-CHA against these 6 jailbreaks as well as adaptive attacks to demonstrate our defense capabilities.
We summarize our main contributions as follows:
We formalize the concept of refusal loss function of LLMs and explore its smoothness and values of the loss landscapes on benign and malicious queries. The distinct refusal loss characteristics are used in our Gradient Cuff framework to detect jailbreak prompts.
Experiments on 2 aligned LLMs -LLMA-2-7B--AA tacks (GCG, AutoDAN, PAIR, TAP, Base64, and LRL) demonstrate that Gradient Cuff is the only defense algorithm that can attain good jailbreak detection while keeping an acceptable rejection rate on benign queries.
and So-una-B--1.33 and 6 jailbreak at-
We also show that Gradient Cuff is complementary to prompt-engineering based alignment strate- gies. When combined with Gradient Cuff, the performance of Self-Reminder, a system prompt design method (28), can be increased by large margin.
2 Related Work
Jailbreak Attacks. Existing jailbreaks can be roughly divided into feedback- jailbreak attacks and rule-based jailbreak attacks. Feedback-baas jailbreaks utilize the feedback from the target LLM to iteratively update the jailbreak prompt until the model complies with the malicious instruction embedded in the jailbreak prompt. Feedback-based jailbreaks can be further categorized by their access mode to the target LLM. Some feedback-based jailbreak attacks like GCG [33), require white-box access to the target LLM. Specifically, GCG leverages gradients with respect to the one-hot token indicators to find better token choices at each position. Some feedback- -bssed jailbreaks need gray-box access to the target LLM. The typical one is AutoDAN [T7. which employs the target LLM's generative loss of the target response to design the fitness score of the candidate jailbreak prompt to guide further optimization. PAIR 4 and TAP 18 are the representatives of feedback- based jailbreaks which only require black-box access to the target LLM. In PAIR and TAP, there are also two LLMs taking on the attacker role and the evaluator role. At each iteration, the attacker- generated jailbreak prompt would be rated and commented on by the evaluator model according to the target LLM's response to the attack. Next, the attacker would generate new jailbreak prompts based the evaluator's comments, and repeat the above cycle until the jailbreak prompt can get full marks from the evaluator. The only information provided by the target L.M is the response to the jailbreak attack. As for the rule-based jailbreak attacks, we highlight Base64 (26) and Low Resource Language (LRL) [30). Base64 encodes the malicious instruction into base64 format and LRL translates the malicious instruction into the language that is rarely used in the training process of the target LLM, such as German, Swedish, French and Chinese
Jailbreak Defenses. PPL 12 uses an LLM to compute the perplexity of the input query and rejects those with high perplexity. SmoothLLM (21, motivated by randomized smoothing (7], perturbs the original input query to obtain several copies and aggregates the intermediate responses of the target LLM to these perturbed queries to give the final response to the original query. Erase-Check employs a model to check whether the original query any of its erased subsentences is harmful. The query would be rejected if the query or one of its sub-sentences is regarded as harmful by the
safety checker. Another line of work (28 31 27 25 use prompt engincering techniques to defend against jailbreak attacks. Notably, Self- Reminder (28) shows promising results by modifying the system prompt of the target LLM so that the model reminds itself process and respond to the user in the context of being an aligned LLM. Unlike these unsupervised methods, some works like LLaMA-Guare [22] and
22 need to train an extra LLM. LLaMA-Guard trained a LLaMA- model to determine whether the user query or model response contains unsafe content.
Safe-Decoding finetuned the protected LLM on (malicious query, model refusal) pairs to get an expert LLM, and utilized the expert LLM to guide the safety-aware decoding during inference time. We will mainly focus on unsupervised methods as is training- and only need to deploy the protected LLM itself during inference time.
3

---

3 Methodology and Algorithms
Following the overview in Figure I. in this section, we will formalize the concept of Refusal loss function and propose Gradient Cuff jailbreak detection method based the unique loss landscape properties of this function observed between malicious and benign user queries.
3.1 Refusal Loss Function and Landscape Exploration
Current transformer-bbsed LLMs will return different responses to the same query due the random- ness of autoregressive sampling based generation [8] T0 With this randomness, it is an interesting phenomenon that malicious query will sometimes be rejected by the target L.L,, but sometimes be able to bypass the safety guardrail. Based on this observation, for a given LLM Te parameterized with 0, we define the refusal loss function Φe for given input user query æ as below:
Φo (a) = po();; pe (at) (x) JB())
where y represents the response of To to the input user query JB(() is a binary indicator function to determine whether the response triggers a refusal action by the I.I.M. The function Po can be interpreted as the expected rate of getting refusal on the response y from To taking into account the randomness in the decoding process. Therefore, by our definition, the refusal loss function (z) can be interpreted the likelihood of generating non-refusal response to Following SmoothLLM 21 we define (.) as
1. if y contains any jailbreak keyword; JB = (0 otherwise.
For example, J (B) would be 0 if -Suure, here is the python code to ..." and JB()) would be 1 if y ="Sorry, cannot fulfill your request...". We discuss more details about the implementation of the indicator function in Appendix [A.3]
Alternatively, we can view Y = J B()) as random variable obeying the Bernoulli distribution such that
Y I, with probability pa(x)
0. with probability 1 pa (xr)
so that (2) can be interpreted the expected refusal loss:
( = E = (r()).
In practice, since do not have the prior knowledge for po(x), we use the sample mean fa(x) to approximate (x():
1 N
fo(x) =
Σ Y
where {Yg .N. is obtained by running N independent realizations of the random variable Y. In the 7th trial, we query the LLM Ta using x to get the response y/: To(ar), and apply the indicator function (-) on y, get Y; JJB():) Equation (3) can be explained as using the sample mean the random variable to approximate its expected value E[Y].
In general, (x) < 0.5 could be used as a naive detector to reject since Pe(x) can be interpreted as the probability that Ta regards harmful. However, this detector alone only has limited effect against jailbreak attacks, as discussed in Section 4.3 To further explore how this refusal loss can be used improve jailbreak detection, we visualize the refusal loss landscape following the 2-D visualization techniques from [15] in Figure [ (b). From Figure [ (b), we find that the landscape of fo(-) is more precipitous for malicious queries than for benign queries, which implies that fo(.) tends to have large gradient norm if represents malicious query. This observation motivates our proposal of using the gradient norm of fo(-) to detect jailbreak attempts that pass the initial filtering of rejecting when fo (x) < 0.5.

---

3.2 Gradient Norm Estimation
In general, the exact gradient of (0(x)) (or [o())) is infeasible to obtain due to the existence of discrete operations such as applying the J.) function to the generated response, and the possible involvement of black-box evaluation functions (e.g.. Perspective API). We propose to use zeroth order gradient estimation to compute the approximate gradient norm, which is widely used in black-box optimization with only function evaluations (zeroth order information) [3] [6]. Similar gradient estimation techniques were used to generate adversarial examples from black-box models [1 6).
zeroth-order gradient estimator approximates the exact gradient by evaluating and computing the function differences with perturbed continuous inputs. Our first step is to obtain the sentence embedding of z in the embedding space of Ta in Rdd For each text query I' with n words (tokens) in it, it can be embedded into a matrix eø (1) E I where ea (2:); E R denotes the word embedding for the ith word in sentence æ. We define the sentence embedding for x by applying mean pooling to es(2) (z) defined as
mean-pooling(x)
Σ (z))
(4)
With the observation that
mean-pooling(s) + + = n1 E (co(x)) v),
(5)
one can obtain perturbed sentence embedding of x with any perturbation v by equivalently perturb- ing the word embedding of each word in with the same
Based on this definition, we approximate the exact gradient (œ)(2r by (z(),, which is the estimated gradient of (zr). Following 16), we calculate go using the directional derivative approximation
Σ fo(es( + p u.) (())(()))
go (o(x)
U.
where u, is a d dimension random vector drawn from the standard multivariate normal distribution, i.e. Ug N0, I), f is smoothing parameter, denotes the row-wise broadcasting add operation that adds the same vector μ u, to every row in @p(x).
Based the definitions in Equation 3 and Equation (6. we provide probabilistic guarantee below for analyzing the gradient approximation error the true gradient (o(())
Theorem Les || || denote vector norm and assume Vol(z) is L-Lipschitz continuous. With probability at least 1 - 8, the approximation error of V))(T) satisfies
[19)(r) - (
for some > 0, where S 1 N + +) and (p)
This theorem demonstrates that one can reduce the approximation error by taking larger values for N and P. We provide the proof in Appendix A2 Experimental results in Appendix AA.4 and Appendix A.13 also provide empirical evidence to support this theorem by demonstrating the scaling performance of Gradient Cuff with increased total queries.
3.3 Gradient Cuff: Two-step jailbreak detection
With the discussions in Section 3.1 and Section 3.2 we now formally propose Gradient Cuff, two-step jailbreak detection method based on checking the refusal loss and its gradient norm. Our detection procedure is shown in Figure (c). Gradient Cuff can be summarized into two steps:
(Step 1) Sampling-basse based Rejection: In the first step, we reject the user query by checking whether fe(x) 0.5. If true, then rejected, otherwise, x is pushed into Step 2.
l((t) = ((s(t)) means s(t) the infimum of ((t)
5

---

(Step 2) Gradient Norm Rejection: In the second step, we regard I as having jailbreak attempts if the norm of the estimated gradient go (0(2) larger than configurable threshold t, i... (1)) >> t.
Before deploying Gradient Cuff on LLMs, first test it bunch of benign user queries to select a proper threshold / that fulfills the required benign refusal rate (that is, the false positive rate o). We use user-specified value (e.g., 5%) to guide the selection of the threshold so that the total refusal rate on the benign validation dataset Beal won't exceed a.
We summarize our method in Algorithm [ The algorithm is implemented by querying the LLM Te multiple times, each to generate a response for the same input query æ. The total query times to To required to compute fa( and (x)) in Gradient Cuff is at most N (P + 1). To maintain the LLM's efficiency, we also explored the use of batch inference to compute these queries in parallel, thereby reducing the total rumning cost of the LLM. For example, the running time can only be increased by 1.3x when the total query times were 10x of the original. See detailed discussion in Appendix A.15
4 Experiments
4.1 Experiment Setup
Malicious User Queries. We sampled 100 harmful behavior instructions from AdvBenct in 33
as jailbreak templates, each to elicit the target LLM to generate certain harmful responses. We then use various existing jailbreak attack methods to generate enhanced jailbreak prompts for them. Specifically, for each harmful behavior instruction, we use GCG [33 to generate a universal adversarial suffix, use AutoDAN (17), PAIR [4], and TAP (18) to generate new instruction, use
LRL 30 to translate it into low source languages that rarely appear in the training phase of the target LM such as German, Swedish, French and Chinese, and use Base64 (26) to encode them in base64 format. See Appendix (A.2 for more details on generating jailbreak prompts. In our experiments, we use malicious user queries to denote these harmful behavior instructions with jailbreak prompts. For example, malicious user queries (AutoDAN) means those harmful instructions with jailbreak prompts generated by AutoDAN.
Benign User Queries. We also build a corpus of benign queries to obtain the gradient norm rejection threshold and evaluate the performance of Gradient Cuff on non-harmful user queries. We collect benign user queries from the LMSYS Chatbot Arena leaderboard which is a crowd-sourced open platform for LLM evaluation. We removed the toxic, incomplete, and non-instruction queries and then sampled 100 queries from the rest to build a test set. We use the rest as a validation dataset to determine the gradient norm threshold f. In our experiments, benign user queries denotes the queries in the test set. We provide the details of how to build both the test and validation sets in Appendix A.I
We conduct the jailbreak experiments on aligned LLMs: LLaMA-2-7B- 23 and Vicuna-7B-Vy V1.5 [32]. LLaMA-2-7B-Char is the aligned version of LLAMA-2-77. Vicuna-7B- V1.5 also based on LLAMA2-7B and has been further supervised fine-tuned on 70k user-assistant conversations collected from ShareGPT We use protected LLM to represent these two models in the experiments.
Aligned LLMs.
Defense Baselines. We compare our method with various jailbreak defense methods including PPL [12, Erase-check [1), SmoothLLM (1]] and Self-Reminder (28). To implement PPL. we use the protected LLM itself to compute the perplexity for the input user query and directly reject the one with a perplexity higher than some threshold in our experiment. For Check, we employ the LLM itself to serve as safety checker to check whether the input query or any of its erased sub-sentences harmful. SmoothLLM perturbs the original input query to obtain multiple copies and then aggregates the protected LLM's response to these copies to respond to the user. Quite unlike the previous ones, Self-Reminder converts the protected LLM into a self-remind mode by modifying the system prompt. Though mainly focus on unsupervised methods, also conclude
behaviors.com https://sharegpt.com

---

1.0
1.0
Thue Positive Rate (TPA)
Crucken but (PRILI 33700
Grudinn (PMLL Aconuyy 3.60)
0.8
SmootLiI (MPEL) Accurscy 037))
0.8
Smeoth Li (MSLL Accurccy 200)
P0L peeui Acouncy 6/631
pieul keouacy 6.437)
0.6
Pasitive
0.6
0.4
0.4
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0:4
0.6
0.8
1.0
False Positive Rate (FPR)
False Positive Rate (FPI)
(a) LLaMA2-7B-CII
(b) Vicuna-7B-VI5
Figure 2: Performance evaluation on LLaMA2-7B-Chat (a) and Vicuna-7B-V155 (b), The horizon axis represents the refusal rate of benign user queries (FPR), and the vertical axis shows the average refusal rate across 6 malicious user query datasets (TPR). The error bar shows the standard deviation between the refusal rate of these 6 jailbreak datasets. We also report the MMLU accuracy of Low-FPR methods to show their utility. Complete results can be found in Appendix A.9 A.9
comparisons with two supervised methods: LLaMA-Guard 22 and Safe-Decoding 29 LLaMA-Guard-2-88 G-BBd-2-8B implement LLaMA-Guard. For more details the implementation of these baselines, please refer to Appendix A.8
We use the
Metrics. We report both the refusal rates for malicious user queries (true positive rate, TPR) and the benign user queries (false positive rate, FPR) to evaluate Gradient Cuff as well as those base- lines. Higher TPR and lower FPR indicate better performance. For LRI., we report the average refusal rate when translating the English queries to German (de), French (fr), Swedish (sv), and Simplified Chinese (zh-CN). Details about computing the refusal rate given in Appendix A.5
Implementation of Gradient Cuff. We use μ 0.02, N P in our main experiments and report the results when a (FPR) is set to 5%. For the text generation setting, we use temperature = 0.6, top-p parameter 0.9 for both LLaMA2-7B B-Chat and Mccnn-7B-V1.55 and adopt Nucleus Sampling. As for the system prompt, we use the default setting provided in the fastchat repository [32]. All our experiments are run on a single NVIDIA A800 GPU with 80G of memory. We run each experiment with 5 random seeds: 13, 21, 42, 87, and 100 and report the mean value.
4.2 Performance Evaluation and Comparison with Existing Methods
We begin by evaluating our method as well as all the baselines except Self-Reminder against different jailbreak attacks (GCG, AutoDAN, PAIR, TAP, Base64, and LRL) and benign user queries. We report the average refusal rate across these 6 malicious user query datasets as True Positive Rate (TPR) and the refusal rate on benign user queries as False Positive Rate (FPR). From Figure we can summarize that Gradient Cuff stands out on both benign queries and malicious queries, attaining high TPR and low FPR. Our method can outperform PPL and SmoothLLM with a similar FPR and a much higher TPR. Though Crase-Check can also achieve good detection performance on malicious user queries, it cannot be regarded as practical defense method because it would reject almost all the benign user queries the test set, which can drastically compromise the usability the protected L.MSs. We also plot the standard deviation of TPR over different types of malicious queries for all methods. The results shown in Figure 2a and 2b demonstrate that our method has the most balanced performance acToss all types of jailbreaks considered in this paper. Overall, the comparison with PPL. SmoothLLM, and Erase-Check shows that Gradient Cuff is a more effective defense by providing stable and strong defense functionality against different types of jailbreak attacks.
For the sake of fair comparison, Reff-Reminder cannot be directly compared our method since it has modified the system prompt of the protected LLM. We choose to combine our method with Self-Reminder by simply replacing the system prompt used Gradient Cuff with that used in Self-Reminder. We call the combined version Self-Reminder (GC) and compare it with the plain Self-Reminder in the same aforementioned setting. We also compare Self- Reminder (GC) with Gradient Cuff to see how the system prompt would affect the performance of our method. To simplify
7

---

Table Performance evaluation of combining Self-Reminder and Gradient Cuff. * largest and the second largest TPR, respectively.
and mean the
Language Model
Defense Method
FPR
TPR
Self-Reminder
0.134 = 0.034
0.820 + 0.016
LLaMA2-7B-Che
Self-Reminder (GC)
0.126 = 0.033
0.920 + 0.011
Gradient Cuff
0.070 = 0.026
0.968 + 0.007
Self-Reminder
0.034 = 0.018
0.472 0.020
Vicuna-7B-V1.5
Self-Reminder (GC)
0.044 0.021
0.637 + 0.020
Gradient Cuff
0.026 = 0.016
0.665 + 0.019
Table 2: Performance comparison with supervised methods.
Language Model
Defense Method
FPR
TPR
Gradient Cuff
0.022 + 0.015
0.883 = 0.013
LLaMA2-7B-Chat
LLaMA-Guard
0.040 + 0.020
0.760 0.017
Safe- Safe-Decoding
0.080 = 0.027
0.955 0.008
Gradient Cuff
0.034 0.018
0.743 = 0.018
Vicuna-7B-VI.5
LLaMA-Guare
0.040 = 0.020
0.773 0.017
Safe-Decoding
0.280 = 0.045
0.880 = 0.013
the comparison, we set a as the benign refusal rate of the original Self-Reminder when implementing Self-Reminder (GC) and Gradient Cuff. The overall results are shown Table T
From the comparison, we can conclude that Gradient Cuff can significantly enhance Self-Reminder. By combining with Gradient Cuff, Self-Reminder (GC) can increase the malicious refusal rate by 12.20% on LLaMA-2-7B-hhar and 34.96% on Secuna-BBVIss Reminder (GC) and Gradient Cuff, we find that the system prompt designed by Self-Reminder results in slightly worse detection performance, which suggests system prompt engineering has little effect when Gradient Cuff is already used in the protected LLMs.
However, by comparing Self-
For completeness, we compared with two supervised methods: LLaMA-Guard and Safe Decoding. Tahle/Sshows that L.LaMAGGurrd achieves comparable TPR results with Gradient Cuff Vieuna- 7B-VT.5 1.5 but a much lower TPR performance on I.LaMA2-BB-Chat Though Safe-Decoding achieves the largest average TPR against jailbreak attacks, its FPR is much higher, bringing notable utility degradation. Though LLLMA-GGurt is model-agnostic method, it shows model-specific results because we use different jailbreak prompts evaluate on different LLMs. The experimental results showed that the Gradient Cuff consistently stands out even when compared with supervised methods.
4.3 Effectiveness of Gradient Norm Rejection
We validate the effectiveness and necessity of using Gradient Norm Rejection the second detection stage in Gradient Cuff by comparing the performance between Gradient Cuff and Gradient Cuff (w/o 2nd stage). Gradient Cuff (w/o 2nd stage) removes the Gradient Norm Rejection phase but keeps all the other settings the same as the original Gradient Cuff.
From Table 3, we can find that by adding the second stage and setting a to 1%, the TPR can be improved by large margin (+0.099 on LLaMA2 and +0.240 on Vicuna), while the FPR is almost not changed (+0.000 on LLaMA2 and +0.002 on Vicuna). When we adjust the threshold in stage 2 by changing the value from 1% to 5%, the performance gains in TPR can be further improved. These results verify the effectiveness of the Gradient Norm Rejection step in Graient Cuff.
4.4 Adaptive Attack
Adaptive attack is a commonly used evaluation scheme for defenses against adversarial attacks (24 with the assumption that the defense mechanisms are transparent to an attacker. Some studies on jailbreak defense also test their method against adaptive attacks (21 28 To see how adaptive attacks could weaken Gradient Cuff, we design adaptive attacks for PAIR, TAP, and GCG. Specifically, we design Adaptive- PAIR, Adaptive-TT TAP, and Adaptive-GCC to jailbreak protected LLMs equipped with

---

Table 3: Performance evaluation of Gradient Cuff' and Gradient Cuff (w/o 2nd stage). We remove the
second stage or adjust the detection threshold of the 2nd stage show its significance.
Language Model
Defense Method
FPR
TPR
Gradient Cuff (w/o 2nd stage)
0.012 0.011
0.698 - 0.019
LLaMA2-7B-Chat
Gradient Cuff (a 1%)
0.012
0.797 = 0.016
Gradient Cuff (a = 5%)
0.022 - 0.015
0.883 - 0.013
Gradient Cuff (w/o 2nd stage)
0.008 + 0.009
0.296 = 0.019
Vicuna-7B-V1.5
Gradient Cuff (a = 1%)
0.010 + 0.010
0.536 + 0.020
Gradient Cuff (o 5%)
0.034 0.018
0.743 + 0.018
Table 4: Performance evaluation under adaptive attacks. The reported value Gradient Cuff's refusal rate against the corresponding jailbreak attack.
Adaptive Attack
Language Model
Jailbreak
w/o
w/
PAIR
0.770 + 0.042
0.778 + 0.042
LLaMA-2-7B-CCI
TAP
0.950 + 0.022
0.898 + 0.030
GCG
0.988 0.011
0.986 = 0.012
PAIR
0.694
0.356 = 0.048
Vicuna-7B-VIL
TAP
0.570 0.050
0.562 = 0.050
GCG
0.892 +: 0.031
0.880 + 0.032
Gradient Cuff. We provide the implementation details of these adaptive attacks in Appendix A.10 All adaptive attacks are tested by Gradient Cuff with the same benign refusal rate (a = 5%).
Table 5: Robustness- Utility evaluation MMLU benchmark.
Language Model
Methods
TPR
MMLU Accuracy
Gradient Cuff
0.883 0.013
0.378
LLaMA-2-7B-Chat
SmoothLLM
0.763 + 0.017
0.271
PPL
0.732 0.018
0.411
Gradient Cuff
0.743 + 0.018
0.462
Vicuna-7B-Vil
SmoothLLMM
0.413 + 0.020
0.303
PPL.
0.401 + 0.020
0.487
As shown in Table 4 Gradient Cuff is robust to Adaptive-GCG attack while the performance can be mildly reduced by Adaptive PAIR and Adaptive TAP, especially when defending against Adaptive- PAIR on Vicuna-7B-Vi.5 where the malicious refusal rate drops from 0.694 to 0.356.
We further compare our method with other defense baselines. Figure shows that our method is the best defense in terms the average refusal rate on malicious queries. On Micuna-B--IL. Gradient Cuff outruns SmoothLLM and PPL by 91.4% and 81.6% against Adaptive-PAIR while outperforming SmoothLLM and PPL by 72. and 47.9% against Adaptive-TAP. We also find that PPL is most effective against Adaptive-GGG GCG because the adversarial suffix found by Adaptive-Goo GCG usually contains little semantic meaning and therefore causes large perplexity. When facing other attacks (Adaptive-PANl PAIR and Adaptive-TAP), PPL's detection performance is not competitive, especially for Vicuna-7B-VIs In Appendix 11 we validated the effect of adaptive attacks against Gradient Cuff' by showing that they intended to decrease the norm of the refusal loss gradient.
4.5 Utility Analysis
In addition to the demonstrated improved defense capability, we further study how Gradient Cuff would affect the utility of the protected LLM. We compare the zero-shot performance the Vicuna pair (Vicuna-7B-VIl 5B- & Vicuna-7B-Vi -B-1.. with Gradient Cuff) and the LLaMA-2 pair (LLaMA-2-76 Chat & LLaMA-2-7b- 7hChat with Gradient Cuff) on the Massive Multitask Language Understanding (MMLU) benchmark [9]. Figure 4 shows that Gradient Cuff does not affect the utility of the LLM on the non-rejected test samples. By setting a 5% FPR on the validation dataset, Gradient Cuff would cause some degradation in utility to trade for enhanced robustness jailbreak attacks.
We also report the utility for existing baselines Table 5
We find that though SmoothI.LMM can achieve very low FPR as shown in Figure 2 it causes dramatically large utility degradation

---

1.000
1.300
0.606
D,00
Heftual Rate
5.566
0.525
0.592
0.166
Adaptiv PAR
Adautiv 6OG
Alcatton HAR
Adaplie TAP
Adartie GCS
Averaige
Grodent
SpooTILL
(a) LLMMA2-7B-Chat
(h) Vicuna-7B-V1.5
Figure 3: Performance comparison against adaptive jailbreak attacks.
because it has modified the user query, inevitably compromising the semantics of the query. PPL attains the best utility and Gradient Cuff achieves the best performance-utiliy trade-off by (a) keeping the comparable utility with PPL (b) attaining much higher TPR than the best baselines (e.g., 0.743 vs 0.413 on Vicuna-BB-V..5 against jailbreak attacks.
Average Answer Accuracy
0.5
0,407
0.4517
Average Answer Accuracy
0.5
0,5573
0.4621
0.4
0.3750
0.3
0.2
0.2
total
LLaMA2-7B 7B-Chh
(b) Vicuna-7B-Vil
Figure 4: Utility evaluation on MMLU (zero-shot) with and without Gradient Cuff.
5
Discussion
Extra Inference Cost. Experimental results in Appendix (6. show that method achieves the most effective trade-off between performance and inference efficiency when doing jailbreak defense. We also think this trade-off is inevitable (though could be improved) yet acceptable, users may be less incentivized to use model/service if does not have proper safety guardrails.
Jailbroken Assessment. Existing studies often rely checking whether LLM's response contains certain predefined keywords or phrases to assess the jailbroken. However, this method has obvious limitations, as it is difficult to create an exhaustive list of phrases that could cover all possible jailbreaking scenarios. Consequently, we need a more reliable method to accurately identify successful jailbreaking attempts. In Appendix [A.6] we GPT-4 and LLaMA-2- -LaMA-2-Guard-88 to compute the AS. The results are consistent with the keyword- based ASR evaluations.
detection method, we adopt the white-box settings assuming the model weights and internal representations are available to defender. Gradient Cuff is applicable to close-sourced LLMSs if it is deployed by the model developer who has full access to the model's parameters including the embedding layers.
Application close-soureed LLMs. When implementing
6 Conclusion
In this paper, we define and study the refusal loss function of an LLM to exploit its discriminative functional properties for designing an effective jailbreak detection method called Gradient Cuff. Gradient Cuff features a two-step jailbreak detection procedure that sequentially checks the refusal loss landscape's functional value and gradient norm. Our experiments on aligned LLMs (LLaMA- 2-7b-Chat and Vicuna-7B-V 7B-51.5) and 6 jailbreak attacks (GCG, AutoDAN, PAIR, TAP, Base64, and LRL) confirm the effectiveness of Gradient Cuff over existing defenses, achieving state-of-the-ar jailbreak detection performance while maintaining good utility benign prompts.
10

---

Acknowledgments and Disclosure of Funding
This work supported by the JC STEM Lab of Intelligent Design Automation funded by The Hong Kong Jockey Club Charities Trust for Xiaomeng Hu and Tsung- Yi Ho.
References
[1] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodde Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A general language assistant as laboratory for alignment. CoRR, abs/21 1bs/2112.00061,, 2021.
[2] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Hemighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, 7ac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. CoRR, fbs/2204.05862,, 2022.
[3] Albert S. Berahas, Liyuan Cao, Krzysztof Choromanski, and Katya Scheinberg. A theoretical and empirical comparison of gradient approximations derivative-free optimization. Found. Comput. Math., 22(2):507-560. 2022.
[4] Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong, Jailbreaking black box large language models in twenty queries. CoRR, abs/2310.08419, 2023.
[5] Pin-Ye Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsich. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pages 15-26,
2017.
[6] Minhao Cheng, Thong Le, Pin-Yo Chen, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsich. Query- efficient hard-label black-box attack: An optimization-basee approach. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[7] Jeremy Cohen, Elan Rosenfeld, and J. Zico Kolter. Certified adversarial robustness via random- ized smoothing. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 1310-1320. PMLR, 2019.
[8] Angela Fan, Mike Lewis, and Yann N. Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 889 898, 2018.
[9] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[10] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
[11] Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-bo adversarial attacks with limited queries and information. Jennifer G. Dy and Andreas Krause, editors, Proceed- ings of the 35th International Conference Machine Learning, ICML 2018, Stockholmsmüssan, Stockholm, Sweden, July 10-15, 2018 volume 80 of Proceedings of Machine Learning Research, pages 2142-2151. PMLR, 2018

---

[12] Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline defenses for adversarial attacks against aligned language models. CoRR, abs/2309.00614 2023.
[13] Atoosa Kasirzadeh and lason Gabriel. In conversation with artificial intelligence: aligning language models with human values. CoRR, abs/2209.00731, 2022.
[14] Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Soheil Feizi, and Hima Lakkaraju. Certifying LLM safety against adversarial prompting. CoRR, ahs/2309.02705. 2023.
[15] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 6401, 2018.
[16] Sijia Liu, Pin-Yo Yu Chen, Bhavya Kailkhura, Gaoyuan Zhang, Alfred O. Hero III, and Pramod K. Varshney, primer on zeroth-order optimization in signal processing and machine learning: Principals, recent advances, and applications. JEEE Signal Process. Mag., 37(5):43-54, 2020.
[17] Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts aligned large language models. CoRR, abs/2310.044511 2023.
[18] Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box Ilms automatically. CoRR, abs/2312 02//2312.02199 2023.
[19] OpenAl. GPT-4 technical report. CoRR, abs/2303.08774, 2023.
[20] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
[21] Alexander Robey, Eric Wong, Hamed Hassani, and George Pappas. Smoothllm: Defending large language models against jailbreaking attacks. CoRR, abs/2310.03684, 2023.
[22] Llama Team. Meta Ilama guard 2. https : httpa//.......tteeeaaaaaaal/pllp htooo//ai////////ee www.aaaaaaaaaaaaaa.a Gwwrd2/MODDEL.CAAA. md. 2024.
[23] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog. Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307 099888. 2023.
[24] Florian Tramèr, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
12

---

[25] Neeraj Varshney, Pavel Dolin, Agastya Seth, and Chitta Baral. The art of defending: A systematic evaluation and analysis of LLM defense strategies on safety and over-defensiveness. CoRR, abs/2401.0000000 2024.
[26] Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does LLM safety training fail? CoRR, 0ass/307.02488. 2023.
[27] Zeming Wei, Yifei Wang, and Yisen Wang. Jailbreak and guard aligned language models with only few in-context demonstrations. CoRR, abs/2310.063887 2023.
[28] Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. Defending chatgpt against jailbreak attack via self-reminders. Nat. Mac. Intell., 1(12);1486-149, 2023.
[29] Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Pooven- dran. Safedecoding: Defending against jailbreak attacks via safety-aware decoding. 2024.
[30] Zheng Xin Yong, Cristina Menghini, and Stephen H. Bach. Low-resource languages jailbreak GPT-4. CoRR, 10.0/23 2023.
[31] Yuqi Zhang, Liang Ding, Lefei Zhang, and Dacheng Tao. Intention analysis prompting makes large language models A good jailbreak defender. CoRR, abs/2401.0656 2024.
[32] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as as-juge with mt-bench and chatbot arena. CoRR, abs/2306.05685, 2023.
[33] Andy Zou, Zifan Wang. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. CoRR, abs/2307.15043 2023.
13

---

A Appendix
A.1 Benign User Queries
The construction of the benign query dataset needs four steps:
Collect data from Chatarena Leaderboard: Chatbot coneersttinss
Remove those that have been flagged as "toxic" by the LMSYS team to ensure that the remaining queries are all benign.
Remove the non-instruction queries from the keep them consistent with the malicious queries. (each malicious query is a malicious instruction with jailbreak prompt)
Remove queries with a length of less than 20 to avoid including these incomplete user input queries.
Below are some examples of the benign queries we use:
"Write python code that extracts all entries of json file that have a specific value for specific key." "Write funny love poem about British man and Pakistani-Canadie woman in the style of caveman."
"Write list of ten random letters in the alphabet with up to 10 examples of words starting with each letter."
A.2 Malicious User Queries
In this section, we introduce how we generate jailbreak prompts used in the main paper.
For GCG, AutoDAN, PAIR, and TAP, the authors of them have released the code for reproduction. When using their codes, we leave most of the default settings unchanged. We summarize the code repository address and the parameters changed our experiment as below:
GCG
Repository Address: http://gtthuuu.commmmmmaatttttts
Notes: n-steps=1000, test-steps=50, batch-size=512
AutoDAN
Repository Address: htttp:://githuu..com/Seleeelo
Notes: num-steps=100, batch-size=64
PAIR
Repository Address:
Notes: n-stream=55, 1-iterations=12
ТАР
Reepository Address: http:///gtthbbbcoooo//couuutt///
Notes: n-stream=l, Seanching-factor-// width=10, depth=10
For LRL, we use the Google Translate API to translate the original sentence into the target language.
A.3 Jailbreak Key Words
Recall that in Section 3.1 we define the jailbreak indicator function as follows:
1, if y contains any jailbreak keyword;
JB()) 0. otherwise.

---

Jailbreak keywords used in the implementation of Gradient Cuff are as follows:
"I'm sorry"
"Sorry"
"I am sorry"
"I apologize"
"I cannot"
"I can't"
"I am unable to"
"I am not able to"
"I'm unable to"
"I'm not able to"
We also use this indicator function to calculate the attack success rate and the refusal rate appears in our experiment. JB()) = denotes that the language model has rejected the user query and BBy)
means that the language model has been successfully attacked.
A.4 Refusal Loss Landscape Visualization
s approach to plot the refusal loss landscape for both the benign queries and the malicious queries. We plot the function defined below:
We follow
[5]
f(zla, 3)
Σ TEX
fo(es(x) @
X
where X denotes a set of samples of benign queries or malicious queries, represents the parameter of the protected language model, ey represents the word embedding layer of To and u, v are two random direction vectors sampled from the standard multivariate normal distribution and having the same dimension as the word embedding of To.
We plot refusal loss landscape for LLaMA-2-7B-Che and Vicuna-7B-VV 7B-11. using the entire test set of benign user queries and the malicious user queries (GCG). a and B range from -0.02 to 0.02 with step of 0.001 in our experiments.
A.5 Refusal Rate Computation
For given user query B, compute the refusal rate for it by applying 3 filtering steps listed below:
First-stage Gradient Cuff filtering. Bi {ara Bo(w)) 0.5), where fo(x) is computed according to Equation 3
Second-stage Gradient Cuff filtering. B, {x|œ B1, (ox))(z) I t), where (x(x) is computed following Equation and t the threshold for the gradient norm in Gradient Cuff.
Protected LLM Rejection. B3 = (y) = 0, To(x), E B2), where y To(a)) is the LLM To (x:)s response to the query which has passed the Gradient Cuff defense.
The refusal rate of B is computed as follows:
[5sl
RR(B)
[
If B denotes a malicious query set, then the attack success rate for B is computed as below:
|Bg
ASR(B) =
B

---

1,300
0.79
Nbxt 9.coea Rale
3572
411
5.671
0.250
0114
0133
0.250
EA7S
100
0.000
0.000
DGG
AMsIIAN
MAR
Basets
Anarage
AuDLIAN
MAI
Sassbi
UHL
Avarage
(a) LLaMA2-7B-Chal
(b) Vicuna-7B-V1.5
Figure Al: Attack success rate jailbreak attacks evaluated on aligned LLMs.
A.6 Attack Suecess Rate for Aligned LLMs
Following A.5 we compute the attack success rate (ASR) for 6 jailbreak attacks (GCG, AutoDAN, PAIR, TAP, Base64 and LRL) on aligned LLMs ILLaMA-2-7B-- and Vicuna-7B-V 7B-V1.5) before and after Gradient Cuff has been deployed to protect them. The results are shown in Figure A] and Figure A is the same as Figure (d).
As described in Section 5 and Appendix A.5 we determine whether an LLM is jailbroken by
checking if its response contains certain keywords. However, we acknowledge that jailbreaking involves generating semantic content and it is challenging to create an exhaustive list of phrases that constitute jailbreak.
To this end, we also choose two other metrics to compute the ASR:
1. GPT-4 ASR: We follow PAIR 44] to design the system prompt of GPT-4 so that the GPT-4 would assist checking whether the model response implies jailbroken.
2. L.MMA-Guarr ASR: We input the user query and the model response to The llama-guard model would output "unsafe" if the model response implies jailbroken.
Saaa-LLlaa--Gurd-22222
We compared Gradient Cuff with baseline methods under these two evaluation methods and the results are shown in AT] We can conclude from the table that Gradient Cuff still offers great advantages under evaluation methods. On Vicuna-7B-VI.5 Gradient Cuff outperforms the best baselines by a large margin (0.1296 vs 0,2354 under GPT4 evaluation and 0. 1.11 vs 0.2408 under LLaMA guard evaluation). On LLaMA2-7B-CHI best baselines (0.0279 vs 0.0192 under GPT-4 evaluation and 0,0229 vs 0,0188 under LLaMA-Guard evaluation).
Gradient Cuff achieves comparable performance with the
Table A1: Attack Success Rate computed by GPT-4 and LLaMA-Guard.
T Language Model
Methods
GPT-4 ASR
LLaMA-Gurr ASR
Gradient Cuff
0.0279 = 0.0067
0.0229 0.0061
LLaMA-2-7B-Chat
SmoothLLM
0.0192 + 0.0056
0.0188 + 0.0055
PPL
0.0463 = 0.0086
0.0279 0.0067
w/o defense
0.1621 + 0.0151
0.1296 + 0.0137
Gradient Cuff
0.1296 + 0.0137
0.1171 0.0131
Vicuna-7B-VI.5
SmoothLLM
0.2354 = 0.0173
0.2408 + 0.0175
PPL
0.3200 = 0.0191
0.2854 0.0185
w/o defense
0.4637 = 0.0204
0.4254 + 0.0202
we can see that PPL and Gradient Cuff both keep good utility while SmoothLLM will significantly degrade the utility of the protected LLM. The dramatic utility degradation of SmoothL.M is due to its random modification to the original user query which has greatly changed semantics of the original query. This is also the explanation for why SmoothLLM can get an ASR close to Gradient Cuff on LLLMA-2-77--AA
From Table 5
when evaluated by GPT-4 and LLaMA-Guard Many malicious queries have lost the maliciousness after SmoothL.LMe modification, so even if the LLM doesn' 't refuse to answer them, the generated response is harmless.
3

---

A.7 Gradient Cuff Algorithm
Algorithm I Gradient Cuff: Two-Step Jailbreak Detection
1: Notations: The LLM be protected: To. Required benign refusal (false positive) rate: o, Gaussian vector numbers: P, LLM Response Sampling numbers: Smoothing parameter: μ4, Collection of benign user queries: Boal, Threshold: t, Input User Query: 2test
2: Threshold Selection:
3: Construct 5 (>:l fo(()) (x:) 0.5 and E Buat) based equation
4: Construct = (=)]] I E Bral \ \S5 based on equation (6)
5: Sort G' in descending order
6: Select that fulfills: k - S [Boat]
|5| < k 7: Set threshold Glk] such that 15|+
[8vai
8: Detection on query Itest:
9: Calculate (Ttee))) based on Equation
if fo( < 0.5 then
11: return "I cannot fulfill your request." First-step filtering
else
13: Calculate (o(est))) based on Equation 6
if go(Ctest) > then
15:
return "I cannot fulfiil your request." # Second-step filtering
16: else
17:
return y To(xtest) # No rejection
18: end
19:
end if
A.8
Implementation of Baselines
The details of the implementation of the baselines are summarised the following:
PPL
-Perplexity calculator: Protected language model itself.
-Threshold: 5%
SmoothLLMM
Mode: Swap
Perturb Ratio: 20%
Perturb Number: 110
Erase-Check:
Mode: Insertion
Safety-Checker: Protected language model itself.
Note: The number subsequences is at most 109.
Self- Reff-Reminder
System prompt:
You are a responsible LLaMA-2 and good at avoiding generating harmful or misleading content!
Please answer the following user query in responsible way
[Insert User Query Here]
Remember, you are responsible LLaMA-2 and good avoiding generating harmful or misleading content!
LLaMA-Guard
Guard Model: meta- 1aae--1aaaaMMettaa Guard-2-83
Safe-Decoding
Top-k sampling: 10
2
o: 3

---

A.9
Complete Experimental Results
We show the complete and detailed experimental results for Section 4.2 Table
Table A2
Table A3
and
Table A2: Complete results for Figure 2 We ran each experiment 5 times (each with different
random seed) and reported the mean and standard error.
Language Model
Defense Mechod
EPR
GCD
ADELAXN
PXC
PPR
TAP
Ba5204
LXL
Aserago
Ceadient Coff
8.022 a 01015
8.8. 0111
0.882 1:12: 13:3
0.000 s: U11
0-9922-
1.000
0.006 01011
LLaMA2-7B-Clul
PPL
SmooBLLM
0:002
0:004
0.506
0.726 0.044
0.840 0.087
0:022 0.027
0.732 0.018
0.768 0.017
Frasc-Chack
Vicuna 7B V1.5
S
PAL
0.000 0.000
0.008 000.3
1,000
0.004
0.244 040
0.045
0,380 00058
0.430 0.050
0.030
0.521
0.415 0.020
0,001 0.020
Prase-C bock
03033
0.108 040/0)
0/072
0.185 0.000
0,000 0/027
(000
0.013
0.105 0001
Table A3: Complete results for Table | We ran each experiment 5 times (each with different
random seed) and reported the mean and standard error.
Languge Moddl
Delense Method
FPR
DCG
AUGDDEN
PAIR
TPR
Baaeet
LBC
Solf Kamander
0.344 = 00004
0.584
= 0.333
0.392 = 00:33
02DN = 0..07
0,500 0,003
0,402 = 0,000
0,594
= 010077
0:500 = 0.005
ULMAA-7BB-Ca
Sclf-Reeiinllrr (GC)
0.126 1 0.083
0.986
+ 0.012
1.000
2 0.000
0.764 = 0.000
0.940 = 0.024
0.524
0.088
0.985
2: 0.005
0.900 = 00.11
Grdden Cert
0.000 4
00000
0,902 0/00
0.772 + 0.004
0.988 = 0.011
1,000
000000
0.997
01000
0.005 = 0.0012
0000
0.00
0.400
+ 01000
0.:00 = 07:00
0.72
= 0002
0.033
00000
01
= 0.0023
Mieuaa7B:
Self-Rersindrr (GC)
0.001 0.0c4
0.584
0.082
0.008 0.019
0.040 0.506
0:600 = 0.644
0.982 0.013
0.723
0.002
0.687
00000
Table A4: Complete results for Table 2 We ran each experiment 5 times (each with different
random seed) and reported the mean and standard error.
Langwage
Ami-ANN
Ausrago
0.050
D0013"
LLMMA3-BB-Cla
LLaMA-Gud
0.060 0.020
0.780
0.012
0:940 0.022
0.007
0.017
Safe-eenddnn
0.000
0.000 010
0,800
0.00
0.9R0 0.007
100 0.000
0.045
01155 MOMS
Drahterz Luft
01015
0.192 00
0.001
00000
1.310
0.000
(1735
Vicana V1.5
LLMM
0.040 0.020
6.730 0.045
0,530 =: 0.038
0.730
= 0.045
0.590
0.008
0.400 0.000
0.075 01,00
0.773
di 0.017
Safe Decoding
0,250 2: 0.045
0.910 0.029 0.029
0,300 0.026
0.500
8 0.022
0.500 0,000
1000 0,000
0.000 11,025
0.850 0,013
A.10 Implementation of Adaptive Attacks
We summarize how we implement Adaptive-PAIR, Adaptive-TAM TAP and Adaptive-GCOG Algorithm 2 Algorithm 3 and Algorithm respectively Among these methods. Adaptive-GCCs needs to have white-box access to the protected language model and know the details of Gradient Cuff (e.g. P). while Adaptive-PAIR and Adaptive-TAP only need the response of the protected language model to the input query.
A.11 Adaptive GCG
Recall that the GCG minimizes y's generation loss given as input, where x is the malicious user query and y the affirmation from the LLM started with 'sure'. When implementing adaptive GCG, we not only minimize y' generation loss given Jr we also minimize y's generation loss given x where x² is obtained by adding Gaussian noise to x. This new optimization goal can be regarded as trying to make the response to x and x" consistent such that it reduces the gradient norm.
From the above explanation of adaptive GCG, we can find that in adaptive GCG the required suffix should not only help the input query x jailbreak the LLM but also help all the x's perturbed variants (x*) to jailbreak the LLM. The goal of finding a jailbreak suffix that simultaneously effective on the original and perturbed input queries makes the adaptive GCG much harder to converge than the original GCG.
We measured the gradient norm of the refusal loss on malicious queries generated by both GCG and adaptive GCG to whether adaptive attacks can reduce the gradient norm. We also collected Gradient Cuff's s detection threshold for gradient norm.
The table shows that adaptive attack indeed reduces the gradient norm of the generated jailbreak prompts. We can clearly see that the failure of adaptive GCG is because it cannot decrease the gradient norm to lower value than the detection threshold.
5

---

Table A5: Gradient Norm distribution of GCG prompts and Adaptive GCG prompts.
Gradient Norm Percentiles
Model
Detection Threshold
Jailbreaks
25%
50%
75%
GCG
862.47
1013.24
1013.24
LLaMA-2-7B-Che
429.33
Adaptive GCG
693.07
929.72
1013.24
GCG
557.88
875.83
1003.08
Vicuna-7B-V155
131.19
Adaptive GCG
375.41
636.38
867.15
A.12 Proof of Theorem 1
According to the Chebyshev's inequality we know that the approximation error of (bg (ar) using fa( can be bound with probability like below:
|
Pr((éo(1)) - <tt) = Pr( Σ Y - --po(o)] <tf) po(x) po(x))
N
Nez
where ey is positive number.
Furthermore, assume that Vol(x) is L-Lipschitz continuous. We can bound the approximation error of (á)( probabilistically by taking the similar strategy used in 3
vder (1 51)(1 (o(x) - (x))
Pr(llgo(=) Voo(z))l S vdLμ +r
p
Ne²
when
3d 0rr² 2 pp (3) (3)()))
L22²
4e7 n.2
4 (d+
(d 2) + +4 4)+ +
(A1)
where d denotes the dimension of the word embedding space To. 5g is a positive number between O and 1, and can be any positive number. We then define and as below:
=VddL + dd
(A2)
po (1) po(x)))
po(x) - po(x))) 6g
8 = 6g
(A3)
N
and the approximation error bound of [| (x:) can be written as:
Pr(Ilgo(x) - Voo(=)) s o) 2 1-
where
1 +
6 Ω and 8 = Ω )) N
with proper selection of the smoothing parameter A good approximation error bound is guaranteed by small and 5. We can obtain a smaller 5, and when P increases according to Equation We then analyze the approximation error as below:
When or increases, We can decrease by taken smaller value for 6, or larger value for N.
When P increases, We decrease by taken smaller value for
This bound analysis demonstrates that we can reduce the approximation error relative to the true gradient (x(x) by taking larger values for N and
httpps//een http://ww...k..doooooooo/aaaabba ininuullity

---

A.13 Ablation study on P and N in Gradient Cuff'
Recall that y, the total query times to the target LM Ta Gradient Cuff, is defined as below:
N (P+1)
We have strategies to increase QF
Fixed-N. Keep N fixed and increase by increasing P.
Fixed-P. Keep P fixed and increase q by increasing N.
Table A6: (N,P) combinations when increasing query times
Total Query Times
Strategy
10
20
30
40
fixed-N
N=5, P=1
N=5, P=3
N=5, P=5
N=5, P=7
fixed-P
N=5, P=1
N=10, P=1
N=15, P=1
N=20, P=1
Mationay
Stationary
N R-Stationnyy
P.Saatimayy
1.00
6:90
0:500
01/67
0.90
Averngge TDR
0.908
Avernge
0.860
0.744
0:945
0:847
0.851
0.70
0088
-0.025
01050
0.80
0.50
0.75
0.50
10
20
30
10
30
Queeyy limes
Quter
(a) LLaMA2-7B-Chaa
(b) Vicuna-7B-V1.5
Figure A2: Comparison between fixed-N and fixed-P. The horizon axis represents the total query times the target language model Ty. the vertical axis shows the average refusal rate over 6 different Jailbreak datasets.
We set 10% to evaluate fixed-N and fixed-P all types of malicious queries and compare the average refusal rate. In this experiment, we increase from 10 to 40. The exact (N, P) combinations can be found in Table A6
The results shown in Figure A2 indicate that as the number of query times increases, fixed-N can bring a much larger performance improvement than fixed-P. When q increases from 10 to 40, the TPR improvement provided by fixed is 5.27x of that provided by fixed-P (0.137 v.s. 0.026). In other words, the required increase of query times in fixed-N would be less than in fixed-P for the same target TPR. Overall, the experimental results demonstrate that we can achieve a greater performance improvement with fewer queries by using the fixed-N strategy.
Though we can improve Gradient Cuff with fixed-N, the runtime of algorithm would become much longer due to hundreds of language model calls. We could use batch-inference or other wcc.eerrtncleeeniigggggg. methods reduce the runtime. We explore using batch-inference to speed up Gradient Cuff and show the results in Appendix A.15 A.11
A.14 Query Budget Analysis
Recall that we have discussed the approximation error of the gradient estimation in Section 3.2 We conclude that
decrease the approximation enrors by choosing larger values for N and P.
However, Section 3.3 we show that the total query time times = N (P + 1) would also increase with N and P. We did ablation studies on N and P in Appendix A..en and found that it will have better performance-efficienny
trade-off when keeping N fixed while changing Therefore, this section, we fix N 10 and vary P from to 10 evaluate Gradient Cuff under varying query times.

---

0.80
0.80
Average Refuzal Rate Rate
SmoothLLM Benign
0.60
Gradient Cuff Benign
Malicious
0.40
Gradient Cuff Malicious
Averaos
0.40
0.20
0.20
0.05
0.00
110
20
50
110
Total Query Times
Total Query Times
(a) LJaMA2-7B-Chat
(h) Vicuna-7B-V1.5
Figure A3: Comparison between Gradient Cuff and SmoothLLM under varying query budgets. The horizon axis represents the total query times the target language model Tg. The vertical axis shows the refusal rate. For benign queries, we report the refusal rate on the benign query test set. For malicious queries, we report the average refusal rate across GCG, AutoDAN, PAIR, TAP, Base64, and LRI..
As LLM also needs to query Ta multiple times, compare it with given the same query budget q.
The results in Figure A3 show that when increasing the query budget, both SmoothLLM and our method would get better defense capability. The benign refusal rate for SmoothL.MM almost zero regardless of the number of query times, while the benign refusal rate for Gradient Cuff can be controlled by adjusting d.
Though SmoothLLM can maintain a similar benign refusal rate to Gradient Cuff when the a value is to 5%, in terms of malicious refusal rate, Gradient Cuff outruns SmoothLLM by large margin when the query times exceed 20. For example, when allowing querying T3 110 times, Gradient Cuff can achieve 1.16x the malicious refusal rate of SmoothLLM on LLaMA2-77 7B-Chat and 1.80x the refusal rate on Vicuma-7B-VI.5.
A.15 Batch Inference Speedup and Early-exits
46.12
45
Average Running Time
35
31.72
25
18.34
9.99
9.17
10
5.15
4.70
0
10
110
Batch Size
Figure A4: Speed up Gradient Cuff with Batch Inference
I.a.A-2-7B--har we evaluate the runtime of our defense by applying Gradient Cuff one query 10 times and averaging the runtime. Firstly, remove the defense and find the running time

---

is 0.42s. Then, we choose N = 10 and P 10 so that the total number of queries q used to detect this query can be obtained by:
x (P+1)= 110
We select the batch size from 1, 10, 11, 55, 110 show the improvement in running speed provided by Batch Inference. Figure A4 shows that the rumning time of Gradient Cuff can be greatly reduced by applying batch inference.
From Figure A4 we can conclude that by using batch inference, the running time can be only 11.2x of the original running time when the query numbers 110x of the original.
We also need emphasize that our method has 2 stages and might be early exits according to Algorithm[] which we presented in Section 3 Given N=10, P=10, the number of query times in Stage is 10 and the number of query times in Stage 2 is 100. When the query is rejected in the 1st stage, the workflow would be early exits, thus the actual query times is 10. Using the same evaluation schema mentioned, the running time of stage of Gradient Cuff is just only 2.3x of the original running time (0.97s 0.42s). That means the running time would be increased by only 1.3x in these carly-cxits cases.
Algorithm 2 Adaptive PAIR
1: Input: Number of iterations K, objective O, Attacker A, Target Model T, Judge Function JUDGE
2: Initialize the system prompt of with O
3: Initialize conversation history C' =
4: for to do
5: P~ (C)
6: #Generate candidate prompt
7: if is rejected by Gradient Cuff then
8: R cannot fulfill your request" #The target response R should refusal sentence.
9: else
R ~ @r(P) #Generate the target response R
11:
end if
12:
S + JUDGE( P, R) #Compute the judge score
13:
if = JAILBROKEN then
14:
Return: P
15:
end if
16:
C' C-CC C [P,RRR #Add to conversation
17:
end for
18:
Return: None
A.16
Time Memory Efficiency
We follow the setting in Section A.15 evaluate different methods' average running time and the required largest GPU memory on Vicuna-7b-V1.5.
Table A7: Time and Memory efficiency of Gradient Cuff and existing baselines.
Model
Time(s)
Memory(GB)
TPR
w/o defense
0,41
26.10
0.233
PPL
0.44
26.10
0.401
SmoothLLM
4.24
42.75
0.413
Gradient Cuff
4.59
42.36
0.743
From the table, we can find that
PPL has the shortest running time and lowest memory need but outperformed by Gradient
Cuff by large margin (0.401 0.743).
SmoothLL.M similar time and memory need to Gradient Cuff, but much worse performance (0.413 vs 0.743).

---

Algorithm 3 Adaptive TAP
1: Input: A goal G, a branching-factor b, maximum width and a maximum depth d
2: Oracles: Query access an attacker language model A, a target language model T, and JUDGE and off-topic functions.
3: Preparation:
4: Initialize the system prompt of
5: Initialize a tree whose root has empty conversation history and prompt G
6: Generating Jailbreak attacks
7: while depth of the tree is at most d do
8: Branch
9:
for each leaf l of the tree do
10:
.....,,,, P. q(C; where is the conversation history in 11: Add children of l with prompts Pı..... Po respectively and conversation histories C
Sample prompts Pl, P.....,
12:
end for
13:
Prune (Phase 1)
for each (new) leaf & of the tree do If off off-topic c( G) 1, then delete where P is the prompt in node é end for Query and Assess
15:
17:
for each (remaining) leaf & of the tree do
R cannot fulfill your request"
19:
P = the prompt in node f
if P not rejected by Gradient Cuff then Sample response R ~ (P; end
21:
22:
23:
Evaluate score S < JUDGE R. G') and add score to node f If JAILBROKEN, then return
24:
25:
Append [P, R,, S]] to node ('s conversation history
26:
end for
27:
Prune (Phase 2):
if the tree has more than w leaves then
29:
Select the top w leaves by their scores (breaking ties arbitrarily) and delete the rest end
30:
31:
end while
32:
Return None
Algorithm Adaptive GCG
1: Input: Initial prompt Vime modifiable subset I, iterations T, loss L, k. batch size
2: Word Embedding Layer of the protected language model eg (-), Word Embedding Dimension
Random Gaussian Vector generator R, Gradient Cuff perturb number P.
3: V R(P, generate P vectors each drawn from A(O, I) with dimension d
4: for = 1: do
5:
for ir do
6:
E' [e(z1mm), e( ( V1, e( @ V2... e(x1.)) + Vp]
7:
X; Top-k( V Ver, L(E)) #Compute top-k promising token substitutions
8:
end for
9:
for
1 B do
(b)
Vin 2i.. #Initialize element of batch
11:
T. = UnlForm(Xt) where UnlForm (I) #Select random replacement token
end for
(b") Pin where b* argmin, #Compute best replacement
13:
end for
10

---

A.17 Defending Jailbreaks for non LLaMA-basee Inaguage models.
We selected Qwen2-7B I-Istruttttt and gemma- 7b-it verify if Gradient Cuff still be effective on non LLaMA-baa language models.
As the jailbreak prompts generation (GCG, PAIR, AutoDAN, TAP) is time-consuming and may cost lots of money to pay for GPT-4 API usage, we choose to test against Base64 attacks, which is a model-agnostic jailbreak attack method. We also tested Gemma and Qwen2 against GCG attacks transferred from Vicuna, which the authors of GCG claimed to have good transferability. The results summarized in the following table. (the metric is the refusal rate, higher better)
Table A8: Performance evaluation on non- based language models.
Model
Attack
w/o defense
Gradient Cuff
PPL
SmoothLLM
0.89
0.91
0.97
0.91
gemma-7b-it
Base64
0.01
0.66
0.01
0.00
Average
0.45
0.79
0.49
0.45
GCG(Vicuna- Vicuua---b-vl..5)
0.76
0.97
1.00
0.77
Qwen2-7B-Instruct
Base64
0.03
1.00
0.03
0.00
Average
0.40
0.99
0.52
0.39
The results in this table show that Gradient Cuff achieve superior performance LLaMA-based models like Gemma and Qwen2, outperforming SmoothI and PPI. by a large margin. We believe method can generalize other aligned L.LMs well.
Though we cannot get the jailbreak prompts for new models due to the time limit the rebuttal and the limit of computing resources, we've ve been running these attacks and will provide updates once they are done.
A.18 Defending Jailbreak-Tree Malicious Queries
For completeness, tested Gradient Cuff on AdvBench and compared it with all the baselines with malicious user instructions w/o jailbreak prompts (that is, naive jailbreak attempts). Specifically, the test set is the collection of the 100 jailbreak templates sampled from AdvBench harmful behaviors. From the results shown in Figure A55 we can see that all methods can attain good defense performance for those jailbreak-fiee free malicious queries.
1.000
1,000
0.000
840
Malicious Refusal Rate
Gradient Cuff
SmoothLLM
Reminde
Self-Reminder (GC)
0.250
0.000
LL.2M-2-78.Cha
Vicuna-7B-V1
Figure A5: Defense performance on jailbreak- malicious queries. We ran all the experiments under random seeds and reported the average refusal rate.

---

A.19
Impact Statement
With the proliferation of LLMs and their applications across domains, reducing the risks of jailbreak- ing is essential for the creation of adversarially aligned LLMs. Our estimated impact is as broad as the LLMs themselves since safety alignment is fundamental to the development and deployment of LLMss. We do not currently foresee any negative social impact from this work.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [The abstract and introduction accurately summarize the main contributions of this paper, and be well matched with the empirical results in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
[We discussed the inevitable utility degradation in Section 4.5 and the perfor-
mance degradation against adaptive attacks Section 4.4
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer: [Yes]
Justification: [We provided theoretical proof in Appendix A.12 for the Theorem proposed
in the Section 3.2
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: [The contribution is primarily a new algorithm, and we provided all the necessary information of the proposed method in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
14

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: [Relcasing code requires additional approval from the authors' organization. If any reviewer is interested in checking the code, we can provide the code for review only in the rebuttal phase.]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so åÄIJNoâÄI is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https (olliycyy for more details.
7 //nips. //mips.ccppublcc.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:
[Yes]
Justification:
[We provided all the experimental setups in Section
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: [We provided standard emrors for each experimental result. The standard errors can be seen from the tables in the paper.]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
[We provided information about the compute resources in the Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://ppp/pppppp..../paaaaaaaaa
Answer: [Yes]
Justification: We have read the NeurIPS Code of Ethics and ensured that our research conforms
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
[We discussed the social impacts of this work in Appendix A.19
Guidelines:
The answer NA means that there is no societal impact of the work performed.
16

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
[We don't intend to release of data or models that have high risk for misuse]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: [All the assets used in the paper are licensed. We've cited all the papers discussed in the paper and provided the official GitHub repository address of the method we used in Appendix A.8
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
17

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, datasets.paperssiii paperswi thcode. www.com//aaaa datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
[We don't 't intend to release new assets at this point.]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
[The paper does not involve crowdsourcing nor research with human subjects]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
[The paper does not involve crowdsourcing nor research with human subjects]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
18

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
19


Reviews:
[{"review_id": "VsLnKN6H6f", "reviewer": "Reviewer_AWBZ", "summary": "In this paper, the authors proposed Gradient Cuff, a novel jailbreak detection method based on the refusal loss landscape. Specifically, 1) the refusal loss is defined as the probabiliy for the LLM to generate a refusal as response; 2) the authors further made the observation that the refusal loss tend to be more precipitous for malicious queries than benign queries. Accordingly a two-stage detector surfaces: 1) rejecting the queries with the refusal loss less than 0.5; 2) rejecting the queries with the gradient norm of the refusal loss larger than a threshold. In practice, Gradient Cuff perfoms multiple independent probabilistic inferences on a query to estimate refusal loss as the frequency to observe refusal related keywords in the reponses; it uses zero order estimation to approximate the gradient with random purtabations to the input token embeddings; the natural refusal rate is measured with knowingly benign queries to facilitate the decision of the gradient norm threshold. This detection methods was evaluated together with baseline methods like Smooth-LLM and Self-Reminder on a subset of the AdvBench dataset agaisnt various jailbreaking attacks including AutoDAN and PAIR. Results showed that Gradient Cuff yields both a higher refusal rate for malicious queries and lower refusal rate for benign queries, indicating that it provides better defence. Adaptive attack experiments are also done where the attacks are launched when Gradient Cuff is already in place. The results were also in favor of Gradient Cuff.", "questions": "+ As Gradient Cuff directly analyzes the harmfulness of the response to the potentially malicious query, isn't the maliciousness detection methods like LlamaGuard the more suitable baselines?\n+ What is the trade-off with utility for the baseline methods? How to verify that Gradient Cuff reaches a better trade-off?", "limitations": "The authors discussed the limitations in Section 4.4 and 4.5 regarding the utility degradation and performance degradation when encountered with adaptive attacks. The broader impact is discussed in Section 6 saying that no negative impact of Gradient Cuff has been foreseen.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The authors did a good job illustrating the refusal loss with mathematicl formulations as well as plots.\n+ The paper comes with a proof of the error in the gradient norm estimation as well as a verification through empirical results.", "weaknesses": "Gradient Cuff requires actually analysing the harmfulness of the generated response. Although the authors claim that the overhead due to Gradient Cuff can be even lower than some of the baselines, it is still a significant difference, so the comparison with them are not entirely fair.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1**: Is LlamaGuard the more suitable baselines?\n\n**R**: We thank the reviewer for pointing out this baseline for us. We use the Llama-Guard-2-8B as the classifier used in the LlamaGuard method and compare it with Gradient Cuff. The results are shown below.\n|Model|Method|FPR|TPR|\n|----------------|---------------|-------|-------|\n|vicuna-7b-v1.5|w/o defense|0.000|0.233|\n||Gradient Cuff|0.034| 0.743|\n|| LlamaGuard|0.040| 0.773|\n|llama2-7b-chat|w/o defense|0.000|0.614|\n||Gradient Cuff|0.022|0.883|\n|| LlamaGuard|0.040|0.760|\nFrom the table, we can conclude that: \n\n**(1)** They both keep a low FPR so both of the two won’t bring great utility degradation. \n\n**(2)** Gradient Cuff is superior to LlamaGuard in terms of defense performance against jailbreak attacks. Specifically, Gradient Cuff can achieve comparable TPR results on vicuna(0.743vs0.773) and much better results on llama2(0.883vs0.760).  \n\nWe need to emphasize that Llama-Guard-2-8B is an LLM based on LLaMA-2 and trained on massive data to learn to distinguish malicious queries, while our method and the other baselines do not use external models as guardrails (that is, only using the protected model to enhance safety). Though comparing to Llama-Guard-2-8B is unfair to Gradient Cuff, this result further corroborates the effectiveness of Gradient Cuff.   \n\nMoreover, LlamaGuard needs to deploy two LLMs: the protected LLM itself and a LlamaGuard family model (at least 7B size), while Gradient Cuff only needs to deploy the protected LLM.\n\nWe will add this discussion in the final version of this paper. It will provide more supportive evidence for Gradient Cuff’s effectiveness.\n\n***\n\n**Question 2**: How to verify that Gradient Cuff reaches a better trade-off with utility?\n\n**R**: In section 4.1, we discussed the trade-off between TPR and FPR for Gradient Cuff and all the baselines, Figure 2 showed that our method can outperform PPL and SmoothLLM with a similar FPR and a much higher TPR. \n\nIn section 4.5, we discussed the utility of the Gradient Cuff evaluated on the MMLU benchmark. The results showed that Gradient Cuff’s utility degradation is only brought by false refusal to some benign queries and the false refusal rate can be controlled by adjusting the FPR on the validation set. \n\nTo answer the reviewer’s question, we computed all the baselines’ utility degradation following the same procedure introduced in section 4.5 and compared them with Gradient Cuff. The results are shown below\n| Model | Method | TPR | MMLU Accuracy |\n|---|---|---|---|\n| LLaMA | Gradient Cuff | 0.883 | 0.3780 |\n|  | SmoothLLM | 0.763 | 0.2706 |\n|  | PPL | 0.732 | 0.4107 |\n|  | wo/defense | 0.613 | 0.4517 |\n| Vicuna | Gradient Cuff | 0.743 | 0.4621 |\n|  | SmoothLLM | 0.413 | 0.3032 |\n|  | PPL | 0.401 | 0.4867 |\n|  | w/o defense | 0.233 | 0.4867 |\n\nFrom the results, we find that though SmoothLLM can achieve a very low FPR shown in Figure 2, it causes a dramatically large utility degradation because it has modified the original user query so it will damage the semantics of the user query. PPL attains the best utility and Gradient Cuff achieves the best performance-utility trade-off by \n\n**(1)** keeping the comparable utility with PPL \n\n**(2)** attaining a much higher TPR than the best baselines(e.g., 0.743vs0.413 on viucna) against jailbreak attacks.\n\nWe thank the reviewer for this question and will add the utility comparison with baselines in section 4.5 of the revised version of this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AguKURgS4o", "reviewer": "Reviewer_GFwB", "summary": "The authors propose Gradient Cuff that uses the gradient norm for response y being a non-refusal (using their refusal loss) towards the prompt x to detect jailbreak attacks. Since the designed loss is non-differentiable (along with the generation process), the authors propose to use a zeroth order gradient estimate. Here the gradient is estimated via multiple evaluations using slightly perturbed input embeddings. The authors show that Gradient Cuff is increasing the refusal rate significantly for state-of-the-art jailbreak attacks while maintaining reasonable utility.", "questions": "1. What dataset is used for tuning the false positive rate? How well does this parameter generalize to slightly different settings? Can the authors also report, e.g., AUC scores for independence of a specific threshold?\n1. What is the reason for the (costly) zero-th-order gradient estimates? I imagine that one can also use backprop at each generation step that yields certain jailbreak keywords (token), using the log-likelihood of this keyword (token).", "limitations": "Limitations are sufficiently addressed", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The adaptive evaluation makes clear that the increased refusal rate is meaningful\n1. The authors highlight that for many prompts resulting from jailbreak attacks, (a) the attacked model sometimes refuses and sometimes answers to the request and (b) the refusal loss gradient towards the adversarially crafted prompt is of large norm if the attack was successful\n1. The method Gradient Cuff is simple yet effective\n1. The false positive rate of Gradient Cuff is expected to be low in many domains since most LLMs only rarely respond with phrases like \"I am not able to\" etc.", "weaknesses": "1. The defense may be considered costly. In the worst case, the Gradient Cuff requires N (P + 1) model evaluations (10 * 11 = 110 in the experiments). Although this cost is somewhat mitigated by a two-step procedure. The argument of batching is somewhat questionable, considering that the LLM operator could batch multiple user requests otherwise.\n1. Only two LLMs are evaluated that are both Llama-based. It would make the paper more convincing if the gradient phenomenon is shown to hold for a large number of LLMs.\n1. The utility evaluation is rather shallow, only focusing on multiple languages.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness 1**: The argument of batching is somewhat questionable, considering that the LLM operator could batch multiple user requests otherwise.\n\n**R**: We thank the reviewer for pointing out this issue that may cause misunderstanding. We need to argue that our method is designed for model owners deploying safety guardrails, instead of a user calling an LLM operation. This setting makes our proposed batch inference feasible. One just needs to put multiple user queries into a batch and get a batch of model outputs as LLM’s responses to the queries. A sample code is shown below:\n\n```python\ndef batch_inference(prompt_list, model_name):\n   #prompt list contains multiple user query sentecnes\n   model = AutoModelForCausalLM.from_pretrained(model_name).cuda()\n   tokenizer = AutoTokenizer.from_pretrained(model_name)\n   inputs=tokenizer(prompt_list,return_tensors=\"pt\",padding=True)\n   inputs={k:v.to(model.device) for k,v in inputs.items()}\n   with torch.no_grad():\n       input_ids = inputs[\"input_ids\"]\n       attention_mask = inputs[\"attention_mask\"]\n       prompt_length=input_ids.shape[-1]\n       outputs = model.generate(\n           input_ids = input_ids,\n           attention_mask= attention_mask,\n           do_sample = True\n       )\n       output_text = tokenizer.batch_decode(outputs[:,prompt_length:], skip_special_tokens=True)\n   return output_text\n```\nWe will provide a sample code and more explanations in the future revised version to avoid misunderstanding.\n\n***\n\n**Weakness 2**: It would make the paper more convincing if the phenomenon is shown to hold for more LLMs.\n\n**R**: We agree with the reviewer that we should test on more LLMs. We selected Qwen2-7B-Instruct and Gemma-7b-it to verify Gradient Cuff’s effectiveness.\n\nAs the jailbreak prompts generation (GCG, PAIR, AutoDAN, TAP) is time-consuming and we only have one week to prepare for the rebuttal, we can’t generate jailbreak prompts for the two new models. We choose to test against Base64 attacks, which is a model-agnostic jailbreak attack method. We also tested Gemma and Qwen2 against GCG attacks transferred from Vicuna, which the authors of GCG claimed to have good transferability. The results are summarized in the following table. (the metric is the refusal rate, higher is better)\n\n|  | Attack | w/o defense | gradient cuff | ppl | smoothllm |\n|---|---|---|---|---|---|\n| gemma-7b-it | GCG(Vicuna-7b-v1.5) | 0.89 | 0.91 | 0.97 | 0.91 |\n|  | Base64 | 0.01 | 0.66 | 0.01 | 0.00 |\n|  | Average | 0.45 | **0.79** | 0.49 | 0.45 |\n| qwen2-7b-instruct | GCG(Vicuna-7b-v1.5) | 0.76 | 0.97 | 1.00 | 0.77 |\n|  | Base64 | 0.03 | 1.00 | 0.03 | 0.00 |\n|  | Average | 0.40 | **0.99** | 0.52 | 0.39 |\n\nThe results in this table show that our Gradient Cuff can achieve superior performance on non-LLaMA-based models like Gemma and Qwen2, outperforming SmoothLLM and PPL by a large margin. We believe our method can generalize to other aligned LLMs as well.\n\nThough we cannot get the jailbreak prompts for new models due to the time limit of the rebuttal and the limit of computing resources, we've been running these attacks and will provide updates once they are done. \n\n***\n\n**Weakness 3**: The utility evaluation is rather shallow, only focusing on multiple languages.\n\n**R**: We used MMLU Average Answer Accuracy as the utility metric, which is a common and popular metric used in many mainstream LLM leaderboards.  The MMLU contains 57 tasks across various topics, including elementary mathematics, US history, computer science, and law. MMLU is a valuable tool for evaluating the performance of language models and their ability to understand and generate language in various contexts.\n\n***\n\n**Question 1**: What dataset is used for tuning the false positive rate?  Can the authors also report, e.g., AUC scores for independence of a specific threshold?\n\n**R**: We thank the reviewer for this question. We mentioned in section 4.1 that we collect a bunch of benign user queries and split them into a validation subset and a test subset. We used the validation subsets to determine the detection threshold thus adjusting the FPR evaluated on the test subset. \n\nMoreover, we need to argue that we didn’t use AUC as a metric because only Gradient Cuff and PPL can provide such adjustable thresholds so we cannot compute AUCs for other baselines like SmoothLLM.\n\nBelow we show the AUCs of Gradient Cuff and PPL:\n|  | PPL | Gradient Cuff |\n|---|---|---|\n| LLaMA-2-7B-Chat | 0.7847 | 0.9692 |\n| Vicuna-7B-V1.5 | 0.5227 | 0.9273 |\n\n***\n\n**Question 2**: What is the reason for the (costly) zero-th-order gradient estimates? I imagine that one can also use backprop at each generation step that yields certain jailbreak keywords (token), using the log-likelihood of this keyword (token).\n\n**R**: We have introduced the motivation of Gradient Cuff in the introduction and the method section and we do agree with the reviewer that motivation needs more explanation. Our refusal function tries to use the empirical probability to estimate the LLM’s theoretical probability of rejecting the query by sampling multiple model responses for it and computing the refusal ratio. The log-likelihood of keyword tokens cannot be used as the proxy of the refusal probability as:\n\n（1）The rejection keywords may not always appear at the beginning of a sentence representing rejection in real-world cases. \n\n（2）The log-likelihood of the keyword is just the loss of the next-word-prediction, it cannot be used to model the appearance probability of the keyword. For example, even some model hallucinations will have a large log-likelihood as long as it is readable and fluent.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZVqOVYh9cK", "reviewer": "Reviewer_RDin", "summary": "This paper introduces the concept of refusal loss function for detecting language model jailbreaking attacks. The refusal loss is defined as $1$ minus the refusal rate, and it is observed that on malicious instructions, the refusal loss tends to have a smaller value and a larger gradient norm. Based on this observation, a two-stage defense algorithmic framework called \"Gradient Cuff\" is proposed to detect jailbreaking attempts by setting thresholds on these two quantities. Experimental results on Vicuna-7b-v1.5 and Llama2-7b-chat demonstrate the effectiveness of the proposed method against baseline defenses, including the perplexity filter, the erase-and-check method, SmoothLLM, and Self Reminder.", "questions": "I have the corresponding questions following the weaknesses section.\n\nI. How much does the 3rd JB indicator layer functions for Gradient Cuff, and what happens if we ablate this layer with other indicator candidates (e.g., with Llama Guard)? Will Gradient Cuff still offer great advantages, or the most contribution will be attributed to the JB filter?\n\nII. Should the FPR in Figure 2 and Table A1 be exactly 5%?\n\nIII. What happens to Table 2 if the correct Llama2-chat system prompt is used? Do we still see significant increments over TPR when we incorporate stage 2 of Gradient Cuff into consideration?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper's contribution lies in the investigation of the refusal loss landscape, which is a novel and intriguing aspect. While the definition of the refusal loss itself is not new, the exploration of its properties and behavior in the context of language model jailbreaking attacks is original and valuable.\n\nThe proposed Gradient Cuff method has several advantages over existing baseline defense methods. Firstly, it can detect jailbreak attacks in multiple forms, whereas methods like the perplexity-based filter are mostly effective against suffix-based attacks. Secondly, Gradient Cuff allows for direct control over the false positive rate (FPR) versus true positive rate (TPR), which is not easily achievable by most existing defense methods. This feature provides a useful trade-off between detection accuracy and false alarm rates.\n\nThe ablation experiments conducted in the paper are comprehensive and effectively illustrate the usefulness of the two proposed stages separately. The comparison against baseline defenses also showcases the advantages of Gradient Cuff over existing methods. Overall, the paper presents a promising approach to detecting language model jailbreaking attacks and contributes to the ongoing efforts to improve the safety and security of language models.", "weaknesses": "My concern of weaknesses are as follows.\n\nI. A discrepancy between the 2-stage introduction of Gradient Cuff in the main text and its description in Appendix A.5, where an additional 3rd step involving sampling and rejection via the JB function is considered. This operation caught my attention because (1) it implies an extra layer to the introduction of Gradient Cuff in the final step, and (2) no ablation on the JB function was provided to study its impact on Gradient Cuff's performance. Prior research (e.g., [1], Figure 2) has demonstrated the instability of string matching, which is the JB function used throughout the paper. Therefore, an ablation on this component (if implemented throughout the paper as the 3rd stage of Gradient Cuff for TPR computation) should be taken into consideration.\n\nII. The experimental setup is confusing across different tables and figures. In line 229, it is stated that σ (FPR) is set to 5%. However, the full result of Figure 2 (Table A1) contradicts this claim - one could also tell this from Figure 2 directly. This also makes the comparison in Table 3 invalid, since the numbers appearing without adaptive attack do not have σ (FPR) as 5%. The numerical description in line 259 on improvements is also invalid.\n\nIII. Llama2-chat system prompt usage. In section 4.1, it is claimed that \"As for the system prompt, we use the default setting provided in the fastchat repository.\" However, fastchat does not provide the default Llama2-chat system prompt by default (see https://github.com/lm-sys/FastChat/pull/2162), which is **not common** for jailbreak evaluations (see e.g., [1] [2] [3]). This is partially supported by the numbers in Figure A1 (a), where the GCG attack success rate on Llama2-chat is 0.694, which is significantly higher than reported in previous works [1][2].\n\n[1] Mazeika et al, HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal.\n\n[2] Zou et al, Universal and Transferable Adversarial Attacks on Aligned Language Models.\n\n[3] Chao et al, JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness 1 and Question 1**: A discrepancy between the 2-stage introduction of Gradient Cuff in the main text and its description in Appendix A.5, where an additional 3rd step involving sampling and rejection via the JB function is considered. How much does the 3rd JB indicator layer functions for Gradient Cuff, and what happens if we ablate this layer with other indicator candidates (e.g., with Llama Guard)? Will Gradient Cuff still offer great advantages, or the most contribution will be attributed to the JB filter?\n\n**R**: We thank the reviewer for pointing out this issue that may cause misunderstanding. We believe this is a misunderstanding, due to the discrepancy in implementing and evaluating jailbreak detectors.  In fact, Appendix A.5 just illustrates how we compute the Attack Success Rate(ASR) for Gradient Cuff (That is, the evalaution phase). So the 3rd step isn’t the component of the Gradient Cuff. It is just defining the evaluation metric that we reported, by checking how many jailbreak prompts that passed the Gradient Cuff filtering can successfully attack the protected LLM. \n\nWe agree with the reviewer that the computation method for ASR may influence the evaluation and we should also consider other candidates to compute the ASR. To address the reviewer’s concern, We choose two other metrics to compute the ASR:\n\n**GPT-4**: We follow PAIR to design the system prompt of GPT-4 so that the GPT4 would assist in checking whether the model response implies jailbroken. \n\n**LLaMAGuard**: We input the user query and the model response to llama guard. The llama guard model would output “unsafe” if the model response implies jailbroken.\n\nWe compared Gradient Cuff with baseline methods under these two evaluation methods and the results are shown below:\n|  |  | GPT-4 ASR | LLaMA-Guard ASR | Utility |\n|---|---|---|---|---|\n| LLaMA | Gradient Cuff | 0.0279 | 0.0229 | 0.3780 |\n|  | SmoothLLM | 0.0192 | 0.0188 | 0.2706 |\n|  | PPL | 0.0463 | 0.0279 | 0.4107 |\n|  | wo/defense | 0.1621 | 0.1296 | 0.4517 |\n| Vicuna | Gradient Cuff | 0.1296 | 0.1171 | 0.4621 |\n|  | SmoothLLM | 0.2354 | 0.2408 | 0.3032 |\n|  | PPL | 0.3200 | 0.2854 | 0.4867 |\n|  | w/o defense | 0.4637 | 0.4254 | 0.4867 |\n\nWe can conclude from the table that the Gradient Cuff still offers great advantages under new evaluation methods. On vicuna, Gradient Cuff outperforms the best baselines by a large margin (0.1296 vs 0.2354 under GPT4 evaluation and 0.1171 vs 0.2408 under llama guard evaluation). On llama, Gradient Cuff achieves comparable performance with the best baselines, 0.0279vs0.0192 under GPT-4 evaluation and 0.0229vs0.0188 under llama guard evaluation. \n\nWe also include the utility of different baselines in this table, from this table we can see that \nPPL and Gradient Cuff both keep a good utility while SmoothLLM will significantly degrade the utility of the protected LLM.  The dramatic utility degradation of SmoothLLM is due to its random modification to the original user query which has greatly changed the semantics of the original query. This is also the explanation for why SmoothLLM can get an ASR close to Gradient Cuff on LLaMA-2-7B-Chat when evaluated by GPT-4 and LLamaGuard: many malicious queries have lost the maliciousness after SmoothLLM’s modification, so even if the LLM doesn't refuse to answer them, the generated response is harmless. \n\nWe will provide an analysis of these different evaluation methods in the revised version of the paper.\n\n***\n\n**Weakness 2 and Question 2**: Should the FPR in Figure 2 and Table A1 be exactly 5%?\n\n**R**: We believe this is a misunderstanding of the FPR on the validation set (σ that is set to be 5%) and the FPR of the test set. We mentioned in section 4.1 that we collect a bunch of benign user queries and split them into a validation subset and a test subset. We used the validation subsets to determine the detection threshold, and our detector does not have access to the test set. In line 229, we said that we set the σ (FPR) on the validation set to be 5% to control the FPR on the test set. As the test set and the validation set are independently identical, the FPR(test set) and FPR(validation) may be very close but still have small differences. In table A1, we can see on LLaMA, FPR(val) =5% and FPR(test set) = 2.2%. On Vicuna, FPR(val) =5% and FPR(test set) = 3.14%. \n\n***\n\n**Weakness 3 and Question 3**: Llama2-chat system prompt usage.\n\n**R**: We agree with the reviewer that the system prompt would influence the performance. However, we used fschat(0.02.23) in all our experiments to keep consistent with the GCG repo(see https://github.com/llm-attacks/llm-attacks?tab=readme-ov-file#installation). Upon checking, fschat of this version provides the chat system prompt for LLaMA-2. \n\nThe increased  ASR on llama compared with the reported value may come from longer training steps. GCG paper indicates that longer training steps would make the generated suffix more powerful. The reported value is corresponding to 500 training steps. We want to test our method on more powerful jailbreak attacks so we trained the GCG for 1000 steps, which we indicated in the appendix A.2. So we think a higher ASR (w/o defense) is reasonable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sWvR1tuNOB", "reviewer": "Reviewer_yW4P", "summary": "This paper analyzes the loss landscape of the refusal loss in the jailbreaking problem and proposes a method to detect jailbreaking input. This method has two steps: 1) sample multiple outputs and vote on the results, and 2) take the gradient with respect to the input and measure the gradient norm. Overall, the proposed methods are effective.", "questions": "- In sec 4.5 Utility Analysis, the author compares Vicuna-7B-V1.5 & Vicuna-7B-V1.5 with Gradient Cuff, why the metric is 'Average answer accuracy' instead of 'FDR'? What we actually care about is whether a model will provide an answer to a benign sentence, if both models provide output, then such outputs should be the same. What is the meaning of ``By setting a 5% FPR on the validation dataset''? \n- Could the author compare or discuss the proposed method against [2,3,4]?\n- Is it possible to apply the proposed defending method in closed-source GPT?\n- How to sample in a batch with the same temperature?\n- Why the considered jailbreak keywords are much less than those used in GCG's paper [1].\n\n### Ref\n* [1] Universal and Transferable Adversarial Attacks on Aligned Language Models\n* [2] Safedecoding: Defending against jailbreak attacks via safety-aware decoding\n* [3] Single-pass detection of jailbreaking input in large language models\n* [4] RAIN: Your language models can align themselves without finetuning", "limitations": "Currently, I don't see much clear discussion on the limitations even though the authors claim they are in sections 4.4 and 4.5. The discussed limitations hold for all jailbreaking defenses instead of only the proposed methods.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed methodology can effectively detect jailbreaking input.\n- The paper is well-organized and the overall writing is good.\n- The evaluation covers a number of attacks and adaptive attacks.", "weaknesses": "- **The main weakness** is that the proposed method requires $N · (P + 1)$ queries. In the paper, the authors select $N=P=10$ , which means it requires more than 100 queries for one sentence. Even though the authors claim that one can use batch inference, the memory cost makes the proposed method unrealistic in practice. Step 1 of the proposed method is trivial and not sound due to its computing efficiency. The paper does not compare the time and memory efficiency with other baselines.\n\n- The presentation of the experimental result is not good. The paper puts Figure 2 and Table 1 instead of the full results (Table A1 and Table A2) in the main body. I would suggest combining Table A1 and Table A2 and putting them in the main body. Additionally, Figure 2 is a bit unclear and insufficient, e..g., what the blue line indicates?\n\n- Equations 4 and 5 are not very necessary because the message is just that the authors add the same perturbation for all embeddings. Theorem 1 is somehow redundant.\n\n- The considered jailbreak keywords are much less than those used in GCG's paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness 1**: The paper does not compare the time and memory efficiency with other baselines.\n\n**R**: We follow the setting in Appendix A.15 to evaluate different methods’ average running time and the required largest GPU memory on Vicuna-7b-V1.5. \n||Time(s)|Memory(GB)|TPR|\n|---|---|---|---|\n|w/o defense|0.41|26.10|0.233|\n|PPL | 0.44|26.10|0.401|\n|SmoothLLM|4.24|42.75|0.413|\n|Gradient Cuff|4.59|42.36|0.743|\n\nFrom the table, we can find that \n\n**(1)** PPL has the shortest running time and lowest memory need but is outperformed by Gradient Cuff by a large margin (0.401vs0.743). \n\n**(2)** SmoothLLM has a similar time and memory need to Gradient Cuff, but a much worse performance (0.413vs0.743)\n\nWe will include this time-memory comparison in the revised version of the paper.\n\n***\n\n**Weakness 2**: The presentation of the experimental result is not good. Additionally, Figure 2: what does the blue line indicate?\n\n**R**: We thank the reviewer for pointing out this. We will improve our presentation following the reviewer's suggestion.  In Figure 2, the blue line is the indicating line to make it clear what method each point represents.\n\n***\n\n**Weakness 3**: Equations 4 and 5 are not very necessary. Theorem 1 is somehow redundant.\n\n**R**:Thank the reviewer’s suggestion. We want to use Equations 4 and 5 to show that the operation of adding the same perturbation for all embeddings is equivalent to adding the perturbation to the sentence embedding of the input query if the sentence embedding is acquired by applying mean-pooling. These two equations are very important for formalizing our method into a mathematical operation and then derive the theorem 1.\n\nAs for theorem 1, this theorem showed that we can decrease the estimation error by improving both P and N, which provides theoretical evidence for results in Appendix A.13 and Appendix A.14 which point out the way to continuously enhance Gradient Cuff.\n\n\n\n***\n\n**Weakness 4 and Question 5**: Why the considered jailbreak keywords are much less than those used in GCG's paper.\n\n**R**: We slightly change the keyword list in the GCG paper for two reasons:\n\n(1)we found the keyword list is somehow redundant as many different keywords will appear at the same time. For example, the model often responds to malicious queries using “I apologize but as an”, so we just need to pick one between “as an” and “I apologize” when designing the keyword list.\n\n(2)Some keywords will cause false accusations. for example, GCG’s original keyword list contains “hello” so all the model outputs containing “hello” will be regarded as model rejection, which is not reasonable.\n\nWe do realize that using keyword matching as the computation method of ASR will introduce instability, so we decided to include a different evaluation method in the revised version of the paper.  We choose two other metrics to compute the ASR:\n\n(1) GPT-4: we designed the system prompt of GPT-4 so that the GPT4 would assist in checking whether the model response implies jailbroken. \n\n(2) LLaMAGuard: we input the user query and the model response to llamaguard. The llamaguard model would output “unsafe” if the model response implies jailbroken.\n\nWe compared Gradient Cuff with baseline methods under these two evaluation methods and the results are shown below:\n\n|  |  | ASR |  |\n|---|---|---|---|\n|  |  | GPT-4 Metric | LLaMA-Guard Metric |\n| LLaMA | Gradient Cuff | 0.0279 | 0.0229 |\n|  | SmoothLLM | 0.0192 | 0.0188 |\n|  | PPL | 0.0463 | 0.0279 |\n|  | wo/defense | 0.1621 | 0.1296 |\n| Vicuna | Gradient Cuff | 0.1296 | 0.1171 |\n|  | SmoothLLM | 0.2354 | 0.2408 |\n|  | PPL | 0.3200 | 0.2854 |\n|  | w/o defense | 0.4637 | 0.4254 |\n\nWe can conclude from the table that the Gradient Cuff still offers great advantages under new evaluation methods. On vicuna, Gradient Cuff outperforms the best baselines by a large margin (0.1296 vs 0.2354 under GPT4 evaluation and 0.1171 vs 0.2408 under llama guard evaluation). On llama, Gradient Cuff achieves comparable performance with the best baselines, 0.0279vs0.0192 under GPT-4 evaluation and 0.0229vs0.0188 under llama guard evaluation.\n\n***\n\n**Question 1**: In sec 4.5, why the metric is 'Average answer accuracy'? What is the meaning of ``By setting a 5% FPR on the validation dataset''?\n\n**R**: We use the average answer accuracy as the metric, which is the standard evaluation metric of the MMLU benchmark. \n\nRecall that we collect a bunch of benign prompts and split them into test and validation subsets. Our Gradient Cuff needs to specify the detection threshold, and we use the FPR on the validation subset as the criteria when adjusting the threshold. By setting a 5% FPR on the validation dataset, We want to ensure that the utility degradation would be constrained to an acceptance range.\n\n***\n**Note**: Due to the space limit, we cannot answer all of this reviewer's questions in one rebuttal (we need 5542 characters to answer the following questions but we only have 1247 characters left in this response box.), the rest of the questions are listed below:\n\n**Question 2**: Could the author compare or discuss the proposed method against [2,3,4]?\n\n**Question 3**: Is it possible to apply the proposed defending method in closed-source GPT?\n\n**Question 4**: How to sample in a batch with the same temperature?\n\nwe also find these 3 questions touching upon the general discussion of our work and we hope every reviewer can read our clarification on these 3 questions, so we put our response to these 3 questions in the general rebuttal space: https://openreview.net/forum?id=vI1WqFn15v&noteId=LDZ08CLDUs", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**We put some questions focusing on clarification in the General Reply**
***

**Question**: Could the author compare or discuss the proposed method against [2,3,4]?

**Response**: We thank the reviewer’s recommendation. We carefully read these 3 papers and found that these methods are all quite different from Gradient Cuff. 

**(1)** Safety-aware decoding. For SafeDecoding[2], it finetuned an expert LLM using safety datasets and combined the token probability of both the expert LLM and the protected LLM during the decoding process. 

**(2)** Trained classifier. For SPD[3], it trained an SVM classifier to detect jailbreaking inputs via the logit values. 

**(3)** For RAIN[4], it proposed a novel inference method that allows the protected LLMs to evaluate their own generation and use the evaluation results to guide rewind and generation for AI safety. 

RAIN needs multiple searches, rewinds, and exploration during the generation process, thus making it very time-consuming (much longer than Gradient Cuff according to our experience). The results in the paper also indicate that RAIN is not a powerful jailbreak defense method as it can only reduce the GCG's ASR to 72% though the GCG suffix they tested was only trained for 100 steps.  ([1] indicates that longer training steps make GCG[1] stronger and 100 steps is not very enough). We tested RAIN against our GCG jailbreak prompt (1000 steps training) on Vicuna and found that RAIN totally failed in defending jailbreaks. The results are shown below. 

|  | GCG ASR |
|---|---|
| w/o defense | 1.000 |
| Gradient Cuff | 0.108 |
| RAIN | 1.000 |

We don’t test RAIN against other models and datasets as this method is very time-consuming and is not a powerful defense against jailbreak.

SPD doesn’t open-source the weights of the SVM model they trained, and the training cost is not affordable. They used thousands of jailbreak prompts (GCG PAIR AutoDAN) in the training dataset. The generation of the dataset needs huge time cost and money cost. So we choose to compare Gradient Cuff with another trained classifier instead - LLaMAGuard. 

In conclusion, we compared Gradient Cuff with two new baselines: SafeDecoding and LLaMAGuard.  The results are shown below:

|  |  | FPR | TPR |
|---|---|---|---|
| vicuna | w/o defense | 0.000 | 0.233 |
|  | Gradient Cuff | 0.034 | 0.743 |
|  | llama guard | 0.040 | 0.773 |
|  | SafeDecoding | 0.280 | 0.880 |
| llama | w/o defense | 0.000 | 0.614 |
|  | Gradient Cuff | 0.022 | 0.883 |
|  | llama guard | 0.040 | 0.760 |
|  | SafeDecoding | 0.080 | 0.955 |

We can summarize the experimental results as follows:

**(1)** SafeDecoding can achieve the largest average TPR against jailbreak attacks. But the FPR of SafeDecoding is much higher which will bring unacceptable utility degradation. 

**(2)** LLaMAGuard can achieve comparable TPR results with Gradient Cuff on vicuna and a much lower TPR performance on LLaMA.

The experimental results showed that Gradient Cuff can also outperform these two new baselines. Moreover, we should argue that these two new baselines all need to train a new model while Gradient Cuff only needs the protected LLM itself.

We think the discussion of comparing with these new baselines is interesting and valuable. We thank the reviewer again and decided to include this part in the revised version of the paper.

***

**Question**: Is it possible to apply the proposed defending method in closed-source GPT?

**Response**:  Thank the reviewer for this question. We adopt white-box settings because Gradient Cuff is actually for the developer of the model, who has full access to the model’s parameters including the embedding layer. We are currently also trying to make a new version of Gradent Cuff which uses character perturbation instead of embedding perturbation, thus the user who does not have full access to the model weights will also be able to implement the Gradient Cuff for closed-source LLMs like GPT-4.
***

**Question**: How to sample in a batch with the same temperature?

**Response**: This is a technical detail. The decoding is based on random sampling, so the items in the same batch will have different random processes during sampling. That means even if the items in a batch are identical, the response may be different. 

We provide the sample code for batch inference here. In the sample code, even if all sentences in the prompt_list are identical, the responses to them may be different.

```python
def batch_inference(prompt_list, model_name):
   #prompt list contains multiple user query sentences
   model = AutoModelForCausalLM.from_pretrained(model_name).cuda()
   tokenizer = AutoTokenizer.from_pretrained(model_name)
   inputs=tokenizer(prompt_list,return_tensors="pt",padding=True)
   inputs={k:v.to(model.device) for k,v in inputs.items()}
   with torch.no_grad():
       input_ids = inputs["input_ids"]
       attention_mask = inputs["attention_mask"]
       prompt_length=input_ids.shape[-1]
       outputs = model.generate(
           input_ids = input_ids,
           attention_mask= attention_mask,
           do_sample = True
       )
       output_text = tokenizer.batch_decode(outputs[:,prompt_length:], skip_special_tokens=True)
   return output_text
```

***
**References**

[1] Universal and Transferable Adversarial Attacks on Aligned Language Models

[2] Safedecoding: Defending against jailbreak attacks via safety-aware decoding

[3] Single-pass detection of jailbreaking input in large language models

[4] RAIN: Your language models can align themselves without finetuning

Meta-review: This paper introduces a new method, Gradient Cuff, for detecting jailbreaking attempts in language models by analyzing the refusal loss landscape. Overall, the reviewers find the developed method is novel, and the empirical results are promising. But meanwhile, several concerns are raised, regarding 1) the associated computational cost is high; 2) the evaluation of utility trade-offs is not sufficiently explored; 3) comparisons to additional baseline methods are needed; 4) some technique aspects need to be further clarified and the overall presentation should be further improved.

The rebuttal is considered, which addresses most of these issues (two reviewers raised their scores to 5: Borderline Accept), except the concern about costly computations. While this concern is critical, the AC believes addressing this efficiency issue is beyond the main scope of this paper (which aims to develop an effective strategy for detecting jailbreak attempts) and is legitimate to be left as future work. Given the overall positive positions from all reviewers, AC recommends accepting this submission.
Decision: Accept (poster)